How-to
Most engineering teams have a healthy reflex: when a test fails, inspect the code under test. That reflex is usually correct. In LLM-powered products, however, the failure may sit in the test itself. A recent field note from an open-source agent-security project describes cases where a detector looked broken, but the real defect was the scenario or expected result. The model had broader world knowledge than the person who wrote the test, so the test became the thing that needed debugging. For cross-border stores, SaaS platforms, and remote engineering teams, this is a practical warning: your evaluation harness is part of the product, and it needs its own quality bar.
Why an LLM can be a better oracle than your test
A test is a compressed assumption. It says: for this input, this output is correct. That assumption is easy to get wrong when the product touches language, locale, policy, tax rules, product categorization, support tone, or code review. The person writing the test may know the happy path; the model may have seen more edge cases, more phrasing variants, and more regional context.
That does not make the model automatically right. It means the model can be a useful second opinion before you rewrite production code. Treat it as an oracle with a scope, a rubric, and an audit trail - not as an unquestionable authority.
Where this shows up in cross-border and SaaS work
- Address and contact parsing for markets with different formats.
- Product titles and descriptions that must be localized without losing compliance details.
- Refund, warranty, and shipping-policy answers across regions.
- Support macros where tone and legal caution both matter.
- Code review comments where the issue is a missing edge case, not a syntax error.
- Store catalog checks where a category or restriction depends on local rules.
In each case, a test that asserts one exact string can fail even when the model response is acceptable or better than the fixture. The team then spends time fixing a detector that was never the problem.
Figure: Editorial context image from the source page (preview only; rights remain with the publisher) · Credit / further reading: dev.to
A seven-step playbook for LLM-in-the-loop tests
Start from intent, not a golden string. Write what the output must accomplish: identify the correct policy, preserve a required legal phrase, reject an unsafe request, or flag a missing edge case. Use exact-string assertions only for stable, deterministic fields.
Record provenance for every fixture. Note who wrote the scenario, which locale or product category it represents, what source material was available, and when it was last reviewed. A test without provenance is hard to challenge and harder to retire.
Split the failure into three layers. Check the detector or parser, the scenario setup, and the expected answer separately. Ask: did the system under test fail, did the test provide bad context, or was the expected answer incomplete? This prevents the default fix-the-code loop from hiding test debt.
Test the test. Add adversarial fixtures where the expected answer is deliberately thin. Ask a reviewer or a separate model pass to argue why the test might be wrong. Keep a small set of known-good outputs that must pass, and a set of known-ambiguous cases that should trigger review rather than binary failure.
Use two oracles with different jobs. Let deterministic checks handle format, schema, banned phrases, and required fields. Let LLM adjudication handle semantic questions: did the answer satisfy the intent, preserve the policy, and fit the locale? Log both verdicts so you can see disagreement instead of flattening it.
Figure: HEXOPS original illustration inspired by the topic (not a copy of the source article imagery) · Credit / further reading: dev.to
Version eval data like code. Store scenarios, fixtures, rubrics, and expected outcomes in the same review workflow as application code. When a model or prompt changes, diff the eval set and the results. Do not let silent fixture edits explain away a regression.
Build a triage scorecard. For each failure, record: layer suspected, severity, locale, customer impact, reproducibility, and owner. A scorecard turns a noisy eval run into a queue of decisions. It also gives remote teams a shared language for what counts as a real defect.
Design the rubric before you look at the output
A common mistake is to inspect the model output first and then write a rubric that fits it. Reverse the order. Define pass, review, and fail criteria before the run. For cross-border work, include locale-specific requirements: currency format, date format, required disclosures, prohibited claims, and tone. For SaaS work, include permission boundaries, data-handling constraints, and escalation paths.
A practical rubric has four parts:
- Must pass: hard requirements such as schema validity, safety, and required policy language.
- Should pass: semantic quality, completeness, and locale fit.
- Human review: ambiguous cases where the cost of a wrong automated verdict is high.
- Blocked: cases where the test itself is known to be stale or under-specified.
The blocked category is important. It gives the team permission to fix the test without pretending the product passed.
Operational guardrails for remote teams
An LLM oracle is still a tool. Keep a human owner for high-risk eval sets, especially those touching money, legal claims, health, safety, or customer eligibility. Do not let the same model that produced an answer be the only judge of that answer. Use a different prompt, a different model, or a human reviewer for the final call.
Also cap cost and complexity. Run expensive adjudication on a sample, not every case. Cache verdicts with the fixture version. Require a short written reason for each fail so the next reviewer can audit the decision. And keep evaluation runs reproducible: same fixture set, same rubric version, same model version, same parameters.
Key takeaways
- A failing LLM test can mean the test is wrong, not the product.
- Separate the detector, the scenario, and the expected answer before fixing code.
- Use exact assertions for stable fields and rubric-based review for semantic work.
- Version fixtures and rubrics so eval changes are visible.
- Keep a human owner and a blocked category for stale or ambiguous tests.
Next steps
- Pick one LLM-in-the-loop workflow that already has failing tests.
- Label each failure as detector, scenario, expected answer, or genuine product bug.
- Rewrite two exact-string tests as intent-based rubrics.
- Add provenance and a last-reviewed date to every fixture in that workflow.
- Run a small disagreement review between deterministic checks and LLM adjudication.
- Assign an owner and schedule a monthly eval-set review.
Sources
Further reading: Three Times My Test Failed and the LLM Was Right
