AI agents
Chains, Graphs, or Code: A Pass/Fail Checklist for Choosing an Agentic Harness
Most agent incidents you've shipped this quarter were not model failures. They were harness failures — the scaffolding around the model that decides what runs next, what gets retried, and what counts as "done." A harness that accepts any output as success will happily escalate a hallucination into a production write. A harness that can't branch will loop forever on the one input it wasn't designed for.
The industry's answer has evolved in three recognizable generations. Chains are linear prompt sequences: step A feeds step B feeds step C. They are trivially debuggable and terrible at ambiguity — one bad extraction at step two poisons everything downstream. Graphs introduce nodes, edges, conditional branching, retry policies, validators, and human checkpoints. They handle the messy middle but impose workflow-engine maintenance on your team. Code harnesses let the model emit or invoke code and treat execution as the verifier: the test suite passes, the SQL returns rows, the schema validates, the type checker is silent. Execution is the strongest grader available, and also the most dangerous one when the sandbox is weak.
The trap is assuming newer is better. A code harness wrapped around a task that needed four deterministic API calls is an expensive way to buy nondeterminism. Run the gates below in order and stop at the first one that fails — that failure tells you the least complex harness you're allowed to ship.
图示：来源页预览图（仅作编辑上下文；版权归属原发布方） · 来源引用：www.infoq.com
How to use this checklist
Run it as a 45-minute design review before you commit to an architecture, and again whenever your eval pass rate drops below target. Each gate has an explicit PASS and FAIL condition. A gate that fails blocks the harness tier above it; it does not block shipping the tier below.
Gate 1 — Task shape (prerequisite for everything)
PASS if: You can write the task's success condition as a check a stranger could run without reading your prompt. "Returns a JSON object matching schema X." "The failing test passes." "The refund amount equals the invoice total minus credits."
FAIL if: Success is defined as "the output looks good" or "the model says it's done." Stop. You have no harness problem yet; you have a specification problem, and no amount of graph topology will fix it.
Gate 2 — Chain adequacy
PASS if all four hold:
- The step sequence is static and identical for every input.
- Fewer than ~6 model calls, each with one job.
- At most one external tool, and it is read-only.
- You have a golden set of ≥30 inputs with expected outputs, and the chain clears your pass-rate bar on all of them.
FAIL if any hold: Steps vary by input type; you've added a second "just in case" retry loop inside a prompt; a failure at step 1 is only discovered at step 5; or you cannot explain to an on-call engineer which step broke from the logs alone.
Action on fail: Do not add retries to a chain. Retries inside a linear pipeline multiply cost without adding information. Escalate to Gate 3.
Gate 3 — Graph adequacy
PASS if all five hold:
- Every branch condition is an explicit, testable predicate — not "the model decides."
- Each node has a declared retry budget, backoff policy, and terminal failure state.
- Validators sit between nodes and can reject a node's output, not just log it.
- Cost and latency budgets are enforced at the graph level, with a hard kill switch.
- Every run is replayable from a trace: you can reconstruct inputs, outputs, and branch decisions node by node.
FAIL if any hold: Your graph is really a chain with an if bolted on; retries are unbounded; a validator's rejection has no defined downstream path; or adding a node requires editing code in three places.
Action on fail: Fix the observability and retry semantics before scaling the topology. Graphs fail at the edges, not the nodes.
Gate 4 — Code harness readiness
Only proceed if Gate 1 passed with an executable success condition.
PASS if all six hold:
- The verifier is execution, not a model's self-assessment: tests run, queries execute, schemas validate, builds compile.
- Code runs in a sandbox with no ambient credentials, a filesystem it cannot escape, and a network allowlist.
- Every generated artifact is diffable and reviewable before it touches a shared system.
- Blast radius is capped: no production writes, no destructive migrations, no
DROP, no unbounded spend — enforced by the runtime, not by the prompt. - You can reproduce a failing run exactly, including the generated code, and replay it.
- Your eval suite includes adversarial cases: deliberately impossible tasks, where the harness must refuse rather than confabulate a passing result.
FAIL if any hold: The model's "I completed the task" message is treated as the success signal; the sandbox is a subprocess on the same host as your secrets; or you have no test for whether the harness can be talked into declaring success without doing the work.
Action on fail: Downgrade to a graph with a human checkpoint. A code harness without an execution verifier is just a chain with root access.
图示：HEXOPS 原创配图（受选题启发，非复制原文图片） · 来源引用：www.infoq.com
Gate 5 — Reliability and evaluation
PASS if: You have a versioned eval set, a pass-rate threshold tied to a business outcome, and a scheduled regression run on every prompt, model, or topology change. Flaky cases are triaged, not averaged away.
FAIL if: Your evidence is a demo, a vibe check, or a single happy-path transcript. Also fail if your eval set was written by the same person who wrote the prompt and has never been adversarially reviewed.
Gate 6 — Cost and latency envelope
PASS if: p50 and p95 latency and cost-per-successful-task are measured and published; every harness tier has a documented ceiling; and the kill switch is tested, not assumed.
FAIL if: You can quote token cost but not cost per successful task — the only number that matters once retries and validation loops exist. Fail if a retry storm can silently 10x your bill.
Gate 7 — Human escape hatch
PASS if: Any run can be paused, inspected, and handed to a human mid-flight, and the handoff includes enough context for that human to finish the task without reading logs.
FAIL if: The only options are "let it run" or "kill it."
The decision rule
| Your situation | Ship this | Never ship |
|---|---|---|
| Static steps, ≤1 read-only tool, clear verifier | Chain | Graph "for flexibility" |
| Branching, validation, retries, human review | Graph | Code execution |
| Executable success condition + hard sandbox | Code harness | Model self-grading |
| Ambiguous success, no verifier | Human-in-the-loop graph | Any automation |
Score each gate PASS/FAIL, then choose the lowest tier whose gates all pass. Complexity is a loan against future on-call, and the interest is brutal.
The two failure modes worth naming in every review
Premature graph. Someone adds branching for a case that has appeared once in 10,000 runs. You now maintain a workflow engine to handle an edge case a single retry would have absorbed. Test: can you delete the branch and lose fewer than 1% of successful runs?
Unverified code execution. The harness runs generated code but grades it with the same model that wrote it. This is circular and will pass adversarial inputs silently. Test: can you make the harness declare success on a task it provably did not complete? If yes, you don't have a code harness — you have a chain with root access and a confident narrator.
Rollout cadence
Review these gates on a two-week cadence while the harness is new, then at every model swap, prompt change, or topology change. Track pass rate, cost per successful task, and p95 latency on one dashboard. When pass rate drops, walk the gates top-down — the failure almost always appears at a gate you marked PASS six weeks ago and never revisited.
Sources
- Ramgopal, Karthik. "Agentic AI Architecture: The Evolution of the Harness." InfoQ. https://www.infoq.com/articles/agentic-ai-architecture-harness-evolution/
