Verification loops

Probabilistic systems doing deterministic work. The harness intercepts every tool call, validates, executes sandboxed and checks the result: pre-execution before tokens burn, post-execution before anything proceeds.

The escalation gradient: the cost of checking is matched to the risk of being wrong. Level 1 automated, level 2 a second model as judge, level 3 a human hand on the switch. Where the lines are drawn is the harness designer's job.

Check in both directions

Verification loops include all the automated quality checks that validate agent outputs during execution. They are there to ensure that probabilistic systems like agents are doing deterministic work. Rather than relying on the model to be correct, the harness intercepts a tool call, validates its parameters, executes in a sandbox, cleans the output, and injects the result back into context. Verification happens in both directions. Pre-execution, where hallucinated function calls are caught along with invalid parameter types and references to nonexistent APIs, avoiding tokens spent on failed retries. And post-execution, during which the harness validates that what the agent did actually worked: automated testing for code, visual verification for UI changes.

The loop prevents a documented family of failures: premature termination of an incomplete task; infinite refinement, where the agent endlessly "improves" its output while consuming compute; reasoning-action mismatch, roughly 14% of agent failures in the MAST taxonomy, where the agent reasons one thing but does another; step repetition; and hallucination cascades, where fabricated information in one step corrupts every downstream decision.

Mistake-proofing the environment

The design philosophy is to make incorrect behavior mechanically impossible rather than asking the model to be correct. If an agent makes a mistake the engineered environment is updated so that the mistake cannot physically recur, a principle championed by Mitchell Hashimoto. Updated instruction files, tools that force verification, staged checkpoints with automated checks.

Drawing the lines

Not every verification is equal; the escalation gradient at the level above shows the three tiers. How the gradient is used matters: place everything under level 3 and the agent is an expensive approval queue; leave everything at level 1 and mistakes eventually reach the real world. A mixed-methods study of practitioners building software engineering agents found teams converging on the same shape by hand: evaluate the smallest units first, prefer executable checks over a model as judge, and reserve LLM-as-judge for the subjective cases automated checks cannot score. The same study found this practice answers a problem most teams hit first: existing test suites quietly become the default oracle simply because they already exist, even when the agent's output is actually better or the old standard has gone stale. The harness designer's job is to draw the line between what is routine, what needs a second opinion and what needs a human hand on the switch.