The benchmark-to-production gap

Lab benchmarks measure standardized tasks under controlled conditions; production is custom tools, legacy systems and organization-specific rules. The compounding problem makes the gap concrete: 85% per step is roughly 20% over ten steps.

The two mechanisms behind the gap, and the one implication that follows: probabilities multiply, stateless agents repeat their failures, so agents are selected by testing against your own data, workflows and tools.

Why the gap exists

Lab benchmarks determine how an agent performs on standardized tasks under controlled conditions. An agent performing with production data, tools and workflows will not perform the same. The gap is where enterprise deployments fail. Industry analyses document a 37% gap between lab benchmark scores and real-world deployment performance and a 50x cost variation for similar accuracy levels.

The gap exists because in contrast to standardized benchmarks, production throws agents against custom internal tools, legacy systems with undocumented behavior, organization-specific business rules and domain-specific policy validation. Two problems have been identified as a result of the gap between benchmark numbers and production performance.

The compounding problem

The compounding problem is observed because each step's probability multiplies rather than adds: an agent executing, for example, a 10-step workflow with 85% reliability per step delivers roughly 20% overall reliability. A system that looks reliable on any individual step becomes unreliable across a realistic workflow. Additionally, unlike traditional software, agents cannot simply retry a failed step. Autonomous decisions may have already triggered irreversible actions.

Stateless execution

The stateless execution problem is that base agents start each task with no memory of previous runs. They cannot retain failure patterns or learn from inefficient paths. An agent that failed on a specific edge case previously will come across the same edge case again in the future and fail the same way, unless the harness engineers a permanent fix into the environment. This connects to Mitchell Hashimoto's principle: engineer the environment so a mistake cannot recur. Reliability improves through harness engineering and not model upgrades.

A benchmark score, then, is a lab measurement and not a production guarantee. The implication is concrete. Agents should not be selected based on benchmark rankings. Instead they need to be tested against actual data, workflows and tools. Reliability is an infrastructure problem, not a model one.