Measuring what matters

Nearly 90% of CEOs expect measurable agent ROI in 2026; under a quarter of organizations report it. The reinvention gap explains where value actually comes from: the same technology pays 6-9x more when the process is redesigned around it.

Four metrics, one discipline: measure trajectories, not just outcomes. The right answer via the wrong reasoning path is a latent failure waiting for its moment.

The ROI gap

The ROI gap is already well-documented. Nearly 90% of CEOs expect AI agents to deliver measurable ROI in 2026, yet only 29% of organizations report significant ROI from generative AI and just 23% from AI agents specifically. Additionally, AI super-users deliver 5x productivity gains, but the organization-wide picture is far less clear. Evaluation methodology, and not model capability, is the bottleneck.

The reinvention gap

There is also what has been coined the "reinvention gap", which describes where value actually comes from: reinvention versus augmentation. This was quantified precisely by McKinsey in call center data, where Gen AI tools layered into existing workflows resulted in 5-10% improvement. Agent-enabled optimized processes produced 20-40% improvement, while agent-enabled reinvented processes produced 60-90% improvement. The gap between augmentation and reinvention is 6-9x, with incremental deployment producing incremental results. The economics only work when the organization redesigns the process, not by just adding an agent.

Dual-metric evaluation

The standard approach of measuring if the agent completed the task is insufficient. Dual-metric evaluation measures both task completion AND whether the agent followed the correct reasoning path. An agent can achieve 100% tool-call accuracy while violating policy on edge cases. The correct answer via the wrong reasoning path is a latent failure, and it will eventually produce the wrong answer via the same path.

Ideally the key metrics that should be utilized are: success rate (task completion), tool selection accuracy (did the agent choose the right tool at every turn), trajectory quality (whether the agent reasoned correctly and not just arrived at the right answer) and containment rate (the percentage of interactions an agent resolves end-to-end without escalating to a human).

The production reality check, though, remains the benchmark-to-production gap: lab scores overstate real-world performance, success rates drop in customer environments with custom tools, legacy systems and undocumented APIs, and costs vary wildly for similar accuracy levels. Combined automated metrics plus expert human judgment produces the most reliable assessment. Neither alone is sufficient.