What 86 Production Agents Tell Us: Short Loops, Human Gates, Custom Glue

Most talk about AI agents is about autonomy: agents that plan, act and finish long tasks on their own. Most evidence about agents comes from demos. So it's worth looking at what teams actually run in production.

That is what "Measuring Agents in Production" did. Researchers from UC Berkeley, Stanford, UIUC, IBM Research and Intesa Sanpaolo collected 306 survey responses, narrowed them to 86 deployed agents, and ran 20 in-depth case studies with the teams behind them. The picture is very different from the demos.

Short loops, not long autonomy

68% of the deployed agents run at most ten steps before a human is involved. Not hundreds of steps, not open-ended runs: ten or fewer, then a person checks, approves or takes over.

The study names reliability as the top challenge teams face, and short loops are a practical answer to it. Every extra step is another chance for a mistake to compound, so teams keep the loop short and put a human at the end of it.

Custom glue, simple models

Among the 20 case-study teams, 17 (85%) built their agents in-house, calling model APIs directly rather than relying on an agent framework. That is a narrower group than the wider survey, where about 61% of deployed agents use a third-party framework, but the in-depth teams leaned heavily toward their own code.

14 of the 20 (70%) use off-the-shelf models with no fine-tuning. The value isn't coming from a custom model. It is coming from the engineering around it: how the agent connects to real systems, what it is allowed to do, and when it hands over.

Humans still check the work

74% of respondents rely on manual, human-in-the-loop evaluation. About half (52%) also use a model to judge outputs, but every interviewed team that does so also has people verify the results. Automated evaluation adds to human review; it doesn't replace it.

What this means for your agents

• Design for the hand-over. Decide where the agent stops and who it hands to, rather than hoping it finishes on its own.
• Spend effort on integration, not the model. The hard, valuable work is connecting the agent to your data and systems safely.
• Keep a person in the evaluation loop. Use automated checks to scale review, not to remove it.
• Judge it on reliability, not the demo. A short loop you can trust beats a long one you have to babysit.

How we approach it at Zylen

Zylen is an AI optimization partner. We work inside the AI systems companies already run and build agents the way the production teams in this study do: short loops, clear hand-overs, careful integration and evaluation with people involved. Our ten-day AI Optimization Audit is the place to start.

Build the agent that ships: start the AI Optimization Audit →

Source: Pan et al., "Measuring Agents in Production", arXiv 2512.04123, December 2025.