Why Offline Evals Lie: Building Eval Users That Behave Like Real Ones

AI Evals | By the Zylen Solutions Team | 3-minute read

At LangChain Interrupt 2026, Lyft shared a lesson that most teams shipping agents will recognise. To test a customer-support agent, they built a lightweight simulator with an LLM playing the rider. Offline, the agent's success rate hit 90%. Then it shipped, and production performance was poor.

The model wasn't the problem. The test users were. According to 8th Light's recap of the talk, the simulated riders were too polite, too patient and too detailed. Real riders were impatient and answered in one or two words.

Why simulated users are too nice

Language models are tuned to be helpful and cooperative. Ask one to play a customer and you get a helpful, cooperative customer: one who explains the problem clearly, answers every follow-up and says thank you at the end.

Real users don't do that. They type "where is he". They reply "??". They change what they want halfway through, ask for a human straight away, or simply leave. An agent tuned against the polite version learns to handle a conversation that rarely happens, and the eval score tells you it is ready.

Fix 1: write the users who will break it

Don't give the simulator one well-behaved default. Give it personas, each with a clear instruction about how to behave:

• The rusher: one or two words, no greeting, no context.
• The vague one: "It's broken." Nothing more until asked, and sometimes not even then.
• The switcher: changes the request halfway through the conversation.
• The typo-er: typos, slang and mixed languages.
• The skeptic: asks for a human straight away.
• The edge case: an unusual but perfectly valid request.

Fix 2: calibrate the simulator against real transcripts

Personas are a guess until you check them against reality. Take a sample of recent real conversations, with personal data redacted, and compare them with your simulated ones on a few simple measures: how long replies are, how many turns a conversation takes, and where people drop off.

Plot them side by side. If the simulated users write paragraphs while real users write a few words, the simulator isn't ready. Re-tune the personas until the two shapes overlap.

Fix 3: track the gap, not just the score

An offline score on its own says little. What matters is how far it sits from production. Report the two side by side, and alert when the gap widens, not only when a score drops.

Then close the loop: every production failure becomes a new eval case. Over time the eval set stops describing the users you imagined and starts describing the ones you have.

Where to start

You don't need a new framework. Pull a small sample of real conversations, write three or four personas based on what you see, and run your existing eval suite against them. The drop in the score is the most useful number you'll get this week.

How we approach it at Zylen

Zylen is an AI optimization partner. We work inside the AI systems companies already run. Our ten-day AI Optimization Audit starts by measuring each workload on real traffic, including cost, latency and quality, before we change anything. No baseline, no claim.

Find out how your AI performs on real traffic: start the AI Optimization Audit

Source: Shawn DeVries, "Production is the New Prototype: Notes from LangChain Interrupt 2026", 8th Light, May 17, 2026.