Back Now
What We Learned from 2M Runs
The failure modes that actually matter and the patterns that prevent them.
James Wilson
Growth Lead
April 14
Research · 13 min
Every team that ships agents eventually builds a dashboard. Latency, cost, success rate a wall of charts that looks reassuring right up until a customer reports something the charts never caught. The problem isn’t the dashboard. It’s when it tells you.
Dashboards are a trailing indicator. By the time a regression shows up as a dip in your success-rate chart, it has already reached production and, worse, real users. The fix is to move quality checks to the same place we catch every other regression: continuous integration.
The case for evals-as-tests
An eval is just a test with a fuzzier assertion. Instead of expect(x).toBe(2), you assert that an agent’s output satisfies a rubric: did it resolve the ticket, stay in policy, and avoid hallucinating an order number? Run that assertion on every pull request and quality stops being a vibe.
If a regression can reach production, it will. The only reliable place to stop it is before the merge button.
Building evals from real traces
The best eval cases come from production. When an agent run goes wrong, that trace is a perfect regression test capture the inputs, pin the expected behavior, and add it to the suite. Over time your eval set becomes a museum of every bug you’ve ever fixed.
Wiring it into the pipeline
Make the eval suite a required check. On every pull request, the runtime spins up the agent, replays each case, and scores it against the rubric. If the pass rate drops below your threshold, the merge is blocked same as a failing unit test.
Run the full suite on every PR, not just nightly.
Gate merges on a pass-rate threshold you control.
Surface the diff: which cases regressed, and why.
The payoff
Once evals live in CI, the dashboard becomes what it should have been all along: a way to discover new failure modes, not a tripwire for old ones. Every new bug you find becomes a case, and your agent gets monotonically more reliable with each release.
That’s the whole trick. Stop watching for regressions in production. Catch them where they’re cheap to fix — before the merge.
