Mocks make agent tests fast and wrong. Real tools make them right and slow. A sandbox gets you both: real calls, fake consequences, results in under a minute.

Most teams start by mocking every tool their agent can call. It works until the day the real API changes shape, the mock doesn’t, and a green test suite ships a broken agent. The mocks were testing your assumptions, not your agent.

The fix isn’t to drop isolation. It’s to move it: let the agent call real tool code, and put the boundary at the side effects instead.

If your eval can’t fail when the API changes, it isn’t testing the agent. It’s testing your mocks.

— Priya Raman, Founding Engineer

Real tools, fake consequences

Wrap each tool in a sandbox that records every call. Reads go through for real. Anything that writes, like refunds, emails or database updates, lands in a ledger your eval can assert on: refund_order was called once, for $42, with the right reason.

Every tool call, recorded and asserted

Keep it under a minute

Speed is what keeps evals in CI. Run cases in parallel, cache read-only responses per suite, and cap each case at a handful of tool calls. A suite that finishes before the coffee does is one your team won’t skip.

Start with the bug that hurt you last week. Turn it into one case, sandbox the tools it touches, and let the next regression fail in a pull request instead of in production.