Every few weeks someone asks which model they should run their agent on. Our honest answer is “the one that passes your evals”, so we ran the experiment ourselves.

The setup

We took our refund-agent suite, 240 cases covering policy lookups, partial refunds, duplicate charges and angry customers, and ran it against five models with the same prompts, tools and sandbox. The baseline is the model the agent shipped with, at 78%.

The results

The best model passed 94% of cases. Three more landed between 82% and 86%. One dropped to 61%, almost entirely on cases that needed two tool calls in a row: it answered after the first call instead of checking the policy.

The surprise was cost. The cheapest model we tested scored 82%, better than the baseline at a fraction of the price per run. The most expensive wasn’t the top scorer.

Price tells you what a model costs. Your evals tell you what it’s worth.

What we’d do with this

Don’t pick a model from a leaderboard. Run your own suite, look at which cases fail and not just how many, and re-run it every time a provider ships an update. A model that’s right for your agent today can quietly stop being right next month.

The full suite and runner config are in the Halyard examples repo, so you can reproduce the numbers on your own account.