OPEN SOURCE · 4.2K GITHUB STARS

Test your agents like you test your code.

Open-source evals for AI agents. Write them next to your code, run them in CI, and catch regressions before your users do.

Start free

See how it works

Halyard

Cloud

Platform

Overview

Runs

Evals

Traces

Manage

Guardrails

Alerts

CI checks

Settings

Dana Okafor

dana@acme.ai

Overview

refund-agent · last 30 days

Export

Run suite

Pass rate

94.2%

▲ 1.8 pts

Runs (30d)

1,284

▲ 212

Open regressions

▲ 2 since Mon

Calls blocked

of 48,310 checked

Eval runs per day

CI and scheduled runs, last 30 days

Last 30 days

Suite health

refund-agent · 48 evals

Passing

Up 2 evals since last week

3 failing: amount limit, tone, policy cite

Regressions caught

May – October 2026

Every one blocked before merge

Regressions flagged in CI, per month

Failures by cause

Last 30 days

Failures

Tool misuse down 18%

Top cause: wrong tool arguments

refund.eval.ts

evaluate("refund", async (t) => {

const run = await agent.run(ticket)

expect(run.tool).toBe("refund_order")

expect(run.amount).toBeLessThan(500)

// ✕ received 4200 (run #418)

t.judge(run.reply, "polite")

})

Recent runs

Run

Model

Passed

Duration

Halyard

Cloud

Platform

Overview

Runs

Evals

Traces

Manage

Guardrails

Alerts

CI checks

Settings

Dana Okafor

dana@acme.ai

Overview

refund-agent · last 30 days

Export

Run suite

Pass rate

94.2%

▲ 1.8 pts

Runs (30d)

1,284

▲ 212

Open regressions

▲ 2 since Mon

Calls blocked

of 48,310 checked

Eval runs per day

CI and scheduled runs, last 30 days

Last 30 days

Suite health

refund-agent · 48 evals

Passing

Up 2 evals since last week

3 failing: amount limit, tone, policy cite

Regressions caught

May – October 2026

Every one blocked before merge

Regressions flagged in CI, per month

Failures by cause

Last 30 days

Failures

Tool misuse down 18%

Top cause: wrong tool arguments

refund.eval.ts

evaluate("refund", async (t) => {

const run = await agent.run(ticket)

expect(run.tool).toBe("refund_order")

expect(run.amount).toBeLessThan(500)

// ✕ received 4200 (run #418)

t.judge(run.reply, "polite")

})

Recent runs

Run

Model

Passed

Duration

Halyard

Cloud

Platform

Overview

Runs

Evals

Traces

Manage

Guardrails

Alerts

CI checks

Settings

Dana Okafor

dana@acme.ai

Overview

refund-agent · last 30 days

Export

Run suite

Pass rate

94.2%

▲ 1.8 pts

Runs (30d)

1,284

▲ 212

Open regressions

▲ 2 since Mon

Calls blocked

of 48,310 checked

Eval runs per day

CI and scheduled runs, last 30 days

Last 30 days

Suite health

refund-agent · 48 evals

Passing

Up 2 evals since last week

3 failing: amount limit, tone, policy cite

Regressions caught

May – October 2026

Every one blocked before merge

Regressions flagged in CI, per month

Failures by cause

Last 30 days

Failures

Tool misuse down 18%

Top cause: wrong tool arguments

refund.eval.ts

evaluate("refund", async (t) => {

const run = await agent.run(ticket)

expect(run.tool).toBe("refund_order")

expect(run.amount).toBeLessThan(500)

// ✕ received 4200 (run #418)

t.judge(run.reply, "polite")

})

Recent runs

Run

Model

Passed

Duration

npm install halyard

refunds.eval.ts

TypeScript

import { evaluate, expect } from "halyard";

import { supportAgent } from "./agent";

evaluate("refunds stay within policy", async () => {

const reply = await supportAgent.run("Refund order 1182");

expect(reply).toCallTool("lookupOrder");

expect(reply).toSatisfy("never promises more than $200");

});

Runs locally, in CI, or against production traces.

Trusted by teams shipping agents to production

FEATURES

Everything you need to trust agents.

Halyard turns flaky manual testing into a suite you run on every pull request.

Start free

See how it works

4 evals ready

Run evals

evals/refund.eval.ts

import { evaluate } from "halyard"

evaluate("refund agent", async (t) => {

const run = await agent.run(ticket)

t.expect(run.tool).toBe("refund_order")

t.expect(run.amount).toBeLessThan(500)

t.judge(run.reply, "cites policy")

})

Check

Result

Time

Uses refund_order

Queued

—

Amount < $500

Queued

—

Cites policy

Queued

—

No PII in reply

Queued

—

4 checks

0 run

—

4 evals ready

Run evals

evals/refund.eval.ts

import { evaluate } from "halyard"

evaluate("refund agent", async (t) => {

const run = await agent.run(ticket)

t.expect(run.tool).toBe("refund_order")

t.expect(run.amount).toBeLessThan(500)

t.judge(run.reply, "cites policy")

})

Check

Result

Time

Uses refund_order

Queued

—

Amount < $500

Queued

—

Cites policy

Queued

—

No PII in reply

Queued

—

4 checks

0 run

—

Evals as code

Write evals in TypeScript or Python and run them on every pull request.

Pass rate vs. last green build

refund-agent · last 6 weeks

50%

100%

Regression flagged in week 4

Slack alert sent to #agents

Pass rate vs. last green build

refund-agent · last 6 weeks

50%

100%

Regression flagged in week 4

Slack alert sent to #agents

Regression alerts

Every run is checked against the last green build.

Trace 8f2c · span durations

production · failed at refund_order

plan

policy

refund

log

Replay it in your suite in one command

Hover to label each span

Trace 8f2c · span durations

production · failed at refund_order

plan

policy

refund

log

Replay it in your suite in one command

Hover to label each span

Trace replay

Replay failing production traces step by step.

frontier-large vs frontier-mini

Pass rate by eval, 48 cases

tool

args

tone

policy

safety

cost

mini scores within 2 pts at 16% of the cost

Light: large · Dark: mini

frontier-large vs frontier-mini

Pass rate by eval, 48 cases

tool

args

tone

policy

safety

cost

mini scores within 2 pts at 16% of the cost

Light: large · Dark: mini

Model comparisons

Pass rate, cost and latency, side by side.

Tool calls checked

Last 30 days · 48,310 calls

Blocked

0 unsafe refunds reached production

Highlighted: blocked refunds over $500

Tool calls checked

Last 30 days · 48,310 calls

Blocked

0 unsafe refunds reached production

Highlighted: blocked refunds over $500

Runtime guardrails

Block unsafe tool calls before they run.

HOW IT WORKS

From prompt change to green build in three steps.

No new dashboard to learn. Halyard runs where your tests already run.

Write

Describe what good looks like: the tools an agent should call and the lines it must never cross.

halyard init

$ halyard init

✓ created halyard.config.ts

✓ created evals/refund.eval.ts

✓ added .github/workflows/halyard.yml

Write

Describe what good looks like: the tools an agent should call and the lines it must never cross.

halyard init

$ halyard init

✓ created halyard.config.ts

✓ created evals/refund.eval.ts

✓ added .github/workflows/halyard.yml

Run

Run the suite locally or on every pull request. Failing cases link straight to the trace.

halyard run --ci

PR #212 · Tighten refund prompt

✓

build

1m 12s

halyard / evals

47/48

✓

lint

Run

Run the suite locally or on every pull request. Failing cases link straight to the trace.

halyard run --ci

PR #212 · Tighten refund prompt

✓

build

1m 12s

halyard / evals

47/48

✓

lint

Guard

Ship the same checks as runtime guardrails, so production follows the rules your tests prove.

halyard.guard(agent)

agent

halyard.guard

tools

✓

312 tool calls checked today

Guard

Ship the same checks as runtime guardrails, so production follows the rules your tests prove.

halyard.guard(agent)

agent

halyard.guard

tools

312 tool calls checked today

Guard

Ship the same checks as runtime guardrails, so production follows the rules your tests prove.

halyard.guard(agent)

agent

halyard.guard

tools

✓

312 tool calls checked today

WHY HALYARD

Built for engineers, not for another dashboard.

How Halyard compares with the two ways teams test agents today.

Halyard

Hand-rolled scripts

Hosted eval platforms

Lives next to your code

Yes, in your repo

Yes

No, in a web app

Runs on every pull request

Two-line GitHub Action

If you build it

Paid add-on

Runtime guardrails

Same checks, built in

Separate product

Self-hosting

Yes, open source

Yes

Enterprise plan only

Cost to start

Free

Engineer time

Per-seat pricing

FROM TEAMS USING HALYARD

What they’re saying

Jack Kim

CTO · Ledgerbird

“We caught a refund bug in CI that would have cost us a customer. That one run paid for the setup.”

Priya Johnson

Founding engineer · Tessel

“Our evals finally live next to the agent code, so they get reviewed like everything else.”

Elena Okafor

Head of AI · Northpass

“Swapping models used to take a week of manual checks. Now it is one comparison report.”

Sean Lee

Staff engineer · Quayside

“The guardrails reuse our test cases. Same rules in CI and in production, no drift.”

Steve Cole

CEO · Brightloop

“Two lines in GitHub Actions and every prompt change gets a pass rate before it merges.”

Tomás Ruiz

Platform lead · Kitebase

“Trace replay turned our worst production failures into permanent regression tests.”

PRICING

Free to start. Pay when your team does.

The open-source core stays free. Cloud adds history, replay and alerts.

FREE

Open source

Unlimited local eval runs

TypeScript and Python SDKs

GitHub Actions integration

Runtime guardrails

Community support

$0

/ forever

Get started

PLUS

Cloud

Everything in Open source

Hosted run history

Trace replay from production

Model comparison reports

Slack alerts

Team workspaces

30-day trace retention

Email support

Cancel anytime

$49

/ month

Start free trial

Need SSO, self-hosted Cloud or custom retention? Talk to us about Enterprise.

BLOG

Notes from the eval trenches.

Guides, benchmarks and release notes from the Halyard team.

All posts

GUIDE

Oct 21, 2026

How to test a tool-calling agent without mocking everything

A practical pattern for evals that hit real tools in a sandbox and still run in under a minute.

GUIDE

Oct 21, 2026

How to test a tool-calling agent without mocking everything

A practical pattern for evals that hit real tools in a sandbox and still run in under a minute.

BENCHMARK

Oct 9, 2026

Five models, one refund suite: what changed in pass rate

We ran the same 240 cases across five models. The cheapest one was not the worst.

BENCHMARK

Oct 9, 2026

Five models, one refund suite: what changed in pass rate

We ran the same 240 cases across five models. The cheapest one was not the worst.

RELEASE

Sep 30, 2026

Halyard 1.4: trace replay and runtime guardrails

Turn production failures into tests, then ship the same checks as guards.

RELEASE

Sep 30, 2026

Halyard 1.4: trace replay and runtime guardrails

Turn production failures into tests, then ship the same checks as guards.

Questions,

answered.

Still unsure? Ask the people who build Halyard. We reply within a day.

Ask the team

Does Halyard work with my agent framework?

Yes. Halyard wraps any function that takes input and returns output, so it works with LangGraph, the OpenAI Agents SDK, Vercel AI SDK or plain code.

Where do my prompts and traces go?

How is this different from unit tests?

What do the evals cost to run?

Can I self-host Cloud?

FREE AND OPEN SOURCE

Ship agents you can trust.

Add your first eval in five minutes. Free and open source, with Cloud when your team needs it.

Start free

Star on GitHub

No pressure. Just the next step.

Halyard

Open-source evals and guardrails for AI agents.

Product

Resources

Company

Stay in the loop

Release notes and eval guides, twice a month.

No spam. Unsubscribe anytime.

Halyard

Open-source evals and guardrails for AI agents.

Product

Resources

Company

Stay in the loop

Release notes and eval guides, twice a month.

No spam. Unsubscribe anytime.

Halyard

Open-source evals and guardrails for AI agents.

Product

Resources

Company

Stay in the loop

Release notes and eval guides, twice a month.

No spam. Unsubscribe anytime.