Agent benchmark

Can your agent run production?

We give AI agents real jobs on CLI (ship a repo, fix a failing deploy, stay inside a budget) and check what they leave behind. A run only passes if the job is done and no guardrail was broken on the way.

Leaderboard

No results are published yet. The method is below, and the harness is open: you can run it against any model today.

How it’s run

Same jobs, same rules, every agent.

  1. A fresh CLI every run

    Each run starts a new, simulated CLI server with the job's world already in it: a project, a repo, a broken deploy, a budget.

  2. The agent only gets the job

    It's given the same prompt a person would type and CLI's MCP tools. Nothing about how the check works.

  3. The end state is checked

    When the agent says it's done, the harness looks at what's actually running, set and spent. Saying it worked doesn't count.

  4. Guardrails fail the run

    Spending past its limit, deleting what it wasn't asked to, or getting around an approval fails the run, even if the job got done.

Ranking

Agents are ranked by the share of runs they passed. Ties go to the agent that broke fewer guardrails, then to the one that needed fewer steps. Cost is shown when the run recorded token prices; it doesn’t affect rank.

What it isn’t

These are CLI’s own jobs on a simulated CLI, written by the people who build CLI. They measure how well an agent operates this platform, not how good a model is in general.

Run it yourself

# in the CLI repository
npm install
ANTHROPIC_API_KEY=… OPENAI_API_KEY=… \
  npm run evals -- \
  --agent anthropic:<model> \
  --agent openai:<model> --runs 3

Every call, reply and failure lands in evals/results/. More in the benchmark guide.