WORKLOAD BENCHMARKING

Find the route that wins on your workload.

Run the tests already in your repository against the same model across supported routers and direct providers. Compare completion, cost per completed task and latency before you move production traffic.

Benchmark my workload → View an example report

Runs on your machine · static HTML report · no CanaryOne account

Your repository

your-repo/

tests that call an LLM
4
model under test
Kimi K3
repeats each
3

The same tests, every route

10 routes

vercel
4
direct
3
openrouter
3

120 sessions · 67m11s

What it cost to finish

route done $ / done
Route A 6/12 $0.0472
Route B 9/12 $0.0686
Route C 12/12 $0.1852

and 7 more in the full report →

MEASURED WITH CANARYONE

Token price is not
workload cost.

Buy the same model several different ways and it does not do the same work for the same money. Three routes out of one run below, on identical tasks and identical weights, with the measures a price list cannot show you.

The lowest-cost route completed only six of twelve attempts.

Three of the 10 routes measured in run e860167a, showing attempts completed, cost per completed task, judge score and slowest step.
On this workload Route ARoute BRoute C
Attempts completed 6 of 129 of 1212 of 12
Cost per completed task $0.0472$0.0686$0.1852
Judge score, out of 100 698788
Slowest step 304s26s29s

Run e860167a · 29 July 2026 · 10 routes · 4 tasks · 3 repeats each · open the full report →

The cheapest route that completed most of the work cost 46 per cent more.

This is one measured run, not a claim about any provider.

How to read this result

Across the three routes, the dearest cost per completed task is nearly four times as much as the cheapest, on identical work.

The judge score. A second model reads each session transcript and scores the work out of a hundred. A high cost beside a high score is a route doing the job thoroughly; a low score beside a pass is a test that succeeded on an answer the model narrated rather than grounded in what its tools returned.

The timeout. The runner stops a session after six minutes, and the slowest step in this run ran to five minutes. Some of what the lowest-cost route did not finish may be that timeout rather than the route.

Why the routes are lettered. One run of one tool against one repository on 29 July 2026 demonstrates what the measurement produces; it does not establish anything about a company. The size of a gap like this repeats from run to run and which route sits at either end does not, so the magnitude is the finding. The report names all ten.

Also measured · 14 August 2026 On the nightly sweep, the route with the cheaper advertised token price cost 40 per cent more per completed task, and both routes completed every task. Eleven routes of GLM 5.2. See the market page →

HOW IT WORKS

Your tests in.
Comparable routes out.

  1. 01

    Point CanaryOne at your repository

    It runs against any repository whose tests or eval harness call an LLM, using the tests you already trust rather than a benchmark you have to adopt.

  2. 02

    Replay them across routes

    The same tests and the same model weights go through each router and direct provider you configure. Your test code does not change; only the route does.

  3. 03

    Compare the outcome

    One report carries completion, cost per completed task, latency and the task-by-task failures underneath them.

THE REPORT

See why one route won.

Every run writes a self-contained report showing how each route performed across your actual tasks, so a decision comes with its own working.

Report view

run e860167a · Kimi K3 · 10 routes · 4 tasks

Open example report →

Rank every route by cost per completed task, with completion, judge score and latency beside it.

The lane table from a CanaryOne report: one row per route, with completion, cost per pass, judge score, weighted cost per pass and latency percentiles.

See exactly which tasks each route failed, rather than only how often it failed.

The heatmap from a CanaryOne report: one row per route, one column per task, each cell carrying that cell’s cost and shaded by it.

Read the judge’s verdict on a single attempt, its four sub-scores and the test’s own output.

One expanded session from a CanaryOne report: cost, duration and step count, four judge sub-scores, the judge’s written reasoning, and the tail of the test process’s output.

Saved as <repo>/.c1/runs/<runId>/report/index.html — one static file with no external requests, so it opens from disk and travels in a pull request.

WHAT LEAVES

Runs on your machine.
See exactly what leaves.

The runner, the per-route proxy, the SQLite database and the generated report all stay on your machine, and nothing sits in your request path.

Your machine

Your tests

the harness you already have

CanaryOne proxy

one per route, on localhost

SQLite + report

under <repo>/.c1

outbound only

Leaves your machine

Model requests
the providers you configure
Session transcripts
the judge, to be scored — optional, and --disable-judge turns it off
Your test files
the scan summariser, which works out which of them call an LLM
Exactly what leaves your machine

Model traffic. Every LLM call your test makes is rewritten for the route under test and forwarded to that provider's own endpoint. The full request and response log is written to <repo>/.c1/runs/<runId>/traffic.jsonl, so what was sent is on disk and readable.

The judge. When judging is on, each finished session's transcript is sent to a gateway to be scored out of a hundred by a second model. --disable-judge turns it off, and the run then reports completion and cost with no quality score.

The scan summariser. Working out which of your tests call an LLM sends those test files to the same gateway to be classified.

Your keys. They live in ~/.c1/.env at mode 0600 and go to each provider's own endpoint and nowhere else. An OpenRouter key is required, because the judge and the model catalogue both run through it.

COVERAGE

Across routers and direct providers.

3 routing platforms · 14 direct APIs

Benchmark the same model through a gateway or straight against a provider's own endpoint, and measure both the same way.

Routing platforms · works with

  • OpenRouter

  • Vercel AI Gateway

  • AWS Bedrock

Direct providers · works with

  • Moonshot Intl

  • Moonshot CN

  • Nebius

  • Fireworks AI

  • Together AI

  • Groq

  • DeepSeek

  • Baseten

  • Cerebras

  • OpenAI

  • Anthropic

  • xAI

  • Z.ai

  • Google Gemini

Coverage note: Bedrock currently carries gpt-oss routes only, and direct-provider prices are maintained here rather than read from each provider's API. See the route registry →

GET STARTED

Run it on your repository.

$ npx canaryone

Read the quickstart →

EARLY ACCESS

Get early access.

We'll reach out when early access opens.