case study zero · our own agent

23.2% less per passed outcome.

One of our own agents, through the whole loop, after two merged switches: measure, test cheaper models on its evals, open a pull request, merge, measure again. Then the model it moved to was retired, and we did it again.

Captured 2026-10-17 from production

Each switch made a passed outcome cheaper, and every eval still passed.Figure 1

Cost per passed outcome on the agent’s 120-case eval suite: the baseline, then the latest full run on the merged code after each switch.

Before gpt-4o-mini$9.00 per million · 120/120 passed · run #26After switch 1 gemini-2.5-flash-lite$7.24 per million · 120/120 passed · run #48After switch 2 ministral-8b-2512$6.92 per million · 120/120 passed · run #88
n = 120 eval cases per barruns #26, #48, #88, each priced from its own billscaled per million passed outcomes; the tables below give the raw figurereal runs, not simulated · captured 2026-10-17

Source: provenance/case_study_zero.json, held to this page by tests/test_case_study_page.py.

$6.92per million passed outcomes today ($0.0000069 each)run #88 · 120 of 120 passed
23.2%less than the $9.00 per million it started atrun #26 · gpt-4o-mini
21face-offs: 2 shipped, 1 refused, 7 trials not taken, 3 not cheaper, 8 failed before measuringevery one listed below
4lines changed, across two pull requests Margin opened and a person merged#21 · #22

the agent and its evals

One small agent, 120 evals, and a 97% bar.

The agent is oss_ladder/04_fanout/agent.py in our own repository, graded by 120 eval cases. A cheaper model is switched to only if, at 95% confidence, it is no more than 3 points behind the current one. Each refusal’s worst case, and the suite size that would settle it, are in the tables. Face-offs on the one-case smoke baseline are listed and decide nothing.

gemini-2.5-flash-lite retires on 2026-10-20. The agent moved off it before that date; switch 2 below is how.

switch 1 · pull request #21

gpt-4o-mini to gemini-2.5-flash-lite

Baseline run #26 on gpt-4o-mini passed 120 of 120, at $0.0000090 per passed outcome. Every face-off against it is below, failed ones included.

RunCandidatePassedPer taskWorst case behind, 95%Suite that would settle itResult
#27mistral-small-3.2-24b-instruct72 of 120$0.000002952.3 ptsnone wouldRefused
#28gemini-2.5-flash-lite120 of 120$0.0000072Trial: run it live
#29mistral-small-3.2-24b-instruct119 of 120$0.00000483.3 pts~240 casesTrial: run it live

gemini-2.5-flash-lite passed 120 of 120 at 19.5% less per task. Both passed every case, so the evals cannot tell them apart; Margin does not promote on no difference, and opened it for review instead.

Margin opened pull request #21 as a draft (1 line added, 1 removed, in oss_ladder/04_fanout/agent.py), merged 2026-10-09. The repository is private, so here is the whole change:

-MODEL = os.environ.get("MARGIN_LADDER_MODEL", "openrouter/openai/gpt-4o-mini")
+MODEL = os.environ.get("MARGIN_LADDER_MODEL", "openrouter/google/gemini-2.5-flash-lite")
Show every run after the merge (11)
Run after the mergeOutcomePassedPer passed outcome
#30failed: the install step could not find the image the run needsnonenone
#31measured gemini-2.5-flash-lite120 of 120$0.0000072
#34failed: the worker has no microVM configured, so it will not run customer code on its own hostnonenone
#35measured gemini-2.5-flash-lite120 of 120$0.0000072
#38failed: its output held more than one result, so none was readnonenone
#40failed: its output held more than one result, so none was readnonenone
#41failed: its output held more than one result, so none was readnonenone
#42measured gemini-2.5-flash-lite1 of 1$0.0008693
#46failed: the API refused the worker's clone tokennonenone
#47measured gemini-2.5-flash-lite1 of 1$0.0008693
#48measured gemini-2.5-flash-lite120 of 120$0.0000072

The prediction was $0.0000072. The 3 full runs on the merged code matched it exactly, on the same 120 cases. 6 runs failed; their reasons are shown above.

switch 2 · pull request #22

gemini-2.5-flash-lite to ministral-8b-2512

Baseline run #48 on gemini-2.5-flash-lite passed 120 of 120, at $0.0000072 per passed outcome. Every face-off against it is below, failed ones included.

RunCandidatePassedPer taskWorst case behind, 95%Suite that would settle itResult
#32mistral-small-3.2-24b-instruct118 of 120$0.00000484.2 pts~480 casesTrial: run it live
#33mistral-small-3.2-24b-instruct119 of 120$0.00000483.3 pts~240 casesTrial: run it live
#36mistral-small-3.2-24b-instruct119 of 120$0.00000483.3 pts~240 casesTrial: run it live
#37mistral-small-3.2-24b-instruct119 of 120$0.00000483.3 pts~240 casesTrial: run it live
#43claude-haiku-5.5nonenoneFailed: the two runs graded different units
#44gemma-4-26b-a4b-it1 of 1$0.0009754Not cheaper
#45ministral-8b-25121 of 1$0.0008299Trial: run it live
#49gemma-4-26b-a4b-it1 of 1$0.0009693Not cheaper
#50ministral-8b-25121 of 1$0.0008299Trial: run it live
#51gemma-4-26b-a4b-itnonenoneFailed: timed out waiting for the result
#52ministral-8b-2512nonenoneFailed: the two runs graded different units
#53gemma-4-26b-a4b-itnonenoneFailed: timed out sending its request
#54gemma-4-26b-a4b-itnonenoneFailed: timed out sending its request
#55ministral-8b-2512120 of 120$0.0000069Trial: run it live
#56gemma-4-26b-a4b-itnonenoneFailed: timed out waiting for the result
#57gemma-4-26b-a4b-itnonenoneFailed: the two runs graded different units
#58ministral-8b-2512nonenoneFailed: the two runs graded different units
#59gemma-4-26b-a4b-it120 of 120$0.0000080Not cheaper

ministral-8b-2512 passed 120 of 120 at 4.5% less per task. Both passed every case, so the evals cannot tell them apart; Margin does not promote on no difference, and opened it for review instead.

Margin opened pull request #22 (1 line added, 1 removed, in oss_ladder/04_fanout/agent.py), merged 2026-10-10. The repository is private, so here is the whole change:

-MODEL = os.environ.get("MARGIN_LADDER_MODEL", "openrouter/google/gemini-2.5-flash-lite")
+MODEL = os.environ.get("MARGIN_LADDER_MODEL", "openrouter/mistralai/ministral-8b-2512")
Show every run after the merge (3)
Run after the mergeOutcomePassedPer passed outcome
#60measured ministral-8b-2512120 of 120$0.0000069
#84measured ministral-8b-2512120 of 120$0.0000069
#88measured ministral-8b-2512120 of 120$0.0000069

The prediction was $0.0000069. The 3 full runs on the merged code matched it exactly, on the same 120 cases.

Why the second switch happened: OpenRouter lists google/gemini-2.5-flash-lite as retiring on 2026-10-20. The first face-offs chose candidates from a fixed list of older models, and that is how a retiring model was recommended. Candidates now come from the live model catalogue, newest first, and a model with a retirement date is never tried.

scope

Two switches on one small agent.

Margin opened each one as a pull request for review, not as a promotion: both models passed every case, so the evals could not tell them apart and a person chose. What it saves on your workload, Margin measures on your own runs.

Case study two is a different shape: our founder’s own agents, outside this repository, where every merged switch was measured again after its merge: savings that stayed above the bar, one the watch flagged, and one upgrade a person chose to pay for. Read it here.

Case study three runs the same loop on three open-source agents we did not write, graded by public tests: the model changes Margin held back, and one host change that measured cheaper. Read it here.

Three reviews the gate has run on real tasks: one promoted, two refused.

  • 64% cheaperpromoted · the thinking budgetsame model, less thinking · paired eval, not live traffic · parity 1.06 over the 0.97 floor · 115 tasks
  • 69% cheaperrefused · the tier cuta cheaper tier of the same house · quality fell 13.8 pts · 116 tasks
  • 85% cheaperrefused · the rival modelDeepSeek against the Google incumbent · quality fell 29.7 pts · 91 tasks

Case study: our own agent, before and after the merge

Watch the loop run on real spend.

The console runs it on open-source agents we metered ourselves, every number tagged real or sample.