case study zero · our own agent
23.2% less per passed outcome.
One of our own agents, through the whole loop, after two merged switches: measure, test cheaper models on its evals, open a pull request, merge, measure again. Then the model it moved to was retired, and we did it again.
Captured 2026-10-17 from production
Cost per passed outcome on the agent’s 120-case eval suite: the baseline, then the latest full run on the merged code after each switch.
Source: provenance/case_study_zero.json, held to this page by tests/test_case_study_page.py.
the agent and its evals
One small agent, 120 evals, and a 97% bar.
The agent is oss_ladder/04_fanout/agent.py in our own repository, graded by 120 eval cases. A cheaper model is switched to only if, at 95% confidence, it is no more than 3 points behind the current one. Each refusal’s worst case, and the suite size that would settle it, are in the tables. Face-offs on the one-case smoke baseline are listed and decide nothing.
gemini-2.5-flash-lite retires on 2026-10-20. The agent moved off it before that date; switch 2 below is how.
switch 1 · pull request #21
gpt-4o-mini to gemini-2.5-flash-lite
Baseline run #26 on gpt-4o-mini passed 120 of 120, at $0.0000090 per passed outcome. Every face-off against it is below, failed ones included.
| Run | Candidate | Passed | Per task | Worst case behind, 95% | Suite that would settle it | Result |
|---|---|---|---|---|---|---|
| #27 | mistral-small-3.2-24b-instruct | 72 of 120 | $0.0000029 | 52.3 pts | none would | Refused |
| #28 | gemini-2.5-flash-lite | 120 of 120 | $0.0000072 | Trial: run it live | ||
| #29 | mistral-small-3.2-24b-instruct | 119 of 120 | $0.0000048 | 3.3 pts | ~240 cases | Trial: run it live |
gemini-2.5-flash-lite passed 120 of 120 at 19.5% less per task. Both passed every case, so the evals cannot tell them apart; Margin does not promote on no difference, and opened it for review instead.
Margin opened pull request #21 as a draft (1 line added, 1 removed, in oss_ladder/04_fanout/agent.py), merged 2026-10-09. The repository is private, so here is the whole change:
-MODEL = os.environ.get("MARGIN_LADDER_MODEL", "openrouter/openai/gpt-4o-mini")
+MODEL = os.environ.get("MARGIN_LADDER_MODEL", "openrouter/google/gemini-2.5-flash-lite")Show every run after the merge (11)
| Run after the merge | Outcome | Passed | Per passed outcome |
|---|---|---|---|
| #30 | failed: the install step could not find the image the run needs | none | none |
| #31 | measured gemini-2.5-flash-lite | 120 of 120 | $0.0000072 |
| #34 | failed: the worker has no microVM configured, so it will not run customer code on its own host | none | none |
| #35 | measured gemini-2.5-flash-lite | 120 of 120 | $0.0000072 |
| #38 | failed: its output held more than one result, so none was read | none | none |
| #40 | failed: its output held more than one result, so none was read | none | none |
| #41 | failed: its output held more than one result, so none was read | none | none |
| #42 | measured gemini-2.5-flash-lite | 1 of 1 | $0.0008693 |
| #46 | failed: the API refused the worker's clone token | none | none |
| #47 | measured gemini-2.5-flash-lite | 1 of 1 | $0.0008693 |
| #48 | measured gemini-2.5-flash-lite | 120 of 120 | $0.0000072 |
The prediction was $0.0000072. The 3 full runs on the merged code matched it exactly, on the same 120 cases. 6 runs failed; their reasons are shown above.
switch 2 · pull request #22
gemini-2.5-flash-lite to ministral-8b-2512
Baseline run #48 on gemini-2.5-flash-lite passed 120 of 120, at $0.0000072 per passed outcome. Every face-off against it is below, failed ones included.
| Run | Candidate | Passed | Per task | Worst case behind, 95% | Suite that would settle it | Result |
|---|---|---|---|---|---|---|
| #32 | mistral-small-3.2-24b-instruct | 118 of 120 | $0.0000048 | 4.2 pts | ~480 cases | Trial: run it live |
| #33 | mistral-small-3.2-24b-instruct | 119 of 120 | $0.0000048 | 3.3 pts | ~240 cases | Trial: run it live |
| #36 | mistral-small-3.2-24b-instruct | 119 of 120 | $0.0000048 | 3.3 pts | ~240 cases | Trial: run it live |
| #37 | mistral-small-3.2-24b-instruct | 119 of 120 | $0.0000048 | 3.3 pts | ~240 cases | Trial: run it live |
| #43 | claude-haiku-5.5 | none | none | Failed: the two runs graded different units | ||
| #44 | gemma-4-26b-a4b-it | 1 of 1 | $0.0009754 | Not cheaper | ||
| #45 | ministral-8b-2512 | 1 of 1 | $0.0008299 | Trial: run it live | ||
| #49 | gemma-4-26b-a4b-it | 1 of 1 | $0.0009693 | Not cheaper | ||
| #50 | ministral-8b-2512 | 1 of 1 | $0.0008299 | Trial: run it live | ||
| #51 | gemma-4-26b-a4b-it | none | none | Failed: timed out waiting for the result | ||
| #52 | ministral-8b-2512 | none | none | Failed: the two runs graded different units | ||
| #53 | gemma-4-26b-a4b-it | none | none | Failed: timed out sending its request | ||
| #54 | gemma-4-26b-a4b-it | none | none | Failed: timed out sending its request | ||
| #55 | ministral-8b-2512 | 120 of 120 | $0.0000069 | Trial: run it live | ||
| #56 | gemma-4-26b-a4b-it | none | none | Failed: timed out waiting for the result | ||
| #57 | gemma-4-26b-a4b-it | none | none | Failed: the two runs graded different units | ||
| #58 | ministral-8b-2512 | none | none | Failed: the two runs graded different units | ||
| #59 | gemma-4-26b-a4b-it | 120 of 120 | $0.0000080 | Not cheaper |
ministral-8b-2512 passed 120 of 120 at 4.5% less per task. Both passed every case, so the evals cannot tell them apart; Margin does not promote on no difference, and opened it for review instead.
Margin opened pull request #22 (1 line added, 1 removed, in oss_ladder/04_fanout/agent.py), merged 2026-10-10. The repository is private, so here is the whole change:
-MODEL = os.environ.get("MARGIN_LADDER_MODEL", "openrouter/google/gemini-2.5-flash-lite")
+MODEL = os.environ.get("MARGIN_LADDER_MODEL", "openrouter/mistralai/ministral-8b-2512")Show every run after the merge (3)
| Run after the merge | Outcome | Passed | Per passed outcome |
|---|---|---|---|
| #60 | measured ministral-8b-2512 | 120 of 120 | $0.0000069 |
| #84 | measured ministral-8b-2512 | 120 of 120 | $0.0000069 |
| #88 | measured ministral-8b-2512 | 120 of 120 | $0.0000069 |
The prediction was $0.0000069. The 3 full runs on the merged code matched it exactly, on the same 120 cases.
Why the second switch happened: OpenRouter lists google/gemini-2.5-flash-lite as retiring on 2026-10-20. The first face-offs chose candidates from a fixed list of older models, and that is how a retiring model was recommended. Candidates now come from the live model catalogue, newest first, and a model with a retirement date is never tried.
scope
Two switches on one small agent.
Margin opened each one as a pull request for review, not as a promotion: both models passed every case, so the evals could not tell them apart and a person chose. What it saves on your workload, Margin measures on your own runs.
Case study two is a different shape: our founder’s own agents, outside this repository, where every merged switch was measured again after its merge: savings that stayed above the bar, one the watch flagged, and one upgrade a person chose to pay for. Read it here.
Case study three runs the same loop on three open-source agents we did not write, graded by public tests: the model changes Margin held back, and one host change that measured cheaper. Read it here.
Three reviews the gate has run on real tasks: one promoted, two refused.
- 64% cheaperpromoted · the thinking budgetsame model, less thinking · paired eval, not live traffic · parity 1.06 over the 0.97 floor · 115 tasks
- 69% cheaperrefused · the tier cuta cheaper tier of the same house · quality fell 13.8 pts · 116 tasks
- 85% cheaperrefused · the rival modelDeepSeek against the Google incumbent · quality fell 29.7 pts · 91 tasks
Watch the loop run on real spend.
The console runs it on open-source agents we metered ourselves, every number tagged real or sample.