case study two · our founder’s own agents
One merged switch made a passed case 67.9% cheaper, and each of its 51 scheduled evals since scored at or above the bar.
It is a personal agent set: Python, OpenRouter, four agents, a 100-case eval suite, built by our founder for his own use, so not a customer’s. Switch #12 is one of 4 that merged on it. Its cost figure is one run before and one run after, on the same 243 cases; the evals since are the house’s own scheduled runs, each read on its point estimate, and the range is in the table below, beside the switch the watch flagged. The first switch, which cost more, is further down.
Captured 2026-11-13 from production
Cost per passed case on the same 243 eval cases, before the switch and after it: 214 passed before, 213 after.
Source: provenance/pov_house.json, derived from the production capture.
proof of value · every merged switch
What each switch did to the cost of a passed case
Cost per passed case is a run’s bill divided by its passes, for the whole agent set, not only the step that changed. Rows are ordered by what the numbers show: savings that held first, then savings the watch flagged, then upgrades a person chose. Captured 2026-11-13 from production.
Scroll the table sideways to see every column.
| Switch | Cost per passed case, before to after | Passed | Re-measured since merge |
|---|---|---|---|
| #12 | claude-sonnet-5.5 to deepseek-v4.1-flash$1008.46 to $324.18 per million, −67.9% Cheaper per passed outcome, and the re-run's pass rate was at or above the bar (point estimate) | 214 to 213 of 243 (99.5% of before) new model’s rate 83.8% to 90.7% | 51 scheduled evals since merge, 0 below the bar; latest 96.0% against 77.6% on 2026-11-13 |
| #13 | gpt-6.1-sol to deepseek-v4.1-flash$758.38 to $306.13 per million, −59.6% Cheaper per passed outcome, and the re-run's pass rate was at or above the bar (point estimate) | 240 to 238 of 269 (99.2% of before) new model’s rate 84.9% to 91.3% | 44 scheduled evals since merge, 1 below the bar; latest 100.0% against 97.0% on 2026-11-13 |
| #7 | claude-sonnet-5.5 to claude-haiku-5.5$2841.26 to $324.18 per million, −88.6% Cheaper, but the re-run passed fewer cases than before: the watch flagged it | 226 to 213 of 243 (94.2% of before) new model’s rate 83.8% to 90.7% | 65 scheduled evals since merge, 40 below the bar; latest 84.4% against 83.2% on 2026-11-13 |
| #6 | deepseek-v4.1-flash to mistral-large-4-0$80.02 to $399.07 per million, +398.7% A deliberate upgrade: passed more, and cost more | 88 to 91 of 100 (103.4% of before) | a face-off switch is read by runs, named in basis |
Whether a merged change was later reverted is not in the record, so none is claimed or ruled out. The bar is 97% of the before pass rate.
the first switch · pull request #6
It cost 5.0× as much per passed case, for a few more passing evals: a direction, not a proven gain. A person chose to pay it.
For its first switch, Margin tested candidate models on the suite as it was then (100 cases), switched to none of the cheaper ones in that round, and opened a pull request for the one that scored highest.
Cost per passed case on the agents’ 100-case eval suite: the baseline, then each run on the merged code whose bill matches its own calls.
Source: provenance/case_study_house.json, captured from production and GitHub.
the agents and their evals
A 100-case suite, and a bar only a cheaper model has to clear.
The agents call models through OpenRouter, and their own eval suite grades 100 cases per run. They run on one person’s private life, and the repository is private, so this page names no agent, no suite and no file.
What the cases are made of, in rough counts of distinct cases across the five suites the paid runs grade: about 110 are real. Roughly 37 are real repair decisions (25 as the agents logged them, 12 with the failing result swapped for an outside-service error the agents had really logged) and roughly 73 are real error and log lines whose labels were set by hand, a few of which may have been written rather than logged. Another 30 are synthetic, with invented people and no real messages, and about 20 are hand-written. The case counts elsewhere on this page grade each case once per repeat, so they are larger than the number of distinct cases. Written cases can be easier than real ones, so read the synthetic and hand-written share with that in mind.
Margin switches to a cheaper model on its own only if it keeps at least 97% of the current pass rate and the result is strong enough to rule out luck. A model that costs more is never switched to automatically: that trade belongs to a person.
every face-off · baseline run #151
10 face-offs against deepseek-v4.1-flash
Baseline run #151 on deepseek/deepseek-v4.1-flash passed 88 of 100, at $0.000080 per passed case. Every face-off against it is below, including the ones that failed before they measured anything.
Scroll the table sideways to see every column.
| Run | Candidate | Passed | Per task | Result |
|---|---|---|---|---|
| #152 | deepseek-v4-flash-0731 | 85 of 100 | $0.000037 | Refused |
| #153 | gemma-4-26b-a4b-it | 86 of 100 | $0.000025 | Refused |
| #154 | mistral-large-4-0 | none | none | Failed: the worker stopped mid-run; its spend could not be read |
| #155 | deepseek-v4-flash-0731 | none | none | Failed: the two runs graded different units |
| #156 | deepseek-v4-flash | 78 of 100 | $0.000002 | Refused |
| #157 | gemma-4-26b-a4b-it | 88 of 100 | $0.000058 | Trial: run it live |
| #158 | mistral-large-4-0 | none | none | Failed: the worker stopped mid-run; its spend could not be read |
| #159 | mistral-large-4-0 | none | none | Failed: the worker stopped mid-run; its spend could not be read |
| #160 | mistral-large-4-0 | 96 of 100 | $0.000581 | Trial: run it live |
| #161 | ministral-3b-2512 | 74 of 100 | $0.000043 | Refused |
mistral-large-4-0 passed 96 of 100 at $0.000581 per task, against $0.000070 for deepseek-v4.1-flash. Margin opened pull request #6 for it as a draft, and our founder merged it on 2026-10-16.
after the merge · pull request #6
Every run on the merged code
Show every run after the merge (2)
Scroll the table sideways to see every column.
| Run | Outcome | Passed | Per passed case |
|---|---|---|---|
| #166 | measured mistral-large-4-0, mistral-large-4-0-20261006 | 94 of 100 | not quoted |
| #209 | measured mistral-large-4-0-20261006 | 91 of 100 | $0.000399 |
Run #166’s recorded bill ($0.019834) is far below what its own metered calls cost ($0.058828): the key’s usage counter read low. Its cost is not quoted, rather than quoting the low figure.
After the merge, run #209 cost 5.0× the baseline per passed case, lower than the face-off’s 8.2× per task. One visible difference is caching: run #209 read 31,488 prompt tokens from the provider’s cache, the face-off 1,968. Pass counts on 100 cases move by a few points from run to run, so read 94 and 91 against 88 as a direction, not a precise gain.
after the merge · pull request #12
Switch #12, re-measured after the merge
One step moved from claude-sonnet-5.5 to deepseek-v4.1-flash. Our founder opened pull request #12 and merged it on 2026-10-31. After that, 4 commits to the house’s own code changed what that step’s evals measure, so Margin labelled the scheduled re-tests confounded instead of blaming the model.
Margin re-ran both models on one commit and the same 243 cases: deepseek-v4.1-flash passed 213 and claude-sonnet-5.5 passed 214. That is no measurable drop, so Margin keeps the switch. It is not called held: 243 cases are too few to show the new model within the 97% bar, and the old one could still be a few points better. Margin calls a switch held only when that bound clears the bar, or when the scheduled evals after the merge keep their median above it with nothing else changed.
Since the switch, the 1846 live calls on that step cost $1.02; the same calls’ tokens priced at claude-sonnet-5.5’s published rates come to $22.63, which is not exactly what it would have spent. The face-off predicted $0.000146 a call; live calls ran $0.000551, so the prediction was 3.8× too low.
| Measure | deepseek-v4.1-flash | claude-sonnet-5.5 |
|---|---|---|
| Passed, re-run #490 against #482 | 213 of 243 | 214 of 243 |
| The 1846 live calls since the switch | $1.02 | $22.63 |
after the merge · pull request #13
Switch #13, re-measured after the merge
One step moved from gpt-6.1-sol to deepseek-v4.1-flash. Margin opened pull request #13, and our founder merged it on 2026-11-02. After that, 8 commits to the house’s own code changed what that step’s evals measure, so Margin labelled the scheduled re-tests confounded instead of blaming the model.
Margin re-ran both models on one commit and the same 269 cases: deepseek-v4.1-flash passed 238 and gpt-6.1-sol passed 240. That is no measurable drop, so Margin keeps the switch. It is not called held: 269 cases are too few to show the new model within the 97% bar, and the old one could still be a few points better. Margin calls a switch held only when that bound clears the bar, or when the scheduled evals after the merge keep their median above it with nothing else changed.
Since the switch, the 2618 live calls on that step cost $0.68; the same calls’ tokens priced at gpt-6.1-sol’s published rates come to $20.27, which is not exactly what it would have spent.
| Measure | deepseek-v4.1-flash | gpt-6.1-sol |
|---|---|---|
| Passed, re-run #519 against #516 | 238 of 269 | 240 of 269 |
| The 2618 live calls since the switch | $0.68 | $20.27 |
being tested now
Face-offs against the code on mistral-large-4-0
8 face-offs so far, the switch back included; none switched to. 7 of them stopped before measuring anything; each one's reason is in the table.
Show every face-off since (8)
Scroll the table sideways to see every column.
| Run | Candidate | Passed | Per task | Result |
|---|---|---|---|---|
| #167 | deepseek-v4.1-flash | 88 of 100 | $0.000087 | Refused |
| #168 | claude-haiku-5.5 | none | none | Failed: the candidate left 6 of the 55 cases the current model graded without a verdict, or graded fewer times |
| #169 | gpt-6-luna-pro | none | none | Failed: the worker stopped mid-run; its spend could not be read |
| #210 | decision-model | none | none | Failed: the harnesses ran but none gave a pass or fail |
| #211 | claude-haiku-5.5 | none | none | Failed: the candidate left 8 of the 55 cases the current model graded without a verdict, or graded fewer times |
| #212 | gpt-6-luna-pro | none | none | Failed: the candidate left 3 of the 55 cases the current model graded without a verdict, or graded fewer times |
| #213 | deepseek-v4.1-flash-1017 | none | none | Failed: the candidate left 9 of the 55 cases the current model graded without a verdict, or graded fewer times |
| #214 | deepseek-v4.1-flash-1017 | none | none | Failed: the candidate left 16 of the 55 cases the current model graded without a verdict, or graded fewer times |
Switching back to deepseek-v4.1-flash passed 88 of 100, below the bar, so Margin did not recommend undoing the change.
scope
4 merged switches, on one set of agents.
Who opened each pull request in this record, read from the record’s author field:
| Pull request | Opened by | Moved |
|---|---|---|
| #6 | Margin’s app, as a draft (a trial) | deepseek-v4.1-flash to mistral-large-4-0 |
| #7 | the founder | claude-sonnet-5.5 to claude-haiku-5.5 |
| #12 | the founder | claude-sonnet-5.5 to deepseek-v4.1-flash |
| #13 | Margin’s app | gpt-6.1-sol to deepseek-v4.1-flash |
One more switch was re-measured the same way and scored lower, under the 97% bar on the point estimate, so Margin could not show it held. Its re-run could not pin the drop on the model either, so Margin did not revert it.
This is one person’s own set of personal-assistant agents, not a team’s production system, and the numbers are small. It shows the method on a real workload; it does not stand in for how a team’s system would behave.
Margin has watched this project since 27 October 2026. In the first 11 days its scheduled evals ran 23 times, 21 of them without error, and calls reached Margin on every one of those days.
None of them passed Margin’s promote gate: each merge was our founder’s call, and Margin re-measured them afterwards. What a larger workload would save or spend, Margin measures on its own runs.
To apply what this project measured to your own monthly spend, as ranges and with each rate dated, use the projection.
Case study zero is the other shape: our own agent, where each switch went to a cheaper model. Read it here.
Watch the loop run on real spend.
The console runs it on open-source agents we metered ourselves, every number tagged real or sample.