case study three · open-source agents, public benchmarks
Margin tested 4 model changes on three open-source agents and recommended none of them as a switch. The one saving it measured kept the model and changed the host.
aider, crewAI and gptme are public projects we did not write. We ran them on public tests and paid for every run, so this is not a customer’s workload. The gate was allowed to say no, and for every model change it did, or held the change to a trial.
On gptme, pinning its own model to one host passed 146 of 157 specs against 144, on a bill 36% lower. That is one run each, so Margin recommends it as a trial with automatic revert, and says why below.
Measured 2026-11-01 to 2026-11-14 · every cost is the provider’s bill
gptme on 157 of its own eval specs, deepseek-v4.1-flash both times.
Scroll the table sideways to see every column.
| Model | Passed, each run | Pass rate | Billed | Per passed case, per thousand | What Margin said |
|---|---|---|---|---|---|
deepseek-v4.1-flash | 144 of 157 | 91.7% | $0.2182 | $1.515 | the baseline |
deepseek-v4.1-flash | 146 of 157 | 93.0% | $0.1391 | $0.953 | cleared the bar; recommended as a trial, not a switch |
Source: provenance/public_bench/gptme-base.json and gptme-hostpin.json.
the gate said no, or not yet
The model changes Margin held back
Each change swapped the model at the call site and left the agent’s code alone. A change passes when its worst plausible pass rate stays within the bar and it costs less per passed case. Passing the bar still earns a trial, not a switch, while the panel is this small.
aider 0.86.2 on 34 Python exercises from its own polyglot benchmark
Baseline gpt-4o, run 2 times; 4 cases flipped between its own runs, which is the noise every change is judged against.
Scroll the table sideways to see every column.
| Model | Passed, each run | Pass rate | Billed | Per passed case, per thousand | What Margin said |
|---|---|---|---|---|---|
gpt-4o | 8 / 6 of 34 | 20.6% | $2.8449 | $203.210 | the baseline |
gpt-4o-mini | 1 / 1 / 2 of 34 | 3.9% | $0.3339 | $83.464 | refused: cheaper per pass, and still not worth it |
crewAI 0.130.0 on the 164 problems of HumanEval+
Baseline gpt-4o-mini, run 3 times; 19 cases flipped between its own runs, which is the noise every change is judged against.
Scroll the table sideways to see every column.
| Model | Passed, each run | Pass rate | Billed | Per passed case, per thousand | What Margin said |
|---|---|---|---|---|---|
gpt-4o-mini | 130 / 136 / 132 of 164 | 80.9% | $0.0871 | $0.219 | the baseline |
gpt-4.1-nano | 138 / 140 / 136 of 164 | 84.1% | $0.0627 | $0.151 | cleared the bar; recommended as a trial, not a switch |
deepseek-v4-flash-0731 | 155 / 155 of 164 | 94.5% | $0.1512 | $0.488 | better and dearer: worth it only above a value per answer you set |
gptme on 157 of the 179 specs in its own gptme-eval suite
Baseline deepseek-v4.1-flash, run once, so its own run-to-run noise is not measured here.
Scroll the table sideways to see every column.
| Model | Passed, each run | Pass rate | Billed | Per passed case, per thousand | What Margin said |
|---|---|---|---|---|---|
deepseek-v4.1-flash | 144 of 157 | 91.7% | $0.2182 | $1.515 | the baseline |
deepseek-v4-flash-0731 | 142 of 157 | 90.4% | $0.1610 | $1.134 | held: its worst case falls past the bar |
nothing dropped
Every setup we started
Every arm in the tables above ran to completion. These we started and stopped, and none of them is in a table:
- aider on gpt-4o-mini, a first attempt: its command line pasted the web pages behind each exercise’s links into the prompt, which aider’s own harness does not do. We turned that off and ran it again.
- crewAI on Gemini 2.5 Flash Lite: every call failed in under a second, cause not found. We replaced it before any result came in.
- aider on DeepSeek V4 Flash, with reasoning on and then off: aider got no answer inside its own time limits, so both were stopped after the first few exercises.
- gptme pinned to one host: a short smoke check on a handful of specs before the full run.
A third agent, one that fixes real GitHub issues, was not run: its benchmark needs one x86 container per task, and a single pass would cost far more than this study’s budget.
scope
What this does not show
No company ran these agents for us, and no customer’s traffic is in any number here. The tests are public benchmarks, the settings are our choice, and the panels are small.
The host pin is one run against one run. The two differ in host and in ordinary run-to-run noise, and one pair cannot separate them. A change that is truly equal passes this gate on 157 specs about 57% of the time; about 314 would be needed for four in five, and gptme has only 157 that run without a browser or a desktop. More runs on the same specs do not help, so the honest next step is a live trial with automatic revert.
Your own tests on your own work are the only evidence that carries over. Case study zero is our own agent, and case study two is our founder’s.
reproduce
Run it yourself
The runner, the agents’ trees and every arm’s record are in our repository. Each arm runs on its own capped OpenRouter key, minted for the run and revoked after it.
python3 scripts/public_bench.py build
python3 scripts/public_bench.py arm --bench gptme_evals --label gptme-base --cap 0.8
python3 scripts/public_bench.py arm --bench gptme_evals --label gptme-hostpin \
--host deepinfra/fp8 --cap 0.35
python3 scripts/public_bench.py verdict provenance/public_bench/gptme-base.json \
provenance/public_bench/gptme-hostpin.jsonWatch the loop run on real spend.
The console runs it on open-source agents we metered ourselves, every number tagged real or sample.