case study three · open-source agents, public benchmarks

Margin tested 4 model changes on three open-source agents and recommended none of them as a switch. The one saving it measured kept the model and changed the host.

aider, crewAI and gptme are public projects we did not write. We ran them on public tests and paid for every run, so this is not a customer’s workload. The gate was allowed to say no, and for every model change it did, or held the change to a trial.

On gptme, pinning its own model to one host passed 146 of 157 specs against 144, on a bill 36% lower. That is one run each, so Margin recommends it as a trial with automatic revert, and says why below.

Measured 2026-11-01 to 2026-11-14 · every cost is the provider’s bill

Same model, one host: the bill fell, and the pass count did not.Figure 1

gptme on 157 of its own eval specs, deepseek-v4.1-flash both times.

Scroll the table sideways to see every column.

ModelPassed, each runPass rateBilledPer passed case, per thousandWhat Margin said
deepseek-v4.1-flash144 of 15791.7%$0.2182$1.515the baseline
deepseek-v4.1-flash146 of 15793.0%$0.1391$0.953cleared the bar; recommended as a trial, not a switch
n = 157 specs, one run per arm151 tied, 2 passed only on any host, 4 only on the pinned oneworst plausible case 1.3% more failures, against the 2.8% the gate allowshost deepinfra/fp8 · real runs, not simulated

Source: provenance/public_bench/gptme-base.json and gptme-hostpin.json.

−36%the bill with deepseek-v4.1-flash pinned to one host, against the same model on any hostgptme · 146 of 157 passed, against 144
4model changes tested; none recommended as a switchaider, crewAI and gptme, every one listed below
57%how often a change that is truly equal passes this gate on 157 cases with this much flippingabout 314 cases would make it four in five

the gate said no, or not yet

The model changes Margin held back

Each change swapped the model at the call site and left the agent’s code alone. A change passes when its worst plausible pass rate stays within the bar and it costs less per passed case. Passing the bar still earns a trial, not a switch, while the panel is this small.

aider 0.86.2 on 34 Python exercises from its own polyglot benchmark

Baseline gpt-4o, run 2 times; 4 cases flipped between its own runs, which is the noise every change is judged against.

Scroll the table sideways to see every column.

ModelPassed, each runPass rateBilledPer passed case, per thousandWhat Margin said
gpt-4o8 / 6 of 3420.6%$2.8449$203.210the baseline
gpt-4o-mini1 / 1 / 2 of 343.9%$0.3339$83.464refused: cheaper per pass, and still not worth it

crewAI 0.130.0 on the 164 problems of HumanEval+

Baseline gpt-4o-mini, run 3 times; 19 cases flipped between its own runs, which is the noise every change is judged against.

Scroll the table sideways to see every column.

ModelPassed, each runPass rateBilledPer passed case, per thousandWhat Margin said
gpt-4o-mini130 / 136 / 132 of 16480.9%$0.0871$0.219the baseline
gpt-4.1-nano138 / 140 / 136 of 16484.1%$0.0627$0.151cleared the bar; recommended as a trial, not a switch
deepseek-v4-flash-0731155 / 155 of 16494.5%$0.1512$0.488better and dearer: worth it only above a value per answer you set

gptme on 157 of the 179 specs in its own gptme-eval suite

Baseline deepseek-v4.1-flash, run once, so its own run-to-run noise is not measured here.

Scroll the table sideways to see every column.

ModelPassed, each runPass rateBilledPer passed case, per thousandWhat Margin said
deepseek-v4.1-flash144 of 15791.7%$0.2182$1.515the baseline
deepseek-v4-flash-0731142 of 15790.4%$0.1610$1.134held: its worst case falls past the bar

nothing dropped

Every setup we started

Every arm in the tables above ran to completion. These we started and stopped, and none of them is in a table:

  • aider on gpt-4o-mini, a first attempt: its command line pasted the web pages behind each exercise’s links into the prompt, which aider’s own harness does not do. We turned that off and ran it again.
  • crewAI on Gemini 2.5 Flash Lite: every call failed in under a second, cause not found. We replaced it before any result came in.
  • aider on DeepSeek V4 Flash, with reasoning on and then off: aider got no answer inside its own time limits, so both were stopped after the first few exercises.
  • gptme pinned to one host: a short smoke check on a handful of specs before the full run.

A third agent, one that fixes real GitHub issues, was not run: its benchmark needs one x86 container per task, and a single pass would cost far more than this study’s budget.

scope

What this does not show

No company ran these agents for us, and no customer’s traffic is in any number here. The tests are public benchmarks, the settings are our choice, and the panels are small.

The host pin is one run against one run. The two differ in host and in ordinary run-to-run noise, and one pair cannot separate them. A change that is truly equal passes this gate on 157 specs about 57% of the time; about 314 would be needed for four in five, and gptme has only 157 that run without a browser or a desktop. More runs on the same specs do not help, so the honest next step is a live trial with automatic revert.

Your own tests on your own work are the only evidence that carries over. Case study zero is our own agent, and case study two is our founder’s.

reproduce

Run it yourself

The runner, the agents’ trees and every arm’s record are in our repository. Each arm runs on its own capped OpenRouter key, minted for the run and revoked after it.

python3 scripts/public_bench.py build
python3 scripts/public_bench.py arm --bench gptme_evals --label gptme-base --cap 0.8
python3 scripts/public_bench.py arm --bench gptme_evals --label gptme-hostpin \
    --host deepinfra/fp8 --cap 0.35
python3 scripts/public_bench.py verdict provenance/public_bench/gptme-base.json \
    provenance/public_bench/gptme-hostpin.json

Watch the loop run on real spend.

The console runs it on open-source agents we metered ourselves, every number tagged real or sample.