Open-Weight Models Closed the Benchmark Gap. Deployment Is Where They Lose.

On the public leaderboards the argument is over. The best open-weight models now sit within a few points of the best proprietary ones on most published evaluations, and on several tasks they are ahead. Teams that have actually deployed both describe a gap that remains substantial, and it has almost nothing to do with capability.

The difference shows up in the parts of a system that no benchmark scores: throughput under concurrency, behaviour at the edges of the input distribution, and the operational cost of being the party responsible when it breaks.

What benchmarks measure and what they miss

A published evaluation measures single-request quality on a curated input distribution, usually with generous latency budgets and no cost ceiling. Production measures something else.

Four teams running both classes of model in production identified the same divergences.

• Throughput at target latency. A model that matches on quality can require substantially more accelerator time to hit the same p95 latency under real concurrency. That is a cost difference, not a quality difference, and it does not appear on a leaderboard.
• Tail behaviour. Malformed input, adversarial prompts, mixed languages, and very long context are underrepresented in evaluation sets and overrepresented in real traffic.
• Structured output reliability. Teams consistently reported more schema violations from open-weight models when asked for strict JSON, which matters enormously when the output feeds a downstream system rather than a human.
• Change management. A hosted endpoint changes underneath you, which is a real risk. A self-hosted model does not change unless you change it, which sounds better until you own the upgrade, the regression testing, and the rollback.

We did not switch back because the open model was worse. We switched back because we were spending two engineers on serving infrastructure and the vendor bill was cheaper than those two engineers.

The cases where open weights clearly win

The picture is not one-sided, and the teams that stayed on open weights had specific, legible reasons.

Data residency was the most common. If the requirement is that inputs never leave a jurisdiction or a private network, the decision is made before quality enters the conversation.

The second was high-volume narrow tasks. Classification, extraction, and routing at large scale is exactly where a smaller fine-tuned open model is not just adequate but preferable, because the per-request cost difference compounds and the task distribution is narrow enough that tail behaviour is controllable. This is the same insight driving the tiered-routing pattern we documented in our reporting on serving-cost economics.

The third was pricing leverage. Two teams described maintaining a functional open-weight deployment specifically as a negotiating position, and both reported it worked.

What the honest comparison requires

The comparison most teams run is not the comparison they should run. Quality on a held-out set is the easy part. The complete accounting includes accelerator hours at target latency, engineering time for serving and upgrades, the cost of the evaluation harness you now have to maintain yourself, and the residual risk you have absorbed by becoming the responsible party.

Run that comparison and the answer stops being ideological. It becomes a straightforward function of volume, task breadth, and how much engineering capacity you have to spend. High volume and narrow tasks favour open weights. Low volume and broad tasks favour a hosted endpoint. Most organisations have both, which is why most end up running both.

The framing that will age worst is the one that treats this as a single decision with a single answer. Hardware supply shapes it too: as we reported on the packaging bottleneck, the cost of self-hosting depends on an accelerator market that is not loosening as quickly as 2025 plans assumed.