Inference Is Now the Line Item That Decides Which AI Products Survive

For three years the number that mattered in machine learning was the size of the training run. It was legible, it was expensive, and it made for a clean narrative about scale. That number now tells you almost nothing about whether a product works as a business. The number that does is the cost of answering a single request, and six engineering teams who shared their figures with Our Times describe the same uncomfortable arithmetic.

Training is a capital expense you amortise. Inference is a variable cost you pay on every interaction, forever, and it scales with exactly the thing you are trying to grow.

The gap between demo economics and production economics

A retrieval-heavy assistant that costs a fraction of a cent per query in a demo will often cost twenty to forty times that in production. The teams we spoke to attributed the gap to four things, and they listed them in roughly the same order.

• Context growth. Demos use short prompts. Real users paste documents. Attention cost grows faster than linearly in sequence length, so the tail of long requests dominates the average.
• Retries and cascades. A single user action fans out into several model calls once you add reranking, tool use, and a validation pass.
• Idle capacity. Reserved accelerators are billed whether or not they are saturated. One team reported 31% average utilisation against a peak they had provisioned for.
• Guardrails. Safety classification, moderation, and output checking are additional forward passes that nobody puts in the pitch deck.

We shipped a feature with a gross margin that was negative for eight months and nobody noticed, because the cost sat in a shared infrastructure budget rather than against the product line.

That quote came from a platform engineering lead at a company with more than 400 employees, on condition that we not name the product. It was the most common structural failure described to us: serving cost is real, but it is not attributed, so it cannot be managed.

What the teams that fixed it actually did

None of the successful interventions involved a better model. They involved routing, caching, and a willingness to say no to requests.

The most effective single change, reported by four of six teams, was tiered routing: classify the request, send the easy majority to a small model, and reserve the expensive path for cases that need it. The teams that measured it found that between 60% and 80% of production traffic did not need their flagship model at all. Users did not notice the difference, because the requests being downgraded were the ones with unambiguous answers.

Caching came second, and the surprise was how much of it was viable. Semantic caching on normalised queries produced hit rates of 18% to 35% in support and documentation workloads. That is a direct multiplier on cost, and it also cuts latency, which improves the metric the product team actually cares about.

The third lever was the least popular and the most effective: limiting context. Two teams capped retrieved context aggressively and measured no degradation in answer quality, because most of the retrieved material had been noise. One of them had been paying to process an average of 14,000 tokens of context to answer questions that needed 1,200.

Why the accounting matters more than the optimisation

The teams that had cost under control shared one non-technical trait. They had a per-request cost figure that a product manager could see, in the same dashboard as engagement and retention.

Where serving cost lived in a central infrastructure line, it behaved like weather: everyone complained, nobody owned it. Where it was attributed per feature, the optimisation happened without anyone mandating it, because the person who had to justify the feature also had to justify its margin.

This is not a new lesson. It is the same lesson cloud migration taught a decade ago, arriving again with a different bill attached. The difference is magnitude. A wasteful web service costs you a percentage. A wasteful inference path costs you the product.

The part that does not optimise away

There is a floor, and several teams have hit it. If the task genuinely requires a large model over long context with verification, the cost is the cost, and the only remaining moves are pricing and scope. Two of the six teams had raised prices. One had removed a feature entirely after concluding that no plausible efficiency gain would make it viable at the price point customers would accept.

That is a healthy outcome, and it is happening more often. The supply picture matters here too: as we reported in our analysis of the advanced packaging bottleneck, accelerator availability is not improving on the timeline most 2025 capacity plans assumed. Teams that budgeted for cost declines driven by hardware abundance are revising those assumptions.

The teams likeliest to survive the next two years are not the ones with the best benchmark scores. They are the ones who can tell you, to the cent, what a request costs and what it earns. For more on how open-weight alternatives change that calculation, see our reporting on where open models still lose on deployment.