Cost Per Answer Has No Ceiling. Cost Relief Converges at 3.6x.
Four independent optimization lines topped out in the same narrow band this year, which makes the app layer's margin defense contractual rather than technical.
The frontier is scaling the process, not the model
What matters in Turing Post's read of OpenAI's Navier–Stokes write-up is where the gain came from. Performance is split explicitly between a stronger base model and post-prompt compute expansion — more agents, more wall-clock hours, more coordinated tool calls around frozen weights. That is a cost structure, not a capability. And OpenAI published it, so enterprise buyers begin asking for "let it think longer" as a purchasable feature within two to three quarters. Any portfolio company selling reasoning-heavy output on flat-rate or per-seat pricing then owns a cost line with no ceiling inside a contract with no pass-through clause.
The same material names the road not taken. Test-time training — where a model updates its own weights during inference rather than renting ten thousand agents for four days — traces to a public 2020 UC Berkeley paper and has still produced no commercial boom. The reason is structural: batched serving with shared weights and reused attention caches cannot support per-request weight mutation, so no serving stack exists for it. That is a real pre-consensus infrastructure gap, and it drags a compliance problem behind it. If the deployed model mutates, the deployed artifact is provably not the validated artifact, which breaks version pinning, model cards and conformity assessment at once.
The relief lever, method by method
Daily Dose of Data Science's teardown of speculative decoding — where a small draft model proposes tokens that the large model verifies in one pass — is the discipline check on every efficiency deck in your pipeline. Four independent research lines converge in one band. The difference that matters commercially is who must own the weights to capture the gain.
| Method | Reported speedup | Needs model changes? | Who captures it |
|---|---|---|---|
| Two-model baseline | 2x–3x (T5-XXL) | No | Anyone, including API consumers |
| EAGLE | 2.7x–3.5x (LLaMA2-Chat 70B) | Yes — draft module per checkpoint | Teams owning checkpoint plus serving engine |
| Medusa-1 / Medusa-2 | >2.2x / 2.3x–3.6x | Yes — added decoding heads | Teams with training capability |
| LayerSkip | 1.82x–2.16x | Yes — a pretraining decision | Model owners only; cannot be retrofitted |
Two consequences. An application company consuming APIs is capped at the drop-in baseline, which already ships free as assisted generation and carries a real memory tax: a second weight set plus a separate cache. And heavy batching erodes the gains, because the accelerator is already doing useful work at every decode step. Speculative decoding is a latency play for interactive chat and coding agents, not a cost play for high-throughput batch inference. Any model booking material COGS reduction across batch-heavy enterprise workloads has a business case that may not survive measurement.
Where the evidence pulls against itself
Chris Short's ledger supplies the counterweight, and it deserves weight: a Claude Code plugin routing bulk reads and boilerplate to Gemini 2.5 Flash claims roughly 90% token reduction. Treat it as directional. It is a synthetic mean across four scenarios in a single Java monorepo, with named failure modes — broken line numbers on edits, a missed thread-safety bug the premium model caught, and 10–30 second round trips that make small reads a net loss. Margin relief is real, but it is company-by-company engineering behind a quality gate, not a sector-level correction you can assume into a model.
Efficiency is migrating from an inference-time knob into a training-time commitment — which means the companies renting the model capture the least of it.
The question to standardize in diligence is accepted tokens per target-model pass, measured at production batch size and production temperature. Low-temperature code generation produces far higher acceptance than high-temperature prose, so a benchmark run on code flatters a company whose customers write text. Ask for the workload, not the multiple.
What to do
Commission an inference-COGS stress test across every AI application position by month-end: gross margin at 3x, 10x and 100x current compute per active task, plus a list of which contracts allow cost pass-through.
Add accepted-tokens-per-pass at production batch size and temperature to the infrastructure diligence template before the next investment committee.
Open a thesis file on runtime weight-update serving and attestation for mutating models this quarter, sourcing five to eight seed teams and requiring a named design partner.