The Two AI Cost Levers That Never Touch the Model
Both levers are published and replicable, and one expires: free validation access ends September 25, after which the number you could measure for nothing becomes a purchase.
Where the line actually falls
The question is not cheap model versus expensive model. It is whether a given call has a finite output space. Published comparisons put the typed-decision model at 85% on page classification against 56% for the generative baseline, then at 27% against 93% on choosing a crawl root, a task that requires a view of the whole site. Adjacent tasks, inverted results. That inversion makes decision-boundary classification an architecture call, made once per task, and almost no engineering organization has a taxonomy for making it.
| Workload | Evidence | Move |
|---|---|---|
| Agent eval and model-as-judge | 500/500 agreement, $0.34 vs $28.17, lower variance between repeats | Clearest displacement target |
| Routing, triage, guardrails | Output space already typed; multiple questions batched over shared state | Move with generative fallback |
| Bounded classification | 85% vs 56% | Validate per task, not per category |
| Context compaction | Up to 90% token reduction via selective retention | Pilot; reversible |
| Whole-system reasoning | 27% vs 93% | Leave on the large model |
The second lever is the harness, and the vendor default is the inefficient baseline
NVIDIA's SoL-Pi ran an automated search over agent harness configurations instead of models, the harness being the wrapper that manages context, actions and observations around a coding agent. Four techniques survived the search: Action Fusion, Online Context Compact, ObservationPack and an Evidence-Preserving Reducer. Together they cut recorded token traffic 44.7–49% and roughly a third off the hourly API bill at approximate performance parity, measured against native Codex and Claude Code defaults. The caveat is material: 51 tasks on EdgeBench. Directionally important, evidentially thin. Replicate it on a local task distribution before booking the saving.
Why this matters
OpenAI published the demand side of the same equation. Non-engineering teams went from roughly 0% to 90% Codex adoption in four months, and rising PR volume pushed a 10x load surge through CI in six months. In Gergely Orosz's synthesis of the nine-stage loop, a human defines the outcome and the agent does "pretty much everything after that." OpenAI explicitly declines to judge the output: "whether this is good quality or not, well, we'll see." A skeptic reads the adoption curve as a productivity story, and on its own that reading holds. Set beside the CI surge it becomes a cost-relocation story: the cost moved to CI compute, token traffic and human verification. None of those scale with headcount, and headcount is what capacity models still measure. OpenAI's own incident agent diagnoses but cannot mitigate, so on-call stays staffed.
The layer commoditized before it matured
Within days of the typed-decision launch the pattern was cloned: kev (Apache-2.0, built on a 0.5B base model, runs on Apple Silicon), LocalJev on local inference, Cua's 2.8MB form-filling model, and an open-sourced reinforcement-learning decision family spanning 100+ languages. The leader's residual advantage is reported at roughly 19 points out-of-domain, measured against clones that shipped in days. The adjacent structured-output vendor drew six clones in two days, and its "output tokens free" line appears in a launch table, not a price list. Vercel's 13% of paid teams inside 24 hours measures trial during a free promotion, not retention. Incumbents should be expected to answer with native cheap-decision endpoints within a quarter or two. Teams that abstracted the layer get price competition. Teams that hard-wired a specialist get a migration project.
Cost savings here are table stakes competitors also capture. The defensible part is a product experience that could not exist above 300 milliseconds.
What to do
Run your existing eval corpus through a hosted typed-decision model before September 25 and bring the agreement-rate and cost delta to the next executive staff meeting.
Require a provider-agnostic decision-layer interface — hosted, open-weight and local backends as a configuration change — before any production traffic moves this quarter.
Instrument tokens per completed agent task and cost-per-merged-PR this quarter, then replicate the four harness techniques against vendor defaults on 200+ internal tasks with paired success-rate measurement.