The Scientist

The entitlement was the lever

Nothing in the price list moved: no dollar list price changed. Every budget model keyed to cost-per-seat now understates cost per unit of completed work, and a team without a tokens-per-completed-task denominator has no instrument capable of detecting the next adjustment. I expect more entitlement changes on roughly this cadence. Agentic coding inference economics are unresolved, and the entitlement is the cheapest lever a vendor holds.

The exposed workloads run weekly with no human in the loop: nightly eval sweeps, repo-wide refactors, LLM-as-judge scoring runs, synthetic data generation. Those are also the lowest-difficulty tasks in the portfolio, which is where substitution is viable. Any eval sweep or agent backfill calendared past September 14 should be re-planned against base × 1.25 rather than current measured capacity.

The vendor meter drifted 10–50%

The second failure lands on anyone reconciling spend against vendor dashboards. Unwind AI reports OpenAI resetting paid Codex and ChatGPT Work quotas after fixing token-burn bugs across image compaction, memory workers, subagents, rolling summaries, and MCP result handling, defects that silently consumed 10–50% of customer quota. That is a calibration failure in the only counter most teams have. The control is an independent per-run token ledger reconciled nightly against vendor-reported usage, with the discrepancy tracked as a monitored quantity instead of discovered at a quarterly true-up.

The substitution decision, with numbers

Terminal-Bench 4.0, a widely watched agentic benchmark, publishes resolution rate, tokens, and dollars together. Per Unwind AI's read, ranking by cost per resolution point reverses the order:

Two readings matter. First, Fable 5's 44.5% overlaps GLM-5.3's 41.8% inside the published error bars at higher cost. A 2.7-point gap you cannot resolve statistically is not a product feature, and it hands procurement a defensible "that difference isn't real." Second, the cheap model burns 34% more tokens to get there. The thing that figure doesn't tell you is latency and context-window pressure, which never shows up on the invoice. In a context-constrained agent loop the per-token discount erodes faster than the dollars-per-task ratio implies.

The honest framing is best open-weights model per dollar for this task class, tested on in-house traces rather than a public suite. GLM-5.3 is fine-tunable through Tinker, so a task-specialized adapter trained on your own agent traces is a plausible way to beat a generalist frontier model on a narrow distribution. That is the substitution that absorbs a capacity cut.

A 25% limit increase arriving as a 50% boost expires nets out to a cut. Agent spend that cannot be expressed as tokens-per-completed-task will hide the next one too.

What the trace actually buys you

No new model, no new retriever. Under a function-calling or MCP-style interface the tool-call log is already being written, so the work is a join: score each answer against whether the retrieval call fired at all, and whether the returned span supports the text produced. One metric becomes three, and each catches a failure the single number scored as correct.

The headline number will fall, and the size of the fall is the measurement worth having. It quantifies how much of last quarter's reported performance was the model recalling the public web rather than reading the corpus, and that fraction grows as models memorize more of it.

Grade the evidence before presenting it. The 50 cheats were planted, which means synthetic and non-adaptive. At that sample size there are no confidence intervals worth quoting. A model optimized against a trace-aware scorer will not behave like one that never saw it. The detector is a floor on detectability, not a solved problem.

Contested facts and the cost of self-correction

Techpresso's coverage carries a second reproducible result: frontier models surface both sides of a genuinely disputed fact about 52% of the time, defaulting to the more common version and dropping the alternative without saying so. That is epistemic bias, not hallucination, and accuracy metrics are blind to it because the majority answer scores as correct. A 30–50 item contested slice in the regression suite, scored on whether multiple positions appear, is an afternoon of work.

The third result moves money. A text-to-SQL cost-benefit audit found that self-correction is the only add-on that reliably pays across systems, and that at a fixed inference budget the pipeline beats a pricier model tier. Execute the generated SQL, capture the error class, feed it back with the schema slice, retry once. Then score on dollars-per-correct-query rather than exact-match, which never priced the retry.

One accuracy number fits too many different systems

Unwind AI's provenance analysis names a failure mode in the same blind spot: copied evidence counted as independent sources. Duplicated or syndicated retrieved documents inflate apparent corroboration the way correlated features inflate an ensemble's certainty. The fix is near-duplicate detection plus per-chunk lineage metadata, and no accuracy metric in the current harness will surface it.

Import AI pushes the logic one layer up. If the promotion gate is an LLM-judge or reward-model score and the system under test can infer that score, the metric stops estimating quality. Goodhart's law with a search budget attached. The prescribed control is a held-out shadow scorer the training and selection loops never see, with a pre-registered divergence threshold on primary-versus-shadow rank correlation.

The thing a single accuracy number doesn't tell you is which of many possible systems produced it. All three sources land in that spot. They disagree only about which column to add first: traces, lineage, or an independent scorer.

A grounded-QA eval without tool-call logs cannot tell you whether you bought retrieval accuracy or a good memory of the internet.

How the number propagated

The figure moved from a security vendor's scan to a news report to a think-tank analysis to downstream commentary, acquiring the phrase "China's diffusion advantage" along the way. Nobody in the chain re-queried the source. Same mechanism as a metric definition sitting in a BI layer with no owner and no as-of timestamp attached at the point of citation.

Both figures come out of the same badly specified instrument. An internet-wide scanner sees only publicly reachable hosts. Nothing behind NAT, a corporate firewall, or bound to localhost shows up at all. Geo-IP attribution breaks the moment deployments move from local machines to one-click cloud hosting, where an instance lands in the provider's region rather than the user's country. The observed flip therefore cannot be attributed to adoption. A U.S. lead in publicly reachable instances is equally consistent with China moving faster off local hosting and onto cloud regions that geo-locate elsewhere.

A metric that counts internet-reachable instances is a misconfiguration census. Roughly 35,700 terminal-capable agents were discoverable by a third-party scanner to produce it. Adoption is a different measurement.

The same gap, closer to home

The other reports each supply a live instance of the observed-versus-claimed gap, all of them cheap to close.

• Tenure. Refactoring's account of Graphite deleting one random employee account per day is chaos engineering aimed at the human cold-start path. The modeling analog: offline eval sets sampled from historical logs are dominated by tenured, feature-rich users, while every new user arrives on a day-0 vector of defaults daily. Aggregate AUC or NDCG holds while the sparse-feature slice rots. Half a day of SQL adds a no-history cohort with its own regression threshold.
• Actor type. TLDR Fintech's read on agentic risk decisioning sharpens it. Behavioral fraud features (session velocity, device entropy, typing cadence, dwell time) were engineered when non-human meant abuse. Funded agent-payment and agent-identity products break that correlation for a subpopulation. Retraining does not smooth this, because the label's meaning changed: non-human traffic no longer implies fraud. Global AUC looks fine while the agent-origin subgroup's false-positive rate blows out.
• Revenue weight. The onboarding checklist in Scarlet Ink states it plainest: learn whether revenue sits with a few key accounts or spreads evenly. Under concentration, unweighted aggregate metrics are dominated by long-tail volume, so mean AUC improves while the accounts that pay regress. Simpson's paradox with a P&L attached.

All three fixes have the same shape: report per-class, not pooled. Per-feature PSI or KS on the class split with alerting above 0.2, TPR at a fixed 1% FPR per traffic class, and expected calibration error per group rather than one number for everyone.

Provenance costs that stay unbudgeted

ChinAI's buried item is a data-engineering problem in humanities costume. Quotes from a decade-old report could not be located in the CNKI originals, either from a lossy translation or from post-publication editing of the source articles. Mapped onto eval sets and RAG corpora, the consequence is concrete: sources are mutable and third-party extractions are lossy. Store URLs and parsed text instead of raw bytes and content hashes, and a benchmark delta six months from now is unattributable between model regression and corpus mutation.