The Scientist

Start with the denominator

Forty percent of queries is not 40% of tokens, and it is a long way from 40% of dollars. Summarization prompts are short. Agentic code generation is long-context and output-heavy. The disclosure carries its own tell: developers still send generation work to frontier models and use the cheap open models to summarize previously submitted code, the highest-volume and lowest-value slice of the traffic. Query share can rise forty points while spend barely moves.

The cost baseline is softer than the headline too. The Information's "at least hundreds of millions annually" is extrapolated from public list prices. Work it back: 45B tokens/day is roughly 16.4 trillion tokens a year, which implies a blended rate above $12 per million tokens. That is frontier output pricing with no enterprise discount and no prompt caching. A business case anchored to that multiple will not survive the first invoice.

The router is a production classifier, and nobody evaluated it

Routing "based on the complexity of the task" means a learned or heuristic classifier is sitting on the hot path with no reported eval. Treat it as any other production model: a labeled routing set, a confusion matrix with explicitly asymmetric misclassification costs, a confidence threshold that escalates instead of guessing, and drift monitoring on the input distribution. Onboard a new team or a new repo and the difficulty distribution shifts, so the decision boundary degrades quietly. The only symptom is a slow rise in retries.

The levers that compound underneath it

One of these is an active bug rather than an opportunity. Prompt caching on OpenAI and Anthropic matches on an exact byte-for-byte prefix. Reorder two cached documents and both become misses. Cache document A and document B separately, then query them together, and B misses because its key-value state was computed without ever seeing A. In production the per-block hit distribution is bimodal: a small fraction of blocks serves almost all the hits while the rest are dead weight, so any saving reported off a mean hit rate needs re-deriving. A reranker exists precisely to vary chunk order per query, which is structurally incompatible with prefix caching. CacheBlend recomputes only the cross-boundary tokens and reuses the rest, the first design here that reconciles the two. The thing its 2-4x doesn't tell you is the hardware, the document-length distribution, or the quality metric, since none of the three is named. The mechanism justifies a spike. The number is yours to reproduce.

Change the unit before you change the tier

ARC Prize is now quoting cost per task rather than cost per token. Gemini 3.7 Flash lands 84.6% on ARC-AGI-2 at $0.25 per task. For agentic work that is the only honest denominator, because a cheap model that burns three times the rollouts is not cheap. Rebuilt on that axis, the Pareto curve may name a different default tier than the one a token-price dashboard picked.

A 56% cost cut for a 2% quality drop is only a good trade if you can prove the 2% is not concentrated in your hardest 5% of tasks.

The mechanism generalises past glucose

Type 1 physiology is more volatile, which means conditional variance differs by cohort. A loss function minimising average error will happily trade accuracy on the smaller, harder, higher-variance subgroup for accuracy on the easy majority, and a single-number scorecard records that trade as an improvement. Swap in new tenants against mature tenants, low-activity against high-activity users, or rare-event against common-event segments. The arithmetic does not change.

The most credible number here is the unflattering one

Recovering roughly half of the accuracy lost to a missing modality is the strongest result in this set because it is disappointing. Papers reporting full recovery from a missing modality are usually leaking the signal back in through correlated features. Hold ~50% as the realism benchmark the next time a vendor argues that their imputation layer makes heterogeneous data availability a non-issue.

The productivity split needs the opposite treatment. The working paper reports no sample size, no standard errors, and no identification strategy, and it has not been peer reviewed, so take the design lesson and do not cite the point estimates. The design lesson is worth more than the estimates anyway: the assisted metric and the capability metric moved in opposite directions in the same population. Census self-report data rhymes at enterprise scale — 55% of US workers use AI, and 13% of users report no time savings or a net workload increase. Ticket resolution time, analysis throughput, and review latency are all measured with the assistant in the loop, and the delta gets booked as capability uplift. The thing those metrics do not tell you is what happens when the assistant is removed. The closed-book exam is the arm almost nobody builds, and the only one that separates leverage from dependency.

What changes in the harness

The engineering is small and the coverage gain is permanent. Enumerate the top five population slices, emit per-slice error alongside the aggregate, and add a max-minus-min gap threshold that blocks the release. Roughly a day of work, and it closes the failure mode that defeated all 33 glucose models.

Two adjacent gates cost about the same. Where a prediction overwrites a field already known with certainty — as in the order-confirmation case, where the verified item title sat in the order record — define a calibrated threshold below which the known value is emitted instead, then track abstention rate and post-abstention accuracy as first-class metrics. Where users see predictions in groups, add modal-label agreement or within-group label entropy to CI. Per-item accuracy and macro-F1 are both structurally blind to incoherent triples on the same basket.

A single-number scorecard cannot fail the subgroup it was averaged over.

The failure class DLP does not cover

In March 2026 an internal AI agent at Meta produced a Sev 1 by surfacing sensitive company and user data to employees with no entitlement to it. Not a jailbreak, not prompt injection. An authorization bypass that the access-control system scored as clean: a service account with legitimate broad access read data and handed it to a human.

That is the reference architecture of most internal retrieval deployments. One high-privilege credential crawls wiki, chat, tickets and the warehouse, one index holds all of it, nearest-neighbour results go to whoever asks. The index applied a lossy transformation to the permission model. Meta has a mature security organisation and a Sev 1 process, and it happened there anyway.

Make leakage a number, not a review item

Build a golden adversarial set of 600-plus (query, principal) pairs across privilege tiers, each with a labeled allow/deny document set, and define Unauthorized-Retrieval Rate as the fraction of returned chunks the requesting principal is not entitled to. Gate it at zero in CI on every index rebuild. Sample size is the whole argument: by the rule of three, zero failures in n trials bounds the true rate near 3/n, so 600 clean probes buys 0.5% at 95% confidence. A 50-query smoke test cannot separate "safe" from "leaks one in two hundred," and one in two hundred is the regime that produces an incident nobody predicted.

Then pay the retrieval bill knowingly. Filtering after approximate nearest-neighbour search spends top-k slots on documents that get discarded, so recall decays quietly as the entitlement pass rate drops, and any code path that skips the filter fails open. Filtered HNSW, per-tenant namespaces, or over-fetch at k' = k / expected pass rate. Then benchmark recall@10 per privilege tier before and after, so the accuracy cost of correctness lands in a design doc rather than a postmortem.

The control plane underneath it is now enumerated

CISA's confirmation of active SSRF exploitation against MLflow servers matters because of what a tracking host usually is: no auth, inside a VPC, holding an instance role that reads the artifact bucket. Instance role to artifact bucket to registry write to a poisoned model on every serving pod. Patch, add an authenticating proxy, enforce IMDSv2 with hop-limit 1, default-deny egress from tracking hosts. The same inventory question applies to every operator console that ships unauthenticated: Ray's job API on 8265, MLflow on 5000, tokenless Jupyter, Airflow, Label Studio, OpenAI-compatible vLLM or TGI endpoints. An open Ray job API is arbitrary code execution on your GPU fleet with a credential-rich environment attached. Hours of inventory, not a quarter of project.

The AI reviewer in front of it has unmeasured recall

Wiz's Red Agent exploited a vulnerable GitHub Actions workflow in a Snowflake repository that AI-based security checks had already cleared, and ended holding credentials to an unrelated SaaS. That is n=1 vendor research with no model named, no defect taxonomy, no recall data, so it establishes no rate. It does establish that the rate is unknown and non-zero, which is enough to reword the control in audit docs. Producing the missing number takes days: 50-100 labeled vulnerable workflow configs (over-permissioned tokens, unsafe triggers, unpinned third-party actions) through the reviewer, false-negative rate computed. Until that number exists, the model is a screen sitting under deterministic policy-as-code, not a gate.

A vector index is a permission-laundering layer until a number proves otherwise.