The Scientist

The mitigation stack is the transferable artifact

ClinReg's authors measured judge bias, then engineered around it. Four judges emit enumerated defects rather than scores; a defect counts only if at least 3 of 4 judges confirm it. A separate chairman model assigns severity and produces the number. The code-generation task used median-of-3 critics drawn from three distinct model families, scored against an FDA-reviewer rubric across four axes.

The statistics travel further than the roster. A continuous 0-10 holistic judgment is a high-variance regression problem. Defect enumeration with a confirmation quorum is lower-variance discrete detection, and severity assignment isolates into one auditable step. That generalizes to any rubric-scored generative eval.

The number Airbnb left implicit

Airbnb's published playbook sets the calibration bar at high-80s agreement between LLM judges and human reviewers, found after hand-reviewing 100 prototype outputs to enumerate real failure modes. Both numbers need arithmetic attached.

• Coverage. At n=100 random samples, a 3%-prevalence failure mode appears at least once about 95% of the time; at 1% prevalence, ~63%. A well-calibrated budget for taxonomy discovery, a bad one for rare-mode assurance.
• Resolution. High-80s agreement is 12-13% disagreement with ground truth. At judge pass rates near 0.85, detecting a 3pp quality delta at 80% power in an unpaired design needs on the order of 2,200 items per arm. Most suites run 50-200 and call a 2-3 point move a win.

If you cannot state your judge's agreement rate with human labels, every eval number you shipped this quarter is a point estimate with an unknown error bar around it.

Where the sources agree, and where they diverge

All three accounts agree the rejection signal dominates the generator. Spotify is the cleanest evidence: static codemods grew to thousands of lines of edge cases, early LLM rewrites without a gate barely beat them, and only agent-plus-judge-plus-verification reached ~80% PR success. The divergence is attribution. Spotify rebuilt its test automation while adding the judge, so that lift is confounded between a deterministic gate and an LLM one. Copy the judge alone and expect a fraction of the effect.

ClinReg admits a worse confounder. Several results moved more with harness, validator strictness, stopping criteria and retry behaviour than with which model was running. The intended independent chairman judge refused the prompts on biological-safety grounds, forcing arbitration onto Opus 4.8, a family under evaluation. Refusal rate belongs on the eval scorecard, with a deterministic fallback chain for both generation and evaluation roles.

The two metrics worth stealing outright

ClinReg split "hallucination" over a 108-field intermediate extraction step into two deterministic rates. Omission rate counts a present field marked missing to dodge penalty. Fabrication rate counts a wrong value emitted for a present field. Boilerplate citations are excluded from fabrications, silent source fields from omissions. Opus 4.8 showed higher omission and lower fabrication than GLM 5.2 at a similar holistic score. A single 0-10 rubric would have averaged that distinction into invisibility. That distinction decides which model routes to a filing and which to a triage queue.

Do the exponent arithmetic before the discount arithmetic

Each turn of a tool-using agent arrives as a fresh stateless request carrying the full history plus the full tool schema. Cumulative input volume across an N-turn session therefore grows as O(N-squared), not O(N). At roughly 2K tokens of tool output appended per turn, turn 30 replays about 60K tokens on that turn alone, and the session has read on the order of 900K tokens of history. Prompt caching prices replayed context at about 10% of the standard input rate. That divides the coefficient by ten. The exponent does not move.

The corollary is the part that transfers across stacks: any token you emit as output becomes an input token you pay for on every remaining turn. Output-side controls compound. Input-side controls do not. That is also why MCP schemas matter more than they look, since ten servers with fifty tools can cost up to 16,000 tokens per turn, re-serialized every turn, and zero-call installs keep loading schemas until someone removes them.

Where the measurement gets loose

Weight this carefully. The token taxonomy is not mutually exclusive: "prior assistant context = 30-45% of input" and "tool results = 23% of input" cannot be summed without category definitions that were never published. The headline 86% is n=1 at the org level, one deployment, with no confidence intervals reported. The thing that number doesn't tell you is the task mix, which was not disclosed. Claude Code's own /context command reportedly has token-count calculation bugs, so the baseline measurement may be wrong before anything changes. The one figure with a real protocol behind it is Comet's own dogfood: median output cost $229 to $181 per million output tokens, a 21% cut, with "zero change in development velocity" asserted and no velocity metric attached to it.

The two experiments that pay for themselves

The repair-loop result is the one that contradicts prevailing agent design. Most frameworks append the failed attempt plus its traceback to context on every retry. That is a growing per-iteration token tax and an anchor holding the model on a bad trajectory. Even a 2x replication of the claimed effect is the largest inference-cost lever available without changing models.

Fix the denominator, then argue about the model

Cost per token is the wrong unit, which is why aggregate dashboards mislead. Report cost per resolved task and cost per merged PR, computed as (input tokens x price in + output tokens x price out + tool calls x tool price) x expected attempts, where expected attempts is roughly the inverse of success probability. That composite reverses rankings routinely: a model at twice the per-attempt cost that lifts pass rate from 55% to 82% delivers outcomes about 35% cheaper. The per-token leaderboard and the production winner are rarely the same ranking. It also guards against the failure mode where a 40% token cut buys a 15% increase in review iterations that nobody attributes back to the config change.

Four teams, four different escapes from the same wall

The most useful admission comes from Granola, a context company valued at $1.5B: Claude connected to its corpus over MCP does a "relatively dumb job," effectively reading roughly the 20 most relevant meetings out of a multi-year archive. That is not a complaint about the model. It is a plain description of the recall ceiling of top-k semantic retrieval over a long-tail corpus, published by the team whose entire thesis is being good at exactly this. Their fix is a pre-computed compressed profile, "apple juice from concentrate." It started as a founder's nightly cron job and reportedly beat live retrieval over the same data.

Pinterest hit the same ceiling from the user side. One embedding per user is a mean-pooled vector that collapses multimodal intent, so they replaced it with per-user agglomerative clusters over the last 500 engagements, with a dynamic cluster count and lifecycle metadata tracking recency and frequency, feeding retrieval, ranking and blending, with lifecycle stage gating which utility signals get weighted.

The engineering costs nobody in these accounts prices

• Serving cost scales with cluster count. k clusters means k retrieval queries, so candidate-generation QPS becomes a function of a per-user hyperparameter.
• Dynamic k is a drift surface. The stopping criterion wobbles day to day and shows up as representation churn that reads exactly like model drift. Monitor the cluster-count distribution as a first-class metric.
• Concentrate layers go stale. Compression is bounded by refresh cadence, and the failure mode is a confident answer from an over-compressed profile. Nobody publishes a post-compaction degradation number, which is decent evidence nobody computes one.

Granola supplies the lineage lesson for free. They planned to delete transcripts after generating notes, then reversed, because transcripts are the source material needed to regenerate notes when the prompt, model or output schema changes. Build the post-compaction regression eval before approving any summarize-then-drop retention policy.

Where the sources genuinely disagree

The commerce teams reached three incompatible designs for one problem, and reported business KPIs rather than benchmark deltas.

DoorDash bought hallucination immunity structurally: an attribute the catalog does not stock becomes impossible to emit, with no downstream validator required. Instacart bought the gap DoorDash left, routing only rare tail queries to a fine-tuned Llama-3-8B under a hard sub-300ms budget. That is the clearest published build-versus-buy signal here, because a task-fine-tuned open-weight model beat the frontier-API path on a latency-bound workload. These are engineering-blog accounts: no confidence intervals, no ablations, no traffic allocation, no duration. Pinterest likewise coupled clustering, lifecycle metadata and ranking-weight changes into one result, so the clustering alone may not carry the lift.

The sharpest analytical implication is Pinterest's nonlinearity claim. If the retention effect is threshold-shaped in use-case breadth, the average treatment effect is the wrong estimator. Pre-register segmentation by distinct-intent count and test the interaction. The alternative is reporting a diluted mean and concluding the change did nothing.