The Scientist

Start with the arithmetic nobody published

Composio ran the same Kimi K3 weights through three agent harnesses. Kimi Code 22/28, Hermes 21/28, Claude Code 20/28. That is a two-task spread. An unpaired two-proportion test on 22/28 versus 20/28 returns p ≈ 0.53, and Wilson intervals on the top arm span roughly 60–90%. Resolving a 7pp difference at 80% power unpaired takes about 570 tasks per arm. So the honest read is not that Kimi Code wins. Harness choice here looks accuracy-neutral and definitely not cost-neutral, with Kimi Code cheapest and most token-efficient and Hermes fastest.

That changes how the recursive-improvement results reported by AINews should land. Cline's harness was tuned against Terminal Bench and then scored on Terminal Bench, which is a hyperparameter search whose search agent can see the validation labels. The −37% cost reduction is far more likely to generalize than the +11.3pp accuracy gain, because token efficiency is a property of the scaffolding, while benchmark accuracy can be a property of memorized task structure. OpenAI's serving numbers (20% lower cost, 15%+ token-generation efficiency from speculative-decoding work) have the inverse profile: no benchmark to overfit, but no disclosed baseline, workload mix, or measurement window either.

The same failure with no agent involved

ByteByteGo's interview with the Codex engineers supplies the cleanest instance of a cost regression that no accuracy metric can see. Codex stored MCP tool definitions in a hash map with no ordering guarantee, so identical tools serialized in different order across requests. Prompt caching matches token-by-token on the prefix, so one reordered token near the front invalidates everything after it. The tasks still completed. They just cost more, with nothing throwing an error.

The most expensive bug in an agent stack will not throw an error. It will quietly stop hitting the prompt cache.

The single metric that catches it is cached_tokens / prompt_tokens per turn, plotted per route. The three prefix rules are worth adopting verbatim: deterministic serialization order, strictly append-only history, and volatile runtime state such as approval policies and feature flags applied at tool-run time rather than edited into the prompt.

The same failure again, one layer down in the data loop

Devshot's ledger closes the pattern from a third direction. Autonomous data-researcher agents improved a model's training data in about 58% of test settings, and in roughly 78% of those successful runs they overshot and finished worse than their own best intermediate attempt. That is not a capability ceiling. It is an unregularized optimization run with no early stopping, where the agent optimizes its own judgment of progress past the point the true objective peaked. The field solved this in 1995 with a held-out metric and argmax over the trajectory. Nobody ported it into the agent loop.

Where the sources agree, and where they diverge

All three converge on one denominator change: report dollars and tokens per successful task, not pass@k. They diverge on how far to trust reported gains. ByteByteGo's engineers note that offline wins routinely regressed under real traffic shapes, and that fixating on one favorite technique caused underinvestment elsewhere. AINews treats the cost half of the self-improvement result as credible and the accuracy half as unverified. Nothing in either source supports the stronger claim now circulating, that agents reliably improve the systems that run agents.

The channel split

The instruction-hierarchy benchmark reported by Cyberpresso covers 37 models and lands at 98.2% for user prompts and 20.5% for tool outputs. That is nearly 78 percentage points of spread, and the collapse sits on the channel nobody has spent two years hardening: the text that the retriever and the function-calling loop push back into context. Most production jailbreak suites perturb the user turn. If yours does, the number on the dashboard is measuring the axis that already works.

The useful part is that this converts into a model-selection criterion you can compute. Split the injection suite by channel, user turn, retrieved document, tool response, and report compliance per model per channel with a promotion floor attached. Then stop asking the model to hold the line at all. A 20.5% model behind hard tool allowlisting is safer than a 98.2% model with none, because the residual 1.8% tail is where the data leaves.

The precision that nobody computed

Figma's security team, covered in TLDR Design, gated its agents at 70% precision before letting them comment on pull requests, and later reached 80%. The reasoning is measurement, not security: historical bug corpora are a positives-only dataset. Recall against confirmed past vulnerabilities is boundable. How often the agent flags something that was never a problem is not, because the corpus contains no sampled negatives. Every false positive is invisible to the primary benchmark, and false positives decide whether anyone opens the tool a second time.

Their recovery mechanism is worth copying. Recall lost at the high-precision gate comes back through a separate offline adjudication pass rather than by loosening the threshold. Two declared operating points beat one averaged one. Developer dismissals get treated as labels instead of noise, which makes them the cheapest hard negatives available and the only in-production signal for precision drift after a prompt or model bump.

The metric that rewards a broken model

Techpresso has the sharpest version of the same disease. Across eight popular AI resume screeners, tools passed fairness audits because they could not judge role fit at all. Near-constant scores produce near-zero measured disparity. That is the null-discriminator artifact, and fairness-only review certifies it as neutral.

A fairness metric with no predictive-validity gate in front of it will always crown your least useful model as your fairest.

The same applies anywhere a disparity dashboard is the gate: fraud triage, lead scoring, moderation prioritization, collections. The fix is an ordering change rather than new tooling. No disparity number is reportable until AUC, calibration error, and lift-over-random clear a floor on a holdout.

The pattern across all four

TLDR IT closes it from the retrieval side. If the bottleneck has moved to lineage and permissions, unfiltered recall@k is a vanity metric. An agent at recall@10 = 0.94 that surfaces one out-of-scope HR document is a compliance incident, not a strong retriever. The replacements are permission-filtered recall@k per identity class and an unauthorized-retrieval leak rate with a target of zero. Four sources, four domains, one shared error. The quantity being optimized is not the quantity that fails in production.

Two decisions, same direction

The academic result Techpresso covered is narrow and specific. On textbook-grounded tutoring QA, search-based retrieval beat graph-based methods at 99.4% correct-page retrieval, and the relative gain was largest for cheaper, weaker generators. The second clause is the one that matters. It says retrieval quality behaves as a substitute for parameter count, which makes this a serving-cost argument before it is a quality argument.

The production decision arrives independently, from the opposite kind of source. OpenAI's Codex team told ByteByteGo it uses BM25 lexical ranking rather than embeddings for tool discovery, on the stated reasoning that an LLM is already intelligent enough that machinery placed in front of it often adds complexity without value. The same logic explains keeping exactly one compaction path instead of a configurable menu. A Codex session can expose hundreds of tools once integrations and MCP servers are counted, and every schema is prefix tokens on every call. So shell, file editing, and a single tool_search entry point stay in the prompt, and the rest is deferred.

The caveats that decide whether it transfers

Page-level retrieval is not answer accuracy, and a single-hop textbook corpus is the friendliest case lexical matching will ever see. Neither source reports sample sizes, ablations, or confidence intervals. That makes this a hypothesis generator for a local harness, not a result to cite in a design doc. It also makes the bake-off cheap: BM25 or hybrid dense-sparse against the existing graph or hierarchical index, on the real corpus, scoring top-k chunk recall, end-task answer accuracy, ingest cost per 1,000 docs, and p95 latency, with a small cheap generator in the matrix. The outcome is either a deleted index and a deleted embedding dependency, or the first quantified reason to keep them.

The routing tier, back-solved

The cost story continues one layer down. Microsoft's Project Perception, which TLDR IT reports entered public preview August 3 gated to MDASH customers, runs in-house MAI-Cyber-1-Flash and invokes GPT-5.4 only for the hardest tasks, claiming 96% on CyberGym and nearly 50% cost reduction. All vendor-reported. Back-solving is simple: if the specialized tier costs roughly 5% of the frontier tier, total cost index = 0.05 + p, where p is escalation rate. Hitting "nearly 50%" requires p ≈ 0.45.

A real-world cascade escalates something like half of hard-domain traffic. Not the 5–10% teams assume when they pitch one in a design review.

Two things are missing from that claim, and both matter. No per-tier accuracy breakdown, and no router error rate. A hard task misrouted to the weak tier is where tail incidents come from. The 96% also sits on a security benchmark derived from public CVEs and OSS repositories, about as contamination-prone as datasets get, since patches and writeups land in every frontier training corpus. Absent a post-cutoff temporal split, the figure is as consistent with memorization as with capability.

The hardware floor is also moving

Liquid AI's LFM2.5 encoders, noted in Simplifying AI, are bidirectional and claim to stay fast at long context on CPU. If that holds on real data, the embedding and rerank tier leaves GPU entirely, the highest-volume, lowest-margin line in most RAG stacks. No published metrics accompany the claim. Hold pooling strategy constant, because mean versus CLS versus late chunking will move recall more than the checkpoint does, and measure nDCG@10, recall@50, and CPU tokens/sec at 512, 2k, and 8k context.