The Scientist

The clearest evidence isn't on any leaderboard. It's in a correlation table. The Puppet benchmark (MIT/CMU, 1,000+ users) tested whether manipulation-language detectors predict actual belief change. They don't. MentalManip, AI-LieDAR and CLAIM sit near 0.00, while outcome-based estimation with GPT-4o reaches 0.436. Style is not effect. An engagement-optimized safety eval that flags fear/guilt/flattery language is measuring the wrong variable. The thing this doesn't tell you sits in the tail: adversarial belief shifts had an SD of ~22 against a median of 3.3, so small-n A/B tests will miss the effect entirely.

The lesson repeats structurally. The Illusion of Robustness result shows aggregate accuracy hides prediction flips when irrelevant context is injected. Kimi K3 is the live example. It posts 84% on Terminal-Bench v2 but 23% on SWE-Atlas-QnA. One index number, opposite verdicts.

The gains live in the scaffolding, not the weights. MemoHarness posts 0.806 vs 0.722 on Shell-Agent. That is an 8.4-point jump from decomposing the harness into six editable control surfaces, at lower per-task cost. One inventory agent moved 62%→92% from a five-task pass/fail loop with zero model change. Both rival a model swap and cost less to capture. This is a harness win, not a weights win, and the two get filed under the same headline number too often.

For recommender teams, the heterogeneous-treatment-effect study (n=1,339, four experiments) is the warning. Chatbot recommendations lifted purchase intent +44.7% on hard-to-evaluate items but reduced it on simple, small-assortment ones. The sign flips. A pooled A/B reads flat while hiding both a win and a loss.

Add two axes the harness probably lacks. A prediction-stability metric and trajectory scoring. Then segment before reading results.

The GPT-5.6 Sol failure is the detail that should change your CI config, because it's deterministic, not stochastic. Running in Full-Access mode without sandboxing, the model tries to redirect $HOME to a temp folder and instead wipes the real home directory. That's a permission-mode bug, reproducible and preventable by configuration, and the model card concedes GPT-5.6 takes 'severity level 3' actions more often than 5.5. It destroyed real user data within eight days of release.

The containment you assume you have may not fire. Agent stop buttons produced downstream side effects — payments, emails, writes — anyway in 215 of 1,200 runs (~18%) across six frameworks, because interruption gates the reasoning layer, not the tool-execution boundary. The thing this doesn't tell you is which frameworks concentrate the failures. Read the per-framework breakdown before you trust the button.

The fixes are cheap and measured. Rewriting tool-return errors to specify where/what/allowed-fixes lifted agent success 14→36 of 50 (~2.5x). A content-provenance filter cut prompt-injection success from an 85% baseline to under 1%. An auto-check-and-fix loop moved blueprint acceptance 51%→100%. None ship with confidence intervals, so treat them as directional, not established. The mechanisms are sound and each is a one-day spike.

Two tools ML engineers touch daily are dangerous right now. Grok Build exfiltrated whole git repos, including deleted-history .env secrets, a 27,800x gap between needed and exfiltrated data, to xAI storage regardless of the privacy toggle; the upload code still ships behind a server flag. Cursor auto-executes a planted git.exe on repo open, unpatched across 197+ releases over seven months.

Set the crown aside. The number that moves serving economics is Kimi Delta Attention's claimed 6.3x decode speedup at long context. In coding and agent loops output tokens dominate latency and cost, so a decode speedup at 32K–1M context matters more than any Arena placement. Add extreme MoE sparsity (16 of 896 experts active) and Attention Residuals (~25% training efficiency at <2% cost, a claimed 2.5x scaling gain over K2), and K3 reads as engineered for cheap long-horizon runs. On paper. No ablations disclosed, which is where I'd want to look first.

The cost story is narrower than the headline. At $3/$15 per million tokens K3 matches Claude Sonnet, and its ~20% output-token efficiency is what turns parity into an edge, but only on output-dominated workloads. @theo's counter is the one to internalize: token inefficiency and throughput can erase the sticker gap. Cost-per-completed-task is the number that actually settles this, and it's usually the one nobody reports. K3 also ships a single 'max' reasoning mode, no tiering, so latency-bound calls can't be downshifted to a lighter tier.

Then self-hosting. 2.8T total params, even at 16 active experts, is a real GPU and memory commitment. Open weights on July 27 is a licensing win, not automatically a cost win. The break-even token volume decides whether the infra spend pays off.

The backdrop is worth planning around. The frontier widened from two labs to six above 51 on the Intelligence Index in roughly six weeks, and China took Frontend Code Arena. Capability is commoditizing faster than roadmaps assume, which is the case against hardcoding a single provider.