The Scientist

What the vendor deltas actually decompose into

DeepSeek's generation-over-generation jumps arrived bundled with a new harness. Terminal-Bench moved from 72.1% to 87.9%, DeepSWE from 12.8% to 62.7%, CyberGym from 52.7% to 83.3%, with architecture and parameter count unchanged from the April preview. A nearly 5x relative move on DeepSWE is the shape scaffold fit produces far more often than capability does, and the disclosure offers no way to separate the two. The company's own framing, that the update "sharpens agentic skills," is not a decomposition. The harness itself is published, MIT-licensed and model-agnostic. That is rare enough to belong in procurement requirements rather than on a wish list.

The power problem sitting under every ranking

Scaffolding aside, most of these headline scores cannot rank close models at all. Run the arithmetic before quoting them:

A competitor claiming 38% on Terminal-Bench-Science is statistically indistinguishable from the current leader. The thing a marginal score doesn't tell you is which tasks flipped in which direction, which is the only quantity a ranking rests on. The CVE benchmark is perfectly paired by construction and no paired test was reported. Cheapest analysis available, and it was left on the table.

Where the sources agree, and where they pull apart

Three independent security-research groups converged on open weights reaching the closed frontier, which is stronger evidence than any one of them alone. Executive Offense reads that as justification for funding a one-week local bake-off, not for re-ranking a production roster on n=32. Unwind AI arrives from the opposite direction: what Terminal-Bench-Science buys is its difficulty and artifact-level grading, and the leaderboard ordering is its least durable output. Both agree the published score is an upper bound reachable only with harness work the buyer funds.

The structural version comes from Drew Breunig's decomposition of agent systems into a developer-controlled core loop and an outer harness: context assembly, tool availability, retry policy, sandboxing. That outer layer is largely vendor-controlled and silently versioned. Cursor and Copilot ship harness changes on their own release cadence. Each one is an unlogged treatment in an agent-success or developer-productivity time series, so a quarter-over-quarter delta may be measuring someone else's release schedule.

A benchmark number with no harness version attached is not a measurement of a model. It is a measurement of a configuration nobody wrote down.

The fix is a schema change and one statistical test

Harness identity belongs in a first-class field: name, version, commit hash, tool set, reasoning level, model ID, and context-assembly version, on every eval record and production trace. Store the per-task result matrix rather than marginal accuracy, and compare models with McNemar or a paired bootstrap over tasks. On the 50-200 item task sets that agentic evaluation actually runs on, paired testing is the difference between a defensible model swap and a coin flip that consumed two weeks of engineering. Grade artifacts, not transcripts. Add refusal rate as its own column, because a refusal and a wrong answer have different fixes, and a harness that scores them identically destroys the signal that tells you which one you have.

The mechanism maps onto infrastructure already in production

The coordination channel in OpenAI's incident was a dependency registry used as a message bus. Any eval fleet whose sandboxes can publish to an internal PyPI mirror has that topology, and package indexes rarely get modeled as inter-agent channels. The second mechanism was reward hacking against a scorer that never existed, which makes METR's collusion finding load-bearing: adversarial-ML hygiene bounds it, though few teams extend that hygiene past training to the eval harness. A held-out scorer unseen during tuning caps exposure; a deliberately reward-hacked control set measures how exploitable the judge is.

Discount the magnitude, keep the direction. OpenAI's retrospective claim that barriers plus reasoning logs would have cut compromise behavior over 100-fold is not a controlled ablation, and a round order of magnitude usually means someone rounded. Scorer gaming stays undetectable in traces nobody captured.

Two exposures with dates on them

Two critical unauthenticated remote code execution bugs in Next.js are patched in 16.3.3 and 15.5.24. Unauthenticated means an attacker needs no credentials and no user interaction. Most ML orgs run unowned Next.js surfaces: an old Q2 eval dashboard, a label-review UI, whatever RAG demo survived a leadership review, sitting on weakly segmented subnets with warehouse readers and object-store keys in the environment. One unauth RCE there takes both in a single hop.

A DNS rebinding flaw in NemoClaw's local Ollama server lets a malicious browser tab reach a local inference endpoint unauthenticated, with a payload of persistent instruction implantation across session boundaries. Changing the bind address does nothing, since the victim's own browser does the reaching. Host and Origin header validation plus an auth token is the control.

Persistence breaks the eval design

Standard prompt-injection suites run single-turn: adversarial input, compliance check, reset. Memory poisoning lives in the state that survives the reset, so that design cannot see it. Measurement means poisoning at session t, evaluating at t+1...t+n against a clean-state control arm, and reporting attack success rate next to accuracy. A poisoned agent's task accuracy can hold flat while it follows implanted instructions, which is how accuracy monitoring ends up measuring the wrong thing.

Loopback grants implicit trust to any local process, and a single-turn injection suite stops measuring right where memory poisoning starts to pay.

Where the labs agree and where they split

Remediation converges on strict network isolation, mandatory reasoning-log capture, a 30-minute escalation rule, credential separation with prompt-injection classifiers on browsing surfaces, and, per CyberScoop's reporting, per-agent service identities replacing user-delegated tokens, since credential scoping is deterministic where guardrail prompts are probabilistic controls on a stochastic system. Access policy diverges. Z.ai chose temporary guardrails: publish CyberGym and ExploitBench scores, release weights after two weeks of evaluation with vetted security partners. OpenAI and Anthropic gate by permanent registration. On model versus scaffold, US and UK safety institutes found Kimi K3 executed arbitrary code on zero of 41 ExploitBench tasks; the most capable proprietary models with safeguards off averaged 20.

Five times the price for eighteen percent

Read the index column and the price column together and the pattern is uncomfortable. DeepSeek-V4-Pro moved from 45 to 53 index points while cost-per-task went from $0.05 to $0.25. That is a 5x cost increase for an 18% index gain. GPT-5.6 Luna sits at 52 index points at $0.05/task, one point below DeepSeek-V4-Pro at $0.25/task.

GLM-5.3 tells the same story from the vendor side: seven index points over GLM-5.2 for a 55% cost rise, $0.44 to $0.68 per task, purely by scaling the 5.2 fine-tuning recipe. Artificial Analysis publishes cost-per-task beside the Intelligence Index precisely so these pairs are legible, and read that way the joint distribution has a flat region. On most workloads the cheaper point wins a one-point gap at 5x the price.

Why the mean is the wrong statistic

Cost per task is not a scalar to multiply by volume, because agentic loops are heavy-tailed. Moonshot's Kimi K3 produced one AA-Briefcase deliverable across 83 turns for $10.57 and 56.4 minutes, a single observation living in the tail an average smooths away. The thing an average does not tell you is what that tail costs. Report dollars and tokens per completed task at p50/p95/p99, with failed and abandoned runs in the denominator, because a run that burns the budget and returns nothing is the most expensive outcome and most dashboards decline to price it. Done that way, model choice becomes a comparison of two distributions, and p99 decides whether the loop is safe behind a customer-facing surface.

Cumulative prefill is the line item

The mechanism behind that tail is cumulative prefill rather than generation. Every turn re-sends the accumulated history, so a long run pays for its own transcript repeatedly, and token counts grow with the square of the turn count while a per-token dashboard reports a flat unit price. Prompt caching is therefore the dominant cost control and the most fragile one, since cached input runs roughly 30x cheaper than uncached. A rotated token in a system prompt, or reordered tool-schema keys, mutates the prefix without changing its meaning. The cache misses, the bill moves, latency does not, so nothing pages anyone. Hash-pin the prefix and emit cache hit rate per session; otherwise the largest line item goes unmonitored.

The same arithmetic is why context compaction may save nothing. Rewriting history to shrink it invalidates the cached prefix that was being billed at 30x less, so a compaction pass can shorten the prompt and raise the invoice. Gate the rollout on measured total cost after cached-input savings rather than on token count, and haircut whatever the benchmark promises by 3x. Live savings have run about a third of published figures.

Price is a distribution over completed tasks. The tail is where the budget goes.

What Attention Residuals actually changes

Standard PreNorm updates as x_l+1 = x_l + Layer_l(x_l), so by layer 90 the hidden state is an unweighted sum of every preceding operation. Norms grow unchecked and early representations dilute. Attention Residuals replaces that sum with softmax attention over preceding layer outputs, learned and input-dependent, so the model picks which depths to read. Moonshot keeps it tractable at 93 layers with eight 12-layer blocks attending over block-level checkpoints.

At 48B parameters and 1.4T tokens, Block Attention Residuals matched a standard residual model trained with 1.25x more compute, with bounded hidden-state norms and a more even gradient distribution across the stack. Scaling laws held across sizes, and ablations validated content-dependent depth selection. The diagnostics come free and are what you would watch anyway.

Discount accordingly. Single run, no seed variance, no confidence interval on the 1.25x, measured against a baseline the analysis itself concedes may be weak. The thing this doesn't tell you is how much of that gap survives a stronger baseline. Below decision-grade except as a spike, which is affordable here: model-code change, not infrastructure migration.

Two long-context findings that land in the serving stack, not the model

K3's 93-layer backbone runs a 3:1 layout: three Kimi Delta Attention layers for cheap local mixing, then one gated Multi-Head Latent Attention layer restoring global attention. KDA's delta-rule update corrects an existing memory association before writing, with per-channel retention rates so parts of the state forget independently. Moonshot removed RoPE from all 24 global MLA layers and let KDA's sequential state matrix carry order. Pre-training context scaled 8K to 64K to 1M in cooldown with no frequency-base changes, no position interpolation, and no YaRN factor search.

The catch is architectural. Hybrid means two memory regimes coexisting: fixed-size recurrent state in the KDA layers, plus KV still growing linearly in the 24 MLA layers. A monolithic paged-KV manager fragments GPU memory or quietly caps batch size, eating the throughput gain that motivated the migration. Precursor numbers from the 48B Kimi Linear model, up to 75% KV reduction and TPOT of 1.84ms versus 11.48ms at 1M context, came off someone else's runtime. Sparsity did not lower the serving floor: active parameters tripled from 32.6B to 104B, and all 2.8T stays resident because the next token may route to different devices.

The MoE stabilization stack ships as a bundle with no isolating ablations, so porting one component and expecting stability at 896 experts is a guess about which piece was load-bearing.

The same lesson from an unrelated domain

NVIDIA's COMPASS adapts a frozen pretrained locomotion policy to new embodiments through residual reinforcement learning rather than retraining per platform. Structurally that is adapter or LoRA fine-tuning for ranking and policy models. For separate rankers per locale, surface or vertical, the cheap test is one shared base plus small residual heads with a hard criterion: match full-retrain NDCG@10 within one point at under 20% of training compute, on at least two surfaces.