The Scientist

The New Axis Your Eval Harness Is Missing

Accuracy-only model evaluation is now actively misleading for production decisions, and this week's evidence is specific enough to act on. The invoice is where model choice gets decided this quarter, not the leaderboard.

IBM's Granite 4.1 8B spent 4M output tokens on the Artificial Analysis Intelligence Index against 78M for Qwen3.5 9B, a 19.5× efficiency gap at comparable capability. That gap does not appear on any accuracy leaderboard. It appears on the invoice. Factory AI's 13-model code-review bakeoff found a $1.25/PR model beat one costing more than 2× on real pull requests, with token spend uncorrelated to review quality on their harness. Harness engineering, iterating on prompts, tools, and middleware with the model held constant, lifted Terminal-Bench 2 pass@1 from 69.7% to 77.0% in 10 iterations, beating the human-designed Codex-CLI baseline at 71.9%.

The harness, not the model, is where this quarter's agent economics live.

The Cost Crisis Is Already Here

Across 15 companies surveyed by The Pragmatic Engineer, AI coding token spend has grown 10–15× in six months. Individual developer burn is hitting $500/day, and one developer burned $10K in a week because of a caching bug. A 2,000-person SaaS cut costs 30% by changing the default model from Opus to Sonnet, which is a config change rather than an architecture change.

On the eval side, evaluation compute has quietly crossed from overhead to bottleneck. Individual eval runs now cost tens of thousands of dollars. DeepMind's ProEval pattern, surrogate models scoring checkpoints cheaply with full evals reserved for release candidates, is the direction of travel. Netflix published the most portable eval blueprint of the quarter: ~600 expert-labeled golden examples with tiered reasoning and consensus scoring, landing at 83–92% accuracy across four quality dimensions.

The awkward finding: a community benchmark showed the two-word prompt "be brief" matched a purpose-built compression plugin on both token reduction and output quality. Trivial baselines embarrass sophisticated prompt frameworks more often than practitioners admit. The thing this doesn't tell you is whether the plugin still wins on tasks the benchmark didn't cover, which is worth checking before ripping it out.

What This Changes

The binding constraint has shifted. Eval harnesses that report accuracy without token cost are no longer decision-grade instruments. Every model candidate should produce (accuracy, tokens-consumed) pairs plotted on a Pareto frontier. The Granite vs. Qwen gap only surfaces when token counts are logged per task. The Factory AI finding only surfaces when cost is a first-class eval dimension.

Microsoft's cloud margin dropped 5 percentage points to 56%, attributed explicitly to inference-heavy AI workloads including GitHub Copilot. Consumption pricing makes token-per-request a product metric rather than an infra footnote. Teams without per-request token accounting will learn their cost structure through a Finance escalation, not a dashboard.

Disclosure-to-Exploit Has Collapsed to Hours

One seven-day window is a small sample, but the signal is hard to ignore. LiteLLM's pre-auth SQL injection (CVE-2026-42208), which fronts most teams' OpenAI, Anthropic, and Bedrock keys, was exploited within 36 hours of disclosure. LMDeploy's SSRF went from advisory to working exploit in 12.5 hours, and no public PoC was available. The plausible mechanism is attackers feeding CVE advisories into LLMs and getting usable exploits back.

Patch-window SLAs were calibrated for a world where writing the exploit was the slow step. On this data, it is not the slow step anymore.

Pickle Deserialization Is the 2026 Pattern-of-the-Year

Three independent ML frameworks shipped the same bug class this week, unsafe deserialization of attacker-controlled artifacts, each at CVSS 9.8:

On top of that: Claude Code's CVSS 10.0 sandbox escape via symlinks (CVE-2026-39861, patched in v2.1.64), NVFlare Dashboard's auth bypass (CVE-2026-24178, CVSS 9.8) in federated learning deployments, and 73 GlassWorm-linked fake extensions on Open VSX in April alone. That is the marketplace behind Cursor, VSCodium, and most VS Code forks.

The Supply Chain Cascade Is the Structural Threat

The TeamPCP/UNC6780 chain is worth tracing end to end. They poisoned checkmarx/kics:latest on Docker Hub. Bitwarden's Dependabot automation pulled it into CI. The malicious code shipped as @bitwarden/cli 2026.4.0. That is the exact automation topology most modern ML teams run: tag-based image pulls, bot-driven dependency updates, CI runners holding model registry credentials.

In parallel, DPRK's HexagonalRodent is targeting ML and Web3 engineers through fake-recruiter coding assessments that abuse VSCode tasks.json auto-run. The scorecard so far: $12M exfiltrated across 2,726 developer machines in Q1 2026.

LiteLLM deserves separate attention. Running it as a cost and routing gateway funnels every upstream provider key into a single database that sits behind a pre-auth endpoint. The working assumption should be that any keys transiting the service during the vulnerability window are burn-worthy. Rotate first, investigate second.

Two Lever Fixes Cover Most of the Surface

One policy change removes most of this week's blast radius: ban tag-based image pulls and pickle-loads-from-untrusted-sources in CI. Both are enforceable with a linter and a registry policy, and between them they would have blocked the majority of the ML-relevant exposure listed above. The thing this doesn't cover is the pre-auth SQL and SSRF class. That one is still a patch-speed problem.

The 93% Problem

Anthropic disclosed that 93% of AI agent actions are auto-approved by human reviewers. At that rate, the approval signal is not a safety control. It is a gauge that happens to have a human attached. The signal carries near-zero mutual information with risk. An agent that only ever takes the same three actions in risky contexts is not safe. It is under-explored, and this metric cannot tell the difference.

Anthropic's own recommendation is to move from per-action approval to continuous policy monitoring. The metrics that carry actual signal are override rate on high-risk actions, policy-violation count per 1,000 traces, and action entropy per agent. A new agent version that pushes auto-approval from 93% to 96% is not an improvement. It is the same broken gauge reading slightly higher.

An agent eval harness without override rate, cost-per-task, and tool-boundary violations is measuring demos, not agents.

Courts Just Made Autonomy a Design Parameter

A Northern District of California court ruled under Rule 10b-5 that when a platform's AI exercises 'ultimate authority' over assembled content, the platform is a 'maker of fraudulent statements'. This is the first U.S. ruling to reject the premise that autonomous AI output is legally equivalent to user-generated content. Section 230-style shields do not automatically extend to agent decisions.

Most production agentic stacks today ship in the third row and log like the first. The thing this gap doesn't tell you, until discovery does, is whether a human actually closed the loop. Observability stacks capture latency, token spend, and output quality. They rarely capture a first-class authority_level field attesting to human approval.

Offensive Capability Is Outpacing Defensive Measurement

MOAK's agent built working exploits for 174 of 178 KEVs published after model training cutoffs, using publicly available Opus 4.6 and GPT-5.4 behind ordinary scaffolding. XBOW reports GPT-5.5 cut its miss rate from 40% to 10%, with black-box performance now exceeding what GPT-5 managed with source access. Persistence-on-failing-paths halved. That is a planning improvement, and planning improvements transfer to any long-horizon agent task.

HackerOne paused its Internet Bug Bounty because AI-driven submission volume is outpacing remediation bandwidth. The same queue-overflow dynamic will replay inside every org: PR review, model-card approvals, security alerts. Any human-review queue in the ML stack is one capability jump from the same asymmetry.