The Scientist

A benchmark with 30% malformed tasks doesn't degrade gracefully. Overly strict tests, underspecified prompts, and misleading instructions create systematic bias toward models that overfit broken cases, and 30% contamination is enough to reorder a leaderboard. So every model-selection or release-gating decision citing SWE-Bench Pro this quarter, yours or a vendor's, is noise dressed as signal. Grok 4.5's 61.5 and Meta's coder numbers on that suite are unverifiable until the task set is fixed.

The evidence this week converges on one point:

• Saturation: GPT-5.6 reportedly crushes standard benchmarks while faceplanting on ARC-AGI-3. Public suites measure memorized capability; divergence from novel-reasoning holdouts is the tell.
• Wrong axis: Raschka's reasoning-budget curves and Databricks' pass-rate-vs-cost-per-task plot on a multi-million-line codebase both show the top-scoring configuration is not optimal under a fixed budget.
• Harness dominance: Pi ran 1.20x and 2.08x cheaper than native tools at comparable pass rates. Scaffolding moves cost more than most model swaps.

The unit of evaluation is no longer model score on a public benchmark. It's (model + harness) position on a private cost-quality Pareto frontier. That artifact can't be bought. Every vendor number this week is a point estimate with no CIs, no sample sizes, and a demonstrated 30% chance the underlying task was broken.

Caution: the retraction came from OpenAI auditing a benchmark where competitors were posting wins. Credible and specific, but the exact 30% is their number, not an independent audit. One more reason to own your denominator.

The build is days, not months. A workable version is about 100 tasks from your own repos, test correctness verified by hand, cost-per-task and tokens-per-task logged next to pass rate, wired into CI as a canary. Then every future vendor claim is testable in an afternoon.

The arithmetic is unforgiving. Agent tasks fire many sequential calls — planning, tool use, analysis — with accumulating context, but the prompt cache is keyed to the model. Route call #7 to a cheaper model and the entire context re-bills cold, forfeiting the ~90% discount. The counterintuitive result: a cache-hot expensive model can cost less per task than a cache-cold cheap one. Per-call cheapest-model routing, the default in most agent stacks, is a cost regression that no per-call price sheet will show you.

The production fix is a 4-stage pipeline that evaluates every request but commits the model once per session:

1. Guardrail filter screens requests.
2. Tiny router model — Arch-Router, 1.5B on HuggingFace, trained on human preference data (what developers actually pick per task type), routing preferences in plain English. The size constraint is the point. Routing that costs as much as the call it routes saves nothing.
3. Selection policy picks by cost or speed from dynamic pricing catalogs, keeping a runner-up on standby.
4. Model affinity — session-ID pinning preserves cache warmth across the multi-call task.

Plano (open-source, Katanemo) implements all four as a local proxy: one YAML config, zero agent code changes, plus a per-request cost console showing which model answered and what it cost. It reportedly cut Hermes agent usage 2x. Caveat: single agent, no controlled A/B, no task mix disclosed. Read it as an order-of-magnitude signal, not a benchmark.

Instrument the failure mode before adopting. Pinning locks a bad initial routing decision in for the task's lifetime, so you need post-commit quality monitoring and a runner-up fallback. The thing the 2x doesn't tell you is how often mid-task switches trigger cold re-bills on your workload. Most teams have zero visibility into that number, which is exactly why measuring it first is nearly free to justify.

The cost assumptions set at the late-2025 trough are now materially understated. Silicon Data's H100 one-year contract index rebounded from $1.70/hr to $2.35/hr, 38%, with spot up 10% YTD. Two independent series moving together is a credible demand signal. The thing this doesn't tell you is the index's tenor and region mix, which are undisclosed. Even granting that, a 38% swing flips build-vs-buy and reserved-vs-on-demand math that looked settled in Q4.

The memory leg compounds it. Micron's gross margins exceed their 2018 peak, memory stocks are up +200% to +700% YTD, and 2023 factory-investment shutdowns put new capacity years out. If you run HBM-backed GPU tiers, high-RAM serving nodes, in-memory feature stores, or managed vector DBs, this spike is already in your bill.

The contradiction worth acting on: hardware unit costs are rising while frontier API sticker prices fall. xAI's and Meta's new pricing is explicitly subsidized land-grabbing. That spread does not persist. Either API prices reprice upward when the traction phase ends, or entrants exit. Hard-wiring today's $1–2/M input pricing surviving 18 months is a structural risk, not a forecast.

The negotiator's tell sits in SpaceX's S1: a 3-year infrastructure deal with 90-day cancellation terms. In a genuinely runaway-demand market, providers don't hand buyers 90-day exits. Sophisticated buyers are paying today's prices while hedging supply relief. Structure the same way instead of locking rigid multi-year commits.

Rising per-hour costs also raise the payoff on right-sizing. A smaller model plus better retrieval that holds quality on your eval harness is worth more at $2.35/hr than at $1.70. The cheapest GPU-hour is the one you stop needing.

The most transferable methodology this week came from security and SRE, not ML research. Autonomous hackbots surfaced 126 real vulnerabilities in five months despite heavy hallucination, using a generator-verifier split. One agent hunts. A second rejection-biased adversarial critic tries to kill each claim before output, with one variant routing rejects to a literal 'hallucination bin.' That is self-consistency restructured as an explicit two-role pipeline biased toward false-positive suppression, which is the opposite of chasing recall. Anywhere a wrong answer is expensive, meaning extraction, RAG grounding, or agentic tool calls, it is a cheaper first move than fine-tuning a classifier. Caveat: no precision/recall was published — rejected-claim volume is the real efficacy metric, and it's undisclosed.

Second: OpenAI fixed an 18-year-old bug by analyzing core-dump signatures across the whole fleet, not one crashed instance. Call it debugging as epidemiology. Most drift detection alerts per-request or per-batch. The thing that framing doesn't tell you is where systematic failure lives. One misprediction is noise. The same failure concentrated in one tenant, feature bucket, or geography is signal. Aggregate first, drill second. That is where the stale-encoder or poisoned-cohort equivalent of an 18-year-old bug hides.

Third, the authority gap. Reviewers are asked to catch errors precisely when they have the least context. Approving an LLM answer without the retrieved passages, confidence, or provenance is not oversight. It is rubber-stamping that manufactures false safety assurance. The fix is interface design, not headcount. Surface what the model saw at decision time.

Put together: verifier agents suppress bad outputs upstream, population-level slicing catches systematic failures midstream, context-rich human review handles the residual downstream. No new models. Only re-plumbing what you already run.