The Scientist

The Convergence

Six independent sources this week — a security benchmark vendor, an enterprise agent company, a QA engineering team, a vulnerability researcher, and two conference keynotes — landed on the same conclusion: the scaffolding around the model contributes more to task performance than the model itself. This stopped being a vibes claim. It has numbers now.

Twenty points of eval accuracy live in the scaffolding, not the model checkpoint.

The Data Points

Why This Matters Now

Sonnet 5 shipped this week with a 5-point agentic coding gain (58.1%→63.2%), and the instinct is to upgrade. If Semgrep's result holds — and the mechanism (endpoint context, retrieval, structured framing) generalizes — then a 5-point model gain is noise against a 14-22 point scaffolding gain nobody has captured yet. The thing the model leaderboard doesn't tell you is where the remaining accuracy actually sits.

The zero-day researcher's methodology is the cleanest read here. He used a non-SOTA model as a high-recall candidate generator, then applied human precision via manual review. The model surfaced irregularities. The human wrote the PoCs. That's an asymmetric-role pipeline where model tier moves recall at the margin but not final output quality, which is gated by the human filter and the harness design.

The LinkedIn Pattern Worth Stealing

LinkedIn's QA Agent runs a System 1/System 2 hybrid: fast deterministic replay when the UI is unchanged, slower agentic reasoning only when it drifts. This is inference cost and latency optimization dressed as architecture — reserve expensive inference for the tail of novel cases. Most agent calls are repetitive, and routing them through an LLM pays frontier pricing for deterministic work.

The Contrarian Check

Sierra's claim is self-serving — they sell the orchestration layer. Semgrep authored the benchmark and sells the winning pipeline. The zero-day researcher's ablation is informal with no sample size. None of this is peer-reviewed. The convergence across independent, differently-motivated sources raises confidence in the direction, not the magnitude. The magnitude is vendor-favorable and needs local validation on your own traffic before it earns a migration.

Where This Breaks

The scaffolding thesis holds best on narrow, well-scoped, repetitive tasks (Sierra cites ~100K identical interactions/week). On novel reasoning, long-horizon planning, and creative generation, model capability still dominates. The correct mental model isn't 'scaffolding always wins.' It's 'for the 80% of traffic that's repetitive, the model is over-specified and the harness is under-built.'

The Per-Task Inversion Is Now Quantified

Wednesday's briefing made the general point that per-token price can diverge from per-task cost. The specific numbers for Claude Sonnet 5 landed today, and they run worse than the general case implied. Three mechanisms compound to take a model that looks 33% cheaper on the price sheet and make it cost 2x more per completed task:

1. Output token inflation: ~69k output tokens/task, 40% more than Sonnet 4.6
2. Agentic turn explosion: 3x the turns on AA-Briefcase/GDPval-AA; max-effort mode uses 6x more turns than low-effort
3. New tokenizer: English text costs 1.4x more at the same list price; Spanish 1.33x; Mandarin roughly flat

Net effect: $2.29/task vs $1.15 for Sonnet 4.6 and $1.99 for Opus 4.8. The mid-tier model is the most expensive option once you price the whole task.

Sonnet 5's price tag dropped and its bill went up. Migrations priced on list will land on a number the bill no longer honors.

The Pricing Clock

The intro window is cheap experimentation budget with a hard deadline. Any cost model built on $2/$10 breaks by 50% on September 1. Budget against the post-promo rate. Treat the discount as eval runway, not a floor.

Where Sonnet 5 Actually Wins

The capability picture is uneven. Sonnet 5 is a coding/agentic workhorse, not a reasoning-frontier model:

• CursorBench: 57% vs 49% (Sonnet 4.6) — clear coding win
• FrontierCode Extended: 53.8% — beats Opus 4.8
• Terminal-Bench: +9 over Sonnet 4.6
• AA Intelligence Index: 53 — only +6, tied GPT-5.5, behind Opus 4.7/4.8
• CritPt physics: 17% — behind GLM-5.2, Opus, Fable, GPT-5.5

Caveat: CursorBench, FrontierCode, and Terminal-Bench are vendor-reported with no independent ablations or error bars.

The Cache Offset Strategy

The 90% cache-hit discount ($0.30/M) is Anthropic signaling that they expect heavy repeated-context loops. For agent scaffolds with stable system prefixes, high cache-hit rates partially offset the per-task blowup. Cline reportedly hits Opus-level Terminal-Bench at 5-minute TTL. If think time between calls exceeds 5 minutes, the cache invalidates and the next turn pays full price. The thing the cache discount doesn't tell you is what your actual hit rate will be under production traffic.

The effort-level system (5 levels including new 'xhigh') behaves like a hyperparameter to sweep, not a fixed setting. A 6x turn multiplier at max effort left ungated is a silent budget hazard.

The Pattern

Three unrelated domains this week exposed the same failure: a pass/fail threshold set by convention rather than decision consequence that produces an inverted signal. The specific metric says 'good.' The downstream outcome says 'useless' or worse.

Case 1: Molecular Structure — 2Å Is 'Slop'

Genesis Molecular's CTO (ex-Meta, led Llama 2/3 training) argues the academic community's 2Å RMSD 'good pose' threshold is deceptively generous. Hydrogen bonds have a 0.6Å valid range, so a model scoring 1.8-1.9Å looks passing on the leaderboard and is operationally useless for drug discovery. The real bar is 1Å. Their PEARL model beat all public models on 802 never-before-seen co-complexes with zero fine-tuning — validated by a temporal-cutoff holdout where the template PDB was released after training.

Case 2: Attribution — Revenue Rose While the Metric Collapsed

A brand's first-click attribution model reported post-click conversions falling from ~80% to ~20% after increasing video spend. The metric triggered panic. Incrementality testing showed revenue actually increased. The attribution model structurally cannot observe view-through effects, so any channel whose value is impression-based (CTV, upper-funnel video) is systematically undercounted. A biased label with the wrong sign drove teams to kill what was working.

Case 3: Auto-Merge Rate — Volume ≠ Quality

Warp's Oz platform reports auto-merge rate climbing 20%→60% as companies gain trust. But auto-merge measures how often the loop closes without a human — not whether the merged PRs were correct. No revert rate, no post-merge defect rate, no coverage delta. A rising merge rate can mean the system improved or the review gate got more permissive. Same metric, opposite conclusions.

The Transferable Lesson

A model that scores 1.8Å looks like it passed and is still useless; your benchmark threshold, not your architecture, is usually the thing lying to you.

The common failure mode across all three: aggregate scores hide per-slice failures that concentrate exactly where stakes are highest. The 5.1-point Sonnet 5 gain (58.1%→63.2%) could sit entirely in short, well-scoped tasks while long-horizon agentic runs barely shift. The 12,000-response citation study found a 5x gap between single-brand queries (39% mention-to-citation) and open-ended category queries (7.2%). Aggregating those is reporting fiction.

Your Audit Framework

For each pass/fail threshold in your eval harness, answer: does this threshold map to a real downstream decision consequence, or to a community convention? Document where an 'acceptable' aggregate could still produce operationally useless outputs on the slices that matter most.