Science & Analytics

The Scientist

The Signal

Nvidia's wrapper carried Claude Opus 5 from 30% to 100% without touching a weight.

Harness-Bench ran the experiment from the other direction, holding one model fixed across 106 tasks and still watching scores span 52.4 to 76.2. That means every ranking you have been using to choose a model was measuring model times harness and crediting the weights. And the ceiling score comes from the public set only, so read it as saturation rather than intelligence.

In Play

  1. Harness Outswings Weights in Agent Evals

    Nvidia's AVO wrapper carried Claude Opus 5 from 30% to 100% across all 183 levels of ARC-AGI-3's 25 public games, per Techpresso, with no weight change. Latent.Space reports the same effect isolated: Harness-Bench held one model fixed across 106 tasks and scores still ran 52.4 to 76.2. Any model ranking you built on end-task success measured model x harness and credited the model. The 100% is public-set only, so read it as evidence that scaffolding can saturate an eval.

  2. Four Agents Lose to One Long Context

    Anthropic ran the hidden-profile test - a group task rigged so the evidence everyone shares points to the wrong answer - on four-agent groups. Most model families reached the right decision in 17-36% of runs, while one agent holding the same full evidence base got it right nearly every time, per Exponential View. Blending several models' answers preserved only about 25% of a single model's good ideas. The control arm your orchestration eval lacks is also the cheaper configuration.

  3. Judge Ensembles Carry One Effective Vote

    One prompt sent to twelve frontier models, including GPT-5.6, Claude Opus 5, Gemini 3 Pro, DeepSeek V4 and Qwen 3.8 Max, returned substantively identical answers by the fifth reply, per Box of Amazing. At the implied error correlation, ten answering models collapse to roughly 1.1 independent judges under k/(1+(k-1)rho). Every consensus threshold in your eval, RAG verification and safety pipelines has been scoring training-distribution overlap. The sweep is n=1 with no seeds, so trust the direction only.

  4. Kubernetes 1.35 Restart-Loops Model Servers

    Kubernetes v1.35.0 fires liveness probes before the startup probe succeeds, so any pod that needs minutes to become ready restart-loops. Chris Short reproduced it on k3s, minikube and kind. Model servers pulling multi-gigabyte weights never reach ready, and CrashLoopBackOff's five-minute ceiling makes it look like a flaky deploy while accelerators bill in full. GitHub's 7h47m outage on August 17 is the paired lesson: Copilot's client-side retry loop kept pushing load into a recovering system.

  5. Depreciation Life Beats the Price Hike

    Server OEMs have told customers that Nvidia's rack-scale GB300 and VR200 systems rise about 17% for 2027 delivery, adding at least $5B per gigawatt, The Information reports from two anonymous sources. Chip systems are 60-70% of fully loaded compute cost, so that arrives as roughly 10-12% on amortized $/GPU-hour. Useful life matters more: at four years instead of six, annual cost per unit of compute runs about 76% above today's baseline. Nobody published perf-per-dollar for the new generation.

Deep Dives

  1. The Wrapper Is the Treatment and the Model Is the Control

    Two independent results put a bigger score swing in the orchestration layer than in the weights, and the scaffolding that produces it is being absorbed upstream while you build it.

    Both designs are paired, and neither used the statistics that buys Granularity check first, because it decides what you can compute on Harness-Bench. Neither 52.4 nor 76.2 is an integer multiple of 1/106 , where one task is worth 0.943…

    3 action items

  2. Your Four Agents and Twelve Judges Are One Correlated Voter

    Two very different experiments converge on a single parameter - mean pairwise error correlation - and it decides whether aggregation adds reliability or deletes the evidence that settles the answer.

    One parameter governs both failures Ensembling reduces error variance by the factor (1/k)(1+(k-1)rho) , where rho is mean pairwise error correlation. At rho=0 with k=4 that is a 75% reduction. At rho=0.6 it is about 30% . Exponential View supplies…

    3 action items

  3. Two Config Values Decide Whether Your Inference Tier Stays Up

    A probe regression and an unbounded retry budget produce the same symptom - a healthy-looking rollout burning accelerator hours - and neither appears in any model card or benchmark table.

    Why this regression selects for model servers specifically The Kubernetes defaults that matter: failureThreshold 3 , successThreshold 1, a 30-second termination grace period, and CrashLoopBackOff starting at 10 seconds and doubling to a five-minute ceiling . Under v1.35.0, a pod…

    3 action items

The edition continues

Take the signal into the room.

Sign up or log in to read all 3 deep dives in full, plus the final take.

Read the full edition

Continue with LinkedIn