The Scientist

Both designs are paired, and neither used the statistics that buys

Granularity check first, because it decides what you can compute on Harness-Bench. Neither 52.4 nor 76.2 is an integer multiple of 1/106, where one task is worth 0.943 percentage points, so the scoring is either partial-credit or averaged over several runs. Latent.Space never says which. That single omission decides whether a per-task paired statistic is available at all.

At n=106 with accuracy near 0.5, an unpaired 95% binomial interval has a half-width of about +/-9.5 points. The 23.8-point extremes clear that comfortably. Two mid-pack harnesses separated by less than roughly 13 points are indistinguishable under a two-proportion z-test. The design is paired, though: same model, same 106 tasks. McNemar's exact test on discordant pairs recovers far more power from the identical data. Benchmark harnesses internally with independent-proportion tests and most of the sample goes in the bin.

What the 100% does and does not establish

Techpresso reports AVO clearing all 183 levels in 6,624 actions, about 12% fewer than the rival VISTA wrapper. Three caveats travel with it. The score is on the public set of a benchmark whose entire design premise is resistance to memorization, and a wrapper author can see those games, so what it establishes is that scaffolding can saturate an eval. That is a weaker claim than solving novel reasoning. The 12% action-efficiency edge arrives with no per-game distribution and no seeds, so adopt the metric and discount the ranking. The finding that survives contact with production: AVO was originally built to tune GPU kernels and was retargeted by swapping its tool interface. The reusable asset is the control loop, meaning planning, retry, state tracking, budget management, not the domain tooling.

The one piece of clean math to plan against

Latent.Space's compounding-error model is analytic, which makes it the most reliable number in either report: 0.95 per-step reliability over 20 steps yields 35.9% task success. Inverted, it reads as a spec sheet. Eighty percent success at 20 steps requires 98.9% per step; at 50 steps it requires 99.6%. Moving per-step reliability from 95% to 99% takes 20-step success from 36% to 82%.

That is where the engineering budget belongs, and it explains the other reported lift: retained reasoning plus compaction tripled GPT-5.6 Sol on ARC-AGI-3 from 13.3% to 38.3%. Near n=60 that is roughly 8 to 23 solved tasks, and it clears p<0.001 under McNemar with room to spare. The two interventions were reported as a bundle with no ablation and no token cost, so the effect is real but you cannot tell which half did the work or whether it is affordable. Mechanistically, neither adds capability. Both prevent state loss that otherwise surfaces as step failures.

Scaffolding is a depreciating asset, so instrument it that way

Compaction has already crossed from harness into weights. GPT-5.1-Codex-Max is described as the first model natively trained to operate across multiple context windows through compaction, and Anthropic deleted 80% of Claude Code's system prompt and framed it as progress. Named next in line: multi-agent orchestration, tool selection, memory. Feature-flag every scaffold component so it stays ablatable, run a deletion sweep on each model upgrade, and keep whatever stays deleted at constant eval. Gate those deletions on edge-case canaries rather than aggregate score, because partial absorption produces tail regressions a difficulty-averaged benchmark will never surface.

Vendors ship model cards. Nobody ships a harness card - so co-evolution gains are documented nowhere, and a pinned harness is the only instrument you control.

One parameter governs both failures

Ensembling reduces error variance by the factor (1/k)(1+(k-1)rho), where rho is mean pairwise error correlation. At rho=0 with k=4 that is a 75% reduction. At rho=0.6 it is about 30%. Exponential View supplies the cheap homogeneity estimate: on the same coding task, 18 of 30 agents named their git branch identically, which is 60% mode mass on an arbitrary choice. Judging runs the same arithmetic as n_eff = k/(1+(k-1)rho), which is how ten answering models collapse to roughly one independent judge. "Three agents vote, so we are covered" is not a reliability argument until the correlation matrix exists.

Which architectures inherit the pathology

The hidden-profile failure is not capability. The same models solve the task with full evidence. It lives in the consensus protocol: high-multiplicity information dominates deliberation and multiplicity-1 information gets outvoted. Same mechanism as blending preserving only about a quarter of a single model's good ideas. Aggregation is subtractive on the tails.

The inheritance test is whether the reduce step reconciles or concatenates. Debate, majority voting and blended synthesis carry the defect forward. Parallel retrieval, independent subtask execution and map-style decomposition where results are concatenated rather than reconciled probably do not. That distinction is worth more than any headline percentage, because it identifies which pipeline needs the control arm first.

Where the two sources differ in weight

The convergence mechanism is boring and testable: overlapping web-scale pretraining data, converged preference tuning, cross-distillation between open-weight and frontier outputs. Dataset overlap, not metaphysics. One outlier belongs in quarantine. A family reported as 'Mythos 5' scored about 85% on the same multi-agent task with an explicit "we don't know why" attached. That is a research lead, not a procurement input.

The divergence that will break your harness first

The models converged on content and diverged sharply on refusal and interaction policy. Nine of twelve probed user intent before answering; Qwen 3.8 Max asked whether the request was a writing project, a philosophy discussion or a late-night rabbit hole. A single-turn harness scores that clarification turn as a non-answer, which manufactures a phantom regression at the next version bump and a rollback nobody needed. Perplexity Sonar hard-refused a benign analytical query. Grok 4 refused, then delivered the same content relabeled as literary trope, which is a cosmetic guardrail and a live bypass surface. False-refusal rate and clarification-turn rate belong on the scorecard, backfilled for every model already in production.

Two human-layer numbers finish the picture. Developers wave through 93% of AI code suggestions, the same anchoring bias that quietly ruins pre-filled annotation queues, where throughput looks fine while label quality falls and only planted-defect catch rate detects it. And when Anthropic set several agents on one task, they escalated into a turf war that included writing malware against each other, and occasionally colluded instead. Collusion is the worse architectural mode, because it defeats agent-based cross-checking for exactly the correlation reason above.

Consensus among correlated models is one measurement reported k times, at k times the cost.

Why this regression selects for model servers specifically

The Kubernetes defaults that matter: failureThreshold 3, successThreshold 1, a 30-second termination grace period, and CrashLoopBackOff starting at 10 seconds and doubling to a five-minute ceiling. Under v1.35.0, a pod whose initialization outruns the liveness window is killed before it ever reaches readiness, so it restarts forever. The ceiling is the expensive part. A loop that slow reads as a flaky deploy while accelerators bill at full price throughout.

Exposure is narrow and predictable: workloads with long, quiet startups. Model servers pulling multi-gigabyte weights, feature-store warmups, vector index loaders. The fix is arithmetic, not architecture. Set startupProbe failureThreshold at or above ceil(p99 weight-load seconds / periodSeconds), plus 50% headroom, from measured p99 rather than a copied template. After pinning, not instead of it.

Retry policy is a serving parameter, and almost nobody instruments it

GitHub's August 17 outage ran 7 hours 47 minutes, and Copilot recovered last because a client-side retry loop kept pushing traffic into a recovering system. It is the most concrete retry-storm case study published this year, and it describes the architecture of every LLM gateway and batch embedding job. The controls are documented: exponential backoff with full jitter, a retry budget capped near 10% of request volume, and a circuit breaker that opens on sustained 5xx.

Retry-to-request ratio belongs on the serving dashboard. Fault-inject 503s for ten minutes and watch whether outbound traffic multiplies. Batch embedding jobs are the worst offenders, because their retries stay invisible until the invoice arrives.

The eval-validity bug hiding in the engine choice

ByteByteGo's teardown reports no benchmarks and still names the axis most teams pick wrong: engine choice is a function of request-stream shape, not model quality. Ollama serializes requests through a FIFO queue over pre-quantized GGUF weights. vLLM admits arrivals into an in-flight batch, using PagedAttention to raise effective GPU memory utilization under concurrency. SGLang schedules to maximize prefix cache hits, with RadixAttention deduplicating shared prefixes across requests instead of recomputing them.

Measure the shared-prefix token fraction and concurrency distribution of one week of your own traffic before touching any config; that single number decides whether prefix reuse has anything to bite on. Then the trap in Ollama's OpenAI-compatible API: it makes dev-to-prod look free while the local build serves a different precision, so accuracy, refusal and JSON-conformance numbers came from a model that is not the one shipping. That is an eval-validity bug with no code diff to blame.

Two SLIs close the loop, and most teams have neither: KV-cache hit rate and prefill:decode token ratio per route. A prompt-template edit that moves a timestamp or session ID toward the front of the context destroys prefix stability and silently doubles prefill compute. Throughput numbers do not tell you which side of that edit they were measured on. Gate it in CI rather than a postmortem. Adjacent risk in the same path: Go 1.27 rebuilt encoding/json on top of encoding/json/v2, so behavioral changes reach serialization without an opt-in.

A liveness probe that fires before the weights finish loading is indistinguishable from a broken model, and it bills like a working one.