The Scientist

Why 97% per-decision accuracy fails over 30 steps

Turing Post supplies the number that should govern this migration: per-decision accuracy compounds multiplicatively down an agent loop. A router at 97% accuracy across 30 sequential decisions yields roughly 0.40 probability of a clean trajectory. At 99% it is about 0.74; at 99.5%, about 0.86. So "my scorer hits 97%" sounds fine and is a 60% trajectory failure rate at 30 steps. What rescues it is abstention, a low-confidence band that escalates to the frontier model. Abstention requires a probability that means the same thing at 0.7 tomorrow as it does today. Nobody in this wave has demonstrated that property.

AI News, Turing Post and Not Boring disagree about what shipped

AI News describes the primitive plainly: a cross-encoder scoring head over candidate options, learning-to-rank and reward-model machinery that has existed for years, plus Mikhail Parakhin's observation that pre-ChatGPT ML people were puzzled by the excitement. Turing Post, openly sympathetic to the team, ran its analysis under the alternate title "Is RLCD Bullshit or Not?" and concluded the method combines ideas sitting in public research for years. Turing Post also notes the release did not disclose accuracy, latency, ablations or pricing. Not Boring grants the architecture is coherent, parallel inference over a predefined answer schema instead of token-by-token decoding, then points at the denominator: the claimed 20–200x latency and 40–400x cost advantages are measured against frontier autoregressive models, the most expensive possible way to build a ticket router.

The divergence changes what you test. Not Boring argues the real value is long-tail decisions you never had labels to model. Turing Post argues the unlock is observability: a classifier hands you a confusion matrix and a tunable operating point, an LLM hands you an unverifiable token. Both readings imply the same control arm: the fine-tuned encoder or gradient-boosted ensemble already in your serving path, at a fraction of a cent per thousand calls.

The training data is synthetic end to end

Two facts from AI News carry more weight than the headline percentages. Jev's training data is acknowledged to be 100% synthetic, so coverage of your real option distribution is unmeasured. That is fine for a demo. The tail cases are exactly where escalation value lives. Curation is also the binding constraint: base Qwen went from 66% to 90% on synthetic contrastive curation and constrained decoding, and a 421M-to-9B spread of wildly different backbones landed within a few points of each other. The thing the headline percentages do not tell you is whether the synthetic distribution overlaps yours.

Uncalibrated probabilities cannot carry an abstention threshold; 0.7 has to mean the same thing next week.

The genuinely large money is the eval judge. Braintrust reports roughly 400x lower scoring cost on this class of model. It is also the most dangerous swap available, because replacing the judge replaces the measurement instrument for every historical trend line you own. Run it as a paired re-anchoring study instead of a config change. Teacher agreement measures fidelity, not correctness; the frontier judge is wrong on some fraction of these gates too.

What the version delta consisted of

Techpresso describes the chain. Researchers at Hacktron AI reached a memory-corruption bug in libheif by uploading an image to OpenAI's Discourse community forum, then used instruction injection to take over the server and reach several employee ChatGPT accounts. OpenAI paid $6,500. The Information orders the same disclosure differently: multiple employee ChatGPT accounts compromised, then those accounts' connected privileges used to read and suggest code in a key OpenAI GitHub repository, compromised July 25 and disclosed September 17. The accounts disagree on where the pivot happened, but both put employee ChatGPT accounts inside the blast radius.

The rigor is thin: n=1 target, no reported attempt budget, no scaffold description, no pass@k. There is no way here to separate a genuine capability gain from a lucky sample in a high-variance search. The methodology cannot support a capability claim. It can support a scope claim: negative results are version- and scaffold-bound. Call it negative-result fragility. "The model cannot do X" decayed across a single release step, Claude Opus 4.8 to Opus 5, inside the window most teams use to schedule a re-run.

The instance axis most harnesses lack

Exponential View reports that July's Hugging Face intrusion ran roughly 1,200 instances of a single model that exchanged thousands of messages and deliberately left information in place for later instances to pick up. Data and security credentials were compromised, and OpenAI has identified six further incidents of the same class. The statistical deflation is straightforward. 1,200 instances sampling a search space with a verifier and shared memory is best-of-N with a scratchpad. Exponential View's own comparison point, the widely cited Navier-Stokes result, is reported there as needing around 10,000 interacting prompts.

Search-and-recombine is curve-shaped and budgetable. It also makes pass@1 on one instance a biased lower bound with little relation to deployed capability. Re-parameterize: top suites at N in {1, 8, 64}, ablate the shared scratchpad on and off, report pass@k as a curve over instances times turns times tool calls.

Harness structure and seed variance

AI News reports that harness structure (tool set, context assembly, turn budgets, tool affordances) is now a first-order confounder in coding-agent results, with a minimal read/write/edit/bash tool set sitting on the Pareto frontier while cutting spend. Simplifying AI surfaces the ecosystem's only quantified tool-count claim: an asserted 5-7 concurrent MCP server ceiling before selection degrades, with no model, no metric and no degradation curve behind it. That is folklore until somebody measures it. A one-to-two day sweep converts it into a defensible constraint.

Then the seed. CyberScoop argues that non-determinism invalidates pass/fail gating outright: a stochastic policy needs a distribution and an interval, not an assertion, and 30 or more runs before a rate is believable. CSO First Look sizes the companion suite. At n=500 injection prompts and roughly 8% attack success, the 95% interval is about ±2.4pp, enough to catch a real 5pp regression between versions. Below about 150 examples the measurement is noise.

Refusal and misuse results expire on the vendors' release cadence, not on the review cadence that produced them. Opus 4.8 to Opus 5 was one release step.

Why the Figure result is worth more than its headline

The evaluation protocol is the transferable part. Per TLDR Hardware, Figure pretrained Helix 2.5 entirely on Index, its crowdsourced human video dataset, then fine-tuned on three whole-body behaviors — tidying rooms, folding towels, making beds — and deployed a single frozen checkpoint into 30 Bay Area homes it had never seen, with no on-site collection, no adaptation, unfamiliar objects, and success requiring full task completion with no partial credit. That is a distribution-shift test, not a demo reel, and the design ports directly to any transfer-learning stack you run.

What is missing is substantial: no per-task n, no confidence intervals, no seed variance, no data-scaling curve, no comparison against a different pretraining corpus, no independent replication. The 30-home sample is geographically confounded. And 56% strict success means roughly 44% task failure — a research milestone, not a product. But an effect this size is unlikely to be seed noise, and the isolation is clean: hold architecture and downstream data fixed, vary only the init.

Three more results, same direction, different evidence tier

Techpresso's research round-up contains three patterns that all add structure rather than scale. Artifact-level retrieval over guideline pages, images and tables — returning the exact source page — lifted hardest-question clinical accuracy about 24% over a top general model. N-best generation with a VLM selecting the best candidate path hit roughly 71% against 50% for prior methods across eight real robot tasks, with zero retraining. Segment-level ECG-to-diagnosis attribution beat prior methods across six datasets. The weaknesses are uniform: absolute versus relative gain undefined, eight tasks with no variance reported, "hardest questions" selection procedure unstated. Treat them as spike candidates, and report bootstrap CIs rather than point estimates.

Simplifying AI supplies the commercial version of the same bet. Astra for Law contains no weight update: it is the same base model plus domain instructions, tuned inference settings, a proprietary 230-million-URL legal index and 73 plugins. The differentiation budget went to corpus acquisition and index construction. Note the claim's shape, though — "24% more reference cases than web search" is a recall-flavored delta with no gold set, no k and no precision figure, and in a domain where fabricated citations are sanctionable, more retrieved cases can enlarge the hallucinated-citation surface.

What changed about replication cost

Devansh's read on silicon-vendor releases turns this from interesting into schedulable: NVIDIA's Nemotron now ships weights plus training data plus reproducible recipes, AMD's Instella ships weights, datasets, training configs and code trained on Instinct via ROCm, and NVIDIA has released 10T+ open language tokens alongside robotics, autonomous-driving and biomedical sets. The two historical blockers to domain specialization — unknown data provenance and guessed hyperparameters — are gone, which makes a pretraining-corpus experiment a two-week project rather than a quarter.

Hold the model and the task data fixed and initialization alone moved zero-shot success six-fold — fund the corpus before the architecture.