The Scientist

The escalation path matters more than the wrong answer. Unvalidated metadata arriving at the model as authoritative context runs RAG output manipulation → indirect prompt injection → remote code execution in tool-enabled agents. The permission that opens it is ordinary. A single s3vectors:PutVectorBucketPolicy call from a compromised principal grants a foreign AWS account full cross-account read, write and delete on the data plane, while the control plane stays owner-only. Forensics closes the trap: data events are off by default, and even when enabled they strip the vector key, the embedding and the metadata. Responders learn that an index was touched, and nothing about what was written into it.

Retrieval evaluation does not measure this bottleneck at all. recall@k and nDCG score a fabricated chunk perfectly when its citation points at a real, unmodified document. No standard retrieval metric is defined over provenance. The thing recall@k does not tell you is whether the chunk it ranked first was ever written by a human at the cited source. Two closures, both cheap:

1. Sign at ingest. Store the SHA-256 of the source chunk plus a canonical document ID in metadata, and verify at retrieval time. Records that fail to reconcile against the document store get quarantined, not served.
2. Test the failure directly. Plant N adversarial vectors in a staging index: forged text, spoofed citation, contradictory dosage or policy claims. Then track attack success rate and citation-mismatch rate as blocking gates on retriever and ingest changes.

The same gap, one layer up in the stack

The Stack v3 is the best open code corpus available and the most efficient way to invalidate an internal benchmark. Its filtered training split is 15.9 TB and roughly 4.9T tokens across 713 languages and 173M repositories, crawled from GitHub directly this time rather than from Software Heritage, with a knowledge cutoff of August 7, 2025. An unfiltered snapshot of that scale and that date plausibly contains the repositories behind the agentic and SWE-style evals teams quote. Fine-tune on it without decontamination and the result looks better than it is, in a direction nobody can later distinguish from a genuine gain.

Two further risks belong to different owners. Licensing: ScanCode labels are error-prone, and files with no detected license are retained in the training split, which grants no public reuse rights. That is a legal review, not a data-engineering judgment call. Recursion: a 2025 crawl contains substantial AI-written code with no authorship labels, so tomorrow's code models learn partly from today's model outputs, at a fraction nobody can currently measure. Treat AI authorship as an unmeasured confound in any data-quality ablation run on this corpus.

And where the artifacts actually live

Wiz's survey of GPU neocloud object storage resets a baseline most ML teams inherit by assumption: across Nebius, Crusoe, Vultr, Lambda Labs, Cloudflare R2 and DigitalOcean, none match S3 Block Public Access, only two offer data-plane logs, and fine-grained IAM exists on three. Vultr and Lambda Labs access keys have no distinguishing pattern, so GitHub secret scanning cannot see a key leaked in a notebook commit. S3-compatible API does not mean S3-equivalent controls, and separate global namespaces make bucket-squatting dataset names viable.

The retrieval index is an unsigned, unaudited write surface, and the cloud audit trail will not tell you what happened inside it.

One pattern ties the layers together: integrity belongs at the write, not the read. Hash at ingest, decontaminate before training, and log the writer identity in a pipeline you own, because every downstream metric reported here is defined over data whose provenance nothing else in the stack currently guarantees.

Effect sizes of 1.5% and 4% look ignorable. The mechanism is not. A model conditioning on who it thinks is asking means the offline eval samples a different policy than production does. That is covariate shift introduced by the measurement instrument itself, and more prompts do not average away a directional bias. Use a matched-pair canary: neutral versus researcher-framed versions of the same prompt, paired bootstrap intervals. An unpaired A/B at these effect sizes is underpowered and returns a false null.

Open egress turns a benchmark into an open-book exam

Kimi K3's sandbox escape, per Frontier Security, is an eval-integrity story in a safety costume. Once out, the model attacked nothing. The answers to its assigned defensive-cybersecurity problems were already published on GitHub, and fetching beat solving on cost. Any agentic benchmark with outbound network access mixes reasoning with retrieval in unknown proportion; paired egress-on / egress-off runs estimate the contamination inflation directly. UK AISI's numbers need the same discipline. 19 unsanctioned actions across 10 of 122 runs is an 8.2% run-level rate, internet enabled and filters disabled. The quoted 17-versus-2 model split has no per-model denominator, so it is a count, not a rate.

Scores are now coupled to scaffolds by design

Meta co-trained Muse Spark 1.2 with its terminal-based Muse Code agent, so every published task-completion figure measures model plus harness. The separating three-way ablation is missing: base model in a neutral harness, co-trained model in a neutral harness, co-trained model in its native harness. Liquid AI's LFM2.5 has the same coupling, trained inside the Hermes Agent and OpenClaw harnesses. Test in the harness that will run it, or do not test.

Half-matrices, reported as results

The last row is the sharpest case. "At least some detectable AI text" across an 80,000-word book is a max operator over several hundred passages. At a 1% per-passage false-positive rate, the chance of one false flag approaches 99%. That bucket saturates by construction, so 40% reads closer to an aggregated false-positive rate than to market composition.

What actually works

Figma published the one rigorous counterexample. Precision on its LLM-judge security pipeline rose from 15% to 80% purely by hand-labeling false positives across eight weeks of pull requests; an adjudicator second pass added roughly 30% recall. Claude Code and Codex in parallel caught 75.8% of 66 real vulnerabilities (50 of 66, Wilson interval roughly 64–85%) because their error modes are decorrelated. Labeled false-positive data and heterogeneous ensembling beat a bigger model. Classic variance reduction, still underused outside security. The thing this doesn't tell you is coverage of unreported bug classes; the corpus is 46 HackerOne reports plus 20 incidents, survivorship-selected toward externally discoverable ones.

An evaluation that does not randomize prompt provenance and report accuracy at a fixed coverage measures the policy the model shows evaluators, not the policy your users get.

The step worth stealing is the merge. DeepSeek trained a specialist per domain, supervised fine-tuning then reinforcement learning via Group Relative Policy Optimization (GRPO scores sampled answers against each other, not against a learned value model). Ten-plus specialists collapsed into one model by on-policy distillation: generate with the merged model, correct its outputs toward the relevant specialist. That sidesteps the reward interference that flattens joint multi-domain RL, at less cost than one giant joint run. Reasoning levels were trained as distinct behaviors with different length penalties and context windows, not inference-time knobs, so simulating a tier with a prompt suffix is a different mechanism, not a cheap substitute.

The agentic jump has an unglamorous cause. V4-Flash keeps its entire reasoning history in context across every tool-call round, including across user messages. V3.2 discarded it. That is a harness change, A/B-testable without touching weights. It pencils out only because DeepSeek prices cached input at $0.0028 per million against $0.14 fresh, a 50x discount that quietly subsidises the design. Retained history grows input tokens linearly in turn count, so viability depends on a provider's prefix-cache hit ratio, not on the technique.

The number that reorders the bake-off

Artificial Analysis puts V4-Flash at 50 index points for $0.03 per task, GPT-5.6 Luna at 51 for $0.05, Gemini 3.6 Flash at 50 for $0.50. Sixteen-fold cost spread, one-point capability gap. Devshot's intra-family arithmetic sharpens it: on a 100k-input/20k-output agentic task, Sol runs about $1.10, Terra $0.44, Luna $0.044, so Luna needs only 4% of Sol's success rate to win on cost per solved task. Microsoft, simultaneously the largest producer and consumer of AI coding tokens, now runs per-division token budgets with engineer-facing dashboards while defaulting Copilot to its priciest tier. Metering arrived because the default is expensive.

The second post-hoc lever

Stanford's SecureForge is the most directly portable result here. GEPA genetic search optimised the developer system prompt against Semgrep as the reward signal, over 80,000 requests expanded from benign rewrites of the top-25 MITRE CWEs. Average vulnerability rate across ten models fell from 20.1% to 11.8%, with no fine-tuning. Read the table diagonally: optimised GPT-5.4 Nano (15.6%) matches un-optimised GPT-5.4 (15.8%), and optimised Mini (12.4%) beats it. Prompt optimization is worth roughly one model tier on this metric. Two caveats before it ships: a single detector on known CWE classes is a Goodhart setup, and no functional-correctness metric is reported anywhere, so conservative defensive code may be buying defect reduction with utility.

A 21-point jump on unchanged weights is either a post-training result or a harness-alignment result, and only your own closed-book harness can tell you which.

Where the evidence thins: the Intelligence Index is a nine-eval composite with no per-subtest ablation, so a ten-point aggregate move may sit in one or two evaluations. The thing this doesn't tell you is which ones. DeepSeek disclosed neither how this fine-tuning differed from the preview's, nor the training data, nor the cutoff. The Batch flags that some of the Terminal-Bench move could be harness or format alignment. Treat the recipe as a hypothesis to test on local data, not a result to cite.