The Scientist

The detail worth internalizing even if you never run several thousand GPUs

Meta shards two parameter populations under two different regimes: 2D FSDP plus Expert Parallelism for dense parameters, and Fully Sharded 2D Model Parallelism for trillions of sparse embedding parameters. Both are mapped topology-aware onto a multi-tiered fabric with SM-free collectives. The transferable finding is narrower than the architecture diagram suggests: one uniform parallelism strategy across a model with a huge embedding table and a modest dense trunk leaves throughput on the floor. At this scale the jobs are comms-bound, not FLOPs-bound, and adding GPUs to a comms-bound job buys more all-reduce, not more steps per second.

Portability, ranked

Where the disclosure thins

No baseline MFU is stated. The ~10–12% prior is an inference from "2x to 20–25%", not a number Meta published. The 4x FLOPs scaling is entangled with the efficiency work, so any model-quality improvement cannot be attributed between more compute and better utilization. Correlated, plausibly causal, not established. And "no CTR/CVR regression" from MXFP8 arrives with no confidence intervals and no slice-level breakdown. The thing that claim doesn't tell you is where the errors went, because low precision fails first on rare slices and on probability calibration, and neither shows up in an aggregate conversion rate.

Meta stated on the record that the honest ceiling for sparse recommenders is low. That is negotiating leverage in your next hardware request, not an excuse.

What to do with it

The cheapest measurement available is the padding ratio. Take one real batch from the highest-cost sequence model in production and compute sum(true_seq_lens) / (batch_size × max_len). Under 0.6 means up to half the attention FLOPs are multiplying zeros. The fix is a varlen kernel, with no quality conversation attached, which is rare enough in this field to be worth doing first. MFU belongs on the training dashboard next to loss and AUC, because without a baseline there is no arguing for or against more GPUs.

Before any linear-attention migration, run a truncation-depth ablation on user-history length: sweep 64 / 256 / 1024 with full attention and plot the AUC curve. BlockAttention's value is entirely conditional on longer history carrying signal in your domain, which is a research-leaderboard property, not a production one. If the curve flattens at 256, O(L) attention buys approximation risk and nothing else. For MXFP8, gate on paired runs with identical seeds and data order, scoring log-loss and expected calibration error across at least five traffic slices including the lowest-volume decile.

The four failures, and why each metric was blind by construction

Start with the cleanest case. Code-authorship attribution reached 92.6% accuracy identifying 1-of-10 programmers on coding-contest submissions and collapsed to near-zero on real classroom coursework. Same task, same architecture class, different distribution. Contest submissions are short, homogeneous, single-author, stylistically idiosyncratic. Coursework is templated, collaborative, assignment-constrained. The model fit the curation, not the construct.

Second, the same failure in the field with a person on the other end. A Flock automated plate reader misread plates in 71% of the alerts it sent to police in one California town, per MIT Technology Review. Nearly three in four alerts carrying a misread is not an edge case. It is an operating threshold tuned on clean imagery and never revalidated against angles, weather, plate wear and motion blur. One town, no confidence interval, no disclosed sample size, which makes it a directional alarm rather than a population estimate.

Third, a metric that cannot represent the target at all. Memory systems get evaluated on recall@k, MRR and nDCG, all of which score items. When the answer is a relation between episodes, one delayed task blocking three people, the conclusion was never stored as an item, so the metric has no mechanism to register a miss. Widening retrieval scope does not move it. Neither does tuning the reranker or raising top-k.

Fourth, and the one that reaches production soonest: rewriting only the agent's stated rationale, actions untouched, dropped detection from roughly 95% to under 11%. Rationale and behavior are decoupled channels, and only one of them is what the system did.

Where the sources converge

Independently they land on one rule: the score is valid only over the object it is defined on and the distribution it was fit on. They diverge on fixability. The authorship and ALPR gaps are empirical, and an out-of-distribution holdout closes them. The recall@k and rationale-monitor gaps are structural. No tuning makes an item-level metric see a relation, and no adversarial training makes a spoofable channel authoritative. Different urgency, same harness.

If the insight lives in how the facts connect, no reranker will retrieve it — and no recall@k will ever tell you it is missing.

The arithmetic that comes before any detector ships

One more case, the one most likely to reach a decision this quarter. Grant a generous detector at 90% true-positive rate and 5% false-positive rate. At 5% prevalence of the positive class, precision is 0.045 / (0.045 + 0.0475) = 48.6%, a coin flip. At 1% prevalence it is 15.4%. AI-writing detectors also show systematically higher false positives on non-native English writers, which turns an accuracy problem into an asymmetric-loss problem with a person attached. The Bayes table at real prevalence goes on the table before anyone wires a detector to an irreversible action.

The move

Three slices, one harness. An out-of-distribution holdout sampled from the actual deployment population as a mandatory launch gate for every production classifier, with the field-versus-offline false-positive delta logged and alerted. A multi-hop slice of 50–100 gold questions whose answers exist only in relations between episodes, reported separately from recall@k. And agent safety scored on action traces, meaning tool calls, arguments and egress destinations, with an adversarial holdout where the rationale is rewritten benign while actions are held fixed.

Why a request that fits in HBM can still crawl

Prefill is compute-bound; decode is memory-bandwidth-bound, because every emitted token re-reads the whole stored cache out of HBM. That is a bandwidth bill, not a storage bill. Sweeping 42GB of cache per token against one H100's ~3.35TB/s is ~12.5ms of pure memory traffic, an ~80 tok/s ceiling before a single multiply runs. That roofline is our arithmetic, not a published measurement. It is still the number that should govern the next hardware request, because if TPOT is the latency problem, FLOPs are not measuring the bottleneck.

What each optimization actually attacks

• Paged attention. Fragmentation waste from 60–80% down to under 4%, 2–3x throughput, essentially zero quality cost. On vLLM or SGLang that gain is already priced in and does not arrive twice.
• Prefix / prompt caching. 50–90% cost and latency reduction on hits, per OpenAI's and Anthropic's own figures. Realized saving is (hit rate × discount), and hit rate is a function of prompt hygiene alone. One timestamp or session ID near the top of a system prompt forfeits the discount on every call.
• 8-bit KV quantization. 2x, at a claimed well under 1% accuracy cost. 4-bit. 4x, with measurable degradation on multi-needle retrieval specifically, which is the RAG and long-document-agent profile.
• MLA (DeepSeek-V2/V3) holds ~70KB/token against 192–328KB for GQA peers, roughly 4–5x the concurrency at fixed HBM. Locked at training time, awkward with standard attention kernels.

The second lever: stop paying max on easy queries

Medical-reasoning models that vary how much they think per question cut compute 4.7x to 6.4x while holding accuracy "nearly the same." No effect size, no confidence interval, one domain: a hypothesis with a known verification cost. The mechanism holds up, since a fixed thinking budget provisions for the hardest input in the distribution and bills it on every request, and required compute is heavily skewed. A cheap difficulty classifier or self-consistency confidence gate up front, escalating only on low confidence, is the whole design. Pre-register the non-inferiority margin before anyone sees results, because "nearly the same accuracy" is where a three-point regression on a tail segment hides.

What this changes downstream

First, context length and concurrency draw from one budget, since cache is linear in tokens and linear in batch. No configuration gives 128K context and high concurrency for free, and the unmade choice gets made by the scheduler as an OOM. Put the capacity equation in admission control: max concurrent sequences, max context per tier.

Second, the eval harness needs a compression canary. Cache-compression damage is task-localized, so an aggregate benchmark will pass a config that broke mid-document fact extraction. Build a position-stratified multi-needle eval, needles spread across the context rather than clustered at the end, run it paired against the 16-bit baseline, and ship 8-bit only when the interval on the delta excludes a meaningful loss. Then audit for sliding-window eviction anywhere in a RAG path and turn it off there. Token relevance depends on a query that has not arrived, so eviction is non-causal by construction, and a default window in a retrieval path reads as model weakness rather than the config bug it is.

Shrink bytes-per-token and share prefixes aggressively; quantize to 8-bit only behind a multi-needle canary; never evict on retrieval workloads.