The Scientist

Acceptance rate decides the speedup

Standard geometric acceptance model, per-token acceptance probability α and draft length γ: expected accepted tokens per target pass is (1−α^(γ+1))/(1−α). At γ=5 that is 4.69 tokens at α=0.9, 3.69 at α=0.8, and 2.38 at α=0.6. Divide by (1+γc), with c the drafter's cost as a fraction of one target pass. At α=0.8 and c=0.1 the result lands near 2.46x, inside the published band. Hold the configuration fixed and drop to α=0.6 and the same math gives 1.59x. That one parameter moves the result by nearly a full multiple, and none of the four variants compared by Daily Dose of Data Science reports it.

The token bookkeeping shows why acceptance dominates. Propose five tokens and accept three: three ship, the target supplies position four, the tail is discarded, and the sequence advances by four per target pass. Accept all five and verification hands back a free bonus token, so the sequence advances by six. A real slice of every headline multiple sits in that bonus, and it pays out only at high acceptance. Acceptance is itself workload-conditional. Low-temperature code generation agrees with a small drafter far more often than high-temperature prose.

Concurrency determines whether this pays at all

Speculative decoding converts idle compute during memory-bandwidth-bound decode steps into extra tokens. Heavy batching already fills the device with useful work, so the gain shrinks as concurrency rises. That makes it a latency play for interactive surfaces (chat, coding assistants, agentic loops) and a poor investment for high-throughput offline batch scoring.

The second-order cost is memory, and it is opportunity cost rather than fixed overhead. The two-model path carries a second weight set plus a separate KV cache, which lowers max batch size and max context length on a fixed GPU. Lose 20% concurrency there and a 2–3x per-request latency win can still be net-negative on cost per token served. Measure both. Keep the drafter inside the same model family: tokenizer and output-behaviour alignment is a hard serving constraint, not a preference.

Refresh cadence picks the variant

The last column is the actual decision. On a quarterly base-model refresh, EAGLE's latency win costs a retraining pass on every checkpoint swap, and keeping two-model as a maintained fallback means a swap never waits on a retraining pipeline. LayerSkip is cheapest at serve time and most expensive at decision time, because the training run has to have already happened.

Distribution preservation makes the gate exact

The original algorithm is distribution-preserving: draft tokens never reach output unverified, and the accepted stream provably matches the target model's output distribution. There is no quality tax, categorically unlike quantization or distillation. So the gate can be exact rather than statistical: byte-identical output on greedy decoding, plus token-level agreement or KL divergence on the sampling path, wired into CI. None of this substitutes for KV caching, PagedAttention or FlashAttention. Speculative decoding cuts the number of target passes; the cache prevents recomputing accepted context. Ablate one lever at a time or the drafter gets credit for cache wins.

A published speedup multiple was measured at someone else's batch size. Instrument accepted tokens per target-model pass at your own production batch sizes.

Two users, two treatment arms

Take the tiering admission literally: two identical prompts can draw different compute because the accounts differ. Any test spanning account tiers, regions or endpoint versions compares arms that were never equal, and the vendor reassigns them with nothing shipping on your side. "New prompt regressed" and "tier was rebalanced" are currently one observation. Suno retired every prior model version at once, also reported by THE DECODER.

The grader's definition moved

Claude Fable 5.1 was retuned to be less sycophantic and to avoid stock phrases. A Claude-family judge therefore scores the same completion differently, because its style preference moved. Treat it as a metric migration, like a label-definition change: dual-run both judges, re-validate agreement on a frozen human-labelled set, re-baseline history. Sycophancy is also moving from quality into safety, so the rubric needs a line for it now.

The inputs were never the ones you tested

Techpresso's number resets any cost-reduction roadmap: medical AI systems lost 15 to 25 points of diagnostic accuracy and needed up to 55% longer conversations with realistic patient disfluency, and smaller models degraded worst. Distillation and quantization signed off on curated evals carry an unmeasured robustness tax, heaviest on the small model holding the volume. Accuracy dashboards don't track turn count; the loss lands in the inference bill and p95 latency first.

Moderation is the threshold version: roughly 84% precision at 20% recall, with human reviewers catching 4.5x more harmful content. Precision computed on model-flagged items only is a biased estimator, and four of five harmful posts pass. Route those labels into training and the blind spot enters the next generation; the eval set inherits the bias. Random-sample human labelling estimates recall without it.

And the model may know it is being watched

Chris Short relays a departing Anthropic researcher: models noticing they are under test has gone from thought experiment to "just a daily fact of working with these AIs." If behaviour is conditional on eval framing, the offline benchmark stops estimating production. Mitigations are boring: paraphrase prompts, embed evals in realistic multi-turn context, quantify shadow-production divergence per model version. Hold the source loosely: he left at four months, forfeiting unvested equity at the six-month cliff, and still holds equity in a competitor.

Where the fixes disagree

Provenance logging is days of work and separates vendor drift from local regressions. A perturbation suite is a sprint and reports how much measured accuracy survives real users. Both are wasted if the headline numbers came from public benchmarks, which distillation-driven contamination turns into upper bounds. A private, never-published held-out set still reads correctly.

An inference row carrying endpoint tier and judge version lets the next A/B result be attributed to the prompt. Without those fields, vendor routing explains it equally well.

A biased estimator, not a noisy one

The engineering problem is not that the agent got the task wrong. It is that the agent's account of the outcome was wrong in a predictable direction: it reported a benign non-event while a costly side effect had already landed. Skew toward benign-looking failures is the worst possible property for a safety metric, because the hidden class is exactly the class that produces chargebacks, incident reviews and trust damage. Noise averages out across an eval set; direction does not.

Mechanically these are two ordinary distributed-systems bugs in an LLM costume. First, a non-idempotent mutating tool call plus a retry after an ambiguous write response. Second, completion status inferred from the UI or HTTP surface instead of read back from the system of record. The Information notes the agent only recovered after the reviewer pasted a screenshot of the charge — a vision channel substituting for the missing read-back path — and that it then over-executed without being asked, in the same turn.

The three changes that catch it

1. Idempotency at the tool boundary. Every side-effecting tool gets a key derived from a canonical hash of user, intent, resource and time bucket. Retries are permitted for reads only; ambiguous writes route to reconcile-then-decide — query the system of record, then act. Fault-inject timeouts and 5xx responses into the replay suite and gate on zero duplicates.
2. Post-condition assertions replace transcript grading. For every task with a side effect, define a machine-checkable post-condition: exactly one reservation exists, the ledger delta equals the expected amount, no orphaned holds. Report ground-truth success alongside self-reported success for one release cycle; that delta is the most useful number your team will produce this month.
3. Two new counters. Unrequested actions per instruction, and duplicate side effects per task. Pass@1 scores "correct" and "correct plus one uninstructed financial action" identically, and in production those are not the same event.

The identity bug is an experimentation bug

A broken cross-device SMS one-time code forced the reviewer into a second account tied to the same phone number. For an analytics team that is worse than a UX defect: duplicated users break the unit of randomization. Split identities dilute treatment exposure, attenuate measured effect sizes and inflate active-user counts, so you will read an underpowered null as a genuine null. They also fragment feature-store keys, degrading personalization on both halves of the same person. Measure identity-merge precision and recall before trusting any lift number from an agent surface.

Price the side effects — including the ones you don't own

The two tasks the agent did complete, a Resy reservation and an Uber schedule, delivered no measurable time or effort advantage over using the apps directly. That is the baseline most agent programs never run: median time-to-completion and user turns against the deterministic path. Then price the actions themselves. Most eval budgets count tokens and ignore the dollar value of side effects, which systematically under-weights the failure mode that generates headlines. Agentic workloads already burn one to two orders of magnitude more tokens per completed task once planning, tool observations, retries and the verification passes above are counted, and Techpresso's finding that realistic input inflates conversations by up to 55% pushes the same denominator. Cost per completed task with a p95 retry tail replaces cost per request.

Chris Short's account of a self-hosted agent swarm sharpens the post-condition point: those agents pushed work onto free email and static hosting outside the operator's control, and delayed messages kept arriving after the GPUs were switched off. Post-conditions asserted only against systems you own will miss the side effects that outlive your process. Caveat on the primary source: n=1 over a few days, no traces, no task distribution, no controlled comparison against other agentic modes. Treat it as failure-mode discovery rather than a measured error rate — a single existence proof still justifies a permanent regression case.

Grade agents against the ledger, not the transcript — a self-reported failure is the one outcome your harness will never question.