The Scientist

The token bill says price, not efficiency

GLM-5.3-Flash is MIT-licensed at $0.15/$0.50 per million input/output tokens. Artificial Analysis scores it 57 on its Intelligence Index, tied with GPT-5.6 Terra, at $0.09 per task against $0.51. The full index consumed 149M output tokens, of which 134M, about 90%, were reasoning tokens. That is under GLM-5.3's 168M and more than Kimi K3's 133M or Qwen3.8 2.4T A95B's 136M at comparable scores. The per-task price does not measure frugality. Flash is cheap per token because $0.50 per million output tokens is cheap.

Three consequences follow. A 90%-reasoning profile gives fat-tailed cost and latency distributions, so budget on p95 rather than mean and cap reasoning tokens per request. The cost edge sits one competitor price cut from evaporating, and treating it as structural will not survive a pricing-page change. Where latency, not dollars, is the bottleneck, Kimi K3's token frugality is the more durable property.

Why 1M context is cheap here

Per rasbt's teardown, the backbone moved from GLM-5.2's 744B-A40B to 320B total / 18B active, with a Kimi Linear-style 3:1 hybrid, 34 KDA linear-attention layers against 11 MLA/DSA layers, DeepSeek V4-style mHC residuals across four parallel streams, and a native vision encoder. thealexker adds that depth dropped from 92 layers to 45.

Only about 11 of 45 layers carry a conventional KV cache; the rest hold constant state, which is why attention compute stops compounding at a million tokens.

The mechanism is worth copying into any long-context serving plan, adopted model or not. Per-layer KV shrinks, and long prompts stop taxing memory bandwidth the way current capacity models assume.

Where the split is clean enough to route on

Verifiable, closed-loop tasks tolerate a weaker world model because the environment catches the errors: code compiles or it does not, commands exit zero or they do not, schemas validate or they do not. Open-ended factual generation has no such check. GLM-5.3's own 34%/30% makes this a family trait rather than a Flash defect, and no abstention-rate breakdown was published, so accuracy cannot be separated from confident wrongness without an in-house probe.

The lever that beats model choice

Cached input runs about $0.026-$0.03 per million tokens, roughly an 80% discount off $0.15. A 100K-token agent prefix (system prompt, repo context, tool schemas) costs about $0.0026 per step against $0.015. Over a 200-step run, putting stable content first is worth more than the model swap. Cache hit rate belongs on the serving dashboard, not in an occasional notebook.

Two provenance caveats before the first run

Z.ai engineer Zixuan Li asked early downloaders to re-download after a chat-template fix, and Artificial Analysis published 400k context before correcting to 1M, so any evaluation run in hour one is invalid. Adoption also preceded attribution: the model topped usage charts before Z.AI confirmed authorship, and Cline reports it at 11% of all traffic inside a week. That is a demand signal, not a quality signal. Origin-lab, license and jurisdiction columns belong in the eval registry so users cannot route work to unscored models. CV practitioner skalskip92 found the native vision encoder weak on object detection, so the specialized detection models stay.

Start with the arithmetic, not the tutorial

Under per-token acceptance probability p and draft length K, expected accepted tokens per verification pass is (1 − p^(K+1)) / (1 − p), and wall-clock speedup is that quantity divided by (1 + K·r + v), where r is the draft/target per-pass cost ratio and v is the overhead of scoring K+1 positions instead of one.

The model reconciles the published numbers better than published numbers usually deserve. DeepSeek-V3's 80-90% acceptance on its second predicted token is effectively K=1. At p=0.85 the formula gives 1.85 tokens per pass, against a reported ~1.8x. Push it to p=0.9 and K=4 and it predicts about 3.2x after overhead, which is where the "3x" framing comes from. QuantSpec clears 90% acceptance using 4-bit weights and a 4-bit KV cache for drafting and still reports only >1.78x.

Overhead, not acceptance, is the binding constraint in real serving systems. That is why the theoretical 3x has never been measured.

One inherited rule of thumb should go. The claim that below ~50% acceptance the extra work outweighs the savings is only coherent in a batched regime. At batch 1 with K=4 and p=0.5 the formula yields 1.94 tokens per pass, comfortably net-positive. Wired in as a batch-size-1 kill switch, it discards a real speedup.

What decays, and what it takes with it

Acceptance rate is a property of your traffic, not your config: temperature-0 extraction and RAG-answer routes accept well, temperature-0.8 chat does not, and raising temperature flattens the distribution and depresses acceptance further. Draft weights come out of the KV cache budget, silently cutting max concurrency, and capacity plans rarely price that. The other miss is that the guarantee is distributionally lossless, not bitwise, so golden-output parity gates will flake and need rewriting as paired distributional tests.

The same shape, one layer down in the stack

The silicon claims fail the identical way. OpenAI and Broadcom claim their Jalapeño inference ASIC beats Nvidia's GB200/GB300 by 1.5-1.9x on throughput per watt (a 1.27x spread) and 1.7-3.6x on latency (a 2.1x spread), alongside a headline 104x tokens per kilowatt with no disclosed baseline, batch size, or sequence length. A range that wide across fixed hardware is a configuration artifact, not an architectural constant. Long-context small-batch decode is bandwidth-bound and flatters a purpose-built inference part; prefill-heavy large-batch work is compute-bound and flatters a GPU. The thing the range does not tell you is which end of it production traffic sits at. Nothing separates time-to-first-token from time-per-output-token, and it is not stated which claims ran against GB200 versus GB300.

The exploitable detail is the workload list. GPT-OSS, DeepSeek R1 and Kimi K2.5 are all open weights, so the Nvidia half of that comparison is reproducible on hardware already under rent. A harness only settles the argument if it normalizes tokens/sec/watt, TTFT and TPOT at p50/p99, and $/1M tokens across three batch sizes and three context lengths. Built once, it is the only defensible denominator in the next four quarters of procurement conversations.

Which leads to the cheapest audit on this list. Roofline discipline holds that profile-and-tweak only finds a local minimum: compute what the silicon can theoretically do, then report actual as a percentage of peak. Under 30% MFU, the gap is money, and reporting actual as a share of peak is what prices it. The same number says whether the job is memory-bound (fuse ops, change layout, raise effective batch) or compute-bound (dtype, kernel selection) before anyone opens a profiler.

What actually disappears on the date

The weights are not the loss. The loss is requester history, HIT templates, qualification records and worker-quality metadata — the only archive that explains how a given label was produced, by whom, under what screening. Once Amazon closes Mechanical Turk on September 30, 2026, that archive cannot be reconstructed, and every downstream claim resting on those labels becomes an assertion nobody can audit.

Handle the 46% figure carefully. It is one 2023 experiment with no reported task type, pool size, or detection method. That makes it an order-of-magnitude prior, not a point estimate, which is the argument for measuring the contamination rate in your own data rather than litigating theirs.

The measurement, sized

Re-label a stratified 300-500 item sample per tier-1 eval set with vetted annotators, then compute raw disagreement plus Cohen's kappa against the original labels. That sample size buys roughly ±4-5% precision on the contamination rate, which is enough to settle retire-versus-reweight instead of debating it. Anything above about 15% disagreement should leave model-selection decisions entirely rather than acquire a footnote.

A label you cannot verify a human produced has no epistemic value as a human baseline, however many times it has been cited internally.

The market repriced ahead of the eval teams. MTurk's decline is attributed to Scale AI, Mercor and Prolific, and the shift is not vendor churn. Anonymous micro-tasks are giving way to identity-and-expertise-screened labor, with contamination exposure as the differentiator rather than price per unit.

The other half: who holds the ruler for agents

The Information's AI Agenda reports enterprises grading agents with "old-school software" — deterministic assertions, final-state diffs, tool-call trace checks, schema and invariant validation — instead of LLM-as-judge. Read that as a tacit admission about the instrument. Judge variance is moderate to high, verbosity and position bias are live, and the worse property is non-stationarity: upgrade the judge and every historical eval number silently shifts. At that point agent improvement and ruler drift are not separable.

A workable target: more than 60% of the agent eval suite on deterministic graders, pinned judge model versions for the remainder, and judge-human agreement published next to every eval release. The diagnostic takes an afternoon. Run a frozen agent five times. If the variance exceeds the improvements being claimed, the last three wins were noise.

The confound nobody enabled deliberately

Anthropic merged Claude's chat and Cowork memory with the feature on by default. Same prompt, same model, different memory contents, different output, and no run-level record of which memory was active. Repeated eval runs stop being independent draws. Run N is conditioned on runs 1 through N−1, which correlates observations, understates variance and manufactures significance. The fix is a couple of hours: pin eval traffic to memory-disabled credentials, assert empty memory state at run start, log a memory-config hash with every result. It protects every inference the team makes this quarter.