ClarityBeta

Science & Analytics

The Scientist

9 min read · 1,831 words · 3 moves to watch
The Signal

Quantizing more of GLM-5.2 bought Baseten 20% throughput with zero quality loss.

Per-layer errors can cancel, so quantization scope stops being monotonic risk and becomes a variable you can search. The transferable piece is the objective rather than the result: KL divergence against full-precision logits gives a per-token fidelity signal, and it catches regressions well before an accuracy delta clears its own noise floor. If the quantization eval you run is a benchmark score, it confirms the average held and says nothing about which slice paid for it. That slice is usually the one in production.

In Play

  1. Quantization Scope Is a Search Problem

    Baseten selectively quantized MORE layers of GLM-5.2 and reported 20% higher throughput than existing NVFP4 configs at matching downstream quality, per Latent.Space. Fidelity was measured as KL divergence between quantized and full-precision logit distributions rather than benchmark deltas. That inverts the usual monotonic-risk framing of quantization scope. It also hands you a per-token, low-variance fidelity signal that surfaces regressions on far fewer samples than accuracy scores do.

    Ask Clarity
    Try

  2. Per-Token Price Versus Per-Task Cost

    Artificial Analysis put DeepSeek V4-Flash at roughly $0.03 per benchmark task against $1.86 for GPT-5.6 Sol and $3.15 for Claude Fable 5, per Turing Post. Devshot's hands-on testing found V4-Flash's default reasoning level produces weak code, and that the usable 'high' setting multiplies token cost about 6x. Nobody discloses which reasoning level the index was scored at, so the headline price and the ranked configuration may not be the same object.

    Ask Clarity

  3. Distilled Generators Lose Coverage Silently

    Two distillation papers surfaced by Turing Post argue that accelerated generative models hold visible quality while losing output diversity. DistillAlign attributes it to a mode-covering versus mode-seeking objective mismatch during distillation; Parallel Decoding Distillation sidesteps adversarial and variational training, whose instability produces motion loss and mode collapse. Perceptual metrics and human preference scores reward good samples, not broad coverage, so this regression is invisible to most shipped eval harnesses.

    Ask Clarity
    Try

  4. Agents Clear Engineering, Fail Framing

    A multi-institution shadow evaluation handed Claude Opus 4.8, running in the OpenClaw harness, the central questions from two unpublished NeurIPS 2026 papers — one of them a distribution-shift detector for tabular foundation models. The original authors graded the output Reject (2) and Strong Reject (1), per Import AI. Turing Post reports the same conclusion from a separate study: agents complete the engineering and fail at scientific judgment. Treat hypothesis selection and novelty calls as human-owned this quarter.

    Ask Clarity
    Try

  5. Open Weights Hit a Deployability Cliff

    Moonshot's Kimi K3 needs at least 16 H200 GPUs just to load its weights, and Moonshot's own recommended production configuration is a 64-accelerator super-node priced near 17 million RMB drawing 45 kilowatts at full load, per ChinAI's translation of a Chinese deployment writeup. Enthusiasts ran a compressed Kimi K2 on a Mac Studio one generation earlier. Any on-prem or air-gapped tier premised on frontier open weights eventually fitting your hardware is now working from a falsified assumption.

    Ask Clarity
    Try

Deep Dives

The Fidelity Objective You Should Be Optimizing Is KL Divergence

Three failure classes your harness is currently blaming on model weights are actually engine bugs, kernel races and hardware nondeterminism — which makes cluster topology an experimental variable.

Joshua Hill, a Baseten research intern, formalized the layer-selection problem using Fourier Analysis on Groups. The paper reportedly ran 72 pages before being cut to 45, which is a reasonable signal that the cancellation effect resists any one-line heuristic. The part that transfers is the scoring function. Measure KL divergence between the quantized model's logit distribution and the full-precision model's on a held-out prompt distribution matched to production traffic, and the result is a per-token, low-variance signal. Benchmark deltas are coarse and noisy by comparison: thousands of samples to detect a regression that a divergence measure surfaces in hundreds.

Quantization is also the only lossy step in the standard speedup stack, per Latent.Space's account. BF16 to FP8 to NVFP4 is roughly 2x combined. A speculator adds about 2x and is lossless. Prefill/decode disaggregation adds about 2x and is lossless. Baseline unoptimized 1-trillion-parameter serving runs 30-50 tok/s, and the honest reported outcome after stacking is 4-6x, not the aspirational 10x. The thing the 4-6x figure doesn't tell you is where the fidelity risk sits. Speculation and disaggregation carry no fidelity-eval burden, so the entire quality-validation budget can go to the quantization step alone. Most serving roadmaps do not draw that decision boundary.

Three failure modes eval harnesses routinely mislabel

  1. Repetition and mode collapse are software bugs, not weight failures. Models repeat the same token even at temperature 0.9, traced to deterministic inference-engine issues, with SGLang implicated and sometimes fixed by upstreaming TensorRT-LLM image changes. Tuning a repetition penalty here treats a kernel bug with a sampling parameter.
  2. Identical weights diverge across clusters. A kernel race condition, a missing barrier for example, stays latent until node-to-node KV cache transfer runs over a slower interconnect on one cluster than another. Same weights and config, different outputs. Any model A/B that spans clusters is confounded.
  3. Temperature 0 is not deterministic. Hardware nondeterminism means greedy decoding does not reproduce exactly, so every exact-match golden-output regression test will flake.

The operational consequence is a run-metadata change. GPU SKU, engine build and interconnect topology belong alongside seed and model version in every eval record. Quantify cross-cluster variance as an explicit noise floor before claiming significance on any model delta.

Quantization scope is a search problem with a measurable objective, not a safety dial you turn down when you are nervous.

Two adjacent wins with real leverage

Speculative decoding is now release-packaged rather than a research trick. DeepSeek's V4 Flash ships it as an attached production module, per TLDR AI. Before training anything, check whether the model ships a native MTP head for self-speculation. EAGLE is the sensible default otherwise, and acceptance rate is traffic-dependent, so a draft trained on the local request distribution beats a generic one. Second, KV-cache hit rate deserves to be an SLO. Mooncake, the Tsinghua MADSys architecture credited with raising Kimi K3's cache hit rates, is open, and most inference gateways still do not dashboard cache hit rate at all. That is why the cheapest available prefill saving usually goes unclaimed.

Caveat worth stating in the design doc: this is a single detailed primary account with no independent replication, and no ablation table has been published for the layer-selection method. The scoring change is safe to adopt whether or not the specific 20% figure reproduces on a given checkpoint.

What to do

  1. Re-run your quantization layer-selection sweep this sprint with an inverted prior — test configs that quantize more layers — scored on KL divergence against full-precision logits rather than benchmark accuracy, excluding modulation and out-projection layers for image models.

  2. Add engine build, GPU SKU and cluster interconnect topology as required logged fields in every eval run record before your next model comparison, and measure cross-cluster output variance as a stated noise floor.

  3. Instrument a repetition guard in the serving layer this sprint that cuts generation when a non-special token repeats four or more times, and log the offending token IDs instead of tuning sampling penalties.

A $0.14 Model That Costs $0.84 in the Config You Would Ship

Reasoning effort, provider identity and checkpoint precision are three undisclosed experimental conditions behind the price war — and each one moves the number more than the model choice does.

DeepSeek shipped V4-Flash as open weights, 304B parameters in a 167GB footprint, which works out to roughly 4.4 bits per parameter, per Devshot. FP8 would want about 304GB, so this is a compressed release. That has two practical effects: a self-hosted evaluation is not apples-to-apples with an API-measured composite index, and 167GB does not fit two 80GB cards once KV cache is budgeted. Three H100-class GPUs is the floor. Precision parity gets checked before any quality gap is attributed to the model.

Three confounders, each controllable in an afternoon

Undisclosed conditionMeasured spreadWhat it corrupts
Reasoning effort level~6x token cost for the usable settingEvery price-vs-quality table; the index config may not be the priced config
Serving provider identityGLM5.2 served at under 30 to 129 tok/s on OpenRouter, identical weightsEvery latency A/B routed through a marketplace
Checkpoint precision~4.4 bits/param in the open releaseSelf-hosted quality comparisons against API scores

The 4x-plus serving spread on identical weights is the quietest of the three and the most damaging. A p-value computed on "model B feels snappier" through an unpinned marketplace route is measuring infrastructure and reporting it as model performance.

The per-task numbers, and what they are made of

Artificial Analysis places V4-Flash near $0.03 per benchmark task against $1.86 for GPT-5.6 Sol and $3.15 for Claude Fable 5, per Turing Post, with quality below the leading OpenAI and Anthropic models and roughly comparable to Gemini 3.6 Flash. Techpresso's cost-per-index-point math puts V4-Flash near $0.0006 against roughly $0.0151 for Kimi K3, a ~25x efficiency gap for a 7-point quality delta on a composite index. The thing those figures do not tell you is that cost-per-task is not a model property. It is tokens multiplied by steps multiplied by retries, on one benchmark under one harness. A 60x headline delta could land at 8x or 200x on production traffic depending on how often the cheap model needs a second pass.

Token-direction asymmetry decides which workloads actually move. Qwen3.8-Max at $2/$6 per million in/out against Kimi K3's $3/$15 is 33% cheaper on input and 60% cheaper on output, per The Information. An output-heavy agent step at 5K in / 25K out saves roughly 59% on the swap. A retrieval-heavy 20K in / 2K out call saves about 42%. Same swap, different budget consequence.

A model's headline token price is a hyperparameter, not a fact. Measure cost-per-resolved-task, or negotiate against a number that does not exist.

What actually changed in the architecture

OpenAI cut Luna 80% to $0.20/$1.20 while Terra fell only 20% to $2/$12, per TLDR Dev, widening the cheap-to-mid spread from roughly 2.5x to 10x. Blended cascade cost against an all-Terra baseline is p + (1-p)/10 for escalation rate p, so at p=0.30 the bill is 37% of baseline where the old spread got you to only ~58%. Cascade routing is the default now, not an optimization, and every escalation threshold tuned last quarter is too conservative. The counterweight comes from TLDR Founders: Manifest ran a learned LLM router for four months across 7,000 users and deprecated it. As price floors converge, the arbitraged spread shrinks toward noise while the added latency and eval surface stay permanent. Static per-workload assignment, refreshed monthly from offline evals, captures most of the upside.

What to do

  1. Add reasoning_effort and provider_id as required logged fields on every eval run this week, and pin explicit provider IDs in any marketplace-routed benchmark.

  2. Re-report your model-selection benchmark on cost-per-resolved-task and tokens-per-resolved-task, swept across reasoning levels, with bootstrap confidence intervals on at least 300 held-out tasks per arm before any migration decision.

  3. Re-derive cascade escalation thresholds against the new 10x cheap-to-mid spread by replaying one week of production traffic through the updated cost function, and shelve any learned-router work in favour of static per-workload assignment.

Where the Verifier Is Cheap, Agents Are Frontier-Grade

Two results from the same model generation landed at opposite ends of the credibility scale, and the variable separating them is not capability — it is whether a mechanical checker exists.

The shadow evaluation's reviews carry more signal than its grades. Both graded outputs name the same three failures: poorly motivated data and experiments, no novel contribution, and impenetrable prose, per Import AI. The behavioural modes transfer better, since each is locally testable: early commitment to a narrow set of research paths, not responding to synthetically generated feedback about improving experimental design, and difficulty reversing out of unpromising approaches. Hand an agent two in-flight unpublished internal analyses, have the original owner blind-grade the output, and the result is a local base rate rather than a vendor claim.

The other end of the range. OpenAI's internal Astra system closed ten open problems across eight fields including high-dimensional geometry, lattice cryptography and extremal combinatorics, with Columbia's Henry Yuen independently vouching for their significance, per Import AI. Each result shipped with a Lean 4 certificate, and the run reportedly cost about $2,000 of inference, per AI Breakfast. Anthropic researcher Levent Alpoge then reported that Claude Fable reproduced 5 of the 10 proofs in 24 hours with no internet access and no custom prompting. One day from announcement to parity points at the training regime, not proprietary scaffolding.

The certificate is the copyable part, not the mathematics

Strip the number theory and Astra is multi-agent decomposition plus a machine-checkable certificate. Lean 4 makes unreviewable output reviewable in constant time. Most production stacks have the decomposition and not the certificate; the acceptance gate is an LLM judge scoring a rubric, itself an unverified model output. The analogues exist already: schema contracts, dbt or Great Expectations assertions, property-based tests, statistical invariants on output distributions, replayed backtests with pre-registered metrics. Auto-adjudicate the large majority of agent output, humans on an escalation queue only. Cheap generation with expensive review is not cheap.

Two measurement patterns worth stealing

  • Report a per-stage conditional funnel, not an end-to-end pass rate. The offensive-security paper Import AI covers reports ~80% vulnerability detection, ~53% exploitation and 88% self-replication, multiplying to about 37% end-to-end. Exploitation is the bottleneck, visible only because the funnel was instrumented per stage. Three 85% stages compose to 61%. A single composite gives nothing to prioritise against.
  • Independent-trajectory retries are best-of-k in disguise. Hard targets go to different replicas, each sampling a fresh reasoning trajectory. Run the binomial: 1 − (1 − 0.373)^k gives roughly 90% at k=5 and 99% at k=10. A 37% agent becomes a 99% system for 10x compute and zero model improvement. Test at k = 3, 5 and 10 on the least reliable task, plotting measured success against the curve. That plot is the independence check.

The same paper's role-scoped reasoning graph is a days-of-work refactor with measurable payoff. Plan, Judge (reviews the plan against command history), Action (phase-appropriate tool subset only), Summary and Progress nodes stay separate, each seeing only role-relevant prompts and tools, which caps context growth per sub-goal. An agent running one 40-tool schema and an ever-growing scratchpad should A/B against it on tokens-per-task and loop-timeout rate. Caveat: five graph nodes are redacted in the source, so what reproduces is a partial architecture without ablations.

Agents are frontier-grade wherever a cheap verifier exists and unpublishable wherever one does not. The marginal return sits in verifier and harness work, not prompt tuning.

Turing Post independently reports the same ceiling from a separate study: agents complete most of the engineering and fail at scientific judgment, backtracking, creativity and deciding what counts as a meaningful contribution. Scope automation to ablation sweeps, feature-engineering search and sandboxed pipeline debugging, and keep hypothesis selection and stopping rules human-owned.

What to do

  1. Replace end-to-end pass-rate reporting with a per-stage conditional success funnel on every agentic pipeline this sprint, publishing the multiplicative composite alongside it.

  2. Replace one LLM-judge acceptance gate with a machine-checkable one on your highest-volume agent workflow this quarter — schema contracts, property tests or statistical invariants — and report the share of output auto-adjudicated.

  3. Write into usage policy this week that agents do not own problem framing, hypothesis selection or novelty judgment on research work, and do not cite the ten-problem math result as evidence they can.

The bottom line

Almost nothing in these announcements was actually a property of a model. Every gain and every price arrived attached to a configuration nobody wrote down: which layers were compressed, which effort setting was active, which operator served the weights, which interconnect carried the cache. That breaks the working assumption that a checkpoint name identifies an experiment, and it means most of the comparisons in your harness are underpowered by construction rather than by sample size. Make configuration a logged and gated field this week, so any number reaching a decision without its serving and effort settings recorded gets rejected rather than debated.