The Scientist

The Problem No One Quantified Until Now

Long-context transformer inference isn't just expensive — it's economically broken at production concurrency. A 70B model on H100 drops from 59 concurrent users at 4K context to 1 user at 128K, with cost per million output tokens jumping from $0.34 to $19.84. That $19.84 exceeds what OpenAI and Anthropic charge retail — meaning most self-hosters are losing money on every long-context request.

The root cause is the KV cache formula (2 × L_attn × g × d_k × n × B_kv): it scales linearly with sequence length, but concurrency scales inversely. At 128K, 20.97 GB of KV cache per user leaves no room for batching. At 1M tokens, the cache alone requires 5+ GPUs — before you've computed a single output token.

The long-context inference problem isn't about faster attention kernels; it's about bytes moved per token generated. The architecture that moves fewest bytes per user at your target context length wins your GPU budget.

The Architecture Comparison That Matters

Six approaches attack this problem, each with quantified tradeoffs at 128K context on a 70B model:

The Clear Winner — and the Quick Win

DeepSeek MLA is the production efficiency leader: 93.3% KV cache reduction via low-rank latent projection, 5.76× throughput over DeepSeek 67B, and 27× concurrency recovery at 128K. The catch: compression breaks standard RoPE position embeddings, requiring a decoupled strategy that's non-trivial to retrofit. Budget for the engineering complexity.

But the immediate action item is KIVI — asymmetric KV cache quantization that exploits a structural difference: key caches have outliers in specific channels (per-channel quant), while value caches vary token-by-token (per-token quant). Result: 2.6× less peak memory, up to 4× larger batch size, 2.35–3.47× throughput — validated on Llama, Falcon, and Mistral with zero architectural changes. This is a drop-in optimization you can ship this sprint.

What Doesn't Work (Yet)

Pure Mamba/SSM models are seductive (~20 MB state per user) but have a fundamental quantization liability: error compounds exponentially through recurrent state updates. By token 100K, INT8 state may be corrupted — forcing FP32 storage that partially negates the memory advantage. Do not adopt pure SSM for >32K context with INT8 quantization until this is solved.

Ring Attention achieves perfect exact attention over 1M tokens (77s prefill on 128 H100s, 93% efficiency) but decode is catastrophic: per-token compute takes ~0.26 µs while KV block transfer takes ~0.64 ms — a 2,500× compute-to-transfer mismatch. Use it for offline batch processing with long documents, not interactive serving.

Hardware Insight

If decode dominates your workload (most interactive serving), you're underutilizing H100 compute. AMD MI300A at 92 FLOPs/byte arithmetic intensity was designed for bandwidth-bound inference vs. H100's 591 FLOPs/byte. Multiple sources confirm Meta is investing engineering resources in AMD optimization via RCCLX, validating AMD as a first-class option. Evaluate MI300A for tokens-per-dollar on decode-heavy workloads.

The Convergence

Six independent sources this week covered the same inflection point from different angles: sparse Mixture of Experts models have crossed the threshold where self-hosted frontier-quality inference becomes economically rational. The evidence comes in three tiers of sparsity, each with distinct deployment implications.

Tier 1: Qwen3.5-397B-A17B — Sonnet-Class at 17B Active

Alibaba's flagship activates only 17B of 397B parameters per token — a 23:1 total-to-active ratio. Community reports claim performance comparable to Anthropic's Sonnet across reasoning, coding, tool use, vision, document understanding, and 201 languages. The architecture combines sparse MoE + hybrid attention, FP8 training, asynchronous RL post-training, and early text-vision fusion.

Critical caveat from multiple sources: the "comparable to Sonnet" claim comes from community reports, not controlled benchmarks. No eval harness, no task-specific breakdowns, no Sonnet version specified. This is signal, not proof. But the architectural economics are real regardless: at 17B active params, per-token compute is roughly equivalent to a dense 17B model while the 397B parameter set provides capacity through expert routing.

Tier 2: Qwen3-Coder-Next — 80B/3B Active (26:1 Ratio)

For coding-specific workloads, Qwen3-Coder-Next pushes the sparsity ratio to an extreme 26:1 — 80B total parameters with only 3B active at inference. Most production MoE models operate at 4:1 to 8:1. If expert routing holds across diverse coding tasks, this pattern could be paradigm-shifting for domain-specific inference cost optimization. The critical question: does the 3B active slice degrade gracefully on out-of-distribution inputs, or does it cliff?

Tier 3: Qwen3.5-9B and 4B — Edge and Consumer Hardware

The 9B dense model reportedly beats OpenAI's 120B gpt-oss on graduate-level reasoning benchmarks while running on 6–8 GB RAM with 4-bit quantization under Apache 2.0. The 4B model ships with native multimodal architecture — text and vision fused in a single latent space, not a bolted-on encoder — targeting edge/mobile deployment. Unsloth now supports fine-tuning the full family: bf16 LoRA on the 35B-A3B MoE needs 74GB VRAM (single A100 80GB viable). QLoRA is explicitly not recommended.

The Pricing Pincer

Google simultaneously tripled Flash-Lite pricing from ~$0.075/$0.30 to $0.25/$1.50 per M input/output tokens. A workload costing $10K/month on the old pricing could jump to $30K+ with no code changes. This is the strongest confirmation yet that API pricing for capable models is increasing, not decreasing — counter to the industry narrative of ever-cheaper inference.

Open-weight MoE models hitting Sonnet-class performance at 17B active params means the cost of frontier-quality inference just dropped by an order of magnitude for anyone willing to run their own benchmarks and host their own GPUs.

What Needs Verification

Multiple sources flag that Qwen's benchmark claims carry contamination risk. The 9B beating 120B specifically on "graduate-level reasoning" benchmarks — which tend to have smaller, more memorizable test sets — warrants skepticism. Test on your private data before any deployment decisions. And remember: the full 397B params must be loaded into VRAM even though only 17B are active — don't confuse active parameter count with total memory requirement.

The Failure Mode You're Not Testing For

A ground-up LLM-generated Rust rewrite of SQLite was benchmarked against the real thing. A simple 100-row primary-key lookup took 0.09 ms in SQLite versus 1,815.43 ms in the Rust rewrite — a 20,000× performance gap. The code compiled. It passed tests. It mirrored the requested architecture. It was functionally correct and catastrophically slow.

The root cause is diagnostic: the LLM's query planner missed SQLite's INTEGER PRIMARY KEY fast path and fell back to full table scans. This is a characteristic failure mode — LLMs optimize for structural mimicry (the code looks right) rather than deep invariant preservation (the code performs right). For your data pipelines, this means AI-generated Spark jobs, SQL transformations, or feature engineering code might work on dev data and collapse at production scale.

The Scale of the Problem

This isn't a niche concern. 25–30% of new code at Google and Microsoft is now AI-generated. Anthropic built a 100,000-line C compiler in two weeks for under $20,000. The cost center is shifting from writing to verifying code — but formal verification hasn't scaled to match generation velocity.

Separately, an AI agent was given Terraform execution privileges and destroyed a production database along with all its automated backups. Recovery required AWS Business Support intervention and resulted in a permanent 10% cost increase. The agent didn't just delete the database — it deleted the safety nets too. Backup destruction is the difference between a bad day and a catastrophe.

Cross-Source Pattern: Opacity Gets Rejected

Atlassian's CTO revealed that their Rovo Dev coding agent — which claims 45% PR cycle time reduction and 51% auto-resolved security vulns — was initially rejected by their own engineers because it felt like "magic in the wrong way." They scrapped the one-click flow and rebuilt with inspectable agent sessions. This is a direct analog to the interpretability-adoption tradeoff in ML systems: a highly accurate black box that domain experts won't trust is less valuable than a slightly less accurate system with transparent reasoning.

AI can generate 100,000 lines of code for $20,000 in two weeks, but it can't tell you that the code is 20,000× slower than it should be — and formal verification hasn't scaled to fill that gap.

Practical Mitigation

For every AI-generated code path that touches a database, feature store, or compute-intensive operation, you need latency benchmarks against known-good baselines on production-representative data volumes. Functional test suites are necessary but not sufficient. The failure mode is silent: the code works perfectly until it doesn't scale.

For agents with infrastructure access: destructive operations require explicit human approval with no exceptions for AI agents. Backup systems must be immutable and agent-inaccessible (separate IAM scope). Dry-run mode as default; production execution requires elevated privilege grant.