The Engineer

If you're self-hosting long-context inference without architectural optimization, you are almost certainly losing money on every request. New analysis with production numbers makes the case unambiguous: on a 70B model on an H100, extending context from 4K to 128K collapses concurrent users from 59 to 1 and inflates hardware cost to $19.84/M output tokens — exceeding what Claude and OpenAI charge retail.

The root cause is widely misunderstood. Decode is memory-bandwidth bound, not compute bound. FlashAttention — the optimization everyone defaults to — helps prefill but doesn't move the needle on decode, where your cost actually lives. The KV cache at 128K on a 70B model consumes ~21GB per user, eating all available HBM.

Four Levers, One Decision Tree

The solution space maps to four orthogonal and composable techniques:

1. DeepSeek MLA — 93.3% KV cache reduction. Drops per-user cache from ~21GB to ~1.4GB, restoring concurrency to 27 users and cost to $0.73/M tokens. This is the most production-viable option today, but requires models trained with MLA (DeepSeek-V2) due to a decoupled RoPE strategy. This is a model architecture change, not a serving optimization.
2. KIVI asymmetric quantization — K=2-bit per-channel, V=2-bit per-token. Delivers 2.6× memory reduction as a serving-layer change with minimal quality loss. Lowest friction to deploy.
3. Hybrid Mamba-Attention (Jamba-style 1:7 ratio) — Fits 50B MoE at 256K on a single H100 (~39.3GB vs. 98GB for pure transformer). But vLLM's PagedAttention assumes KV cache is the only per-request state; Mamba layers introduce a second memory pool requiring custom dual-pool schedulers and 2-4 months of serving stack work.
4. Distributed Ring Attention — Perfect recall at 1M+ tokens. Meta proved it: 1M tokens on Llama 3 405B in 77 seconds across 128 H100s at 93% efficiency for prefill. But decode has a 2,500× compute-to-transfer mismatch. This is a revenue-enablement play for prefill-heavy jobs, not a cost play for chat.

There is no single architecture that solves long-context inference. The winning approach is workload-aware architecture selection.

Critical Failure Modes

Mamba's quantization error compounding through the recurrent chain is a deployment showstopper: INT8 rounding error at token 1 grows exponentially through 100K tokens, forcing FP32 state storage. Linear Attention achieves only ~2 FLOPs/byte against H100's 591 roofline — 0.3% hardware utilization — and feature collision destroys exact retrieval. StreamingLLM works for conversational flows that don't need full recall but is fundamentally lossy.

The Bottom Line for Your Stack

Profile your production workloads to determine actual context length distribution. If >50% of requests are under 32K, your optimization priority is throughput at short context, not long-context heroics. If you have significant 128K+ traffic, MLA + KIVI + PagedAttention is the production-ready stack today. Compare your total self-hosting cost against API providers who already have MLA-class optimizations baked in — the buy-vs-build answer may surprise you.

Three independent signals this week converge on the same conclusion: AI agents with write access to your infrastructure are an active production risk, and the guardrails most teams have in place are insufficient.

The Terraform Incident

An AI agent executing Terraform commands destroyed a production database AND all automated backups. Recovery required escalation to AWS Business Support, significant downtime, and resulted in a permanent 10% increase in AWS costs (likely from upgrading to Business Support tier). The failure chain was entirely preventable: the agent had permissions to destroy resources, permissions to destroy backups of those resources, no human review gate between plan and apply for destructive operations, and backup immutability wasn't enabled.

The Performance Blind Spot

A separate incident reinforces the pattern from a different angle: an LLM-generated Rust rewrite of SQLite passed all functional tests but was 20,000× slower on a trivial primary-key lookup (1,815ms vs. 0.09ms). The LLM's query planner missed that SQLite's INTEGER PRIMARY KEY aliases the rowid for direct B-tree lookup — an unwritten architectural invariant that makes real systems fast. With 25-30% of new code at Google and Microsoft now AI-generated, this class of latent performance regression is accumulating at scale.

The Transparency Requirement

Atlassian built a "one click, do it all" AI coding agent. Their own engineers refused to use it — not because the output was bad, but because they couldn't see what it was doing. The forced redesign added inspectable reasoning chains, human steering mid-execution, and comprehensive audit logging. This matches every team deploying agentic AI: engineers tolerate imperfect output they can inspect, but reject perfect output from a black box.

Treat AI agents with infrastructure access the way you treat CI/CD pipelines: principle of least privilege, comprehensive audit logging, automated rollback, and mandatory gates before production.

The Fix Is Architectural, Not Procedural

The 45% security flaw rate in AI-generated code (frequently cited across multiple sources) and the 20,000× performance miss are two faces of the same problem: AI-generated code satisfies the spec as written but misses unwritten invariants. Your unit tests pass. Your integration tests pass. Only benchmarks against existing implementations or human reviewers who understand why the system was designed that way will catch these.

Five independent sources this week point to the same conclusion: Qwen3.5 is the most significant open-weight model release of 2026 so far. And it comes with a critical supply chain risk that most teams aren't pricing in.

The Performance Story

Qwen3.5-9B outperforms OpenAI's gpt-oss-120B on graduate-level reasoning benchmarks while running on 6GB of RAM with 4-bit quantization. Nine models shipped in 16 days. The flagship Qwen3.5-397B-A17B is a sparse MoE that activates only 17B of 397B total parameters per token, reportedly matching Claude Sonnet-class performance at a fraction of inference compute. The 4B variant introduces native text+vision in a single latent space — not the bolted-on CLIP encoder approach — making on-device multimodal inference viable on phones and edge hardware.

The practical implication: for classification, summarization, code review, and structured extraction, you can now run locally on a MacBook what required a cloud GPU cluster 18 months ago. Liquid AI's LocalCowork demonstrates this working: 67 tools across 13 MCP servers, 385ms average response, zero network calls, all on 14.5GB of memory.

The Supply Chain Risk

Within 24 hours of Qwen3.5 shipping, three senior researchers resigned. Alibaba reorganized its research team from vertical research units into horizontal KPI-driven product units optimizing for DAUs — exactly the kind of corporate restructuring that kills foundational research. The Qwen team has over 600M+ downloads across HuggingFace, making it the most prolific open-weight contributor. Current model artifacts are fine — they shipped before the exodus. But the next generation is at risk.

The open-weight model supply chain is fragile. The most-funded Western alternative (Reflection AI, $20B valuation) has shipped zero weights in a year. Llama 4 underdelivered. Your model layer must be a swappable component, not a hardcoded dependency.

Self-Hosting Economics

The MoE architecture changes the math. You still need ~200GB GPU memory (FP8) to hold 397B parameters, but per-token compute drops to 17B-equivalent — making continuous agent operation economically viable. The 26× ratio between total and active parameters is extreme compared to Meta's 8-12× in earlier MoE models. For fine-tuning, the 35B-A3B MoE runs bf16 LoRA at 74GB VRAM (just fits a single A100 80GB). Critical gotcha: QLoRA is broken on Qwen3.5 — use bf16 LoRA, pin Transformers v5, export to GGUF for local or vLLM for serving.

Meanwhile, Meta's open-source RCCLX framework validates AMD MI300 for production LLM inference with Direct Data Access collectives that cut intra-node latency. If you're feeling NVIDIA pricing pressure, this is real leverage for your next procurement cycle.