The Engineer

The Benchmark Lie Is Now Quantified

METR's research delivers the hardest number yet on coding agent reliability: roughly 50% of AI-generated pull requests that pass SWE-bench's automated grading would fail human code review. The failure modes aren't exotic — they're the exact things that distinguish demo-ware from production engineering: code quality issues, broken surrounding code, and core functionality that the test suite simply doesn't cover. When a vendor tells you their agent scores 48% on SWE-bench, your real-world merge rate is likely closer to 24%.

SWE-bench scores are roughly 2x overstated. Every coding agent evaluation you've done using benchmark pass rates as the primary metric needs recalibration.

This finding is independently corroborated by NVIDIA's internal experience. Their chip-design team built a fine-tuned domain expert model in 2023 and it failed completely — not because the model was bad, but because hardware engineers couldn't trace or verify answers. Adoption only happened after they rebuilt with source-traceable retrieval, making every response auditable. The pattern is clear: generate with the LLM, verify with deterministic rules, cite sources for human trust.

A separate research finding reinforces this architecture: a reasoning verification system achieved 92% accuracy on mathematical proofs versus 67% for raw LLM output by checking each step against formal rules. That 25-point gap is the difference between a toy and a production system for any formally verifiable output — code, SQL, logical assertions.

The Production Playbook Is Now MIT-Licensed

LangChain's Open SWE codifies the internal coding agent pattern already running at Stripe, Ramp, and Coinbase into an MIT-licensed framework. The architecture: Slack ticket pickup → sandboxed code execution → PR creation with human review gates. Two design decisions stand out:

• AGENTS.md — a convention file encoding your team's architectural decisions, testing standards, and style preferences into every agent run. This is the difference between generic AI code and code your team would actually approve.
• ~15 focused tools — following Stripe's internal principle that 'tool curation matters more than tool quantity.' Every additional tool is another decision branch where the agent can fail.

Pluggable sandbox backends (Modal, Daytona, Runloop) give deployment flexibility, and live message injection middleware lets you clarify requirements mid-run via Slack without restarting the agent.

Cursor Composer 2 Changes the Cost Equation

Cursor shipped Composer 2 with a novel technique: compaction-in-the-loop RL. The model is trained via reinforcement learning to compress its own context to ~1,000 tokens mid-task, enabling coding sessions spanning hundreds of actions at $0.50/M input tokens — 10x cheaper than Opus 4.6 while matching its benchmark scores. Given METR's findings about benchmark inflation, validate this equivalence claim on your actual codebase before trusting it.

Prefill vs. Decode: Two Problems, Two Hardware Profiles

NVIDIA's GTC 2026 made the industry pivot from training to inference economics explicit. The key architectural insight: the prefill phase (processing prompt tokens in parallel) is compute-bound and GPU-friendly. The decode phase (generating tokens sequentially) is memory-bandwidth-bound, leaving thousands of CUDA cores idle while waiting on memory fetches. NVIDIA's answer — Vera Rubin + Groq delivering 35x throughput per megawatt over Blackwell — signals heterogeneous inference architectures are going mainstream.

If you're designing inference serving infrastructure today, disaggregating prefill and decode onto different hardware backends is no longer a research curiosity — it's the direction NVIDIA itself is heading.

NVIDIA valuing Groq at $20B and integrating inference-specialized silicon alongside GPUs confirms the 'GPU for everything' era in inference is ending. Any major inference infrastructure commitment made today should have a clear migration path to heterogeneous compute — or you're building on a cost structure with a known expiration date.

Demand Paging: OS Primitives Applied to KV Cache

A new technique applies operating system-style demand paging to LLM context tokens: only materializing tokens in the KV cache when the attention mechanism actually needs them. The claimed numbers — 90% memory reduction within 1% accuracy loss — are extraordinary. A 128K context inference currently requiring 4× A100 80GB could potentially run on a single card.

Critical unknowns to validate: What's the latency profile of 'attention page faults'? Does it compose with continuous batching and speculative decoding? Does the 1% accuracy number hold on adversarial needle-in-a-haystack tasks? Worth a deep read of the paper before making infrastructure bets.

The Multi-Agent Token Explosion Is Real

One documented power user's progression tells the story: 150K tokens/day (heavy ChatGPT usage, summer 2024) → 870M tokens/day (multi-agent system with 4 sub-agents, March 2026) — a 5,800x increase in under two years. Even discounting this as an outlier, the architectural pattern is instructive: hub-and-spoke agent topology where each sub-agent gets its own full context window, full system instructions, and full output. There's no shared context optimization — every sub-agent pays the full token tax.

The infrastructure investment picture reflects this demand curve: Fal at $8B valuation for AI model deployment, Hosted.ai at $19M seed for GPU workload pooling, Pado AI at $6M for data center orchestration. The inference stack is disaggregating into funded, specialized layers — the same pattern Cloudflare followed in CDN.

The Supply Ceiling

NVIDIA has locked 70% of TSMC's 3nm capacity. ASML's EUV lithography is hard-capped at ~700 units/year. A 44 GW power shortfall persists through 2028, driving Meta to secure 6.6 GW via TerraPower nuclear. GPU scarcity persists — inference optimization and semantic caching aren't optional, they're survival strategies.

The Paradox, Articulated by Someone Who Lived It

Raffi Krikorian — who ran Uber's self-driving car program — crashed his Tesla and wrote what should be required reading for anyone building human-in-the-loop systems: 'We are asking humans to supervise systems designed to make supervision feel pointless... a machine that works almost perfectly? That's where the danger lies.'

A system that works 97% correctly doesn't train humans to catch the 3%. It trains them to stop looking.

This isn't abstract philosophy — it's the failure mode of every human-in-the-loop architecture we build. Your deployment pipeline that auto-approves 97% of changes correctly. Your AI-assisted code review that catches most bugs. Your monitoring system that auto-triages with 95% accuracy. The human 'supervisor' isn't supervising — they're rubber-stamping. And the 3% that needs their attention sails through because the system trained them to stop paying attention.

Three Production Examples This Week

The DLSS case is technically instructive: NVIDIA's loss function and users' aesthetic judgment were measuring different things. Jensen told gamers they were 'completely wrong.' The product got rolled back anyway. If your AI pipeline optimizes for quantitative metrics but produces output humans interact with qualitatively, you need user studies, not just dashboards.

Design Patterns That Actually Work

The fix isn't 'add more humans' or 'add more AI.' It's designing systems that acknowledge attention decay:

1. Mandatory confirmation friction for high-consequence actions — not dismissible modals, but friction that requires engagement (e.g., type the resource name to delete it)
2. Randomized adversarial injections to keep operators calibrated — periodically insert known-bad items that require human catch. Track the catch rate as your real supervision metric.
3. Honest classification of your human-in-the-loop: if the approval rate exceeds 98%, you don't have a human-in-the-loop — you have a human-in-the-rubber-stamp. Either add verification friction or remove the human and invest in automated guardrails instead.

Microsoft's Copilot lesson applies to every team shipping AI features: AI features should follow the pull model (user seeks them when they have intent) not the push model (ambient integration everywhere). Instrument actual usage rates before expanding surface area.