The Scientist

The Physics You Can't Optimize Away

A first-principles breakdown of transformer inference economics reveals the complete FLOP cost formula: 24nd² + 4n²d per layer. The quadratic attention term (4n²d) crosses the linear projection term at n=2d — for d=2048, that's exactly 4,096 tokens, explaining why 4K was the standard context length for years. Beyond this crossover, costs explode: at 32K context, the quadratic term accounts for 73% of total compute. At 128K, it's 92%.

But the deeper problem is the prefill/decode regime split. Prefill (processing your prompt) runs compute-bound at ~4,096 FLOPs/byte. Decode (generating each token) runs at just 1 FLOP/byte at FP16 — catastrophically memory-bound. The H100's compute-to-bandwidth threshold is 295 FLOPs/byte, meaning your GPU sits at ~0.34% utilization during token generation. You're paying for 989 TFLOPS and using 3.4.

Compute power compounds at 3x every two years while memory bandwidth grows at roughly half that rate. The decode bottleneck gets structurally worse with each hardware generation.

KV Cache: The Concurrency Killer

KV cache is the binding constraint on GPU concurrency, and the numbers are stark. A 7B INT4 model on an H100 (80GB HBM) serves 278 concurrent users at 4K context but only 8 at 128K — a 35x cost increase per user from $0.009/hr to $0.31/hr. Double the context, halve the concurrent users — it's a direct linear relationship.

This table should be on every ML team's wall. Your 128K context feature isn't just expensive in FLOPs — each long-context session evicts other users from the GPU.

The Architecture Evaluation Litmus Test

Every serious architectural innovation of the last two years attacks exactly two numbers: bytes of KV cache per token, and bytes of weights loaded per decode step. Apply this filter ruthlessly:

• GQA (Llama 3.2): 4x KV cache reduction. ✅ Moves number 1.
• MoE (Mixtral 8x7B): 47B total params but ~13B active. ✅ Moves number 2.
• Hybrid attention/SSM: 6 attention layers instead of 16 = 192 MB vs 512 MB at 32K. ✅ Moves number 1.
• FlashAttention: Optimizes memory access, does NOT reduce FLOPs. ❌ Moves neither number.
• INT4 quantization: Quadruples arithmetic intensity from 1 to 4 FLOPs/byte — the single largest software-side decode improvement.

If a new architecture paper doesn't clearly move one of these two numbers, it doesn't change your inference economics regardless of benchmark scores.

Cross-Source Tension: Long Context vs. RAG

Here's where today's intelligence gets interesting. Claude Sonnet 4.6 ships with a 1M-token context window (beta), which theoretically lets you skip RAG entirely for documents under ~750K tokens. But the inference economics above show why this is expensive: at 128K context you're already at 92% quadratic compute share and 8 concurrent users per GPU. Scaling to 1M context would be economically devastating at self-hosted scale. The implication: long-context models make sense through API providers who absorb the utilization problem, while self-hosted deployments should invest in RAG with simple chunking (see next deep dive) and aggressive context management.

The raw compute floor for a well-optimized 14B deployment is ~$0.004/M tokens at full utilization. API pricing runs $0.10-$1.25/M tokens — an 8-40x markup that covers redundancy, SLAs, and the engineering team you don't hire. But hidden self-hosting costs range from $125K-$190K/year (minimal) to $6M-$12M+ (enterprise-scale).

RAG Chunking: 512 Tokens Wins

FloTorch's 2026 benchmark evaluated multiple chunking strategies for RAG pipelines. The winner: recursive character splitting at 512 tokens — the most basic approach in the toolkit. It beat semantic chunking (embedding-based boundary detection) and proposition-based chunking (LLM-extracted atomic claims) on accuracy while producing 3-5x fewer vectors, directly translating to lower vector DB storage, query latency, and infrastructure costs.

The directional signal aligns with practitioner intuition: semantic chunking introduces its own error modes (embedding model quality, boundary sensitivity) that can degrade retrieval more than they help. Proposition-based chunking fragments context that retrieval needs to reconstruct. Simple splitting preserves local context windows naturally.

Methodological caveat: the benchmark doesn't disclose dataset composition, query type distribution, embedding model choice, or the specific accuracy metric (recall@k? MRR? answer correctness?). The cost finding (3-5x lower vector counts) is mechanically robust — fewer chunks means fewer vectors — but the accuracy claim needs your own evaluation harness to validate on your domain.

Agent Scaffolding: Harness > Model

LangChain's coding agent jumped from Top 30 to Top 5 on Terminal Bench 2.0 with only a harness change — same underlying model, no fine-tuning. The key techniques: self-verification (having the agent check its own outputs before submission) and structured tracing (logging the reasoning chain for debugging). A Top 30→Top 5 jump is dramatic enough to suggest most teams are leaving significant performance on the table by focusing exclusively on model selection.

This converges with OpenAI's published prompt caching mechanics: restructuring prompts so identical prefixes are shared across requests enables KV reuse, cutting both latency and input token costs. The pattern is the same — engineering the scaffolding around the model delivers outsized returns.

A/B Testing: Your Lifts Are Inflated

Large-scale replications from Bing, Amazon, and Talabat show that trustworthy experiment lifts are typically below 1% — not the 5-15% gains that populate internal case libraries. The mechanism:

1. Underpowered tests (below 50% power) can only detect effects much larger than the true effect
2. When an underpowered test reaches significance, the estimated effect size is necessarily inflated (winner's curse)
3. Published case studies are doubly selected: only significant results get published, only impressive results get shared
4. Even very large experiments often lack power for revenue-per-user and purchase rate, forcing reliance on surrogate metrics

If your experimentation program regularly reports lifts above 1% on core business metrics, you probably have a power problem, not a winning streak.

The Convergent Pattern

Three independent domains — retrieval, agents, experimentation — all point to the same conclusion: the dominant failure mode in production ML is over-engineering. Teams invest in complex semantic chunking when simple splitting works better. They swap models when scaffolding changes deliver 5x the improvement. They trust inflated experiment results because they never audited statistical power. The highest-ROI work this sprint isn't adding complexity — it's removing it.

MCP as the Standard Protocol

Multiple signals this week confirm Model Context Protocol (MCP) is solidifying as the integration standard between AI agents and external tools. Figma now accepts work from Claude Code via MCP server, producing fully editable design layers — demonstrating MCP handling complex structured output (vector layers with hierarchy and constraints), not just text or simple function calls. Separately, Agoda built a zero-code tool that converts any REST or GraphQL API into an MCP endpoint using automated schema introspection and in-process DuckDB for context-limited summarization.

For data scientists building agentic systems, MCP adoption by a $20B+ company (Figma) validates it as enterprise-grade. If your agents need to interact with internal tools — dashboards, feature stores, experiment trackers — MCP is worth evaluating as the integration layer instead of building custom function-calling adapters.

Security: The Missing Layer

Nono launched as the first kernel-enforced sandbox purpose-built for AI agents, MCP, and LLM workloads. It enforces zero-trust at the kernel level — agent actions (file writes, network calls, credential access) are constrained by explicit capability grants rather than prompt-level guardrails or container boundaries. Meanwhile, analysis of AI agent authorization patterns shows traditional RBAC/ABAC (including AWS Cedar) is inadequate; relationship-based access control (ReBAC) via SpiceDB/Google Zanzibar is emerging as the required pattern for agent-to-data access.

No performance benchmarks or latency overhead measurements have been published for Nono, which is a significant gap for production LLM inference workloads.

The Context Explosion Problem

Here's where agent infrastructure collides with inference economics. Agentic workflows naturally cause context explosion as agents share traces, tool outputs, and reasoning chains. If your agent orchestration passes full conversation history between agents, you're compounding the KV cache problem multiplicatively. At 32K context, you're at 34 concurrent users per GPU on a 7B model. Multi-agent traces pushing to 128K drop you to 8.

The solution is architectural: design agent communication protocols that summarize rather than pass raw context. This is where the inference economics deep dive directly informs agent design — every token of context you pass between agents has a concrete dollar cost in KV cache memory and evicted concurrent sessions.

GitHub Agentic Workflows

GitHub's Agentic Workflows entered technical preview — developers describe automations in plain Markdown, and a coding agent executes them within GitHub Actions. The dream for ML teams: describe outcome-based CI/CD ("run integration tests, check model metrics don't regress beyond 2%, auto-approve if passing") without writing YAML. The reality: no reliability metrics, latency benchmarks, or failure mode documentation published. The gap between "agent triages GitHub issues" and "agent reliably orchestrates model retraining with rollback" is enormous. Test on non-critical workflows first.

Netflix's LLM Post-Training Stack

Netflix open-sourced their LLM post-training infrastructure, revealing a production stack worth studying:

The most technically interesting detail: on-the-fly sequence packing with document masking for skewed distributions. Sequence packing concatenates variable-length examples to fill GPU memory efficiently (eliminating padding waste), while document masking ensures attention doesn't cross document boundaries within packed sequences. No throughput benchmarks or convergence comparisons were shared — this is architectural intelligence, not a reproducible result.

Magma: Sparse Updates Beating Dense Optimizers

Google's Magma optimizer uses momentum-aligned gradient masking to improve pre-training efficiency. The counterintuitive finding: randomly masking parameter updates can outperform dense adaptive optimizers like AdamW. When momentum and current gradient disagree on direction, masking that update avoids noisy steps that waste compute. A masked RMSProp variant reportedly exceeded recent state-of-the-art methods.

We only have a summary-level description — the actual paper is needed to assess scale, ablations, and evaluation protocol. But the experiment is straightforward: swap your optimizer on a representative fine-tuning run, hold everything else constant, compare convergence curves.

YOLO26: NMS Elimination for Edge Detection

Ultralytics released YOLO26, eliminating Non-Maximum Suppression via a dual-head architecture. During training, a one-to-many head provides rich gradient signal from multiple box assignments per object. At inference, a one-to-one head outputs exactly one prediction per object — no post-processing, no threshold tuning, no platform-specific cleanup.

Important context: DETR-family models have been NMS-free since 2020. YOLO26's contribution is bringing end-to-end inference to the YOLO architecture specifically, which matters because YOLO dominates real-time edge deployments where DETR models are too slow. However, zero quantitative benchmarks were published — no mAP on COCO, no latency comparison against YOLOv8 + NMS, no comparison against RT-DETR (which is NMS-free under Apache 2.0, not AGPL).

Pinterest's Auto-Healing Spark

Pinterest's Auto Memory Retries for Spark implements progressive escalation: first increase CPU allocation (many OOM failures are contention-induced, not genuine memory exhaustion), then scale memory at 2x/3x/4x profiles. Result: 96% reduction in OOM failures with compute cost savings. The CPU-first retry insight is directly implementable if you run Spark at scale.