ClarityBeta

Science & Analytics

The Scientist

9 min read · 1,729 words · 4 moves to watch
The Signal

Google's Gemma 4 ships the most aggressive KV cache engineering in any open model

If your team is evaluating Gemma 4 on H100s this week, you're benchmarking the model at ~9 tok/s when it's capable of 124 tok/s on Blackwell. Stop the eval until vLLM ships per-layer kernel dispatch — or accept you're testing the wrong thing.

In Play

  1. Gemma 4: 83% KV Cache Reduction, 14x Serving Penalty

    Gemma 4 stacks 5 independent KV cache compression techniques for 83% reduction, but 512-dim attention heads break FA2 on all pre-Blackwell GPUs — throughput drops from ~100 to ~9 tok/s. Edge and server models are architecturally divergent, not scaled versions. The vLLM per-layer dispatch fix is still an open issue.

    Ask Clarity

  2. Opus 4.7 Migration Crisis + K2.6 Open-Weight Price Disruption

    Opus 4.7 deprecates budget_tokens, prefilled responses, and inflates multi-turn costs — a migration event, not an upgrade. Moonshot's K2.6 matches Opus 4.6 on 4/6 benchmarks at $0.95/M input tokens (vs $5/M) under Modified MIT. If you're on Claude, you face a two-front decision: migrate forward or evaluate an open-weight alternative at 5x savings.

    Ask Clarity

  3. Context Engineering Beats Model Selection: Production Evidence

    Three production systems independently prove context engineering dominates model choice. STCLab runbooks lifted agent quality from 3.6→4.6/5 without model changes. Cloudflare's 7-agent code review hit 85.7% cache hit rate across 120B tokens at $1.19/review. Shopify's Tangent agent achieved 5.25x QPS gains via automated optimization. Token economics fragmentation (8+ types, 15x reasoning inflation) makes routing the primary cost lever.

    Ask Clarity

  4. Continual Learning Taxonomy: Where RAG Hits Its Ceiling

    a16z maps the full continual learning spectrum from EWC to TTT-Discover, arguing RAG has a hard ceiling for tacit knowledge, adversarial adaptation, and agent coherence beyond ~20 steps. SDFT (self-distillation fine-tuning) is closest to production. SSMs aim to extend agent coherence from ~20 to ~20,000 steps. An 8B model with knowledge modules reportedly matches 109B on targeted tasks. Parametric approaches remain evaluate-only.

    Ask Clarity

  5. ML Serving Stack: Two Unpatched RCEs

    protobuf.js has a CVSS 9.4 RCE (GHSA-xq3m-2v4x-88gg) affecting every gRPC-based serving stack — TF Serving, Triton, @grpc/proto-loader. Patch to 8.0.1/7.5.5 today. Separately, SGLang has an unpatched RCE via malicious GGUF model files with zero vendor response. If you load models from external sources on either framework, you have an open code execution vector.

    Ask Clarity

Deep Dives

Gemma 4: Five KV Cache Innovations, One Serving Catastrophe

Why This Matters Right Now

Google shipped Gemma 4 with a design choice more consequential than any benchmark: the edge and server models are architecturally divergent, not scaled versions of the same core. E2B exploits flash/DRAM asymmetry with Per-Layer Embeddings (46% of its 5.1B params are static flash lookups); server models skip PLE entirely because H100's 80GB HBM has no such asymmetry. This signals the end of the "one architecture, scale it up" paradigm.

The KV Cache Attack Stack

Gemma 4 stacks five independent compression techniques for an 83% KV cache reduction at 8K context:

  1. Interleaved sliding-window attention: 80% of layers pay O(n) instead of O(n²) — 5x attention speedup on edge
  2. Grouped-Query Attention: 8:1 on edge (most aggressive in family), differential 2:1/8:1 on server
  3. Cross-layer KV sharing: 20 of 35 E2B layers skip KV projection entirely, reusing from earlier layers — back-loaded and type-matched to prevent attention contamination
  4. K=V weight sharing (server only): Global layers compute key once, reuse as value with RMSNorm — halves global KV cache on top of GQA
  5. Wider MLPs on shared-KV layers: MLP width doubles from 6,144→12,288 where KV is shared — a compute-for-memory swap revealing quality loss from sharing is real

For comparison, DeepSeek's MLA achieves 93.3% within-layer compression vs. Gemma's 83% cross-layer sharing. These are complementary — cross-layer and within-layer — and could theoretically be combined.

Partial RoPE: The Long-Context Fix

Standard RoPE rotates all attention dimensions, but positional encoding overwhelms semantic content at long ranges. Gemma 4's 512-dim global heads split: 128 dims (25%) get theta=1M rotation; 384 dims (75%) are pure content channels with zero rotation. Result: the 31B model jumps from 6.6% to 86.4% on tau2-bench Retail — a 13x improvement. Google published no ablation isolating partial RoPE's contribution from training data changes.

The FA2 Crisis: Your GPU Generation Matters More Than Your Model Choice

This is the deployment fact that overrides everything else: 512-dimension global attention heads exceed FlashAttention-2's hard limit of 256. On every pre-Blackwell GPU, Gemma 4 falls back to unoptimized Triton kernels:

HardwareThroughputStatus
Blackwell124 tok/sOptimized kernels available
H100 / A100 / RTX 4090~9 tok/sFA2 fallback — 14x penalty

The fix — per-layer backend dispatch routing local layers to FA2 and global layers to alternative kernels — is an open vLLM issue as of April 2026. Until this ships, Gemma 4 is effectively a Blackwell-only model for production serving.

Gemma 4 arrived before its serving infrastructure — if you're on H100s, you're paying a 14x throughput tax until vLLM patches per-layer kernel dispatch.

What to do

  1. Run `nvidia-smi` to confirm your GPU generation before any Gemma 4 evaluation — benchmark only on Blackwell or defer until vLLM per-layer dispatch lands

  2. Profile KV cache cosine similarity across layers in your existing models — if adjacent layers show >0.9 similarity, implement type-matched cross-layer sharing

  3. Implement partial RoPE (25% rotated, 75% content) in your next long-context training run if retrieval degrades beyond 32K tokens

  4. Track vLLM per-layer dispatch issue and re-benchmark Gemma 4 when the patch ships

Opus 4.7 Breaks Three Integration Patterns — And K2.6 Just Undercut It 5x

Two Shifts, One Decision

Two releases landed in the same cycle pulling in opposite directions. Opus 4.7 ships behavioral changes that break existing pipelines — this is a migration event, not an upgrade. Simultaneously, Moonshot's Kimi K2.6 claims Opus 4.6 parity on agentic benchmarks at 5-6x lower cost with open weights under Modified MIT. If you're running production Claude workloads, you face a two-front decision.

Opus 4.7: What Breaks

At least three integration patterns will fail on migration:

  1. budget_tokens deprecated: You must switch to thinking: {type: 'adaptive'}. The model now decides how much to reason — not you.
  2. Prefilled assistant responses deprecated: Already returning 400 errors on Mythos Preview. Any structured output harness using prefills will break.
  3. Per-turn reasoning overhead: Every user message triggers reasoning. Multi-turn pair-programming sessions now cost significantly more.

Additional behavioral shifts: 4.7 follows instructions more literally, won't silently generalize, spawns fewer subagents, and introduces a five-tier effort system (low through max). The upside is real — 11 percentage points better recall on Anthropic's hardest bug-finding eval, low effort on 4.7 outperforms 4.6 at the same level — but you have to earn these gains through deliberate migration.

A subtle trap: 4.7's improved recall plus literal instruction-following creates a paradox for code review. Conservative severity thresholds now suppress the very bugs the model is better at finding. Solution: two-stage pipeline. Stage 1: find everything (high recall, no filter). Stage 2: classify and filter.

K2.6: The Open-Weight Pricing Challenge

BenchmarkK2.6Opus 4.6Delta
SWE-bench Pro58.653.4+5.2
HLE with tools54.053.0+1.0
DeepSearchQA92.591.3+1.2
LiveCodeBench89.688.8+0.8

Critical caveat: These are vendor-published benchmarks with no independent verification. Three of four "wins" are within 1.2 points — likely within run-to-run variance. Only SWE-bench Pro's +5.2 passes the smell test for significance. The 12-hour autonomous demos (4,000+ tool calls, compiler from scratch) are capability ceilings, not reliability floors.

But the pricing is unambiguous: $0.95/M input vs. $5.00/M (5.3x cheaper), $4.00/M output vs. $25.00/M (6.3x cheaper). At these margins, K2.6 only needs ~80% of Opus 4.6's quality to be cost-effective on throughput-sensitive workloads. Modified MIT means self-hosting eliminates per-token costs entirely.

Opus 4.7 is strictly better than 4.6 at every effort level, but treating it as a drop-in replacement will break your pipelines and inflate your costs; K2.6 is the first credible open-weight threat to frontier agentic pricing.

What to do

  1. Inventory every API call using budget_tokens, prefilled assistant turns, or multi-turn agentic loops — these are your three failure vectors for the 4.6→4.7 migration

  2. Run K2.6 head-to-head against Opus 4.6 on 100+ tasks from your actual workload this sprint — measure pass rate, latency, and cost-per-completion

  3. Implement effort-level routing in your LLM orchestration layer: classify incoming tasks by complexity, route to appropriate Opus 4.7 tier, log token usage per tier

  4. Refactor code review prompts into two stages (find all bugs → filter by severity) before migrating to 4.7

Context Engineering > Model Selection: Three Production Systems Prove It

The Convergence

Three independent production systems — from organizations at radically different scales — arrived at the same conclusion this cycle: what you feed the model matters more than which model you feed. This isn't a vague principle anymore; it comes with hard numbers.

STCLab: Runbooks Beat Model Upgrades

A two-person SRE team deployed HolmesGPT for alert investigation and found that structured markdown runbooks (specifying which tools to skip per namespace) lifted quality scores from 3.6 to 4.6 out of 5 — a 28% improvement without changing the model. Wasted tool calls dropped from 16 to 2 per investigation. Total cost: ~$12/month for ~12 daily investigations (~$0.03/investigation). Caveats: single-team case study, unspecified evaluation methodology.

Cloudflare: Semantic Caching at 120B Tokens

Cloudflare built a 7-agent AI code review system that processed 120 billion tokens in month one across 131K reviews, achieving an 85.7% cache hit rate at $1.19/review. The architecture uses circuit breakers and model failback chains. The cache hit rate implies most code review queries are semantically similar enough to serve cached responses — standard for large orgs with recurring patterns. Without caching, costs would be roughly 7x higher (~$1.1M/month vs. ~$156K).

Shopify: Critique Loops Crush Parallel Agents

Shopify's CTO Mikhail Parakhin calls parallel non-communicating agents "almost useless" — an explicit anti-pattern and token-burning waste. Their recommended architecture: critique loops where one model generates and a different model critiques, with the first model regenerating incorporating feedback. The key metric is the ratio of generation tokens to expensive review tokens.

Shopify's Tangent auto-research agent, running on their open-source Tangle platform, improved search index throughput from 800 QPS to 4,200 QPS (5.25x) on identical hardware through automated code optimization. In another run, 400+ experiments over weeks yielded only 1 success — but it found an improvement on a system already optimized for years.

Token Economics: The Hidden Cost Architecture

Underneath all of this, LLM pricing has fragmented into 6-8 distinct token categories: input, output, reasoning, cached, tool-use, vision, structural, speculative — each with different compute profiles and billing rates. The biggest trap: reasoning tokens can outnumber output tokens 15:1, billed at output rates. A 200-token answer may generate 3,000 internal thinking tokens. There is no billing standardization across providers.

Token TypeRelative CostOptimization Lever
Input1x baselineCaching, compression
Output2-6x inputStructured JSON, shorter schemas
Reasoning~Output rate, 10-15x volumeTask routing, model selection
CachedDiscountedStable system prompts

Task routing — matching task complexity to model capability — is the primary cost optimization lever. Sending classification to a reasoning model is, as one analysis puts it, "pure waste."

Context engineering beat model selection by 28% on quality scores — write better prompts before you buy bigger models.

What to do

  1. Write structured context documents (tool-skip lists, domain constraints, task decomposition guides) for your top 3 agent workloads before running any model evaluation

  2. Instrument token-type observability: parse reasoning_tokens, cached_tokens, and completion_tokens separately from every API response and build a per-feature cost dashboard

  3. Evaluate semantic caching for any LLM inference pipeline processing >1K requests/day — target 50%+ cache hit rate

  4. Evaluate Shopify's open-source Tangle for ML experiment orchestration — test content-addressed caching against Airflow/Dagster for cross-team compute deduplication

Two Unpatched RCEs in Your ML Serving Stack — Patch Now

The Threat

Two distinct remote code execution vulnerabilities affect ML inference infrastructure. One has a patch; one has no vendor response. Both are exploitable through artifacts your pipeline routinely processes.

protobuf.js: CVSS 9.4 via Schema Eval

Endor Labs discovered that protobuf.js concatenates unvalidated schema type names into JavaScript source code and evaluates them via the Function constructor — essentially an eval() equivalent. Any application processing untrusted .proto schemas through protobuf.js versions before 8.0.1 or 7.5.5 is vulnerable to arbitrary code execution.

Why this hits your stack specifically: protobuf.js is transitively included via @grpc/proto-loader, Firebase, and Google Cloud SDKs. If you run TensorFlow Serving, Triton Inference Server, or any custom gRPC model serving on Node.js — check your lock files. In most production setups, schemas are baked into containers at build time (limiting exposure), but any configuration endpoint, dynamic schema loading, or dev tooling that touches protobuf.js is at risk.

SGLang: Unpatched GGUF RCE

The SGLang framework — increasingly adopted for high-throughput LLM serving — has an unpatched vulnerability exploitable via malicious GGUF model files. The project has not responded to researchers. GGUF is the standard format for quantized models from HuggingFace and community fine-tunes. If your inference pipeline loads models from external sources via SGLang, this is a direct code execution vector on production infrastructure.

No CVE assigned. No severity score. No patch timeline. The broader lesson: model files are code, and loading untrusted model artifacts is equivalent to running untrusted executables.

Cross-Reference: The Expanding Agent Attack Surface

These infrastructure vulnerabilities compound with the agentic attack surface. Google's Antigravity IDE had a prompt injection flaw where the find_by_name tool executed shell commands before Secure Mode could evaluate them — a TOCTOU vulnerability applied to LLM agents. Microsoft's Azure SRE Agent leaked internal reasoning chains, credentials, and live command streams across tenant boundaries. The pattern is clear: AI systems generate novel information artifacts (reasoning traces, intermediate tool calls) that traditional security boundaries weren't designed to contain.

Model files are code. Schema files are code. Reasoning traces are sensitive data. If your ML serving stack treats any of these as benign, you're shipping exploitable infrastructure.

What to do

  1. Run `npm ls protobufjs` across all Node.js projects in your ML infrastructure and upgrade to 8.0.1 or 7.5.5 today

  2. Audit SGLang deployments for GGUF model file loading from untrusted sources; implement model file validation or sandbox model loading until patch exists

  3. Implement pre-execution allowlists for all LLM agent tool calls that can reach system commands — never post-validate

  4. Audit what crosses tenant boundaries in multi-tenant ML serving: reasoning traces, feature values, retrieved documents, embedding vectors, and credential stores

The bottom line

Gemma 4 shipped the most sophisticated KV cache engineering in any open model — 83% memory reduction, five stacked compression techniques, 128K context on phones — but broke FlashAttention-2 on every pre-Blackwell GPU with a confirmed 14x throughput penalty; Opus 4.7 broke three integration patterns your agentic pipelines depend on while Kimi K2.6 undercut it 5x on price; production evidence from Shopify, Cloudflare, and STCLab proves context engineering delivers 28% quality gains without model changes; and your gRPC serving stack has a CVSS 9.4 RCE right now. The infrastructure layer beneath your models — serving kernels, context management, token routing, and dependency hygiene — now determines your system's cost, speed, and security more than model selection does.