The Scientist

Coding Agent Index and Speculative Decoding, Same Stack

The Artificial Analysis Coding Agent Index and the speculative decoding tutorial both landed this week, pointing at the optimization surface most serving teams have not touched. The Index reports that model+harness pairs with comparable task completion carry >30x cost-per-task variance, >3x token variance, 80-96% cache-hit-rate spread, and >7x wall-time variance on the same workload. The tutorial shows that Llama 3.2 1B as a drafter delivers 2.31x throughput over vanilla autoregressive decoding on a Llama 3.1/3.3 70B target, with mathematically identical outputs and zero quality loss.

The unit of evaluation is the (model, harness, cache policy) triple, and most teams have only optimized one of the three.

Why Small Beats Big on Drafting

The counterintuitive result: the 1B drafter beats the 8B drafter at 2.31x versus 2.08x, despite a lower acceptance rate. Drafting latency compounds at every step, and at 8B the overhead of running the drafter itself eats the verification gains. Cross-tokenizer speculation (UAG) caps at 1.5-1.9x, which means same-family drafting is the only path to the full win.

The verification step is compute-bound like prefill, not memory-bandwidth-bound like decode, so it saturates silicon that would otherwise idle. Best case harvests K+1 tokens from a single target forward pass and worst case is still 1 token, so expected value is non-negative modulo drafter memory.

The Harness Is the Cost Lever

The 30x cost variance from the Index is not a model-selection question. It is a harness-selection question. Opus 4.7 in Cursor CLI led at 61, GPT-5.5 in Codex/Claude Code was close, and the open-weight setups (GLM-5.1, Kimi K2.6, DeepSeek V4 Pro in Claude Code) came in competitive but behind. The thing this doesn't tell you is which harness a given workload actually resembles, but most current defaults sit 5-10x on the wrong side of the cost curve at equal quality.

Separately, TurboQuant, a widely adopted quantization shortcut, is showing empirical failure under the first comprehensive accuracy+latency+throughput study. Teams with TurboQuant in their quantization path should freeze adoption pending internal repro.

The Combined Playbook

These are independent optimizations that stack multiplicatively.

1. Enable speculative decoding with --speculative-model meta-llama/Llama-3.2-1B-Instruct and num_speculative_tokens=5. Measure on real traffic, since chat, RAG, and code show different acceptance profiles. The reported 2.31x will likely land at 1.5-1.8x on production prompts.
2. The harness triple is the first thing worth measuring. Three model+harness+cache combinations, scored on cost/task, tokens/task, cache hit rate, and wall time. The AA methodology is reproducible on an internal repo in a one-week spike.
3. Acceptance-rate telemetry belongs in the request path. Per-request mean-accepted-length and effective-speedup, with an alert when acceptance drops below 60% for a traffic slice. That is where the drafter stops paying rent.

What Happened

Anthropic is in advanced talks to acquire Stainless for $300M+. Stainless is not a model company. It is the OpenAPI-to-SDK generation shop that produces the official Python, TypeScript, Go, Java, and Ruby client libraries for Anthropic, OpenAI, and Google. Every team that has run pip install openai or imported google-genai is shipping Stainless-generated code into production.

One lab now controls the developer-experience layer of its two largest competitors. Even under the most benign post-close conduct, competitor SDK roadmap and release cadence get set by a competitor.

Why This Matters More Than the Price Tag

Stainless's stated direction explicitly includes AI agents as first-class API consumers. In practice that means agent-grade SDKs that handle deterministic tool-schema serialization, structured-output decoding, retry semantics that do not corrupt trajectories, and token-accurate streaming. If Anthropic wants Claude Code and the Anthropic SDK uniquely good for agentic workflows, the fastest lever is Stainless's codegen templates.

The failure mode is not a breach. It is drift. A deprecation here, a lag in supporting a new model parameter there. Drift is harder to measure than an outage, which is the argument for pinning versions and running automated diffs against the upstream API spec.

What Happens Next

The base rate says OpenAI and Google will bring SDK generation in-house within 6-12 months. Expect a period of SDK divergence: subtle behavior shifts in retry logic, streaming parsers, and tool-call schemas. The thing this doesn't tell you is how many eval harnesses silently depend on one provider's conventions. That is where the regressions will show up first, and they will look like model quality changes.

Adjacent signal: OpenAI's $18B Broadcom custom-silicon deal hit a financing snag. Any 2026 unit-economics model that assumed ASIC-driven token-price cuts from OpenAI should push that assumption right by at least a year.

The Defensive Posture

The migration cost is small if done now. Most teams use a narrow slice of the SDK surface: auth, retries, typed requests, streaming parsers. A motivated team replaces the hot path with about 200 lines of httpx. That is the honest upper bound on exposure. The real decision is whether a full abstraction layer earns its keep by enabling provider A/B testing, or whether it is over-engineering for a risk that resolves when the big labs fork.

The Convergent Signal

Ramp's result is the one that should change how teams budget this quarter. Ramp trained a small RL model with Prime Intellect that beats Opus by 4% exact-match accuracy on spreadsheet Q&A at Haiku latency. In the same week, Monday.com disclosed that AI inference is compressing gross margins while shipping per-user credit tracking, and the LLM pricing curve bifurcated. GPT-5.5 and Opus 4.7 moved up while DeepSeek V4 collapsed the floor. Nothing defensible sits between them.

Workloads with a verifier (SQL equivalence, unit tests, exact-match, schema-valid JSON) are candidates for small-model RL distillation today. The Ramp result pulls the distillation frontier inside most teams' cost-of-experimentation budget.

The Middle Tier Is Gone

Any router still defaulting to a mid-tier model is picking a strictly dominated option. The positions that still make sense:

• Frontier tier (GPT-5.5, Opus 4.7): reasoning-heavy, open-ended tasks where accuracy per call matters more than cost per call.
• Commodity tier (DeepSeek V4, distilled in-house models): high-volume structured tasks where cost per completed task is the binding constraint.

Monday.com is the first clean public example of the squeeze. Revenue growth decelerated from 27%+ to 19-20% while the company claimed AI productivity gains, and the stock is off 48% YTD. The market is penalizing unquantified AI ROI narratives, which means the internal version of that story should be better evidenced than the punished public ones.

When It Works and When It Doesn't

The Ramp number has caveats. No confidence interval, no sample size, no eval-set provenance disclosed. A 4% gap on 500 samples is noise. A 4% gap on 50K is a shipping decision. The thing the headline doesn't tell you is how the task distribution was constructed. Narrow RL on a narrow eval is circular unless the eval matches production traffic.

The decision rule is simple enough. Workloads where the input schema is stable and the output is verifiable are distillation candidates. Workloads where the distribution drifts week to week are not. Expect roughly half the reported gain on first pass with in-house data. Half of +4% is +2%, which still clears migration cost for high-volume endpoints.

The Capacity Backstory

Anthropic is running at 80x annual growth against a 10x plan, with ARR tracking $9B to $45B in roughly five months. Claude Code is explicitly compute-starved. Rate limits will tighten. Tail latencies on high-context models, where margins are worst, will degrade further. Teams that measure substitution cost now will have more options than teams that measure it under a rate limiter in Q3.

Coatue identifies HBM as the next infra bottleneck with 5x demand growth over 5 years. Training cost is not the binding constraint. Inference-time KV footprint is. If the Ramp recipe generalizes and more teams run more small models in-house, aggregate HBM demand pushes further in the direction Coatue is pointing.