The Engineer

The Architecture That Survives Agent Traffic

Databricks published the clearest production case study for taking rate limiting off the latency budget. Old path: Envoy → Ratelimit Service → Redis. Two network hops per request. 20-40ms at p99 under contention. New path: in-memory counters with 100ms async batch reporting, server-pushed rejection instructions, zero network calls on the hot path. p99 dropped roughly 10x.

The mechanism is the point. A synchronous shared-store limiter adds a network hop to every request it gates. Under normal load, invisible. When one tenant bursts, Redis becomes the bottleneck for everyone, including the requests you wanted to reject. Stop asking permission on every call. Report counts asynchronously.

A synchronous check against a centralized store is fundamentally incompatible with low-latency requirements once traffic gets large. The network is the bottleneck, not the algorithm.

Why This Matters Now: The Agent Traffic Cliff

Box's CEO projects a 90/10 agent-to-human traffic ratio within three years. Discount the timeline if you want. The direction is confirmed across multiple sources. Limiters keyed on IP or user token degrade under agent load. One agent holding one token issues bursts that look like a DDoS from a logged-in customer. The limiter does what it was configured to do. That is the problem.

Compounding this: reasoning models burn 10-100x the tokens of a plain completion. Personalized prompts break caching because each prompt varies per user. Multi-tenant batching breaks when every call carries user-specific context. The two escape hatches SaaS teams relied on are gone for AI workloads.

The Databricks Design Decisions Worth Copying

Three coupled choices made the system work:

1. In-memory token bucket — CAS is free in-memory and expensive over network to Redis. State location constrained algorithm choice.
2. Batch-reporting on a 100ms timer — spiky inbound becomes constant outbound. Thundering-herd goes away.
3. Server-pushed rejection — "reject key X at Y% until timestamp Z" replaces per-request permission checks.

The ~5% overshoot tolerance is explicit. Their backends already absorb slight over-limit traffic, so the tradeoff is cheap. If your rate limits enforce hard financial or security boundaries, this architecture does not apply. For everything else, which is most traffic, you are paying 20-40ms per request for exact enforcement nobody needed.

Migration Path

They could not modify Envoy, so they ran a localhost sidecar for the batch-reporting client. Before in-memory counters shipped, Redis Lua scripts batched writes as a bridge. Each step shipped value independently. Dependencies forced the order: sharded in-memory first, then batch-reporting to decouple the hot path, then token bucket to leverage in-memory CAS.

Shift 1: Small RL Models Claim Frontier Parity

A 4B-parameter Recursive Language Model claims parity with Claude Sonnet 4.6 by training a shared parent/child policy that recursively decomposes tasks. The mechanism removes inter-model latency from the standard orchestrator-plus-workers pattern. A 4B model on an A10G runs roughly 100x cheaper per token than Sonnet-class via API.

The caveat gets equal weight: "matches" on which eval suite, at which temperature, with how many reasoning tokens. A curated math/code subset is not production agentic work. Verify on your own eval before rewriting a cost model. Parity claims holding for six months and then cracking on harder evals is a well-documented pattern.

Shift 2: Serverless GPU Cold-Starts Collapse

Modal cut inference cold-starts from kiloseconds to tens of seconds. Two orders of magnitude. The likely mechanism is memory snapshotting with lazy GPU state rehydration, not cold container plus cold weight load. Cold starts are the invisible tax forcing most teams onto always-on GPUs at sub-50% utilization. At 30 seconds, scale-to-zero gets viable for bursty workloads.

The failure mode worth watching: 30s average with a p99 in the minutes. Ask for the distribution, not the mean.

Shift 3: Chinese Labs Prove the Floor

DeepSeek V4 Pro serves near-Opus quality at $0.43 per million input tokens at 50-70% gross margin. The margin disclosure kills the "subsidized land-grab" dismissal. Z.ai claims 50% margins on GLM-5 at $1/M tokens. MiniMax reports 70% enterprise margins. The gap against US incumbents is 11-28x.

The mechanism is straightforward. Cheaper accelerators. Denser batching. Architectures tuned for throughput over leaderboard position. Compute scarcity forced these labs to optimize every FLOP, and that discipline now shows up as a structural cost advantage in production serving.

The Combined Implication

Inference pricing assumed two things: frontier quality required frontier-sized weights, and serverless GPU had a cold-start floor that made bursty workloads uneconomical. Both are now contestable. Not broken. Contestable.

OpenAI deprecated finetuning APIs this week, which confirms what the usage data already said. For 99% of use cases, a base model with a decent system prompt and three retrieved examples beats a LoRA. The top 1%, Cursor and Cognition, run RLFT on open weights where they own the training loop. Everyone in the middle using vendor finetuning as a half-measure gets neither side of the tradeoff.

The Honest Cost Math

Cactus Compute's Needle, 26M parameters, no FFN layers, hits 6000 tok/s prefill for tool-calling and beats 10x larger models. This is the shape of a routing layer. A tiny model picks the tool. The expensive model only runs actual work.

Cloudflare Lost 60% Throughput to a 3-Line Port

Linux kernel CUBIC resets the congestion window when a connection goes idle, and it measures idle from the last sent packet. Cloudflare's quiche ported that logic straight into QUIC. QUIC does not ACK like TCP. After heavy loss the cwnd drops to its floor of 2 packets, and the idle check, reading from the wrong reference point, pins it there permanently. 60% of test downloads failed.

Fix: measure from last ACK instead of last send. Three lines. The assumptions underneath are the hard part. Any QUIC stack whose congestion control descends from the Linux kernel is carrying some variant of this bug.

Code moves line-for-line out of a codebase where certain invariants are held by the surrounding system. The port compiles, the tests pass. The invariants are silently gone.

Discord's Idempotent Database Control Plane

Discord replaced fragile bash with a declarative YAML workflow engine driving ScyllaDB operations. Multi-day procedures now finish in hours. The invariant that makes it work: every task checks current state before acting rather than trusting what the previous step claims to have done. The engine can be dumb because the tasks are safe to retry.

The pattern travels. If a runbook contains the phrase "carefully verify" or "wait until you see," that runbook is the candidate. Speed is not the prize. The prize is that the procedure no longer requires a specific senior engineer awake at 3am.

DuckDB Breaks the Embedded-Only Model

DuckDB's new Quack protocol adds HTTP client-server with multiple concurrent writers and token-based auth. Integration is planned with DuckLake and DuckDB 2.0. For a shared analytical backend fronting several services, the math changes. For a single-service analytical workload, embedded is still the right answer.

Redshift RG: AWS Pulls Query Layer Back In-House

Redshift RG runs 2.2x faster than RA3, costs 30% less per vCPU, drops the $5/TB Spectrum scanning fee, and executes Iceberg queries 2.4x faster. No code changes required. The strategic read: AWS is removing the cost incentive to route data lake queries through Athena or an external engine. If queries are currently split between Redshift and Athena for cost reasons, that split may collapse back to a single engine.

The Common Thread

All four items share one shape: assumptions imported from one context silently fail in another. TCP idle semantics in QUIC. Human-speed pacing in runbooks. Single-process assumptions in analytics. Per-scan pricing on data lake queries. Each was a reasonable inheritance that nobody reexamined until the cost landed on a dashboard.