The Engineer

Why a pinned GPU stays pinned while the queue grows

Prefill and decode fail differently. Prefill runs the whole prompt in parallel and is compute-bound. Decode emits one token at a time and is bound by reading weights out of high-bandwidth memory, so the device reports busy at modest arithmetic intensity. The gauge reads saturated long before throughput is exhausted. It reads the same with zero requests waiting and with four hundred. That is the defect: the autoscaling signal is not a function of backlog. The horizontal pod autoscaler adds replicas late and removes them early.

The latency panel beside it is blind the same way. Continuous batching lets one long prefill stall short decodes. The distribution goes bimodal and the mean sits in the empty middle. It looks healthy while tail users time out. Histograms at p50/p95/p99, on time-to-first-token and inter-token latency, are the only view that shows the stalled request.

The replacement scaling signal is the serving engine's waiting-requests gauge, exposed on /metrics. Verify the exact metric name against your vLLM or SGLang version before wiring it up. These names have moved between releases. A silently absent metric evaluates as zero, and an autoscaler fed zero never scales.

Two more panels that mislead the same way

The agent change lands in this tier

Conditional replanning is a serving change in an agent costume. One LLM call per step becomes two, which doubles arrival rate against the same replicas. The two roles have opposite shapes. Planner prompts carry state plus prior plans plus action history, so they are prefill-heavy. Executor calls emit one action, so they are decode-heavy. Mixed into one queue, a long planner prefill delays every executor decode sharing that batch.

Two consequences follow. Splitting model routing by role, strong model for the planner and cheap grounded model for the executor, is a cheaper lever than a frontier upgrade, and it lets the two workloads scale as separate pools with separate SLOs. And a per-request token budget that hard-caps steps is the only defense against one pathological task occupying a replica indefinitely.

Prefix caching is where agent loops genuinely pay off. A stable system prompt plus a fixed plan prefix repeats across every step of a task, which is close to the ideal cache shape. That is good engineering, and it deserves the credit. Measure the hit rate per workload rather than assuming a global win. A retrieval-heavy endpoint with unique context per call gets nothing from it, and a blended average will not tell you which one you have.

Utilization tells you the device is busy; only queue depth tells you the users are waiting.

Sequencing matters. Observability precedes optimization. Prefix caching, quantization and speculative decoding all have workload-dependent ceilings that histograms find and averages hide. Tune after the tail is visible, not before.

The bug is the state object, not the prompt

A ReAct loop writes a thought, takes one action, reads the observation, appends all three to the same prompt. Nothing ever leaves. A failed search at step three still sits in context at step fifteen, competing with the objective for attention. Prompt tuning does not reach this. A larger context window does not fix it either. It slows the degradation and raises the per-step bill.

Plan-and-Act introduces asymmetric context. The planner reads the query and the initial page, then writes high-level steps. The executor reads the plan, the task, its own action history and the current HTML, emits one grounded action, then strips the HTML it no longer needs. Execution context stops growing with step count. A stale step, say a search for "library at CMU" returning nothing, gets rewritten by replanning to "libraries near CMU". The failure is replaced in the plan instead of accumulating in the trace.

Two costs the headline number hides

Replanning is token relocation, not token elimination. The planner's prompt carries current state plus previous plans plus actions taken, so context pruned out of the executor reappears there. LLM calls per step go from one to two. The authors flag this and propose letting the executor decide when a replan is needed, which they leave unsolved. Triggering on empty result sets and missing target elements covers most of it, with repeated actions catching the loop case, rather than firing on every step.

Plan granularity is load-bearing and it fails silently. "Input New York as the arrival city" works. "Input the arrival city" leaves the executor guessing which city. "Analyze the search results" is not a step at all; it hands the reasoning back to the executor, so you pay for two models and get one. A single click is too small and collapses back into ReAct at double the call volume.

The 20.60% run is the one to study

A planner finetuned without exposure to the target environment writes steps that read fine and match nothing on the page. The executor follows them anyway. No standard agent dashboard distinguishes a grounded plan from a fluent one. Task failure is observable. Plan groundedness is not. That is how a 16.37-point regression ships without anyone naming it. The same coverage lays out a staged modeling ladder: heuristic, then simplest model, then tuning within that model class, then deep learning, with each phase's best becoming the next phase's baseline. A plannerless loop is the Phase 1 heuristic. If a sophisticated planner cannot beat it, either planning adds nothing for that task or there is a pipeline bug.

Rollout order that respects the evidence

1. Measure. Context tokens per step and steps per task. Then repeated-action rate, the cheapest stale-step detector available today.
2. Prune observations. Derived fact in state, raw payload in logs. One reducer change. No planner and no new failure surface.
3. Gate the planner on an eval set built from local environment traces, with the plannerless loop as the permanent comparator.
4. Then conditional replanning. Always-replan is the reference implementation, not the production one.

An ungrounded planner is a sixteen-point regression you can ship by accident, because nothing in standard telemetry measures whether a plan matches the page.

Caveat worth holding: 53.94% is still near-coinflip. Harness engineering changed the slope, not the safety threshold. Anything customer-facing keeps a human confirmation step or a reversible-actions-only scope.

Twelve cents is the mechanism; the multiple is noise

An incremental operating margin near 12% is a statement about mix, not sentiment. Bandwidth, inference, and seat-based Zero Trust carrying support load are expensive to serve. No source reports an actual price change, and this is an inference from margin plus multiple rather than an announcement. Worth preparing for anyway, because the preparation is cheap and useful in either branch. Three levers exist: higher list prices, more aggressive metering through new billable dimensions or tightened free tiers, and pruning unprofitable products. All three arrive as changes to primitives already sitting in the critical path.

Exit cost is per primitive, not per vendor

The lock-in classification below is Clarity's engineering judgment, not a reported figure.

Leaving is the wrong response. Find the two rows that would actually hurt in a given stack and put a seam there and nowhere else. Abstracting everything costs the exact primitives that justified adopting the platform in the first place. A Workers deployment behind a generic "edge function" interface is a Lambda with worse cold starts.

Then run the numbers on the lines nobody controls. Model a 2x unit-cost scenario on your three largest usage dimensions, requests, storage, egress or seats, and check whether it breaks gross margin or only the budget approval. Agent traffic makes this urgent regardless of what the vendor decides. Request count per user task and tokens per request both grow once a loop starts calling a planner as well as an executor. The lines most exposed to repricing are the same lines the roadmap is about to multiply.

The differ nobody builds

The most copyable pattern in today's material came from a research vendor, not a platform team. Canary Data surfaced undisclosed McDonald's executive departures by diffing public leadership pages, catching an executive quietly removed from the site before any filing. Mechanically it is four parts: a scheduled fetch, a normalizer that strips navigation and timestamps, a structural diff, and an alert on change or deletion. Two days of work. Unglamorous, and genuinely well chosen. Pointed at pricing pages, deprecation notices, status-page history and SDK release notes for the top ten dependencies, it produces advance warning on breaking changes that no vendor announcement will give. Deprecations land in docs before they land in your inbox.

The corollary runs the other direction. Public surfaces are being snapshotted on someone else's schedule, so removing a pricing tier, a status incident or a page is itself a diffable event.

The switching cost you never estimated is the one your vendor's pricing team is counting on.