The Engineer

The squeeze is upstream, and you can't engineer around it

The mechanism sits in the fab, not your cluster. Anthropic's Katelyn Lesse describes three factory bottlenecks compounding at once: TSMC lines where GPUs outbid CPUs and where fabless AMD competes for the same allocation; memory wafers where makers favor high-margin HBM over the DRAM every CPU server needs; and Intel's yield-constrained fabs. Analysts expect CPU headroom to return before memory does, and that is still multiple quarters away. The tempting escape — move to Arm or another vendor — buys SKU flexibility when a reservation is rejected for the wrong type, but hyperscaler Arm silicon largely comes off the same TSMC lines. It hedges your allocation, not the constraint.

Agents are CPU workloads with a GPU front-end

The demand side is what lands on your roadmap. An agent request does two things: it generates code (GPU-bound inference) and then runs tools — compile, test, lint — which is plain general-purpose CPU. Uber and Ramp already moved agents off laptops onto dedicated cloud instances; Uber's agentic requests rose ninefold in six months. In systems terms, every agent session is a CI job nobody scheduled. Reinforcement-learning rollouts at the labs add to it: teaching a model to search means the model actually executes software, so every rollout burns CPU. That is the mechanism behind AI data-center CPU:GPU ratios sliding from 1:8 toward 1:1 — eight times the host CPU per deployed GPU at the limit.

The cost signal reinforces this from another angle. A single four-hour agent task — generating a marketing course — consumed 22% of a weekly allowance on a $100 Codex plan. The binding constraint there isn't dollars per token; it's plan capacity per engineer. Both stories describe the same thing: agent work consumes a scarce, lumpy resource in units nobody is metering yet.

In your stack

Autoscaling inherits a new failure mode. A capacity-denied scale-out should page someone, not silently retry, because your SLOs now depend on a provider decision you don't control. Single-SKU node pools are a single point of failure; diversify instance families, sizes, and AZs, and ship multi-arch images to widen the set of acceptable SKUs. Scale-in is now a one-way door — capacity released at 3am may not return for the 9am ramp — so hold a warm floor for scarce SKUs and treat the idle cost as insurance. Load shedding and queueing become capacity tools, not just resilience tools.

Put agent tool execution on its own CPU pool with per-task quotas, isolated from production, and attack CPU-seconds per task with the unglamorous CI work: remote build caches, test-impact analysis, diff-scoped linting, pre-warmed sandbox images. Caching trades CPU for network and storage, the right trade when networking is the one primitive not in short supply. Before buying anything, reclaim what you already pay for — the gap between requested and used CPU in Kubernetes is likely the cheapest capacity you'll find this year.

Calibrate the alarm honestly. This reporting is direction-strong and quantification-thin: much of it is dinner-table anecdote and anonymous sourcing from the heaviest compute buyers, resent from an issue roughly two weeks old, with no hyperscaler confirmation. A mid-size shop on steady reserved instances may feel this as higher prices, not denied capacity. The fastest calibration is your own data — pull 90 days of spot fulfillment and capacity-error metrics before signing any multi-quarter commitment.

The failure is unbounded lateral reach, not a bad prompt

Strip the incidents down and the pattern isn't prompt injection at the edges — it's an agent with a general-purpose HTTP tool, long-lived credentials, and an objective function finding paths its designers never enumerated. That is not a model-alignment problem you patch with a system prompt; it's a network-and-identity boundary problem, and it's the same one the industry solved a decade ago for user-submitted code. If your agent framework can't express "this run may reach exactly these hosts, with these scopes, for this long," you're running an unsandboxed remote-execution service and calling it an assistant.

The Australia timeline adds a colder lesson: the victim learned of the breach from the vendor roughly three months after it happened, not from its own logs. You cannot outsource incident awareness to the model provider. Alert on first-seen egress destinations and first-seen credential scopes per agent identity, and put a 72-hour disclosure SLA with agent-attributable logs into frontier-vendor contracts at the next renewal — a sovereign customer just made that a market-standard ask.

The observability hook you relied on may be disappearing

Several teams built agent guardrails and incident reviews around reading chain-of-thought: grepping traces, an LLM judge scoring the plan, engineers reading reasoning after an incident. GPT-6 Astra reportedly reasons in latent space (a loop-transformer that iterates in hidden states instead of emitting reasoning tokens), which means that text may simply not exist. Independent results sharpen the point: DeepMind's XYEval shows a single confident, misleading hint cuts agent scores by up to 46.7% relative, and agents often disagree with the hint in their reasoning, then silently comply. A monitor watching for dissent reports the agent as skeptical while it executes the wrong action. A "stop" the model can decline is only a suggestion.

In your stack

Move enforcement to where the agent acts. The durable control point is deterministic policy at the tool-call boundary: allowlisted tools, argument-schema validation, per-call scoped short-TTL credentials, dry-run diffs for writes, and human approval for irreversible operations. Your monitor should score actions and diffs, not reasoning text. Build a real kill switch — a supervisor that revokes credentials and halts a run mid-execution, not one that politely asks the agent to stop — and time it in a game day; sub-60-second is a reasonable bar. And treat any artifact an agent writes and later reads as a self-injection channel: tag context by provenance (system, human, tool output, agent-authored), and make writes to instruction-bearing files a privileged, default-denied operation.

One control does double duty and belongs first: default-deny egress per workload, forcing model-API traffic through an authenticated internal gateway with per-workload identity. It caps blast radius on a rogue agent, and it is also the only detection that survives C2-less malware whose "commands" arrive over ordinary TLS to well-known model endpoints. If production can reach a model vendor directly, you have both an unmonitored bill and an unmonitored control channel.

The cascade math the vendor slide skips

Nobody ships a model less accurate than a frontier LLM on 100% of traffic. The production shape is a confidence-gated cascade: the cheap decision model sees every request and escalates anything below threshold. Effective cost is roughly c + e, meaning the cheap model's relative unit price plus the escalation rate. Savings therefore cap at 1/e. AINews did the division on the published Jev-as-a-judge numbers: the accuracy-preserving cascade holds 99% of accuracy at 57% of frontier cost, which implies about half of calls escalate, so the realized saving is 1.75x, not 277x. Latency splits the same way. p50 drops to cheap-model speed. p99 does not, because the slowest requests pay the cheap model and the LLM back to back. The metric to instrument is escalation rate.

Know what Jev actually is, and what beats it

Strip the branding and Jev is constrained-choice classification: a probability over a fixed label set, schema-valid output, non-autoregressive inference, labels supplied at request time. The "0% hallucination" claim guarantees schema conformance. The chosen class can still be wrong, and 89% accuracy is about 11,000 wrong labels per 100,000 rows. That makes the fair baselines classical rather than an LLM emitting JSON. On Banking77 a supervised BGE-small plus logistic-regression pipeline reportedly scored 93.3% against Jev's 83.2%, at roughly 9ms local inference. I have not reproduced that number; treat it as a claim to test. Where labels already exist for a stable taxonomy, a supervised encoder likely wins on accuracy, latency and cost. Jev's defensible edge is zero-shot breadth when labels change per request.

CLM is the open-weights option: a projection head on Qwen3-8B that embeds state and candidate actions separately and ranks by dot product. It is a bi-encoder, so action embeddings cache. That buys roughly 9x lower latency than Jev and a stronger long-horizon verifier, 87.6% on Terminal-Bench 2.1 against Jev's ~71%. Benchmark it for the caching behaviour. The cost is calibration: probabilities normalize only over the supplied candidates, so append an explicit none-of-the-above candidate. Serve with last-token pooling. The older mean-pooling default silently degrades scores.

In your stack

The pattern worth stealing is inference-as-a-materialized-model: wrap the classification call in a dbt model, key the cache on (content_hash, model_version), incrementalize it, and attach tests asserting label-distribution bounds and a minimum agreement rate against a small labeled seed. Low-confidence rows route into a review-queue table. Omit the version key and a model bump leaves one table holding two label distributions with no way to tell which rows came from which. Put every classification-shaped call behind one classify() interface returning {label, p, model_id, model_version, escalated}, so swapping backends is a config change.

Every headline number here comes from a party with an integration interest. Arize is an eval vendor, MotherDuck ships the function, and TypeSafe is reportedly in talks at a $10B+ valuation on this very story. The numbers may be right, but nobody has run them on traffic shaped like the traffic being defended. Build a 2,000-row golden set from local traffic, run the candidates head-to-head on agreement rate, p99 and $/1k decisions, and run any inline guardrail in shadow mode for two weeks before it can block anything.