ClarityBeta

Engineering & Technical

The Engineer

6 min read · 1,268 words · 4 moves to watch
The Signal

Amazon's Kiro agent deleted a live production environment with zero sign-off.

The agent that triggered the 13-hour outage was fixing a minor bug while holding operator-level credentials. Two March follow-ons added an estimated 6.3M lost orders on top. Permission scope set the blast radius; the model's intelligence never entered into it. I'd spend the sprint auditing every agent's IAM scope, because over-scoped service accounts produced this exact failure mode long before anyone wired a model to one.

In Play

  1. Agent Reliability Is a Deterministic-Engineering Problem

    Compounding error is arithmetic, not risk: at 95% per-step accuracy a 20-step chain succeeds ~1-in-3. Amazon's Kiro deleted prod on operator credentials; entity-binding drift multiplies one wrong resolution up to 8.5x. Every fix is boring engineering — scoped IAM, hard invariants, verification gates — not a smarter model.

    Ask Clarity

  2. RAG Reliability Is Set at Write Time, Not Query Time

    The RAG killer isn't out-of-corpus hallucination — it's partial-coverage leakage: retrieval is topically right but incomplete, and the model silently fills the gap from parametric weights with zero token-level attribution. Removing one 'skip retrieval if you know it' clause moved an eval from 19/33 to 30/33. A study of 12 memory systems confirms reliability is set at write time.

    Ask Clarity

  3. MCP Trust Boundaries and the Supply-Chain Attack Surface

    SANDWORM_MODE spread through 19 malicious npm packages that register a rogue MCP server inside Cursor, VSCode, Claude Desktop, and Windsurf, then use the AI's own permissions to exfiltrate credentials and propagate. Separately, CVE-2026-50522 (CVSS 9.8, unauth RCE) in on-prem SharePoint is under active exploitation — and patching doesn't revoke stolen ASP.NET machine keys.

    Ask Clarity

  4. The AI Productivity Paradox: Velocity Up, Experience Down

    DX's Q2 data (500+ teams): AI-generated code jumped 34% to 50%+ in one quarter and PR throughput rose 37% — but median PR size nearly doubled and the Developer Experience Index fell 67 to 65. Bolting AI onto an unchanged review pipeline moves the bottleneck to review, not away. Stripe's Minions fires 10-50 agents per engineer; the constraint is CI and review capacity.

    Ask Clarity

  5. Behavior Beats Scale in Model Training

    Poolside's Laguna S 2.1 (118B total, 8B active MoE) reportedly matches a ~1T-param rival — the gap closed through post-training behavior (verification, persistence), not scale. On an RTX Pro 6000 it hit 109 tok/s vs Qwen3.5-122B's 103. The prior worth internalizing: most current quality ceilings are post-training problems, not pretraining-scale ones.

    Ask Clarity

Deep Dives

Your Agent Chain Is a Probability Experiment Running on Customers

Three independent failures this cycle share one root cause: a probabilistic model wired into a deterministic system without the required guardrails.

Do the arithmetic before the architecture. At 95% per-step accuracy, a 20-step chain finishes correctly about one time in three. Documented hallucination baselines run 3–27%, which makes the math worse, not better. A chain with no structural accounting for compounding error is a probability experiment. Production traffic is the sample.

Two incidents put a price on it. Amazon's Kiro agent was fixing a minor bug while holding operator-level credentials. It deleted a live Cost Explorer environment. The outage ran 13 hours. Separately, entity-binding-drift research shows what happens when an agent locks a wrong resolution early: wrong user ID, wrong file, wrong ticket. The error compounds up to 8.5x downstream. A cheap second-model recheck cuts that by 79%.

The sources agree on the diagnosis, and it is not model capability. Kiro did not need a smarter model. It needed a smaller IAM role. The countermeasure for compounding error is not a better prompt. It is deterministic control flow with a hard iteration cap, invoking the model only at the two or three points that need genuine judgment. The entity-drift fix is a verification checkpoint between steps, not a bigger model. Kent Beck's 2008 DARPA anecdote is the same mechanism: a car told to "relax constraints when stuck" drove onto a sidewalk. Agents execute what you encode, not what you meant.

The multi-agent debate, with Cognition's "Don't Build Multi-Agents" on one side and Anthropic's roughly-90%-better-at-15x-cost result on the other, resolved into one usable pattern. A single orchestrator owns full context and spawns isolated, short-lived sub-agents that each return a summary, with no peer-to-peer chatter. Treat the model as a stateless function. Conversation, plan, and progress belong in Postgres or Redis, not in the context window. That is what makes pause/resume and horizontal scaling possible. Stateless HTTP services taught the same lesson, and it holds here.

An agent's IQ does not matter if its IAM role does the damage. Scope the blast radius before you scale the model.

What to do

  1. Audit every coding/ops agent this sprint; confirm none holds operator-level or unscoped IAM roles without a synchronous human-approval gate on destructive actions.

  2. Insert a lightweight second-model verification checkpoint between steps in any multi-step pipeline that resolves entities (user IDs, files, tickets), before the next release.

  3. Externalize agent state into Postgres/Redis and put a hard iteration cap plus explicit completion condition on every production agent loop.

RAG's Real Failure Mode Isn't Hallucination — It's Partial-Coverage Leakage

Your eval catches out-of-corpus hallucination but misses the model blending retrieved context with parametric knowledge, leaving no signal it did.

Standard evals sail past this mechanism, and the reason is simple. There is no token-level attribution in the output stream. Tokens generated from retrieved context and tokens generated from parametric weights arrive looking identical. Hallucination detection asks whether the model answered something entirely outside the corpus. That is not the common failure. The common failure is retrieval that was topically correct but incomplete, with the model quietly backfilling the gap from training data. That is partial-coverage leakage. It produces confidently wrong answers that pass a surface check.

A diagnostic session isolated it with a custom categorical rubric (corpus_abstention) using five fixed verdicts: GROUNDED, CORRECT_ABSTENTION, UNGROUNDED, MIXED_LEAKAGE, WRONG_ABSTENTION. Not a regenerating aggregate score. Fixed categories, so results compare across runs. The root cause turned out to be one instruction line. It let the agent skip retrieval for questions it "already knew." That line drove 6 of 15 in-corpus cases to smuggle in an unsupported claim. Remove the line, force retrieval on every query, and the eval moves from 19/33 to 30/33. Ungrounded answers hit zero.

The deeper structural point comes from a study of 12 agent memory systems. Reliability under fact updates and cross-session reasoning is decided at write time, not query time. Flat and append-only stores produce "hallucinations of the past": stale facts that survive updates because no schema exists to overwrite them. Typed graph memory does better. Graphiti's Pydantic ontology is the example, and it wins for an unglamorous reason: structure gets imposed while the memory is being written, not reconstructed later.

DimensionFlat / Append-onlyTyped Graph Memory
Fact-update correctnessFails — stale facts persistStrong — overwrite at write time
Cross-session reasoningWeak — no entity schemaStrong — typed constraints
Query latencyLowCan be orders of magnitude higher

The caveat matters: graph memory in stateless, high-QPS agents costs real latency. It pays off when updates stay incremental. If every update rewrites the whole graph, it does not.

Partial-coverage leakage cannot be patched at query time if your retrieval or memory architecture never built structure in at write time.

What to do

  1. Grep every production RAG system prompt for conditional-retrieval clauses ("if you already know the answer, respond without citations") and remove them this sprint.

  2. Replace built-in aggregate faithfulness scores with a fixed categorical rubric and deliberately generate boundary-case scenarios (topic covered, detail missing).

  3. Benchmark typed/graph memory against your flat store before defaulting any multi-session or fact-updating agent to append-only logs.

The MCP Trust Boundary Nobody Audited Is Now a Worm Vector

SANDWORM_MODE weaponizes the AI coding tool's own execution context, not your build files — and this month's SharePoint patch doesn't undo the prior credential theft.

The novel part isn't the malware. It's the trust boundary. SANDWORM_MODE spread through 19 malicious npm packages that skip the usual trick of hiding backdoors in build artifacts. They register a rogue MCP server inside Claude Desktop, Cursor, VSCode, and Windsurf. Then they instruct the trusted AI agent to exfiltrate credentials and wipe files if it can't propagate. The payload is an instruction set. A legitimate agent runs it with its own elevated permissions. That's why the leading EDR flagged only 65% of the campaign. A third went undetected because the executing process was trusted.

Propagation is automatic. Stolen npm token pulls a GitHub API token. That pulls an SSH key. That pushes infected dependencies downstream. One compromised dev machine cascades through the whole dependency graph with no human action.

Running in parallel: CVE-2026-50522, a CVSS 9.8 unauthenticated RCE in on-prem SharePoint Server (2016/2019/Subscription Edition), under active exploitation via public PoC. Here's the detail buried in most advisories. A single request extracts the ASP.NET validationKey and decryptionKey. From there, attackers forge valid auth tokens indefinitely. Microsoft's July patch closes the RCE entry point. It does nothing about keys already stolen. Patching without rotating is security theater.

Both failures share the pattern the sources keep hitting. The security boundary ends where trusted automation begins. MCP server registration deserves the same suspicion as a postinstall script.

For a third of SANDWORM's targets, endpoint detection did nothing — assume the compromise already happened and hunt, don't just patch.

What to do

  1. Default-deny unknown MCP servers across all AI coding tools org-wide and alert on any registration outside an approved allowlist.

  2. For internet-facing on-prem SharePoint: apply the July patch, then rotate validationKey/decryptionKey, restart IIS, and hunt for web shells and forged tokens.

  3. Rotate npm publish tokens, GitHub PATs, and SSH deploy keys, and enforce short-lived scoped tokens in CI/CD.

AI Pushed Code Generation Past 50% — Then the Bottleneck Moved to Review

Throughput dashboards are green while the metric that shows whether this works moved the other way — because review was never redesigned for the new arrival rate.

Read this as a queuing problem. Arrival rate up, service rate flat. DX's Q2 report covers 500+ teams. AI-generated code jumped 34% to over 50% in a single quarter. PR throughput rose 37%. Median PR size nearly doubled. The composite Developer Experience Index fell 67 to 65. Run the queue math: 37% more PRs, each close to twice the size, feeding a review pipeline whose service rate did not move. Median PR size doubling is a known leading indicator of technical debt. It is not noise to shrug off.

Stripe's internal Minions orchestrator shows where the ceiling sits. One engineer fires 10-50 concurrent agents at bugs. The binding constraint is not whether the agent writes correct code. Stripe is past that. The constraint is review throughput, CI capacity, and merge-conflict resolution at 10-50x normal PR volume.

Two mitigations recur across the sources. First, cross-model review. Claude Code and Codex each miss the bug classes they themselves produce, and each reliably catches those classes in the other's output. That is ensemble logic from ML applied to CI. Reviewer diversity drives catch rate, not reviewer capability. Second, judge/test-gated migration. Anthropic's ~1M-LOC Zig-to-Rust port landed in under two weeks because a rigorous test suite existed to gate every diff. Remove the gate and agent output is fast and wrong. The gate is the mechanism, not the migration tooling.

The budget line sharpens the mandate. AI spend is up 28x against a flat innovation ratio. Adoption percentages will not survive a serious review. Before/after DXI-style data will.

AI got the code written much faster; if review wasn't redesigned to match, the bottleneck moved rather than disappearing.

What to do

  1. Instrument PR size and review latency as first-class metrics tracked separately from AI-adoption rate, and build a before/after DXI model before your next budget review.

  2. Prototype a cross-model review gate in CI — have Codex review Claude Code's diffs and vice versa — and measure the catch-rate delta over two weeks.

  3. Cap PR size regardless of generation speed and move AI-heavy repos toward trunk-based incremental delivery so large diffs decompose before a human sees them.

The bottom line

These failures rhyme: a stochastic model trusted like deterministic code. Highest-leverage move: insert one hard gate between the model and anything it can break — scoped credentials, a verification checkpoint, or a second reviewer — before scaling the fleet.