Autonomy Ships With an Undo Button or It Doesn't Ship
An insurer, the U.S. Army and OpenAI's own red-team accident converge on the same shortlist of agent controls — and every item on it is expensive to retrofit once autonomy is distributed.
The price of a wrong answer is now quoted
An underwriter sat down and put a number on a hallucination. Testudo, a Lloyd's-backed carrier, is writing generative-AI liability policies with limits up to $10 million for annual premiums of $10,000 to $20,000. That is a rate on line of 0.1–0.2%, which is how you price a loss you believe is unlikely but not zero. The demand trigger in the reporting is narrow, and it sits exactly where most agent roadmaps point: financial institutions evaluating tools that interact with customers or execute transactions, not summarizers or copilots.
What makes an agent insurable is a short, unglamorous list: scoped permissions, deterministic fallbacks, audit trails, human escalation. That is the same list the Army published as procurement language. The compliance backlog and the differentiation backlog are now one backlog.
The Army wrote the security-review answers
CyberScoop's reporting on Project Griffin's IRON solicitation names six requirements. Federal requirements lead commercial security questionnaires by roughly two to four quarters, so this reads less like defense procurement and more like a 2027 enterprise checklist arriving early.
| Requirement | Product translation | Cost if retrofitted |
|---|---|---|
| Master kill switch | Global and per-tenant instant disable of autonomous action | Low if planned, high if autonomy is distributed |
| Undo capability | Reversible actions with a rollback log per agent decision | Very high — requires action-model redesign |
| Manual confidence thresholds | Customer-tunable autonomy: act, suggest, or escalate | Medium — usually hard-coded today |
| Minimized token usage | Cost-per-task metering, routing, caching | Low to instrument, medium to optimize |
Two details matter more than the list. The Army names Microsoft Defender as a Policy Enforcement Point that IRON agents must integrate with, and intends to field multiple vetted solutions. The enforcement layer is incumbent-owned. The orchestration layer above it is contestable. And minimizing token usage is a stated source-selection criterion, which means inference cost has left the margin spreadsheet and entered the evaluation matrix.
Why partial autonomy is worse than none
Michael Dalton's Black Hat framing, relayed by Ben Thompson: automate vulnerability discovery without automating patching and human engineers drown or be inundated. Substitute any AI feature that produces output a human then has to act on. Flagged records, suggested edits, detected anomalies, drafted replies. Each one is a backlog generator unless the same team owns the remediation path.
A detection-only AI feature without an owned remediation path doesn't add capability. It adds queue.
Thompson's underlying asymmetry explains why the offense side got there first. Automated attack has positive expected value: a failed exploit changes nothing, a successful one only has to work once. Automated defense has negative expected value: success preserves the status quo, and one bad patch breaks production. That is an incentive gap, not a capability gap. It is the same gap that separates a startup willing to close the loop from an incumbent keeping a human gate for organizational comfort.
The gate nobody scheduled
On Aug 21, 2026, Hyundai's Korean union struck for a full eight hours, its first in a decade, idling 40,000 workers across Ulsan, Jeonju and Asan. The demand is not wages. It is binding consent authority over any AI or humanoid deployment on the line, while Hyundai plans to run Boston Dynamics' 50kg-lift Atlas at its non-unionised Georgia plant as early as 2028. Where AI touches represented or frontline work, contract signature is no longer the last gate. The forcing function is smaller than it sounds: per-site opt-in, scope limits, an immutable audit log, and an exportable deployment report a worker representative can read. Those are the four things that move a pilot into production.
What to do
Run a containment review on every agentic surface this week: network egress allowlists, read-only filesystems, no shared writeable state across agent runs, and no package-manager internet access inside sandboxes.
Classify every human-in-the-loop gate in your AI features by the end of this sprint into irreversible (gate stays), reversible (replace with rollback plus audit trail), and organizational comfort (remove and instrument).
Spec undo and customer-tunable confidence thresholds as first-class features next planning cycle, then publish an agent-safety page mapped to the Army's six controls for Sales to use pre-emptively.