Prompt Etiquette Is Not a Permission Boundary
Attacker capability has compounded on the same curve as the tooling, while the controls holding most agent deployments together remain sentences a model can be talked out of.
The attacker curve moved with the tooling
Security capability took a discrete step, not an incremental one. Anthropic released a public cryptanalysis tool, per Simplifying AI, and Alex Stamos frames open models as an irreversible trade-off analogous to encryption: the capability that lets defenders find bugs also lowers the threshold for attacking. OpenAI extended its Daybreak initiative with a model scoped to authorized security work, and dates its own urgency to a narrowing window before offensive AI is deployable at scale. A reasonable skeptic would point out that both framings are self-interested. The skeptic is correct. Neither framing is wrong because of it.
The timing is the argument, not the tooling. Credential surface and automation surface are expanding inside most organisations at the same time adversary capability compounds. That combination belongs on the immediate agenda rather than the roadmap.
The one governance model with a track record went the other way
The most scrutinized codebase on earth accepted AI tooling as permanent and still requires a named human to triage and sign off every AI-assisted contribution. That is now the reference model, whether or not anyone volunteered for it. What enterprise customers, auditors and eventually regulators will ask for is not evidence of a tooling choice but evidence of a provenance trail: what generated the change, which model, which human accepted it. Teams instrumenting that today answer the question in a query. Everyone else answers with a six-week archaeology project across merged commits.
Three control types, one of which fails safely
| Control | What it actually stops | How it fails |
|---|---|---|
| Prompt instruction ("wait for approval") | An agent that reads the instruction the way you intended | One injection, one model update, one ambiguous message |
| Enforced permission (vaulted identity, least privilege, branch protection) | Any write the credential does not authorize, regardless of intent | Only through misconfiguration you can audit |
| Stop mechanism plus tested rollback | Damage duration once something already went wrong | Silently, if it is never exercised |
Optiver ships hundreds of production changes a day behind a risk layer that can halt strategies. Knight Capital lost $440M to a single bug. The tradeoff is explicit in both cases: throughput is cheap and reach is expensive. The metric that matters is not how much agents ship, but what an agent can reach and what stops it mid-flight.
The gate precedes the capability
The evidence on autonomy is unflattering and consistent across sources. A survey of roughly 336 papers cited by Simplifying AI finds GUI agents systematically lacking error recovery, safety checks and auditability. Internal correctness probes shift with extraction method, which rules them out as a substitute for tests. Where grounding was added, the numbers move sharply: diagnosis grounded in a service map beat rival AI methods by more than 25 percentage points on first-guess root cause. Grounded and gated works where autonomous does not, so the evaluation layer has to arrive before more capability rather than after it.
Four independent reads converge on authority, not capability, as the binding constraint. They diverge on the remedy. Platform vendors sell more tooling, the kernel's answer was a signature, and the trading floor's answer was a stop button. Two of those three require no purchase order.
A guardrail that lives in a prompt is a request. What an agent actually holds is a credential.
What to do
Convert every prompt-level guardrail on a production-connected agent into an enforced permission this week — vaulted identities with least-privilege scopes, branch protection, and a named human sign-off on merge.
Prohibit remote-script installs of unsanctioned agent frameworks and inventory every agent identity holding write access to code, deploy or storage systems by month end.
Require an automated stop mechanism, a tested rollback, and a passing evaluation gate on every path where agent-generated change can reach production, reported to you by quarter-end.