The Scientist

Read this month as a labeled dataset

Score the September batch as a classification problem and the triage failure becomes measurable: 973 candidates, 2 confirmed-exploited positives. Base rate 0.21%. Both positives — CVE-2026-81963, a Windows Update Stack link-following bug reaching SYSTEM, and CVE-2026-85880, an ALPC heap overflow escaping the AppContainer sandbox with no user interaction — are rated only Important at CVSS 7.8. The 113 patches marked Critical caught neither. Severity is a weak feature and a worse gate. KEV membership is the only signal in this batch with usable precision, and its failure mode is latency: it fires after exploitation is observed.

The feed also carries label noise worth modeling explicitly. SANS's diary says CVE-2026-85880 was not in KEV at time of writing; the same issue's CVE table lists it as KEV since 2026-09-08. That is publication skew, and it leaks into any backtest that does not snapshot the feed with as-of dates.

The registry row monitors cannot see

Of the three KEV rows, the artifact registry is the underrated one. An attacker with write access to JFrog Artifactory does not need to exfiltrate anything. They substitute a checkpoint that clears the aggregate accuracy gate and carries a trigger. No drift monitor catches that. Neither does a PSI check or a rolling AUC, because the input distribution never moved. The thing aggregate accuracy does not tell you is whether the checkpoint evaluated is the checkpoint served. Hash lineage from training run to serving artifact is the only control that answers it, at roughly a day of implementation.

The other two rows map onto owned assets just as cleanly. LiteLLM held every provider API key it has ever proxied, so the exposure is free inference billed to your budget. The patch is incomplete without key rotation and a 30-day reconciliation of token spend against expected workload volume. Kestra gives command execution next to whatever warehouse, S3 and feature-store credentials the DAGs carry.

The unauthenticated services next door

The same cycle lists Aim 3.29.1 at CVSS 9.8 (experiment tracking, usually deployed with no auth and full run history), a FastChat SSRF at 9.4 that reaches instance metadata and therefore cloud account credentials from a serving box, and Elasticsearch 9.5.0 deserialization RCE at 8.8 on the retrieval tier. IBM Instana CVE-2026-19274 (9.6) lets an authenticated Kubernetes tenant hijack another tenant's cluster-level RBAC. On a shared multi-tenant GPU cluster, one compromised notebook namespace reaches cluster scope.

The agent tool layer clusters as well: excel-mcp-server 0.1.8, TEN Framework 0.11.71, Cua computer-server and AutoAgent root RCE all at 9.8, plus OpenAI Codex Desktop CVE-2026-19593 (9.8), which executes code as the signed-in user via Git metadata and workspace settings. Cloning an untrusted model or dataset repo into an agent workspace is an execution primitive now, and research teams clone untrusted repos routinely.

Where the vendor framing thins

Keep one counter-example for the ranker. MOOS-IvP CVE-2026-85438 is CVSS 9.8 with 90 GitHub stars, Exploitability 0%, flagged "NO CUSTOMER ACTION REQUIRED". Maximum severity, near-zero reachability. That single row is the argument for adding reachability and popularity features, and also for discounting Wiz's 13-tier prioritization model published in the same cycle. Telemetry from over 50% of the Fortune 100 is still one vendor's install base, and 13 ordinal tiers with no published base rates, no stated fitting procedure and no holdout validation is unfalsifiable as shipped. Ask for the lift chart and the calibration curve before adopting the ordering.

A severity label predicted exploitation at zero recall this month. Observed exploitation predicted it at near-perfect precision, and shipped with no score attached.

The mechanism: three clocks on one object

An agent flow instance holds state with two different lifetimes. Application state is the current message and the domain fields. Execution state is which nodes finished and what they returned. Reuse the instance and the second call finds its nodes already marked complete, so the graph short-circuits and replays the cached output. The intuitive fix degrades things in two separate ways: class-level @persist() snapshots after every method, so the newest row can encode the middle of a run rather than its terminal state, and kickoff(inputs={'id': ...}) is read silently as a checkpoint restore, dragging the previous turn's message back in.

The framework-agnostic repair separates three clocks and asserts an invariant on each. Run state dies with the run, with the execution record empty at turn start. Conversation history spans turns within a session in an external store keyed by session ID (len(history) == 2 × turn_index). Memory outlives the session through explicit, tenant-scoped writes. Conflating the three on one object is the root cause, and it reproduces on CrewAI, on a LangGraph checkpointer, and on a hand-rolled FastAPI service.

The trace corpus is biased against the sessions worth mining

The quieter finding is about data quality. defer_trace_finalization defaults to True, so any batch never explicitly finalized is never exported. Missing traces are therefore not missing at random. They cluster on timeouts, dropped connections and abandoned sessions, which are the sessions that carry the eval cases and the error taxonomy. The opened-versus-exported trace ratio behaves as a first-class data-quality metric. Teams that wrap the turn loop in try/finally and reap idle sessions recover the missing batches.

Where the sources converge, and where they diverge

Four independent threads in the reviewed source material all argue the scored object is wrong, and they disagree usefully about which direction to fix first.

On statistical power the reporting is unusually concrete. At 85% accuracy with n=50, the standard error is about 5pp and the 95% interval runs roughly 75–95%, so two models separated by a genuine 4pp are indistinguishable. The typical bake-off is a contaminated measurement of a quantity it has no power to resolve. Google DeepMind's double-blind frontier protocol, where neither side sees the other's eval assets, is being positioned as the reference standard; it governs who holds the eval assets during procurement. Paired designs on identical items recover most of the power otherwise bought with three to five times more labels.

Arena's own concession sharpens the promotion rule: under its stated assumptions, overlapping confidence intervals mean two models are statistically indistinguishable. Conformal selective prediction is the missing piece, and it is installable on a team's own judge inside a quarter. Its AutoEval issues Day-1 scores before human votes accumulate, guarded only by weekly recalibration, and asked directly when AutoEval should abstain, Angelopoulos did not answer.

The security thread adds the layer the others skip. The damaging failures cross from generation into retrieval, identity, tool execution and transactions, so a text-only eval loop physically cannot measure them. A 30-prompt red-team session in a Slack thread carries no bounding power at all. Sizing is the thing the accuracy number doesn't tell you: at a 10% baseline attack-success rate, n ≈ 400 prompts gives a ±3pp interval, and 500 prompts with zero successes bounds true ASR near 0.6% by the rule of three.

If your agent evals score turns instead of sessions, you can ship a bot that answers turn one five times in a row and every dashboard stays green.

The experiment harness breaks before the model does

Agent traffic never renders the front end. Any experiment assigned and logged client-side therefore under-exposes treatment for agent-heavy accounts. That produces dilution, non-compliance, and a sample ratio mismatch that looks like an instrumentation bug and is actually a design flaw. Move assignment and exposure logging to the API and agent-gateway layer, and split SRM alerting by channel. Otherwise the lift being measured belongs to the subset of users who still click.

Drift monitoring fails second, and quietly. Global PSI averages channel migration away. Bucket accounts by agent-share quartile, then check per-feature PSI or KS plus calibration, meaning Brier score and reliability curves, inside each bucket. Channel migration degrades calibration well before it dents AUC. Miscalibration in the top quartile is what fires false churn alarms on the best-paying accounts. In session decay, an account that migrated channels and an account that is genuinely dying are indistinguishable. One wants an upsell, the other a CS intervention.

Audit the evidence before importing the narrative

Every quantitative claim here comes from a CEO addressing investors at a Goldman Sachs conference, and the gaps are consistent. HubSpot's assertion that external agent usage "drives better engagement and better retention" is observational, has no control group, and carries an obvious self-selection confound: agent adopters are larger, more technical, already committed. Salesforce's Bill Patterson saying agent access is "increasing the amount of work happening through our CRM" is a volume metric, not a value signal. One agent answer fans out into dozens of chatty retrieval calls.

To earn the retention claim internally, use a propensity-matched difference-in-differences design, matching on account size, tenure, seat count and prior usage, then validating parallel pre-trends. A staged agent-enablement rollout is better. The 350,000 figure arrives with no denominator, no time window, and no definition of "access," so normalize it against the internal MAU base and compute agent share per account per week before it lands in a planning doc.

The revealed position worth copying

Figma is instructive because its stated and revealed positions differ. Dylan Field publicly defends native-tool primacy. Figma has nonetheless opened Claude and ChatGPT access down to frames, components and variants, the structural primitives rather than rendered output, and charges partly on usage, including chatbot-driven usage. The early consumption returns were good enough that he softened to "I don't think it's entirely bearish." The transferable finding is that metered agent access is a viable revenue line, and the artifact that enables it is the one telemetry needs anyway.

Build the per-call ledger once: account, human principal, agent client_id, tool_call, tokens, rows_read, latency, with idempotency keys. It serves consumption billing, cost allocation against the model router, and channel attribution at the same time. Bulk-read monitoring on that same table is the early warning for data-extraction churn.

The lineage layer nobody instrumented

The same week, agents were wired into governed analytics. Alteryx shipped an MCP server plus ChatGPT integration exposing governed datasets and workflows, Writer launched Enterprise Brain with team-level Agent Memory across Slack, Teams, Zoom and Meet, and Claude reached GA inside Excel, PowerPoint and Word. One architecture in all three: a retrieval layer that row-level security has never been red-teamed against. Enforce per-tool dataset allowlists, verify row and column security holds at the agent boundary, and log every tool call. The append-only, hash-chained inference log compliance wants is also the best production-truth eval corpus and drift feed available. Build it once and stop treating audit as a tax.

Session decay and channel migration are the same signal in your dashboards. One of them means a customer is leaving and the other means they finally started using the product properly.