The Scientist

This is a formulation bug, not a feature bug

The temptation is to fix a censored label with better features. It does not work, because the defect is in the problem statement. A model consuming KEV's ransomware-use field as a binary is sampling from a population where an unknown share of the negative class are future positives, then being evaluated against a label table that caught up after the prediction window. That is positive-unlabeled contamination and point-in-time leakage in the same pipeline, and they push error in the same direction: offline precision looks strong, production precision cannot match it, and nobody can explain the gap.

The two defensible reformulations are well established. Treat it as time-to-event with explicit right censoring, where a CVE not yet confirmed is censored rather than negative. Or treat it as PU learning with an estimated label frequency. Either way, every backtest joins against a point-in-time snapshot of the label table, not today's. If you buy a vendor exploitation score, the question worth asking is which of the two they do; the concrete finding here is that the underlying field changes without notification, so a vendor that re-scores history against current labels has the same defect you do.

Rank your label sources by whether you can train on them

A remediation prior worth stealing

Shadowserver counted roughly 11,000 unpatched WatchGuard Fireboxes in December 2025 and about 9,000 nine months later — near 18% remediation on a KEV-listed, CVSS 9.3, ransomware-exploited flaw. Plex still showed 36,000-plus exposed servers a week after its advisory. If your exposure or time-to-remediation model assumes exponential decay, it is badly optimistic; the empirical hazard for edge appliances looks closer to flat. No denominator is published for either count, so use them as relative trend only.

The regime break sitting in the same dataset

Microsoft's monthly CVE volume reached 974 in September against 622 in July, roughly 5x May and 10x February. That is a level shift, not a noisy month, and a rolling-window baseline will smear it into a gentle trend and stop alerting on anything real for a year. Declare a changepoint or reset the window.

The headline count also fails to reconcile across outlets: 974 per Dark Reading and The Register, 972 with 113 critical per CrowdStrike, 966 per BleepingComputer — and CrowdStrike's tidy 45% privilege-elevation / 26% RCE / 18% information-disclosure split rides on its own denominator. ZDI separately counts 2,760 Microsoft CVEs year to date, more than double the prior record with a quarter left. The popular explanation, that AI-assisted code analysis drove the surge, has no ablation separating it from CVE-assignment policy, bounty scaling, or third-party bundling. Ingest the machine-readable vendor feed and persist your denominators, or your severity-mix charts will drift for editorial reasons alone.

A censored label with silent updates is not a data-quality ticket. It means your backtest measured a future you had already been told about.

The only instrumentation that sees it

Aggregate metrics will not move on the Brevo cohort, and neither will input drift tests: headers are well formed, domains are aged, reputation scores are high. The detection instrument is slice-level, label-conditional evaluation — precision and recall reported per sender-authentication stratum and per sender-domain reputation tier, never pooled. Report the authenticated-sender slice as its own line and the AUC delta becomes visible in an afternoon.

Then replace what died. The features that survive an account takeover at the provider are tenant-relative, not artifact-relative: template similarity against that tenant's own prior legitimate sends, list-wide URL novelty, and send-time deviation from the tenant's historical cadence. All three describe the sender's behaviour rather than the message's credentials, which is exactly the axis the compromise did not fake.

When the document is genuine, forgery is the wrong label

The IDScan exposure tied full names to driver's-license and government ID numbers within a dark-web dataset of 153M-plus driver's licenses, 10M ID cards, 3M travel documents, 579,000 medical cards, and 170M-plus individuals across the US and Canada — verified by Brian Krebs pulling his own record. The documents are real. A document-authenticity classifier is therefore optimizing a label that no longer separates the classes: the adversary holds an authentic scan, and the cost of obtaining one collapsed.

This is a label-definition problem, so the fix is a label change plus an eval slice. Build a purpose-built genuine-but-stolen cohort, ablate knowledge-based identity features at a fixed false-positive rate to quantify what they still contribute, and shadow-score device, behavioural, and graph replacements before the degradation arrives so you can distinguish it from noise. AdaptHealth's 4,115,802-person breach points the same way: entry came through social engineering of a third-party contractor's session, not a cracked password, which makes session-trajectory anomalies — device fingerprint change mid-session, impossible travel, unusual API call ordering — the right abstraction over credential-centric features.

A high-weight feature just died quietly

Anthropic's misuse report states plainly that 'sophistication' can no longer be used as an attribution signal: a Russian-aligned autonomous toolkit and undergraduates running an exploit foundry now produce indistinguishable technique complexity. Any model scoring threat, abuse, fraud severity, or account risk on proxies for actor capability — attack complexity, tooling quality, operational polish — just lost discriminative power while those features' distributions held steady.

The recalibration ticket is auditable: PSI or KS on those features across the last two quarters, feature-importance deltas, and a reliability curve. Miscalibration here has a predictable shape — systematic under-scoring of the low-resource segment — and ECE plus Brier make the before-and-after defensible to a reviewer.

Where the sources stop supporting you

All three reports agree on direction and share the same evidentiary hole. The misuse report is conditioned on detection: every case is one the vendor's own classifiers caught, and actors running open weights locally emit no API telemetry, so they are absent from the sampling frame entirely. There is no denominator, no false-positive rate on legitimate biology or security queries, and no counterfactual uplift. That is enough to change a feature set. It is not enough to change a rate in a risk model, and moving a number on this evidence is how a vendor's visibility becomes your prior. Separately, 42% of LG smart TV apps previously carried a residential proxy SDK across roughly 216M units, so re-measure slice AUC on residential-ASN cohorts before trusting IP-reputation features either.

Concept drift with a frozen input distribution is the one failure your drift program cannot detect by construction, because it monitors the data and the definition is what moved.

Where the entitlement predicate sits decides two numbers at once

The security framing of this — agents authenticating as infrastructure rather than as users — is a confused-deputy bug, named decades ago and now sitting in the retriever and the tool router. The measurement consequence is the part nobody writes up. A post-retrieval filter grades your retriever on documents the serving path will discard, so recall@k and nDCG are silently inflated, and the offline numbers you used to pick an embedding model were computed against a corpus that does not exist at request time.

Expect the honest number to come in lower per entitlement cohort. That drop is not a regression; it is the first accurate measurement you have taken.

Adaptive retrieval is selective prediction, and you can build it

Databricks positioned Adaptive Instructed-Retriever as a cost and latency play — deciding per query whether extra search hops are needed — with no baseline retriever, no dataset, no cost-per-query delta, no latency percentiles, and no ablation isolating the routing policy from retriever quality. Efficiency claims without a baseline are unfalsifiable by construction, so the useful move is to name the method rather than evaluate the product.

Methodologically this is selective prediction: a cheap path, an expensive path, and a calibrated policy routing each query to the cheapest path that clears a quality floor. That literature is mature, the evaluation instrument is a risk-coverage curve, and the router can be built from signals you already log — retriever top-1 score margin, score entropy across top-k, generator logprob, or a small classifier over those features trained against adjudicated correctness labels.

The failure mode to instrument is specific. An under-calibrated router does not error out; it quietly sends the hardest few percent of queries down the cheap path and returns confident, incomplete answers. Track expected calibration error on the routing signal, not aggregate accuracy, and freeze a stratified eval set of 300-500 production queries labelled by required hop count with cost-per-correct-answer and p50/p95 latency alongside faithfulness. That single artifact converts every future vendor efficiency claim into a one-afternoon test.

The gate that has no zero

Prompt injection and model-artifact supply chain remain the top entries on the most-critical-LLM-vulnerabilities list, which after years of guardrail products is itself the finding: no defense drives attack success rate to zero. Treat injection resistance as a robustness metric with a number, not a certification. Cover direct injection, indirect injection through retrieved documents, and injection through tool outputs, at 200-plus cases per surface so a five-point movement is distinguishable from sampling noise, and hold part of the attack corpus back so guardrail tuning does not overfit the attacks you already saw. Severity lives at the permission boundary: a 12% success rate against a read-only tool is an annoyance, and the same rate against a tool with production write access is an incident.

If entitlement filtering happens after retrieval instead of during it, you have a data leak and a fake recall number — and only one of them ever shows up in an incident report.