The Scientist

Price the miss before tuning the threshold

Start with the survival curve. Truffle Security re-verified 10,616 AWS keys leaked publicly between August 2022 and August 2026. 88% still authenticate. Of the roughly 9,342 live keys, 768, about 8.2%, carry full administrative control, and the median compromised key had gone roughly five years without rotation. Conditional on a leak, the hazard of a credential expiring on its own is effectively zero. The curve is flat, which leaves time-to-detection as the only controllable variable.

Put those quantities into an objective and the F1-maximising threshold stops being defensible for a secret detector. F1 prices a false positive and a false negative identically. Here the false negative is a live credential with probability 0.88, roughly one in twelve of them admin. The false positive costs analyst-minutes. The defensible operating point is maximum recall at a fixed triage budget, the budget set by what a false alarm actually costs in human time. Re-deriving that threshold for a detector already in production is about two days of work, and among the highest-leverage two days available.

What GitHub supplied, and what it left out

GitHub's four principles are unglamorous, which is the recommendation: decide the product question before picking the metric; run evaluation as a continuously executed integration test rather than a pre-launch gate; keep offline conditions close to production; use targeted error analysis on repeat failure patterns instead of watching an aggregate score. The thing this doesn't tell you is which practice moved anything: no precision or recall deltas, no baseline, no ablation. Credible playbook from a team running a real high-stakes surface. Not evidence.

The estimator defect both reports share

Techpresso's model-scanner comparison hits the same problem from another angle. One tool returned verdicts on all 135 test files; another judged roughly half, and accuracy was reported only over the rows where a verdict was emitted. That is a selection-bias confound: abstentions get laundered out of the headline number, so the score describes a self-selected easy subset. Coverage-adjusted is the right figure, accuracy multiplied by verdict rate, abstentions counted as misses. Most LLM-as-judge harnesses share the defect, wherever an "unclear" verdict is dropped before the rate is computed. Two independent reports, one estimator bug.

An accuracy figure computed only over the rows a detector agreed to answer describes the easy inputs, not production data.

The other half of this is in-house keys

Those base rates describe internal stacks too. Static credentials sit in notebook environments, dbt profiles, Airflow and Dagster connections, MLflow, and feature-store service accounts, and nothing in a data platform forces rotation. Short-lived workload identity is the structural fix; budget alerts and a cost-anomaly monitor cover the window before it lands. One honest limit on transferability: Truffle's population is publicly exposed keys, so 0.88 is conditional on public exposure and should not be applied unchanged to an internal key that never left the network boundary.

The breakeven, written out

The decision rule is a breakeven volume rather than a cost ratio: V ≈ L / (C_frontier − C_finetuned), with L the total labeling plus training spend and the C terms per-token inference costs. A 14x inference reduction leaves a denominator of 13/14 of the frontier price, which puts the threshold near 1.08 × (L / C_frontier) tokens. Practically, the fine-tune has to absorb about as many tokens as the label budget would have bought at frontier prices, plus roughly 8%, before any saving is real. High-volume classification, routing and extraction clear that bar without effort. Long-tail open-ended generation almost never does.

The underlying result is what the literature would predict: information filtering is a supervised classification problem wearing an LLM costume, and supervised fine-tuning on in-domain expert labels wins narrow, label-rich tasks. The ~30% error reduction at one-fourteenth cost, reported through Exponential View from Bridgewater's work with Thinking Machines, arrives with no ablation separating base-model contribution from label quality, no sample size, no task definition and no confidence intervals. The expert-labeled dataset is doing the work here; the base checkpoint is interchangeable. Thomson Reuters shipping its first in-house model on Qwen explicitly to cut costs has the same shape.

Where the spend actually sits

The corroborating evidence comes from invoices rather than benchmarks. Ramp's index across roughly 70,000 companies puts Opus 4.8 at 28% of Anthropic spend in July 2026, while Anthropic's annualised revenue climbed from $47B in May to $65B in July and OpenAI passed $40B; the FT reads that as the best model struggling for users while cheaper tools thrive. Both sources point at routing rather than the frontier checkpoint as the place the cost curve gets set.

They diverge on how hard to lean on it. Ramp measures observable card spend, skewed toward startups and SMBs, and probably misses committed enterprise contracts, which makes 28% a floor on flagship share rather than a measurement of it. Vercel's 62% open-weight token share is a single-day record drawn from a deployment mix weighted toward edge and hobbyist workloads, so it functions as a maximum statistic. The 28% to 62% slope over two months is credible as a trend whose level will not port to enterprise inference.

What to change in the pipeline

Run a three-arm bake-off on the single highest-token-volume narrow task in the stack: current frontier API, a fine-tuned 27b-class open model, and a GLM 5.3 distillation once one exists. Report paired error rates with bootstrap confidence intervals on a frozen private split, and ship only when error parity and a five-fold blended cost reduction both clear. Budget the quarterly re-fit up front, because a 30% accuracy advantage inverts under distribution shift when nobody owns the drift monitor.

The underrated line item is evaluation throughput. A 27b model resident on developer hardware converts judging, synthetic data generation and per-commit regression sweeps into electricity, which is how a 500-sample ablation becomes a 10,000-sample one. One calibration note: Chad Syverson of Chicago Booth finds micro-level AI productivity gains with little aggregate effect — the warning label on any internal savings number reported without a control group.

The cheapest model in a stack is usually the one fine-tuned on labels you already own, and the label budget decides whether the inference saving survives.

Four hazards wearing one coat

A binary churn target collapses at least four generating processes into one event: voluntary disengagement, deliverability failure, identity or account change, and death. The hazard shapes differ, and so do the correct interventions. What breaks the model is observability: three of the four are partially inferable from behavioural features, the fourth never. That floors the error of any engagement-only churn model at the Bayes error, the irreducible error no model beats, set by the mortality rate in the population. More features and more capacity sit above that floor.

The shape is partly recoverable even when the cause is not: high tenure, a very low coefficient of variation on inter-event intervals, then abrupt truncation to zero. Genuine disengagement decays; this is a step function. A CUSUM or Bayesian changepoint score on the engagement series separates those two regimes far better than the 30/60/90-day recency bucket most pipelines use, which discards the shape by construction. The clean formulation is competing risks: cause-specific or Fine-Gray subdistribution hazards, death treated as a competing event rather than the event of interest, plus an external deceased-suppression source for the cases inference cannot reach.

Size it before you build it

On a 25,000-person adult professional list, plausible annual all-cause mortality of 0.2–0.8% implies roughly 50 to 200 deceased addresses per year accumulating in the lapsed pool. That range is an analyst estimate, not a reported measurement. It barely moves AUC, but avoidable incidents are not measured in AUC. The return here is suppression and a human-review gate above a tenure threshold, not modelling lift.

The label you cannot reconstruct later

Mintlify's self-reported analysis puts agent visits above 66% of traffic across more than 20,000 documentation sites; sponsored, methodology unstated, so not citable as a level, though the direction is not in doubt. Agent-mediated commerce is already live: a Beijing dumpling shop published an AI skill so customer agents can read the menu and join the queue. Session definitions, engagement features and randomisation units all assume human behaviour that agent traffic violates.

The test is cheap: re-run SRM checks, sample ratio mismatch, the diagnostic for unequal assignment between arms, with agent traffic excluded, and see whether any live experiment flips sign. Related contamination sits upstream of any social or sentiment feature. X reports 200,000 accounts tied to China, about 200 of them generating anti-data-centre and anti-energy-policy content with generative tooling, corroborated by OpenAI's own June report. Narrow-topic, high-coordination synthetic content survives generic bot filters.

The deadline is structural rather than external. An agent-versus-human flag has to be stamped at the edge, because it cannot be reconstructed from events already stored. Every week without it is a week of readouts that cannot be re-segmented later.

A churn model files a lapsed reader and a dead one as the same event. Model capacity does not separate them.

Reading the mortality case as proof that machines measure while humans interpret is emotionally satisfying and technically incomplete. The defect is a mis-specified target variable with no external ground truth for a competing event, solvable with known tooling: suppression-file ingestion, competing-risks estimation, changepoint detection, and a review gate. The limit is only irreducible while the target stays as-is.