The Scientist

What the four numbers license, and what they don't

The ablation is four point estimates with no confidence intervals, no seeds, and no run counts. WebArena-Lite's standard split runs on the order of 165 tasks scored as binary success. Near a success rate of 0.5, single-arm standard error is about ±4 percentage points, so the unpaired gap between 43.63% for the grounded planner and 53.94% for planner plus dynamic replanning sits around two sigma. Suggestive, not decisive. Both arms run the same task set. A paired McNemar test, the right test when the same items are scored twice, would almost certainly be tighter and might clear p<0.05. It is not in the reporting as summarized. The estimate that survives scrutiny comfortably is the negative one: the naively finetuned planner at 20.60%, 16.37 points below running no planner at all.

Why a wrong plan passes human review

The naive planner was finetuned without ever seeing the target sites. Its steps read fine. A reviewer would approve them. They corresponded to nothing in the page's DOM, and the executor followed them faithfully anyway. That is the hardest bug class in an agent system: a plausible-looking artifact with no grounding, emitted by a component whose output is prose and whose consumer is another model.

A plan that reads well and grounds in nothing is worse than no plan at all — the artifact passes review and the executor obeys it.

The paper's plan-step contract is specific enough to encode as a deterministic check, which is why a linter is the cheapest item here:

• One unit of work per step. "Search for the product in the search box" qualifies. A single click is too small.
• No cognitive verbs. "Analyze the search results" is not a step. It pushes reasoning back onto the executor and dissolves the point of planning.
• Fully value-instantiated. "Input New York as the arrival city," never "input the arrival city." The vague version makes the executor guess which value belongs in the box.

The replanning gain is the expensive one

Dynamic replanning contributes the largest single increment, +10.31 points from 43.63% to 53.94%, by letting the planner see current state, prior plans, and actions already taken. The reported example rewrites "library at CMU" into "libraries near CMU." It also costs one planner call per executor step. The authors flag that cost as unresolved and propose executor-triggered replanning instead.

That is a measurable design. Fire a replan only on concrete failure signals, such as an empty result set, element-not-found, or no state change, and instrument replan-trigger precision: the fraction of replans that actually change the plan. If most fired replans are no-ops, most of the gain survives at a fraction of the token spend. That is an inference-cost win, not only a line item.

The rule both of today's threads land on

Today's forensic-modeling material argues the same operating rule from the other end of the stack: a new layer earns entry only by beating a mandatory baseline that then stays permanently in the harness. Here the baseline is the planner-free loop. There it is a naive rule the complex model must clear. Neither case defends against a crash. Both defend against output that looks right.

So instrument before rearchitecting. Log tokens per step, the share of context occupied by failed or stale observations, and step-level action success. If token growth outpaces step success past some N, the append-only diagnosis has reproduced on live traffic and the horizon limit is located. Short-horizon agents may never reach it, which makes refactoring on faith premature.

Ranking survives a prior shift. Calibration does not.

The mechanism has a name and a testable signature. Prior shift means the base rate of the label moves while the feature-to-label relationship stays where it was. Firms can behave identically while the probability that a given filing gets labeled "enforcement action" rises, purely because a unit with dedicated expertise, focus, and capacity starts opening cases. Rank-based metrics are invariant to that change. So AUC holds and monitoring stays green while the calibrated probability drifts by an unknown factor. AUC measures ordering, and ordering is not the bottleneck when the consumer of the score is a human reviewer or an automated threshold reading a calibrated probability.

The diagnostic is cheap. Fit a reliability curve on the most recent complete label vintage and compare it against a vintage two years older. Slope intact, intercept moved: that is a recalibration job, not a retraining job. The distinction saves a quarter of pointless feature work.

Every recent negative is censored, not clean

Enforcement labels arrive years after the conduct they describe. Every recent-period row labeled 0 is therefore right-censored. The case may well exist and simply not have been filed yet. Training on those zeros teaches the model that the present is safe, which is backwards during an enforcement expansion, and the present is the period users care about most. The repair is a framing change rather than an architecture change: convert binary labels to time-to-event labels with an explicit enforcement-intensity covariate, and refit the calibration layer on rolling quarterly windows instead of once at training time.

Do the precision arithmetic before anyone sees a flag

Assume a 3% annual base rate for material restatement in a large-cap universe. A model at 90% sensitivity and 80% specificity produces 2.7% true positives and 19.4% false positives.

That table is the entire conversation. No architecture choice rescues you from the base rate. A much tighter operating threshold and a corroboration requirement do. Publish the precision-at-base-rate curve next to the flag, because the consumer of a rare-event score cannot infer it and will default to something closer to the recall number.

At a 3% base rate, a 90/80 classifier hands your reviewer eight false accusations for every true one — publish that ratio next to the flag or the flag is not shippable.

Point-in-time discipline, or the backtest is fiction

Screen receivables anomalies against current-vintage fundamentals and the restatement is already inside the data for exactly the cases the model claims to predict. What that produces is a very high reported AUC and nothing that survives deployment. Rebuild on as-first-reported vintages, promote filing timestamps to first-class fields, and enforce one hard rule: no feature may derive from data published after the prediction date.

This is not only a finance problem

Substitute another label factory and the analysis holds unchanged. Content-moderation labels after a policy rewrite behave the same way. So do fraud-operations labels after the review team doubles, and incident-severity labels after a runbook change. Each is a human process whose capacity sets the base rate the model was fit against. If that process just scaled, the probabilities are stale while every ranking metric on the dashboard still reads healthy.

What the pipeline actually costs

Strip the tickers off this and what remains is a five-stage ingestion job a competent team builds in a week. A daily crawl of executive and leadership pages. Normalized name extraction. A DOM diff anchored on stable structure rather than raw markup. Entity resolution against filings. A human-confirmation queue for every candidate removal. The dominant failure mode is not modeling. It is false positives from CMS migrations, CDN variants, and layout changes. Two cheap mitigations do most of the work: hash the normalized set of extracted names rather than the HTML, and require a candidate removal to persist across consecutive crawls before it enters the queue.

The denominator is the whole feature

A four-person cluster at a $195B company with thousands of officer-level roles is plausibly within Poisson expectation. That is the count you would see by chance, given the number of seats and normal turnover. Dave & Buster's is the contrast. Four CEOs since 2024, the latest gone after roughly 13 months, against a denominator of exactly one CEO seat, at a market capitalization collapsed to $357M. Same narrative shape. Opposite statistical content.

A raw-count departure feature is therefore size-confounded. Fed to a gradient-boosted model over a broad universe, it will learn market cap and headcount, then hand that back labeled as risk. The correlation is real. The causal story attached to it is not established, and the model has no way to tell the difference. Fit a per-firm negative-binomial baseline conditioned on headcount, role tier, and trailing turnover, and score the exceedance over that expectation rather than the count itself.

An unnormalized event count is a market-cap proxy wearing a risk label — score exceedance over a per-firm expectation or the model learns company size.

Label quality is the ceiling, not detection

Detection here is easy and the labels are terrible. The outcome attached to a departure is a self-reported reason, which means there is no clean target to train against. The thing a detection metric doesn't tell you is that the bottleneck sits entirely downstream of it. Treat this family as a trigger and a feature, not a supervised target. It fires a human review, or it joins a slower, better-labeled model as an input. A classifier whose ground truth is a press release predicts corporate communications style.

The pattern generalizes past equities

The structural claim is the reusable one. The authoritative structured feed is a lagged, lossy projection of the state that matters, and the unstructured web carries transitions that never reach a form. That holds for vendor status pages, model cards, pricing pages, deprecation notices, API changelogs. Monitoring that subscribes only to structured channels learns about state changes after everyone who diffs the page.