The Scientist

The four operations, and the two that bite

The scoring path is simple: resolve the token IDs for your labels, read the logits at those vocabulary positions, discard every other logit, softmax across the selected subset. Two consequences follow that the tutorial framing understates. First, raw logit magnitudes are model- and layernorm-dependent. The source's illustrative example uses logits of 8.2/5.5/4.8, producing 0.91/0.06/0.03; the actual run on Qwen/Qwen2.5-0.5B-Instruct produced 25.28/24.50/21.19. Every threshold you tune against one checkpoint is therefore non-transferable to the next. Second, normalizing only over declared choices guarantees the distribution sums to one even when the input belongs to none of your classes, so a security incident arriving at a billing/technical/account router gets routed confidently.

The tokenizer is the silent failure surface

Labels must resolve to exactly one token, which is why single letters with the semantics pushed into the prompt ("A = billing questions and payment problems") is the correct pattern — A/B/C resolve to [32]/[33]/[34] on this tokenizer. Two things break it without raising an error. Tokenizers often fold a leading space into the token, so 'A' and ' A' are different IDs. And the chat template can inject whitespace or control tokens at the answer position. A model swap silently re-points your scores at the wrong vocabulary entries and returns a perfectly well-formed distribution. That belongs in CI, not in a code review.

The source also never tests position bias. Letter-labeled multiple choice has a documented preference for early options, and scoring reads the very first next-token distribution, so there is no chain-of-thought to dilute the prior. The fix is uniquely cheap here: k zero-decode forward passes to marginalize over label permutations still undercuts a single generation call.

Where the evidence stops

The reported speed comparison is not procurement-grade. Two lanes of 100 cases each race on the same GPU under continuous batching behind a threading.Barrier(2), requests are issued sequentially within each lane, and the evidence is a video the author states was sped up after 8 seconds. No latency table, no throughput number, no aggregate accuracy — despite labeled expected answers existing in the dataset. The mechanism implies a large latency win (zero decode steps versus up to 32). Treat that as directional physics, not a measurement.

Three sources, one missing instrument

Computerworld's read on the McDonald's drive-thru retirement is the same finding from the deployment side: the system failed not because it made too many errors but because it could not tell which answers it did not trust. For any customer-facing decision model, that makes the metric stack expected calibration error, a coverage-risk curve, and escalation latency — not accuracy. Bloomberg's agent taxonomy adds the third axis: when two retrieved sources disagree on an attribute, that disagreement is a free unsupervised uncertainty signal, and silently picking one passes every faithfulness check you run.

Where the sources diverge matters. The scoring writeup offers a "top above 0.80, margin at least 0.20" heuristic; the deployment evidence says derive the operating point from a measured coverage-risk curve instead. The heuristic is a reasonable shape and an unvalidated magnitude. And before you buy a typed decision model for this — TLDR IT notes one reached nearly 13% of Vercel's paid teams inside 24 hours with no published benchmarks or ablations — run the three-way bake-off on your own traces against a fine-tuned small encoder and schema-constrained decoding on your incumbent.

The inference trick is free. The calibration layer is the product, and it lives in your code where it can be versioned and tuned to your loss function.

What the search actually did

Google's Empirical Research Assistance keeps a tree of past experiments as notebooks, uses a UCB rule to pick which notebook to mutate — optimistically rather than greedily, so even the fifth-best node gets expanded — has Gemini propose roughly ten code mutations per selected node, and shares branch history so leaves learn from siblings. Two details make it worth your time. The repo is public and model-agnostic, so the scaffold is not the moat. And the identical scaffold went from "just not working" on Gemini 2.0 to "working great" on Gemini 2.5, which means any LLM-driven search loop you shelved two model generations ago was a statement about the base model, not the method.

The headline result is automated specification search, not automated modeling. Platt's team already had the heat-trapping half of the contrail radiative-forcing model; the search found a simple model for the reflected-sunlight half that incorporated confounders the humans had not considered. That is the shape of work where these loops earn their compute: omitted-variable problems, uplift and incrementality specifications, anywhere you suspect the confounder list is incomplete.

The arithmetic of best-of-N

Now price the exposure. If an agentic loop scores N candidate notebooks against one fixed validation set with roughly independent noise, the expected best-of-N score is inflated by about σ·√(2 ln N) standard errors — roughly 3.7 SE at N=1000. On a mid-sized validation set, that is most of the reported lift. The reported score is a biased estimator by construction. And the optimistic policy that makes the search good at exploration also makes it good at finding your label bugs: Platt's own $15,000 Kaggle contrail competition was won in part by entrants who spotted a half-pixel label origin ambiguity — corner versus center of pixel — and folded it into their winning approach. Great for the prize, useless for contrails. High-throughput optimizers find artifacts before they find signal, because artifacts are the cheapest gradient in the space.

The same bias, in your online metrics

Two other items in this briefing are the same defect wearing different clothes. Fastly states machine-generated traffic exceeds half of all traffic on its network, with no published classification methodology, so treat it as an order-of-magnitude signal for one provider. The direction is what binds: if bot share is majority and growing, it is a non-stationary nuisance variable sitting in every conversion rate and experiment denominator. Uniform distribution across arms attenuates your lift toward zero; non-uniform distribution biases it in a direction you cannot sign without measuring.

Then the exposure stage. A feature shipped and enabled for 100% of customers had near-zero adoption because a sidebar did not expand by default — shipped was true, findable was not. Under intention-to-treat assignment on flag exposure, if only fraction p of assigned users reach the surface, your estimate converges to roughly p × the true effect among the exposed. At p=0.05, a genuine 10% lift reads as 0.5% and you write "no significant effect." That is a Type II error you shipped and filed.

Two methodology corrections in the paper flow covered here point the same way: a UK legal benchmark that scores whether systems pinpoint the exact relevant paragraph rather than the right document finds the best current systems fall short, and an authorship-attribution result holds only when the setup disentangles true style from topic and format. Document-level recall@k and random splits are both flattering you.

An optimizer with a good prior finds your label bugs faster than your discoveries — fix the holdout protocol before you fix the model.

What Windows ML actually changes

DirectML failed for measurable reasons. It was a low-level op-composition library, GPU-only because it sat on DirectX 12, and its runtime optimizations shipped through driver updates that took roughly six months to reach meaningful adoption. Windows ML wraps the ONNX runtime instead, and each silicon vendor implements optimization behind an Execution Provider that Windows downloads on demand; an RTX card pulls the NVIDIA Tensor RTX EP. The wrapping is cleaner, but the skew problem is the same one DirectML had, relocated. Driver-version skew becomes EP version and kernel-selection skew, and the out-of-band update that cured the adoption lag is also what silently changes the numbers being measured. Opaque server-side model changes do the same work from the cloud direction, which puts both halves of a longitudinal metric on unversioned dependencies.

The demo number should not enter planning. ~40 tokens/sec for Qwen on a pre-release NVIDIA Surface is a rate attached to a configuration in which parameter count, quantization scheme, context length, batch size and the prefill-versus-decode split are all free variables. Qwen spans roughly 0.5B to 235B+ parameters, so the figure is consistent with hardware floors an order of magnitude apart. One inference survives. At 40 tok/s a 4,000-token agent turn costs 100 seconds of wall clock regardless of how hard the task is, so route the local tier on predicted output length.

Containment is an experimental variable

The same release ships MXC, a JSON-policy sandbox API with five tiers from process containment through full VM, already running on Windows, Linux and macOS. Microsoft did not publish latency figures for any tier. The tiers also push measurements in opposite directions: process containment leaves the user filesystem exposed, so leaked state can inflate pass rates, and a full VM can deflate them through timeouts. Change the tier, change the pass rate, for reasons that have nothing to do with the model. Error taxonomy takes damage too. Agents are not told what the sandbox forbids, so policy denials land in traces as model failures.

The hosted-model version of the same risk

The Information reports Anthropic is in early talks to lease up to 1 gigawatt directly from Stream Data Centers. Stream is a 27-year-old developer majority-owned by Apollo Global Management. The sites would be filled with Broadcom/Google-designed TPUs instead of Nvidia GPUs, with reduced cloud-provider reliance as the stated motive. Handle the provenance honestly: every substantive claim traces to anonymous people with knowledge of the discussions, the talks are explicitly early, and the open variables include lease term, price, site list, chip volume, date, and whether the capacity targets training or inference. Calling 1GW one of the largest single-tenant developer commitments is an analyst inference carrying 0.6 confidence, not sourced reporting.

The decision-relevant part is narrower and more useful. Identical weights served on different accelerators, with different kernels, batching schedulers and low-precision paths, can produce measurably different output distributions. Prompt-tuned thresholds, an LLM-as-judge rubric, a strict JSON schema, a downstream classifier trained on a hosted model's outputs: all of those are calibrated against a serving stack that can move without a version bump. The eval harness is the only instrument that will notice.

The local model artifact is half the reproducibility story. The Execution Provider, and on hosted endpoints the accelerator, is the other half, and it belongs in the eval metadata alongside the model hash.