The Scientist

What Happened

OpenAI deprecated its finetuning APIs on a window shorter than any migration plan assumed. For teams that built RLFT stacks on these endpoints, with reward-model loops and policy updates routed through the hosted finetuning call, this is not a config change. The endpoint sits behind abstraction layers and scheduled jobs nobody has opened since onboarding.

The Bifurcation

The market response splits into two camps, and the split is cleaner than usual:

Cursor and Cognition (now at $25B) are increasing open-model RLFT. That validates the approach at the top of the distribution. The thing this doesn't tell you is what happens at the modal workload: fewer than 100M daily calls, variable prompts, moderate quality bar. There, long-context prompting with caching is now the path of least resistance.

The Hidden Risk

The expensive bug is not the API swap. It is discovering post-migration that the reward signal drifted because the reward model was tuned against a specific base checkpoint. The new base is not that checkpoint. Offline evals that tracked online behavior on the old base are not guaranteed to track on the new one, and the cleanest way to confirm is to re-run the correlation study before you cut over.

The replacement that passes the eval harness and fails in production is always the one where nobody re-validated the reward model against the new base.

Cross-Source Pattern

This deprecation lands the same week as three other data points worth reading together: a 4B recursive model reportedly matching Sonnet 4.6 at a fraction of cost, which suggests small-model RLFT has legs; DeepSeek V4 Pro pricing at $0.43/M input tokens, which makes the "just use a bigger model" path cheaper than it was on Monday; and Cactus Needle at 26M params doing tool-calling at 6,000 tok/s. Reportedly is doing work in the first one. Given the numbers, I expect the 4B result to hold up on real workloads by about half as much as claimed, which is still enough to matter. The finetuning decision is now inseparable from the model-selection decision.

The Escalation

The npm supply-chain worm covered in last week's note has picked up a new behavior worth flagging: it wipes the host when the owner rotates the stolen token. The standard detect-revoke-rotate loop now triggers the destructive payload. Five independent sources confirmed the escalation this week. That is enough to update the playbook.

The mechanism: Shai-Hulud plants a gh-token-monitor persistence hook that watches credential files for access-pattern changes. When the monitored file changes, it runs a wipe. The thing this does not tell you is how often the hook fires on benign edits. We do not know yet. Assume the false-positive rate is low enough that the attackers shipped it.

New IR Primitive: Contain Before Rotate

The sequencing change is the point. It will not surface on a dashboard:

1. Snapshot the affected environment (disk, memory)
2. Network-isolate the runner or workstation
3. Search for persistence hooks in .claude/settings.json, .vscode/tasks.json, and credential-monitoring processes
4. Remove persistence from a forensic copy
5. Only then rotate credentials from a clean, separate environment

Blast Radius for ML Stacks

The harvest list enumerates AWS, GCP, Kubernetes, Vault, GitHub, and SSH credentials. For an ML org, that one list covers the feature store, the model registry, training cluster IAM, and deploy keys. The SANDCLOCK stealer from the same group (TeamPCP) is also pivoting to AI gateway credentials. LLM API keys are now on the target list.

The muscle memory of 'rotate tokens first, ask questions later' is now the attack vector. Contain and isolate before you touch a single credential.

The Checkmarx Connection

Separately confirmed, and the part that should change your threat model: the Checkmarx Jenkins AST plugin compromise (CVE-2026-33634, CVSS 9.4) traces to credentials stolen from Trivy in March that Checkmarx never rotated. That is a two-month dwell time on a secret known to be leaked. Same attack group. The correlation with long-lived tokens is not subtle. The causal read is that the entire class needs to move to short-TTL OIDC-federated credentials. If that migration only cuts dwell time in half, it still pays for itself on this one incident.

Three Converging Results

A few recent results point the same way: the axis that matters next quarter is cost-per-useful-token at fixed quality, not peak capability. Three worth evaluating seriously.

1. Recursive 4B LMs Matching Sonnet 4.6

A 4B-parameter recursive language model trained with RL and a shared parent/child policy reportedly matches Claude Sonnet 4.6 on task performance. Treat the parity claim with caution. Last year's 'small model matches GPT-4' cycle did not survive contact with real tool-use traces. The architectural insight still transfers: a shared policy serving both the outer planner and the inner sub-call collapses a router plus executor cascade into one trainable artifact.

The thing the headline doesn't tell you is inference cost. If the recursive model runs 8 passes to match, effective cost is closer to 32B than 4B. Still interesting. Not the same product decision.

2. Compression-Aware Scaling Laws (Bytes, Not Tokens)

Across about 1,300 models, the canonical '20 tokens per parameter' Chinchilla rule is a tokenizer artifact. Compute-optimal scaling is cleaner expressed in bytes per parameter. If your pretraining mix skews toward code or non-English text, both of which have lower bytes per token, your compute budget has been quietly biased. Rederiving in bytes is roughly a week of work. It could reveal systematic over- or under-training.

3. Chinese Model Price Arbitrage

The Exponential View team visited 14 Chinese labs. The pricing picture:

The Chinese cost advantage is not hardware. Huawei's Ascend 950PR only matches 2022-era H100. It is software: MoE routing, FP8 training, speculative decoding, and longer pre-training cycles forced by chip constraints. That part is reproducible.

Chinese open-source went from 1.2% to 30% of global usage in 12 months. An eval harness that still defaults to closed US models is measuring last year's frontier.

The Router Pattern

Operational synthesis: a cost-aware LLM router that dispatches easy intents (extraction, classification, summarization, first-pass rerank) to DeepSeek/GLM-5/Kimi and keeps the hard tail on Opus/Sonnet. Cursor already ran this pattern with Composer 2 on Kimi K2.5. At 28x output-token savings, 28x more LLM-as-judge calls fit in the same budget, or the same judges run over 28x more candidates.

The Architectural Pattern

Thinking Machines Lab shipped a 276B MoE interaction model at 0.40s end-to-end latency with 200ms micro-turns, ahead of GPT-Realtime-2 and Gemini Live on responsiveness. What travels out of this paper is not the model but the shape: a fast-path conversational model paired with an asynchronous background reasoner.

The pattern is a deployable choice, not a research result. Route turn-taking to a small quantized model in the 7–13B range and keep the frontier endpoint for async tool calls and long-horizon planning. The non-trivial piece is the state-sync protocol. The async reasoner arrives late or disagrees with the fast path, and reconciling that disagreement is the new open problem.

The Caching Gap That Multiplies the Win

Datadog's telemetry from 1,000+ organizations shows agent framework adoption roughly doubled while prompt caching stayed underutilized. Agentic workloads carry the highest repeat-prefix rates of any LLM pattern, driven by system prompts, tool schemas, and ReAct scaffolding. The orgs scaling agents fastest are the ones overpaying per call.

Cached input tokens run at roughly 10% of base price on Anthropic and a ~50% discount on OpenAI. For an agent doing 100K calls/day at 4K cached tokens each, the delta is four-figure monthly savings per endpoint.

If agent traffic doubled and cache hit rate did not, the provider is capturing margin that should be sitting in the infrastructure line. The top-5 endpoints typically account for most of the wasted spend.

The Rate Limiter Lesson

Databricks published a reference architecture after their rate limiter collapsed under real-time serving traffic. The fix moved from synchronous Redis per-request to in-memory async batch reporting, trading ~5% overshoot tolerance for roughly 10x tail latency reduction. Optimistic local decisions with asynchronous reconciliation generalizes to any inference ingress where a 10-20ms Redis hop against a 50ms P99 inference budget is burning 20-40% of the latency on bookkeeping.

New Eval Metrics Required

Most current LLM evals do not measure what matters in dual-engine architectures: time-to-first-useful-token, interruption recovery, async-tool completion rate, and micro-turn latency distributions. MMLU averages correlate poorly with turn-taking latency and interruption handling, so rankings on those benchmarks will diverge from production behavior on real-time candidates.