The Scientist

Caching pays for itself at these prices; the cost sits in silent invalidation

Anthropic and OpenAI both price prompt caching at 1.25x the base input rate to write and 0.1x to read. Solve 1.25 + 0.1n against 1 + n and break-even arrives when a written entry is read about 0.28 times inside its TTL, roughly a one-in-four chance. Broad enablement is the correct default, which is why nobody audits it afterward.

Invalidation is where the money goes, and it goes quietly. On a miss, Anthropic walks backward through a capped 20 blocks looking for an older write; put more conversation than that between two calls and the hits stop with no error, and the extra spend gets attributed to traffic growth instead. Daily Dose of Data Science lists the rest of the surface, and every item is routine optimization-sprint work: a timestamp or request ID at the front of the prompt, tool schemas reordered ahead of the system prompt, history summarization rewriting the prefix, and provider toggles — web search, citations, thinking config, tool_choice — that rewrite the rendered prompt text. Truncating tool outputs in place is the one edit that leaves the prefix byte-identical.

The router arithmetic can invert

Unwind AI relays a constraint that should sit in the architecture diagram: cache entries are model-scoped. Route an in-flight session to a cheaper model and the whole accumulated history re-prefills at cold rates. On a 100K-token agent session heavy with tool output, one switch can erase whatever per-token arbitrage the cascade earned. The design correction is that the unit of routing is the session, not the request: classify once at session start, partition context so cheap and expensive models never share a prefix, or accept the re-billing and measure it.

Only the last row can miss and still return a wrong answer with a 200 status code, and it is the only layer that keeps billing an embedding call on every miss. Published similarity thresholds span 0.75 to 0.97, a 22-point spread that reflects local traffic, so a threshold lifted from someone else's write-up carries no information about yours. Some failures never respond to threshold tuning: negated sentences sit close together in embedding space, and two prompts sharing a sentence frame while differing in one operational value score as near-identical because the frame dominates the vector. The metric to track is a wrong-answer-rate-at-threshold curve built on roughly 500 adversarial pairs.

One clean experiment, and the capacity wall behind it

TLDR Data's block-paging result is the rare capacity win with no model-risk surface: fixed-size blocks plus a block table replace contiguous per-request allocation, and attention outputs do not change. A regression suite should return bit-comparable results, so no quality re-validation is owed. The thing the 2–4x spread doesn't tell you is where a given workload lands inside it; no hardware, model size, or sequence-length distribution was disclosed, and the range almost certainly tracks shared-prefix fraction and sequence-length variance. The prefix-sharing ratio in production sets the number.

Capacity planning is where this bites hardest. Decode is memory-bandwidth-bound: the whole KV cache streams from HBM on every decode step, and cache size grows linearly with tokens, about 40GB for a single 128K context on a 70B BF16 model, comparable to the entire model at 4-bit weights. The prefix cache draws from the same GPU pool as the running batch, so a bigger cache buys fewer concurrent sequences, and vLLM's LRU eviction takes the long shared prefixes first. Size on KV memory and bandwidth rather than FLOPs, and benchmark GQA or FP8 cache quantization on a local accuracy suite before enabling it. TheSequence's point lands here too: most teams emit neither MFU nor batch-fill, so they pay full depreciation for a minority of available FLOPs.

A miss on three of the four caches costs money; a miss on the fourth returns a wrong answer, so semantic caching needs its own wrong-answer metric alongside the hit-rate dashboards.

A harness that flushes cannot see this failure class

Per-turn input sanitization and per-session jailbreak detection both assume the attack and the harm land inside the same turn. CSO Update's reporting on InjecMEM relocates prompt injection from the read path to the write path: the instruction is written into long-term memory and shapes responses after the planting session ends. Eval harnesses flush state between trials because flushing is what makes evals reproducible. The blindness here is by construction, not by oversight.

The measurable version is a persistence eval. Inject payloads through each write channel (user turn, tool output, retrieved document, web fetch), clear the conversation, then score behavior on 20–30 held-out probe tasks at turns t+1 through t+k, and report persistence ASR@k with confidence intervals. Attack success is Bernoulli, so roughly 384 trials per cell buys ±5% at 95%. Twelve hand-written adversarial prompts is an anecdote. Pair the result with a benign-task regression score, or persistence gets "solved" by filtering the agent into uselessness. The reporting itself is teaser-grade: no CVE IDs, no attack success rates, no model names. Treat it as a hypothesis to reproduce locally rather than a published result.

The cheapest item here is an afternoon of endpoint auditing

The same edition carries a DNS-rebinding flaw in NemoClaw that hands a visited browser tab unauthenticated access to a local Ollama server. The transferable part is the broken assumption: a service bound to 127.0.0.1 is not unreachable. That assumption sits in the defaults of vLLM and TGI, Jupyter, MLflow tracking servers, Ray dashboards, and local Chroma or Qdrant instances. Requiring an auth token plus Host and Origin validation across that inventory is the highest expected value per hour in this briefing.

The hash you check at load has already passed

The Hacker News reports GPUThor, a Rowhammer variant that defeats ECC on NVIDIA RTX A6000 GDDR6 memory and escalates to host root. ECC corrects single-bit and detects double-bit errors within a codeword; Rowhammer induces multiple flips inside one codeword, pushing corruption past both. Risky.Biz frames the ML consequence plainly: ECC was the defense the industry leaned on for silent-data-corruption safety, and host root from the GPU inverts the assumption that CUDA-level access is a lesser privilege than shell access.

One caveat, flagged by the source itself: the disclosure covers ECC bypass and root escalation. Weight corruption is an inference from bit-flip-attack literature, where a single flip in the exponent bits of an FP32 or BF16 parameter moves its magnitude by orders of magnitude while the process runs cleanly. The thing this doesn't tell you is flip rate, time-to-flip, or cross-SKU behavior. None was published, which leaves H100 and A100 HBM exposure unquantified rather than disproven.

That last row is the whole argument for a recurring canary: a fixed batch of about 32 inputs, expected logits stored beside the checkpoint, tolerance tuned to kernel nondeterminism, run every 15 minutes in long-lived serving processes. Feature-drift detectors and throughput dashboards do not observe the parameters.

Two quieter versions of the same defect

ben's bites notes Anthropic merging memory across Claude Chat and Cowork while keeping Claude Code separate. Methodologically that is per-user, non-stationary hidden state. With memory on, identical prompts stop being identical experimental units, offline eval stops predicting online behavior, and every assistant-quality A/B test acquires an unmeasured covariate. Memory off in all eval runs, memory state logged as a covariate when it must be on, randomization at user level and never request level.

Bloomberg Technology's personnel item is the same problem viewed from the top: two former Google DeepMind researchers left to build a hybrid human-AI safety evaluation system as a nonprofit. No architecture, no false-negative rate, no inter-rater agreement, no cost-per-adjudication has been published, so it is not evidence of anything yet. The copyable structure is automated evals as the wide net with stratified human adjudication as the gate, plus tracking of the disagreement rate between automated scores and human labels. That rate is the only empirical handle on the automated suite's recall.

Until attack persistence is measured after a context flush, and weight integrity after load, your guardrail metrics are scoring the wrong turn and your dashboards are scoring the wrong tensor.

Peer data kills the five-year savings claim and confirms the $4-per-file unit cost

Asana's Enzyme-removal counterfactual prices four engineers at about $300K over five years: roughly 20 engineer-years for 3,000+ remaining files. Airbnb independently estimated 1.5 engineer-years to hand-migrate 3,500 files of near-identical scope, a ~13x gap. The Pragmatic Engineer names the mechanism. The five years was queue latency, not work, the projected drain time of an item parked two years in the "opportunistic" tier of a critical/important/opportunistic backlog. Conflating work-in-progress latency with per-unit throughput is the standard way an AI ROI claim inflates.

The unit repricing holds. Asana's run cost about $12K in inference and infrastructure, roughly $4 per file, over two calendar weeks and about 1.5 engineer-weeks. Uber moved 600,000 unit tests across 15M lines from JUnit 4 to JUnit 5 in four months with two engineers, modifying 1.25M lines. Bun ported 530,000 lines from Zig to Rust in two weeks for $165K in API spend. Against Sentry's 2021 manual JS-to-TypeScript conversion at roughly $21–42 per line, that is a 70–130x shift in unit cost. The sample is four self-selected successes; nobody publishes the LLM migration they abandoned, and no one ran an ablation separating model generation from pipeline design.

The last 3% is where the schedule goes

Airbnb published the only usable throughput distribution. Naive retry loops cleared 75% of 3,500 files in four hours, a purpose-built five-phase refactor pipeline reached 97% over four days, and engineers closed the final 3% by hand over a week. That is a per-item success probability with a heavy difficulty tail, the same shape as weakly-supervised labeling loops and auto-remediation pipelines. The four-hour pass buys 75%; the remaining quarter consumes days four through fourteen.

Why "the suite is green" is a cheap proxy

Enzyme asserts against a component instance; React Testing Library asserts against the rendered DOM. An equivalent counter test shares only its imports. This is a semantic rewrite, so an LLM can return a passing test that asserts strictly less than the original, structurally identical to a model that holds AUC while its calibration rots. The accept function has to be coverage delta ≤ 0 plus a mutation score within tolerance of the pre-migration run, with a sampled human diff review. The same warning covers every LLM rewrite of assertion-bearing code in the repo: dbt tests, Great Expectations suites, schema validators.

The eval-discipline results point the same direction

Two independent reports converge on the denominator. TLDR Data describes GitHub cutting secret-scanning false positives 95% offline by versioning the prompt, model ID, input-construction code, and eval config as one artifact, and by restating the objective as minimize false positives subject to a recall floor rather than maximize accuracy. Without that artifact hash, a prompt tweak and a model swap are confounded interventions and no metric movement is attributable. The caveat is real: offline only, no production A/B, and the recall floor value was never stated.

Applied AI's account of Visa open-sourcing an 11-stage agentic vulnerability harness carries the same lesson on worse evidence. Visa declined to share any cost or time figures, and the circulating 60% cost-reduction number traces to an unnamed July study about a merely "similar" harness, which makes it commentary rather than a planning input. What transfers is the stage DAG: deterministic pre-scan mapping decides which codebase regions to scan before any frontier-model call, and an explicit false-positive verification stage concedes that frontier models alone lack the precision for autonomous action. Retries are where the cost hides. A stage failing 30% of the time with up to three retries roughly triples that stage's spend while every per-call latency and cost dashboard reads normal, which makes retries per completed task, by stage the figure that exposes it. Nobody measured false negatives, the failure mode that hurts when an agent acts autonomously.

AI moved migration cost from headcount to verification rather than removing it, so the team that already knows how to build an eval harness owns the highest-ROI item in its deprioritized backlog.