Science & Analytics

The Scientist

The Signal

Anthropic quietly killed the flat-rate Claude developer subsidy

This eliminates the 70-90% effective discount power users had been getting. OpenAI dropped a 2-month-free Codex enterprise switch promo the same day, and Vercel's production data shows 59% of all tokens are now agentic.

In Play

  1. Anthropic's Triple Squeeze: Metered Credits, 80x Capacity Miss, June 15 Cliff

    Anthropic metered programmatic Claude usage at list-rate API credits, admitted an 80x-vs-10x capacity planning miss, leased xAI's 220K-GPU Colossus 1 cluster, and will split third-party tool billing on June 15. ServiceNow burned its full-year Claude budget by May. Enterprise share flipped to 34.4% vs OpenAI's 32.3%.

    Ask Clarity
  2. 59% Agentic: Eval Harnesses and Cost Models Are Measuring the Minority

    Vercel's AI Gateway puts agentic workloads at 59% of all production tokens. Anthropic captures 61% of spend (via Opus for reasoning), Google captures 38% of volume (via Flash for throughput). Single-turn eval harnesses now score the minority of traffic. Cost models built on 3:1 input-output ratios are off by ~5x for 15:1 agentic traces.

    Ask Clarity
  3. AI Cyber Capability Crossed AISI Threshold — Harness Dominates Model

    Anthropic's Mythos is the first model to clear both AISI simulated attack ranges (full network takeover). GPT-5.5-cyber cleared one of two. Mozilla's agentic harness found 271 Firefox bugs with the same model that found 1 CVE in curl without one — a 271:1 delta proving harness engineering dominates model selection for vulnerability discovery.

    Ask Clarity
  4. Training Efficiency: Three Papers Move the Unit Economics

    Nous TST reports 2-3x wall-clock speedup at matched FLOPs (validated 270M→10B MoE). Datology hit +11.7 pts on VLM benchmarks at 17x less training compute via pure data curation. NVIDIA Star Elastic claims 360x cheaper model-family derivation from a single post-training run. All three change the $/capability math for teams running their own training.

    Ask Clarity
  5. GPU Supply: 4:1 Demand Ratio and the Inference Silicon Fork

    Nebius reports 4+ customers competing for every GPU online, posting 684% YoY revenue growth with $3-3.4B 2026 guidance. Cerebras IPO'd at $56B (+70% day one) backed by OpenAI's $20B commitment. Cisco AI orders jumping $5B→$9B. The inference hardware layer is diversifying away from Nvidia faster than most capacity plans assume.

    Ask Clarity

Deep Dives

Anthropic's Metering Cliff: Re-Price Your Agent Stack Before June 15

What Happened

Anthropic shipped three changes that interact badly for anyone running Claude in production. First, subscriptions now convert to dollar-matched API credits across Agent SDK, claude-p, GitHub Actions, and third-party harnesses. The implicit 70-90% subsidy on programmatic usage is gone. Second, Dario Amodei conceded the company planned for 10x growth and got 80x, which is the read on weeks of degraded Claude Code performance: a capacity miss, not a model regression. Third, starting June 15, third-party tool usage (Zed, Conductor, OpenCode, T3 Code) moves to a separate credit bucket with no rollover and overflow at API rates.

The capacity patch is leasing xAI's entire Colossus 1 cluster — 220,000+ GPUs spanning H100, H200, and GB200 — from a CEO who publicly insulted Anthropic three months ago. Rate limits are being raised in parallel: Claude Code 5-hour caps doubling, peak-hours throttling removed, Opus API limits "substantially raised."


Why Sources Disagree on Market Position

Seven independent sources reported Anthropic overtaking OpenAI in enterprise adoption (34.4% vs 32.3% per Ramp). The thing this doesn't tell you is what the metric measures: credit-card billing share, not token volume, not workload criticality, not large-enterprise invoiced spend. OpenAI's point that $1M+ ACV accounts pay by ACH, not card, is correct and material. Ramp's own economist separately flagged Opus 4.7 tripling image costs and mounting reliability complaints. The crossover is real for bottoms-up developer adoption. It is not yet a statement about the Fortune 500.

ServiceNow burned its full-year Claude budget by May. National Life Group's CIO says Claude is 'not great for companies' wanting per-user monitoring. The frontier model most teams build on has no native cost attribution, no SLAs, and no per-user telemetry.

The Compound Effect

Token consumption in agentic workflows is non-linear. A reflection loop or tool-use chain can 10x spend per task with no proportional quality gain, and the per-task variance is wide enough that an average tells you very little. Remove the subscription subsidy on the same date the vendor still ships no per-tenant attribution, and the failure mode is a silent budget overrun that surfaces in the monthly invoice rather than the observability dashboard. Teams with gateway-level logging in place before June 15 absorb the change. Teams discovering it in the invoice do not.

SurfaceBeforeAfter (May-June 2026)
Agent SDK / claude-pFlat subscription covers heavy useDollar-matched API credits, then list rate
Third-party tools (Zed, etc.)Covered by planSeparate bucket, no rollover (June 15)
Claude Code caps5-hour limit, peak throttledDoubled, throttling removed
Opus API rate limitsConstrained during crunch'Substantially raised' post-Colossus

What to do

  1. Audit every Claude-backed workload (Agent SDK, claude-p, GitHub Actions, batch evals) and reconcile projected token burn against the new credit cap by end of this sprint

  2. Deploy an LLM gateway (LiteLLM/Portkey) with per-user, per-feature tagging and daily budget alerts before June 15

  3. Run OpenAI's 2-month-free Codex enterprise switch promo as a controlled head-to-head on your actual eval harness with matched prompts and tool schemas

  4. Re-baseline all Claude benchmarks (throughput, p95 latency, rate-limit headroom) after Colossus 1 integration stabilizes — do not commit to workarounds built against the degraded period

59% Agentic: Your Eval Harness and Cost Model Just Became Minority Instruments

The Production Reality

Vercel's AI Gateway production index is the only multi-tenant usage dataset with 200K+ teams and 7 months of data, which is the disclaimer up front. It puts agentic workloads at 59% of all token volume. Six months ago that number was under 20%. The composition shift is the fastest since completion-to-chat endpoints, and it happened while most eval harnesses were still scoring single-turn responses against reference answers.

Three structural mismatches follow. First, cost models are off by ~5x: agentic traces run 15:1 input-to-output versus the 3:1 most forecasts assume, with heavy cache reuse on some providers and none on others. The thing this doesn't tell you is which providers, so the 5x is a population estimate, not a per-vendor one. Second, eval harnesses measure the minority: single-turn accuracy at 90%+ hides the planner that burns 40K tokens arguing with itself before giving up. Third, the spend/volume split is the routing signal: Anthropic captures 61% of dollars via Opus (reasoning), Google captures 38% of tokens via Flash (throughput), with no vendor loyalty observed.


The Architecture That Works at Scale

Abridge's disclosed architecture across 80M+ clinical conversations is a second data point on the same pattern: cheap fast model triages, expensive model reasons only when needed, with memory externalized from weights and LLM judges calibrated against human annotators. Two independent sources converging on confidence-gated routing across a constellation of models is enough to call it the de facto production pattern, not a research proposal.

If 59% of your tokens are agentic but 100% of your evals are single-turn, you're flying instruments-out — update the harness before you update the model.

What the MCP Overhead Tells You

Glean reports off-the-shelf MCP burning 30% more tokens than retrieval-tuned knowledge graphs on agentic tasks, losing 2.5x head-to-head on preference. The methodology is vendor-published with no disclosure. Treat the magnitude as a hypothesis. The directional claim, that naive tool listings balloon context windows while reranked snippets do the same work cheaper, matches what SAP and ServiceNow concluded independently this week. All three converge on Knowledge Graph grounding + MCP-governed execution as the enterprise agent reference architecture.

Multi-agent decomposition adds a dimension worth a separate ablation. Microsoft's MDASH (100+ agents) beat Anthropic's Mythos on CyberGym by decomposing tasks into scan → adversarial debate → PoC exploitation stages. The 100-agent count is a design choice. No ablation isolates the staging from the ensemble size, so it is possible but not established that the staging is doing the work. The decompose-debate-verify pattern is consistent with ensemble priors and is the cheap experiment to run on any workload with auto-verifiable outputs.

What to do

  1. Add trajectory-level metrics to the eval harness this sprint: tool-call precision/recall, steps-to-completion, cost-per-successful-task, recovery-from-error rate

  2. Instrument per-node token cost in your agent graph and route utility calls (summarization, JSON extraction, query rewriting) to Flash/Haiku-class models within 2 weeks

  3. Run a 1-hour spike measuring MCP tool-calling overhead vs. a rerank/KG baseline on 100 replayed production agent traces

  4. Add LLM-judge-to-human-annotator agreement (Cohen's kappa) as a tracked SLI in the eval pipeline; re-calibrate quarterly

AISI Range Saturation: Autonomous Cyber Capability Is Now a First-Class Eval Axis

The Threshold Crossing

The UK AI Security Institute evaluated the newest Anthropic Mythos and OpenAI GPT-5.5-cyber on autonomous cyber-offense tasks. Both completed full network takeovers in controlled environments, one capability tier above the prior Mythos generation, which topped out at "advanced persistence." Mythos cleared both of AISI's hardest tests. GPT-5.5-cyber cleared one. AISI is already building harder tests, because the current ladder is saturating.

This is the first time a national evaluator has publicly stated that frontier models can complete an end-to-end attack chain. Neither lab is releasing these variants broadly. Anthropic gates Mythos to select enterprises and government agencies.


The 271:1 Harness Signal

Two teams ran Claude Mythos against C codebases in the same month. Mozilla wrapped a custom agentic harness around existing fuzzing infrastructure and surfaced 271 bugs in Firefox 150, including sandbox escapes, use-after-frees, and race conditions. Daniel Stenberg pointed Mythos at curl with a generic scanner and got exactly 1 low-severity CVE with 4 false positives. His verdict: "primarily marketing."

Same model, two orders of magnitude apart. The variable that moved was the harness. Mozilla's wrapper emits reproducible test cases, scales across ephemeral VMs, and integrates with their security lifecycle. The thing the leaderboard score doesn't tell you is which scaffold the model was wearing. This is the strongest public evidence to date that eval budget dominates model choice by at least 50x on real codebases.

DimensionMozilla + FirefoxStenberg + curl
ModelClaude Mythos PreviewClaude Mythos Preview
HarnessCustom agentic, fuzzer-integratedOut-of-box scan
True positives271 (incl. sandbox escapes)1 low-severity CVE
False positive rateTooling-filtered~80%
Vulnerability discovery just moved from human-weeks to model-minutes. If the patch SLA is not benchmarked against inference time, the defense is tuned to last year's threat model.

Implications for Agent Deployments

Refusal-rate harnesses calibrated on GPT-4-era capability assumptions will produce false negatives against Mythos-class attackers. The miscalibration is structural, not a tuning fix. For anything agentic with tool access, the eval needs a staged rubric covering recon, initial access, lateral movement, persistence, and exfil, not input-side prompt filters scored in isolation. Microsoft's MDASH shipped 16 real Windows fixes off multi-model bug-hunting, which is the load-bearing data point that the capability has crossed the utility threshold for both offense and defense at the same time.

What to do

  1. Add an autonomous-cyber-capability tier to your model eval harness this quarter — include AISI-style attack-range tasks for any model with tool/shell access

  2. Spike a domain-specific agentic vulnerability harness on your own codebase modeled on Mozilla's pattern: reproducible test cases + ephemeral VMs + integration into existing signal pipelines

  3. Instrument agent logs for tool-use patterns matching recon → lateral movement → persistence signatures, not just prompt-injection strings

  4. Compress critical-patch SLA to model-release cadence rather than CVE-publication cadence

Three Training Efficiency Papers That Change This Quarter's Build Math

Three recipe and curation results worth pricing in

This week's drops share a through-line for teams running their own training or distillation: the marginal dollar in model development has moved from raw compute to recipe and curation. Each result shifts unit economics in a direction that matters, and each carries its own replication risk profile.


1. Nous Token Superposition Training (TST)

Reports 2-3x wall-clock speedup at matched FLOPs with no inference-time architecture change, validated from 270M through 10B-A1B MoE. The thing this measures is pretraining throughput. The thing it doesn't measure is whether the speedup holds at frontier scale under independent replication. Single source, but the mechanistic claim is clean. If it holds, it's a free 2-3x on training runs without touching the serving stack.

2. NVIDIA Star Elastic

Claims one post-training run produces a family of reasoning model sizes at 360x lower cost than pretraining a family, 7x better than SOTA compression. Headline numbers of that size always shrink under independent eval. The question is the floor, not the ceiling. A 30x hold would still restructure how size tiers are produced for routing. A 10x hold probably would not.

3. Datology VLM Curation

Hit +11.7 points on 20 VLM benchmarks at 2B params, beating InternVL3.5-2B by about 10 points at 17x less training compute. Produced a near-frontier 4B model at 3.3x lower response FLOPs than Qwen3-VL-4B. The lever was purely data curation, not architecture or scale. This is the clearest evidence this year that curation dominates compute at the VLM frontier.

WorkClaimValidated ScaleInference ImpactReplication Risk
Nous TST2-3x wall-clock at matched FLOPs270M → 10B MoENone — no arch changeMedium
Star Elastic360x cheaper family derivationUnspecifiedProduces size tiers from one runHigh
Datology VLM+11.7 pts at 17x less compute2B and 4B3.3x lower response FLOPsMedium
TST is the one to spike first: it's a training recipe with no inference-side tax. If it replicates at even 1.6x, it pays for itself on the next full run.

Adjacent: SWE-ZERO-12M-Trajectories

Kevin Li released 112B tokens, 12M trajectories, 122K PRs across 3K repos in 16 languages, positioned as the largest open agentic trace corpus. Useful for SFT, reward-model training, and synthetic eval generation. Open releases at this size tend to acquire licensing friction within months. Worth mirroring before that happens.

What to do

  1. Spike Token Superposition Training on a 1B-param continued-pretraining run against a matched-FLOPs baseline this month

  2. Pull SWE-ZERO-12M-trajectories and stand up a preprocessing pipeline (dedup, license filter, language stratification) before licensing frictions accumulate

  3. Run a data-curation ablation on your VLM pipeline: systematic filtering and reweighting against a compute-matched baseline

  4. Benchmark Star Elastic's model-family derivation against your current distillation/quantization pipeline once the paper drops

The bottom line

Anthropic killed the flat-rate Claude subsidy the same week Vercel's production data showed 59% of all tokens are agentic — meaning your cost model is wrong by the subscription change AND by the workload-mix shift simultaneously. The teams that survive this quarter are the ones that meter at the gateway, route by task difficulty, and never let a single vendor's pricing change become a sprint-level emergency because they already have the abstraction layer. If you're still calling anthropic.messages.create directly with no fallback, that's no longer technical debt — it's an unhedged financial position.