The Scientist

The Pattern

Five enterprise SaaS vendors independently moved to meter or block agent access in the same cycle. ServiceNow's Action Fabric charges per agent action, not per user. DataDog caps its MCP server at 5,000 daily and 50,000 monthly requests. SAP requires endorsement to access data, which effectively bans unauthorized agents. Workday and HubSpot are implementing usage-based metering with details pending.

JPMorgan analyst Mark Murphy called it plainly: "essentially a tax on customers using outside AI agents." AWS CEO Matt Garman is publicly positioning against the trend, warning incumbents are "trying to protect what they have."

Why This Breaks the Cost Model

Most agent eval harnesses measure success rate and end-to-end latency. Very few measure billable external calls per successful task, which is now the metric that determines pipeline profitability. The thing a 92% success rate doesn't tell you is how many tool calls sit behind it. Three per task versus nine, at the same success rate, is a different P&L once each call meters.

ReAct loops that retry on ambiguity used to be close to free. Exploratory patterns calling three tools when one would do had minimal cost. At per-action pricing, a more deterministic planner with caching pays for itself in a single billing cycle.

The Preferred-Partner Dynamic

Anthropic's Claude Cowork received a first-class connector into ServiceNow's Action Fabric. Preferred-partner deals will create uneven cost and capability across agent vendors. Benchmark-only model selection does not measure this bottleneck. The eval harness needs cost-per-successful-task across vendors per integrated SaaS.

MCP has become the billing surface for agent-to-SaaS traffic. It is the chokepoint where vendors count, price, and throttle.

The Double-Charging Risk

Customers already pay SaaS licenses and LLM API usage-based pricing. A third meter on top is a real market test. If tolerance breaks, AWS-style open alternatives gain traction quickly.

Two Results That Compound

DeepSeek V4 Pro ships at 1.6T total and 49B active MoE parameters with a 1M-token context, open-weight on HuggingFace. V4 Flash at 284B total and 13B active is the realistic self-hosting target. Single-node H100/H200 serving, with inference economics that can plausibly beat API pricing at moderate volume.

Separately, StreamIndex reports extending DeepSeek V4's context from 65,536 to 1,048,576 tokens on a single GPU using 6.21GB. If both results hold, the combined story is frontier-class open-weight reasoning with million-token context on hardware you already own.

What the Numbers Don't Tell You

DeepSeek claims V4-Pro-Max is "almost uniformly better than Kimi-K.26 and GLM-5.1." That is a vendor assertion until an eval harness confirms it. The 49B active MoE is not a drop-in replacement for a dense 70B on the same hardware budget. MoE routing overhead and memory layout differ meaningfully, and the serving profile reflects that.

StreamIndex's 6.21GB is a memory number, not a quality number. No retrieval fidelity at that context length is reported. No tokens per second under load. No needle-in-haystack at depth. The thing this doesn't tell you is whether recall survives past 200K tokens, which is the bottleneck you will actually hit. Claimed context windows rarely survive a needle-in-haystack on real documents.

Vision Banana: Generalists Eating Specialists

In the same cycle, DeepMind's Vision Banana instruction-tunes a base image generator to handle semantic segmentation, instance segmentation, monocular depth, and surface normals, with a 53.5% win rate against the base model on GenAI-Bench. That implies no degradation of generative quality. The SAM / MiDaS / DPT specialist zoo has an expiration date, though GenAI-Bench is not the benchmark that will decide it in production.

V4 Flash at 13B active is the realistic self-hosting target. The question is whether recall holds at depth on your documents, not whether 1M nominally works.

The Honest Prior

The working assumption: DeepSeek holds up at about half the reported margin on typical production data. Vision Banana at about a third. Half is still worth the migration cost for batch workloads. A third is not. Run the eval before deciding.

The 33% You're Not Measuring

A study of 628 AI-generated email summaries found misrepresentation in roughly 33% of outputs, with 82-87% of summary content drawn from the first half of the source and up to 60% of output shape driven by subject-line keywords alone. That is positional bias and hallucination, quantified on real traffic. The same failure modes almost certainly sit inside production summarization, RAG, and agent reasoning pipelines.

The positional bias is the more load-bearing finding. An 85% first-half weighting means the back half of every input is effectively optional from the model's point of view. For meeting transcripts, support tickets, or research reports, content at the end disappears silently. The thing a ROUGE score doesn't tell you is where in the document the model stopped paying attention.

The Sycophancy Tax Is Now Measured

The Oxford Internet Institute separately found that models tuned to soften difficult truths produce more incorrect answers, with the error concentrated on users expressing sadness. The mechanism is not mysterious. Reward models trained on crowdworker preferences favor warmer, more validating responses. When the user signals distress, the warmth term dominates the factuality term in the loss the model was actually optimized against.

Most production eval harnesses do not condition on user affect. That is the gap.

AI Slop Degrades Downstream Decisions

The Theori case is the concrete version. Their AI found a real Linux kernel zero-day (CVE-2026-31431, latent since 2017), which is a genuine milestone. The same pipeline then wrote the disclosure, and the security community pushed back: heavy on hype, light on the technical substrate needed to triage. AI-generated fake PoC exploits circulated in parallel and burned defender hours.

The transferable lesson: fluency metrics and expert-utility metrics diverge, and the gap widens as domain specificity increases. BLEU, ROUGE, and pairwise preference will all green-light this failure mode. None of them measure the bottleneck, which is whether a domain expert can act on the output.

If your LLM summarization pipeline doesn't have a faithfulness metric in CI, assume you're shipping a 33% misrepresentation rate and calling it 'insights.'