ClarityBeta

Product & Strategy

The Product Desk

6 min read · 1,265 words · 3 moves to watch
The Signal

A team swapped models three times last quarter chasing a four-point eval bump and shipped

The numbers this week argue the harness is the product: Agentic Harness Engineering took Terminal-Bench 2 from 69.7% to 77.0% (past the 71.9% Codex-CLI baseline), HALO pushed AppWorld from 73.7 to 89.5 by rewriting its own harness, and IBM's Granite 4.1 8B matched Qwen3.5 9B on 19.5x fewer tokens.

In Play

  1. Harness Engineering Beats Model Upgrades — By a Lot

    Multiple benchmarks confirm: optimizing prompts, tools, and middleware delivers larger quality gains than switching models. Terminal-Bench 2 went from 69.7% to 77.0% via harness alone. IBM's 8B model matched a 32B MoE through architecture optimization. LangChain launched Harness Profiles for commercial versioning.

    Ask Clarity

  2. Token Costs Hit 10-15x in 6 Months — Anthropic's Extraction Era

    Survey of 15 companies confirms AI token spend grew 10-15x between Oct 2025 and April 2026. Individual devs burning $500/day. Anthropic offers zero discounts even at $5M+/year, is silently nerfing Claude Code, and banning companies. Open-weight alternatives at $1-3/M tokens are the pressure relief valve.

    Ask Clarity

  3. Agents Became Autonomous Buyers — Three Platforms Shipped Simultaneously

    Stripe's Link CLI (agent payment credentials), Cloudflare's self-provisioning (agents create accounts/deploy), and Cursor's SDK (embedded agent runtime) shipped in the same week. An agent can now write code, pay for hosting, and deploy — autonomously. Your user model needs a third row: not human, not service account, but agent with delegated authority and a budget.

    Ask Clarity

  4. AI Dev Tool Security: 8+ Critical RCEs in a Single Week

    Claude Code (CVSS 10.0), Cursor (RCE via git clone), GitHub Enterprise (RCE via push), and 73 malicious GlassWorm extensions in Open VSX all disclosed in one cycle. AI tooling ships with 2005-era security practices. 88% of GitHub Enterprise Server instances remain unpatched. Exploit weaponization now takes under 12.5 hours.

    Ask Clarity

  5. Your Interface Is Becoming Someone Else's Tool Library

    Adobe exposed 50+ tools across 8 apps to Claude as the orchestration layer — explicitly conceding the interface to a chatbot. SaaS products that are 'routing friction' rather than systems of record face existential agent bypass risk. The question isn't 'does our product have an API' — it's whether AI agents reach for your tool when orchestrating a workflow.

    Ask Clarity

Deep Dives

Harness Engineering Is Your New Competitive Moat — Not Model Selection

The Data Is In: Orchestration Beats Intelligence

The traditional assumption — 'wait for the next model upgrade to hit our quality bar' — is now empirically wrong. Multiple independent benchmarks released this week confirm that the harness (prompt design, tool definitions, error recovery, middleware) is a larger lever than model choice for AI feature quality.

Agentic Harness Engineering improved Terminal-Bench 2 pass@1 from 69.7% to 77.0%, beating a human-designed Codex-CLI baseline at 71.9% — and it transferred across model families while reducing token use by 12%.

HALO showed even more dramatic results: AppWorld improvement from 73.7 to 89.5 on Sonnet 4.6 through recursive self-improvement of the harness, not the model. These aren't marginal gains — they're the difference between a demo and a production feature.

The Cost Dimension Makes This Urgent

IBM's Granite 4.1 8B used only 4M output tokens on the Artificial Analysis Intelligence Index versus 78M for Qwen3.5 9B — a 19.5x efficiency gap that's entirely about harness and architecture optimization. Shopify fine-tuned a smaller open-source model on Flow-specific data and achieved higher accuracy, lower latency, AND lower cost than general-purpose frontier models. When your inference costs are growing 10-15x in six months, a 19.5x efficiency difference isn't a nice-to-have — it's the difference between a viable product and a margin crisis.

Commercial Infrastructure Is Emerging

This isn't just research anymore. LangChain launched Harness Profiles — versioning per-model prompts, tools, and middleware configurations. DeepAgents Deploy shipped low-code agent deployment with LangSmith tracing. The pattern is clear: harness quality is becoming a managed, versionable, evaluable artifact alongside your code.

What This Means For Your Team

  • Model selection is now a product decision, not an engineering default. Features where errors cost minutes belong on cheaper models with optimized harnesses. Features where errors cost hours earn frontier models.
  • Your harness — the prompts, tool definitions, error recovery patterns — should be versioned like code and evaluated per model family.
  • Netflix's LLM-as-a-Judge blueprint (600 golden examples, 83-92% accuracy) provides the evaluation framework to measure harness improvements objectively.

What to do

  1. Create a harness versioning and evaluation pipeline for your top 3 AI features this sprint

  2. Run a model-tier routing analysis: classify every AI feature as 'errors cost minutes' vs 'errors cost hours' and assign model tiers accordingly

  3. Build 600 expert-labeled golden examples for your primary AI feature following Netflix's blueprint

  4. Evaluate Shopify's approach: benchmark a fine-tuned small model against your current frontier API for your most well-defined use case

The Token Cost Reckoning: 15x Growth, Zero Discounts, and the Vendor Lock You Didn't Price

The Numbers From 15 Companies Are Worse Than Expected

A staff engineer at a late-stage fintech watched one of her developers burn through a day of Claude Code and realized the bill looked like a second salary. That is the texture behind Gergely Orosz's survey of 15 companies, which is the most granular AI cost data available this quarter. Token spend grew 10-15x in six months at both a large enterprise and a seed-stage startup. Individual developers are spending $500/day on Claude Code, which staff engineers say has 'practically doubled' employee costs. One seed-stage company went from $200/dev/month to $3,000/dev/month.

About half of surveyed companies are on the 'let it rip' plan, which is a pitch-deck word for unforecasted cost. The Series D fintech that called its token spend 'unsustainable' at an all-hands is the preview.

Anthropic's Extraction Era Is Real

Anthropic offers zero enterprise discounts even at $5M+/year spend. Teams describe silent Claude Code nerfing, outright bans, and aggressive price increases, while nearly every company in the survey is Claude-dependent. What customers actually bought is not the API call. They bought every prompt, eval, and guardrail tuned against one model family, and that is the switching cost that keeps them paying.

The Cost Lives in the Wrong Budget Line

The PM decision this quarter is a budget-line decision, not a model decision: do tokens belong in COGS or R&D? If tokens sit in COGS, the product team owns unit economics and a 3x reduction in retrieval spend becomes a roadmap item with an owner. If tokens sit in R&D, nobody owns it and the bill keeps growing. One RAG app consumed 10.4M tokens via naive Supabase MCP integration. The same job ran on optimized infrastructure for 3.7M tokens. That is a 64% reduction from an architecture choice, not a model change.

Open-Weight Models Are the Pressure Valve

Open-weight pricing has collapsed to $1-3/M output tokens (Qwen 3.5 Plus at $3/M, MiMo-V2.5 Pro at $1/$3/M). LangChain staff explicitly state closed models are 'too expensive for many agent workloads.' Features that were economically impossible at $15-60/M tokens become viable at these prices. Three Chinese labs dropped frontier models in a single week: Alibaba's Qwen 3.6 Max, Moonshot's 1T-parameter Kimi-K2.6, and MiniMax M2.7 (fully open-source, 56% SWE-Pro).

ProviderOutput Token CostEnterprise Discount
Anthropic (Claude)$15-60/MNone at $5M+/yr
Qwen 3.5 Plus$3/MOpen-weight
MiMo-V2.5 Pro$1-3/MOpen-weight
MiniMax M2.7Free (OSS)N/A

What to do

  1. Audit every AI-dependent feature for cost model accuracy this sprint — re-forecast COGS using current token pricing at 10x whatever was modeled 6 months ago

  2. Map Anthropic dependency across customer-facing features AND internal dev tooling — identify which capabilities break if prices rise 2-3x

  3. Implement per-feature token budgets with hard circuit breakers and real-time alerting before end of quarter

  4. Benchmark open-weight models (Qwen 3.5, MiniMax M2.7, Kimi-K2.6) against your current provider for your top 3 AI features

Your AI Feature Stack Is a Security Liability — 8+ Critical RCEs This Week Alone

AI Tooling Ships With 2005-Era Security

A security engineer opened her advisory feed on a Tuesday and counted eight AI-adjacent tools with CVSS 9.8-10.0 vulnerabilities disclosed in a single week. Claude Code hit 10.0 for a symlink sandbox escape. Cursor's AI agent allowed arbitrary code execution via malicious git clone. GitHub Enterprise Server exposed millions of repositories via crafted push requests. None of these are novel vulnerability classes. They are hygiene failures shipped at startup speed.

The evaluation question for an AI framework is not which one has the best features. It is which one will not be a 10.0 headline next week.

The Weaponization Window Collapsed

Sysdig observed attacks on LMDeploy within 12.5 hours of disclosure, without a public proof-of-concept, because advisory text alone was enough for an LLM to generate the exploit. MOAK autonomously exploits 98% of Known Exploited Vulnerabilities using publicly available models. Disclosure-to-exploit is measured in hours now. A patching SLA written against a weekly cadence is a patching SLA for a threat model that no longer exists.

Supply Chain Attacks Are Industrializing

73 malicious GlassWorm extensions appeared in Open VSX in April 2026 alone. The TeamPCP/UNC6780 campaign hit npm, PyPI, and Docker Hub within 48 hours. Dependabot carried the attack through Bitwarden CLI's merge gates. The automation did exactly what it was configured to do, which is the uncomfortable part. LiteLLM's SQL injection exposed every downstream API key (OpenAI, Anthropic, Bedrock) via pre-auth attack, exploited within 36 hours.

The PM Responsibility

Separate the thing being pitched from the thing being done. The pitch is "adopt AI tools to move faster." The thing being done is adding a new attack surface with its own incident timeline. 88% of GitHub Enterprise Server instances remain unpatched, and DPRK's HexagonalRodent is using Cursor and ChatGPT to vibe-code malware loaders. The forcing function for this sprint: no AI tool enters the stack without a named owner, a patch SLA measured in hours, and a documented kill switch. Teams that write that policy this sprint do not run the fire drill next quarter.

  • Claude Code: CVSS 10.0 (sandbox escape, patched v2.1.64)
  • Cursor: RCE via git clone (patched v2.5)
  • GitHub Enterprise: RCE via push options (88% unpatched)
  • LiteLLM: Pre-auth SQL injection exposing all API keys
  • Open VSX: 73 malicious extensions in one month

What to do

  1. Verify Cursor is updated to v2.5+ and GitHub Enterprise Server is patched across all environments — escalate to platform team within 24 hours

  2. Add mandatory security review gate to your AI tooling evaluation process — no AI framework enters your stack without a vulnerability assessment

  3. Audit your LLM proxy/gateway layer for credential isolation — specifically verify that provider API keys are not stored in a single queryable database

  4. Implement dependency cooldowns across your CI/CD pipeline and pin container images by digest, not tag

The bottom line

The AI performance lever just moved from 'which model' to 'how you orchestrate it' — harness engineering delivers 10-20% quality gains without any model change, while your token costs grew 10-15x in six months and Anthropic refuses discounts at $5M+/year. Meanwhile, Stripe, Cloudflare, and Cursor simultaneously shipped the infrastructure for agents to buy, deploy, and operate autonomously — and 8+ AI developer tools disclosed critical RCEs in the same week. The PMs who win this quarter are the ones investing in harness optimization over model shopping, building multi-provider abstraction before Anthropic's next price hike, and treating every AI tool integration as an attack surface with a 12-hour exploitation window.