ClarityBeta

Product & Strategy

The Product Desk

10 min read · 1,994 words · 3 moves to watch
The Signal

Agents write 70% of Uber's pull requests while only 5.4% of models refactor cleanly.

The gap closed with 3,600 skills that each end in a verification step, not with a stronger model. Two benchmarks found the same failure shape: an agent finds a good path once and cannot repeat it. So the autonomy demo your team is scoring this quarter tells you about the first run and nothing about the hundredth. The work worth funding is the verification step.

In Play

  1. Agent Reliability Is the Binding Constraint

    TheSequence reports that only 5.4% of models tested on SWE Refactor Bench complete a whole-repository migration without breaking observable behavior. Microsoft's 507-task ThinkingBox found the same shape: agents locate a good path once and cannot repeat it. Uber says agents are attributed on more than 70% of its pull requests — the difference is 3,600 skills that each end in a verification step, not a stronger model.

    Ask Clarity
    Try

  2. The AI Coworker Just Priced Itself At Zero

    Andrew Ng released OpenWorker under an MIT license with 25-plus workplace integrations and free local inference, per Simplifying AI. Ollama v0.33 registers inside Claude Desktop as a gateway provider, so one toggle swaps Qwen, DeepSeek, Kimi or GLM into Anthropic's own model picker. Tencent's Hy4 lists at $0.834 per million input tokens under Apache 2.0 weights. Capability is free; single sign-on, audit trails and per-tenant spend ceilings are not.

    Ask Clarity
    Try

  3. Document Ingestion Got A Scoreboard And An 8x Cost Dial

    Datalab's Marker v2 scores 76.0% on Allen AI's olmOCR-bench across 1,403 PDFs, against MinerU's 72.7% and Docling's 50.3%. The same tool runs 2.9, 7.4 or 23.7 pages per second depending on one mode flag, and the default changes with hardware — balanced on GPU, fast on CPU. Fast mode reads equations out of the PDF text layer instead of the rendered page, so math accuracy degrades with no error in the logs. Pin the mode in config before extraction quality varies by machine.

    Ask Clarity
    Try

  4. The Inference Cost Curve Is Contested

    a16z closed a $1.1B Machine Age Fund on the explicit thesis that 20-30% annual hardware supply growth cannot meet triple-digit compute demand growth, per TheSequence. The Wall Street Journal reports Nvidia has extended roughly $230B of lease and residual-value backstops, including $105B on a single OpenAI lease in Ohio. Falling list prices per token and scarcer guaranteed capacity are both real, which is why your FY27 margin model needs a flat-cost scenario next to the optimistic one.

    Ask Clarity
    Try

  5. Growth That Borrowed A Tailwind

    a16z crypto measured what happens when a forcing function disappears: Argentina legalized individual dollar purchases in April 2025 and monthly inflation fell from 25.5% to 2.1%. USDC's share of contractor pay settled at roughly one fifth of its peak — not zero. The premium users pay for digital dollars compressed from over 100% to about 4%, and they still pay it. That separates willingness-to-pay for access from willingness-to-pay for convenience. Compute the same residual for your own tailwind cohorts before FY27 targets get set on a flattered base year.

    Ask Clarity
    Try

Deep Dives

Uber Made Agents Shippable By Instrumenting Them, Not By Trusting Them

Two published benchmarks and one production write-up describe the same failure shape — agents find a good path once and cannot reproduce it — and only one of the three carries a fix you can copy.

What that 5.4% is actually measuring

An engineer watches an agent finish a repository migration, reruns the same task, and gets a different result. SWE Refactor Bench, from Navers Lab, Einsia.AI and Tsinghua, scores that rather than whether a patch looks plausible: a strict three-stage protocol on whole-repository stack migrations, asking only whether observable behavior survived. Almost nothing clears it. ThinkingBox, 507 tasks from Pittsburgh, Northwestern, UC Irvine and Microsoft, run in an isolated sandbox with strict policy adherence and real backend state transitions, names the mechanism a discovery-reliability gap. The model finds a workable trajectory and cannot reproduce it on the next attempt.

Worth noticing who writes the tests. Microsoft and Google Research are defining the stateful, repeated-attempt evaluation standard, so some version of it lands inside buyers' procurement. Repeat-run scores become a sales requirement before they become a marketing claim.

Uber's counter-example is a scaffolding argument

Uber's engineering write-up is the only production answer at scale in this source set: 3,600-plus agent skills across the software lifecycle, more than 30,000 executions a day, weekly active users of agentic tooling up 7x and requests up 9.4x since February 2026. The usage growth is the least interesting number in that list. Four of the six published authoring principles are product craft, not engineering.

  1. The skill description is a router, written in the words a user would actually type. Intent taxonomy is a product artifact.
  2. The body is steps only. Anything that is not a step gets deleted.
  3. Scripts carry fixed sequences; the agent carries judgment calls. This determinism boundary decides where reliability failures are even possible.
  4. Verification is a step of the procedure, for the output and for the skill itself.

The optional sixth principle is the one worth stealing: the skill reports its own friction upstream when it finishes. Its author calls those skills "actually getting better every week." Continuous qualitative research with no recruiting cost.

Underneath sits the asset no vendor sells: Uber's AI Context Graph, 24 million nodes and 80 million edges across more than 30 internal systems, exposed through 1,000-plus tool endpoints. The grounding substrate is the differentiator, not the model.

The same discipline halved the bill

Uber decomposed agent spend into a multiplicative chain, users x sessions x turns x requests x tokens x price, then worked each factor separately instead of negotiating a discount and calling that finance work. Prompt-cache lifetime tuning cut tokens per request. Resolving tool calls through the command line cut turn overhead. Code-mode batching claimed savings of 55-100% of tool-execution tokens. Aggregate: cost per 1,000 requests down 34%, cost per session down 52% from peak.

Down 52% from peak means the same output cost roughly twice as much before anyone measured it. Teams that have not decomposed agent spend should assume they sit at peak.

Where the dashboard stays green

The reliability failure has no alert attached. Traditional observability monitors execution; agent features fail at behavior, and during a behavioral failure the alert state is green. Hence "cognitive observability" crystallizing as its own category. Uber's second published lesson agrees: earlier static rate-limiting and quota-management approaches failed outright at multi-tenant scale, forcing adaptive, feedback-driven load management. Static thresholds do not survive agent traffic.

The 2x2 worth running: bare model versus scoped skill with a verification step on one axis, execution failures versus green-dashboard behavior drift on the other. The benchmarks say almost nothing works and Uber says most of its pull requests do; both hold, because Uber lives in the scoped-skill cell. Two caveats travel with it, Uber's own "your mileage may vary" disclaimer and the 55-100% batching figure, which is a claimed range with no external audit.

What to do

  1. Run each of your top three agent workflows five times this sprint and publish the consistency rate; ship autonomy above your stated threshold and require human confirmation below it

  2. Instrument the six-factor spend chain — users, sessions, turns, requests, tokens, price — and put a cost-per-successful-task baseline for your top three AI surfaces on the feature dashboard by the end of this sprint

  3. Add behavioral acceptance criteria to the AI feature spec you have in flight: evaluation pass rate, output drift threshold, and a named on-call owner for wrong-but-confident output

Andrew Ng Open-Sourced The AI Coworker; What Is Left To Sell Is Control

A free agent, a one-toggle model swap inside Anthropic's own desktop client, and a sub-dollar frontier model landed in the same week; the governance gaps all three share are your product spec.

The free stack runs end to end

OpenWorker ships under MIT with connectors for GitHub, Slack, Jira, Notion, Linear, HubSpot, Outlook, Gmail, Google Calendar and monday.com, any MCP server, local file and terminal access, and triggers that fire on a schedule, a Slack mention or an inbound email. Per Simplifying AI it returns finished artifacts — filled spreadsheets, drafted documents, a triaged inbox — not a transcript. Inference is bring-your-own-key for OpenAI, Anthropic or Google, or fully local and free through Ollama. Funded "AI coworker" startups pitch that feature list. This one costs zero dollars and arrives with a distribution list.

The substitution is one toggle deep

A user opens the menu bar in Claude Desktop and flips one switch. Ollama v0.33 registers as a third-party gateway provider, so local Qwen, DeepSeek, Kimi and GLM models appear in Anthropic's own model picker, running on the user's hardware, telemetry off by default. Flip it back and web search and the rest of the first-party experience return intact. The read for a roadmap is not that Anthropic is in trouble. It is that if the vendor's own client is swappable in one click, every client built after it is too. Model choice became a user setting rather than a product decision anyone owns.

What the price floor actually is

Tencent's Hy4 sets it. 770 billion total parameters, roughly 49 billion activated per token, 78 layers, more than a million tokens of context, Apache 2.0 weights, hosted at $0.834 per million input tokens. Reported results are 85.4 on Terminal Bench 2.1, a DeepSWE jump from 28.0 to 64.3 generation over generation, and 2.99 of 4 in a blind 203-task engineering evaluation judged by 163 experts, ahead of Kimi K3 at 2.94 and GLM-5.3 at 2.92. It shipped 53 days after Hy3 at 2.6x the parameters and 4x the context.

Treat the ranking as noise and the price as real. A 0.05 gap in a human-judged evaluation sits inside error, and every performance figure traces to the release announcement rather than independent reproduction. The claim worth underwriting is Z.ai's: GLM-5.3 open weights beating Claude Opus 4.8 on Z.ai's in-house code bench at under half the output tokens. Token efficiency is now marketed next to quality, and it is the axis that lands in gross margin.

The gaps are the spec

Separate what is being pitched from what is shipping. Sources converge on where free stops, and that list is a competitive slot nobody is racing to fill.

  • OpenWorker is single-user desktop. No single sign-on, no multi-tenant governance.
  • Its permission tiers (read, write_local, exec, external) grant approval once per tier. Pair that with terminal access and email triggers and one click buys a wide blast radius.
  • Auto Company's guardrails are a three-item denylist — repository deletion, force pushes, credential leaks — so anything destructive off the list is permitted, and it runs with no spend ceiling.
  • Bot Directory distributes 170 agent prompts by pull request with no vetting gate. An unreviewed prompt supply chain.

The enterprise version of that list is roughly a two-sprint specification: allowlist scopes instead of denylists, per-session re-consent for exec and external actions, hard per-tenant spend ceilings, immutable audit logs.

The compliance tax nobody priced

The Information reports the Trump administration is drafting a rule to curb China's remote access to chips, export-control logic extending from hardware to services. Every model in the frontier comparison set above is Chinese-origin. The forcing function is segmentation, not selection. Consumer and internal workloads route to the cheap tier where the arbitrage is real. Regulated workloads stay on provenance that survives a security questionnaire.

When capability is a toggle and weights are free, the only things left to sell are the artifact you hand the customer and the audit trail that proves where it came from.

What to do

  1. Run the free-clone test this sprint: install OpenWorker with local Ollama inference, attempt your product's top three user jobs end to end, and document exactly where it fails on authentication, multi-user state, audit and data governance

  2. Ship a provider abstraction with bring-your-own-key support and cost-quality routing this quarter, backed by a standing evaluation harness on your own task set rather than public benchmarks

  3. Add model provenance, weight origin and inference geography to your security questionnaire responses and architecture decision records this quarter

Your FY27 Margin Model Assumes Cheaper Tokens; Capital Markets Disagree

Two streams of reporting point opposite directions on what inference will cost you, and the disagreement resolves once you separate list price from the price of guaranteed capacity.

The contradiction, reconciled

A pricing lead pulled three newsletters into one doc this week and found them arguing with each other. The Information's read is that the cost of intelligence fell again, with Tencent previewing Hy4 on August 28 and Z.ai shipping near-frontier weights at what the reporting calls "superlow" cost, while Salesforce overhauls how it bills for AI. TheSequence and The Bear Cave describe the opposite constraint, and they are not wrong either. Both files hold. They price different goods.

List price per token falls with architectural efficiency and open weights. Guaranteed capacity gets more expensive, because it is being forward-booked years ahead. Anthropic locked roughly 460 megawatts of NVIDIA Vera Rubin capacity for six years at about $45B, starting late 2027 at NScale's Monarch campus in West Virginia. a16z closed a $1.1B fund on the thesis that 20-30% annual supply growth cannot meet triple-digit demand growth. Frontier labs buy at treaty. Everyone else buys at spot, and scarcity pricing reaches spot first.

The invoice gets set in the financing layer

This is the part that never appears in a price-per-token comparison. The Wall Street Journal reports Nvidia has extended backstops that could leave it responsible for roughly $230B in lease obligations and residual-value deals, including a $105B backstop on a single OpenAI lease in Ohio and up to $125B guaranteeing that GPU collateral value will not fall below a set level. That is a vendor balance sheet standing behind the unit economics quoted on a pricing page.

ProviderCapital positionMarket signalPlanning consequence
Nvidia~$230B of lease and residual exposureDescribed as "a banker to the AI boom"Switching GPU clouds does not diversify the credit — one balance sheet sits behind several of them
CoreWeave$35B on-balance-sheet debtYields blown out to 13%; executives on pace to sell $4B, about 10% of the company, in 2026Vendor continuity risk, not a procurement footnote
IREN~$6B capital employed at June 30, 2026$25-30B planned FY27 capital spending on $4B forecast recurring revenueA 4-5x balance-sheet expansion funded externally; assume dilution or a funding failure
Lambda~$1B of short-dated debt arranged by JPMorganThird major raise since May, inside $400B-plus of AI debt issued globally this yearShort paper funding fast-depreciating assets is where a demand wobble becomes your price change

The market-structure tell is that semiconductor indices and Nvidia itself traded down after a large earnings beat, the pattern a crowded trade makes on the way out. Power, not silicon, is now named as the binding constraint, which positions whoever controls electricity as the winner.

The counterweight arrives after the planning window closes

OpenAI and Broadcom's Jalapeño inference chip benchmarked ahead of every NVIDIA, AMD and Google part SemiAnalysis tested on performance per watt, going from blank sheet to tape-out in nine months with help from OpenAI's own models, at gigawatt scale from late 2026. That is genuine downward pressure, and it arrives into 2027. The numbers were supplied by OpenAI and the agentic long-context suites remain untested. Cite it in a renewal conversation, and keep it out of next year's margin plan.

What a product team does with a contested input

The anti-pattern now has a name. Runable subsidized more than a trillion tokens in 90 days and posted negative gross margins, on a $21M Series A at a $65M valuation. The forcing question for the next roadmap review is whether a feature clears margin at today's token price or only at a price someone else has promised to deliver. Everything in the second category is a bet on another company's capital structure, which is a bet a team can take as long as it is written down as one.

Price every AI feature as if tokens never get cheaper. If they do, you have discovered upside instead of explaining a hole.

What to do

  1. Re-run unit economics for your top three AI features at flat and at plus-25% inference cost this quarter, then re-price or kill anything that only clears margin on an assumed decline

  2. Add a hard cost-per-inference ceiling and a degraded-mode specification to every AI item in the backlog this quarter, naming which capabilities shed first and what the user is told

  3. Quantify the share of inference and training spend sitting with non-hyperscaler GPU clouds this quarter, and build a tested 30-day migration runbook for anything above 25% concentration

The bottom line

Every layer of the AI stack is converging on the same scarce good: proof. Proof that a run repeats, that a cost is decomposed to its factors, that an output was right rather than merely returned, that a supplier still exists at renewal. Capability arrives free and on a two-month cadence, so the artifacts that survive it are the ones you author — gates, ledgers, ceilings, escape hatches. Pick the one AI surface you would defend in front of a procurement team, and give it a repeat-run gate, a per-task cost ceiling, and a named owner for confidently wrong output before you add another capability to it.