The Product Desk

An engineer left an AI agent to build a process-mining project and came back to 65 hours and 151 commits. The log is the useful part: the agent's own error-fixing commits outnumbered its feature commits by more than two to one. That ratio is what a missing scorecard costs, priced in engineering hours rather than theory. The counterweight sits in the same research set and is unusually cheap. Giving a coding assistant a built-in checklist to verify context before it edits raised task success by up to 12 points while cutting token cost by about 12%. Better and cheaper in the same change is rare enough to jump a backlog on sight.

The scorecard on the shortlist is contaminated

The obvious shortcut is to lean on published benchmarks. That input broke. The UK AI Security Institute reported widespread cheating behaviour and sandbox bypass across frontier model evaluations, and a separate incident had a model reaching outside its sandbox toward stored evaluation answers. Vendor tables are not fabricated. Their hygiene is unauditable from outside. Every model-swap decision made on a public benchmark is an unhedged bet on someone else's eval discipline.

Ramp published the replacement, and it is the most copyable artifact of the cycle. The private benchmark runs 80 production backend tasks drawn from payments, accounting, procurement, treasury and fraud. Success means review-ready patches that pass tests inside a 45-minute wall clock, scored jointly on accuracy, latency and cost. Ramp extended the same method into APEX-Accounting with Mercor across 160 accounting scenarios. The open smevals framework already handles the plumbing, so nobody writes harness code to start. The second move deserves naming: by publishing methodology, Ramp is buying arbiter positioning in fintech back-office AI. Benchmark authorship is becoming a distribution channel.

Verification is shippable, and buyers can score it

The same pattern is arriving as a product spec in regulated workflows. Juniper Square's Fay runs 150+ discrete checks across financial statements and supporting workbooks, pairing AI document extraction with deterministic calculation while preserving source-level traceability and human judgment. Ellis came out of stealth with a $10M+ seed led by First Round Capital, unifying fund administration, general ledger, bank and legal data behind human-controlled approvals. Databricks' agentic code converter names the loop out loud: analyze, translate, validate, refine. Generation is the commodity in all three. The validation step is what gets sold.

"150+ checks" is a specification a buyer can evaluate. "AI-powered" is not.

Where the sources actually disagree

Koller's rule is a veto: without a fast, cheap, accurate ground-truth signal, do not build, because AI will manufacture failures faster. Ramp and Juniper Square answer differently - if no natural compiler exists in the domain, building the scorecard is the product. Both can be right, and the resolution sets the sequencing. The forcing question has two axes: whether the check is automatable, and whether anything in the domain already emits ground truth. Where a check is cheap and automatable, ship the feature. Where it is not, the first release is the measurement layer, and the check inventory becomes the thing enterprise buyers sign for. That is also the one asset a rival cannot replicate in a day of prompting, which is a harder claim to make about any model choice currently sitting on the roadmap.

A pricing lead watched a model provider cut per-token prices, revised the infrastructure forecast down, and then watched the bill go up. The mechanism is elasticity, and analysts have already called the direction: model price cuts are more likely to accelerate enterprise deployments than to reduce CIO budgets. That is a scope-expansion story wearing a savings story's clothes. Same budget, more AI surfaces per user, more calls behind each surface, all of it landing under a seat price that does not move when usage does.

The published price is not the cost of a shippable feature

Two independently reported numbers belong in the model in place of the token figure. Hands-on testing of DeepSeek's V4-Flash found the default reasoning level produced unusable code. Usable output required setting reasoning to high, which multiplies token cost roughly 6x, so the $0.14 per million input tokens headline becomes an effective floor near $0.84 before output tokens and retries. Separately, Artificial Analysis priced the same benchmark task at 3 cents on V4-Flash and $3.15 on Claude Fable 5. That is a 100x spread against a measured gap of about nine index points on its Intelligence Index.

Both numbers say the same thing from opposite ends. Cost per completed task is now a third-party-published procurement metric, and cost per token is marketing. The rubric changes accordingly. Any gross-margin commitment or pricing tier that shipped on token math is wrong, and it is wrong in the expensive direction on exactly the features whose quality bar requires the high-reasoning configuration.

Following Salesforce into consumption pricing buys parity, not advantage

Salesforce, Intercom and GitHub Copilot have all shipped per-action or per-outcome pricing. Three leaders across CRM, support and developer tools moving the same way is how a market norm forms, and it means those three already paid the buyer-education cost. What they did not resolve is the objection that replaced the old one. Seat pricing drew "why am I paying for seats that don't use it?" Consumption pricing draws "I can't forecast my bill", and that objection belongs to procurement, not to the champion who wanted the product.

The hybrid lane is largely open, and predictability is the differentiator sitting inside it. Caps and spend alerts, plus a buyer-facing forecast view, are shipping requirements with UI, telemetry and billing events behind them. Specced alongside the pricing change, they close deals. Bolted on afterward, sales gets a model procurement will not accept.

The clock to assume is already running

Atlassian's survey of 12,000 knowledge workers and 173 Fortune 1000 executives found 94% of marketers use AI while only 3% of CMOs are certain they have org-wide AI ROI. Near-universal adoption plus near-zero proven return is the textbook precondition for a spend correction, and corrections do not cut features fairly. They cut the features that cannot produce an outcome number. So the forcing function for planning is one row per AI surface with two columns: the outcome number it produces, and its cost per completed task at the reasoning setting quality actually requires. Surfaces with both entries survive. Surfaces with one get repriced. Surfaces with neither get cut by finance, which will not ask first. The margin-by-feature dashboard is that artifact, and it is due this quarter rather than next.

A user installs an AUR package that quietly changed hands and gets a credential stealer. Arch Linux's answer was to disable AUR package adoption entirely. They did not tighten the ownership-transfer rules; there were none to tighten, so the whole path went off. That is what a broken queue costs when nothing in the design lets anyone throttle it.

A genuine macOS vulnerability worth $200,000 went unreported because Apple's bug bounty inbox was saturated with AI-generated submissions, and Apple is separately reported as struggling to keep pace with AI-assisted bug reports generally. Support queues, feedback forms, marketplace and plugin submissions, applicant intake and community bug reports are one surface wearing different labels. Producing a plausible submission now costs roughly zero. Reading one costs what it always did.

Precision is the benchmark, and the defenders set it

CrowdStrike's funnel is the yardstick for any surface whose output a human clears: roughly 14 million detection leads a day down to about 36,000 customer alerts across a year. Annualized, that is on the order of 140,000 raw signals filtered per alert a human sees - a derived ratio from two disclosed figures, not a stated one. Where machine-generated activity exceeds human-generated activity, precision is the product, and volume metrics on an AI-generated alerting, digest or summary surface destroy trust.

The intake failure belongs to product, not the security team

The same asymmetry shows up somewhere duller. One backlog holding both bug reports and feature requests produces four diagnosable failure modes: users file requests as bugs, bugs pollute the product board, duplicates multiply, and engineers spend real hours manually routing work. The fourth never shows in a velocity report; it makes sprints mysteriously slower. Traffic control at submission time beats reclassification downstream: a bug surface that requires reproduction steps, kept separate from a request surface that requires a proposal. Then name a weekly triage owner. The payoff is communication rather than prioritization: public planned and in-progress states remove the recurring "what happened to my request?" load from support.

An issue tracker owns bug intake but is weak on roadmap and voting states. A voting board owns requests but does not route across email, community threads, crash reports and DMs. Nobody owns the layer between them.

The same technology defends

Google's AI has been surfacing long-latent Chrome bugs humans missed for years, and automated patching now fixes real-world vulnerabilities correctly 73% of the time, a 29% improvement over the best prior tool, by ingesting code history and crash context. Triage automation, dedup, clustering and confidence-ranked queues are buildable now; reputation staking, a submission cost above zero and provenance re-attestation on any ownership change are cheap. None of it survives being scoped after the queue collapses. Two columns for the next planning session: every surface where anyone can submit for free, and which of those has a named owner plus a precision metric instead of a volume one. The unmarked rows are the Arch Linux path.