The Product Desk

Why instruction tuning never wired this

An engineer files a ticket: the agent answered a question that had no answer. The model had the signal. A linear probe reading internal activations separates solvable from unsolvable problems at 0.939 mean AUC across 11 models, and that geometry is largely present before instruction tuning, per The Sequence's read of the USC and Arizona State work. Refusal was trained on harmful requests, so the unanswerable-question signal never got wired to it. The recognition direction and the safety-refusal direction sit at a mean cosine similarity of roughly 0.087, effectively unrelated.

Steering the recognition direction moves abstention by 33 to 52 percentage points. That turns a qualitative epic into two numbers a release can be held to: abstention rate on a labeled set of impossible or under-specified queries pulled from real logs, and a hard regression ceiling on answerable queries. Teams tell themselves this is a research quarter. The honest scope of the first test is one engineer and three days, and the output is a metric a reliability OKR can name.

Structure is substituting for model capability

The same pattern appears one layer up. Procedural Graphs, from Google with Georgia Tech and Peking University, store editable (procedure, relation, procedure) triples: a "what to do" graph where a knowledge graph stores "what is." They soft-bias a ReAct agent instead of hard-constraining it, and they self-evolve offline behind a validation gate that vets added, deleted and updated topology. PG is model-agnostic. It runs on whichever vendor a team already pays. A competitor can adopt it on the same contract this quarter, so the only advantage left is packaging speed. PG-guided Claude, Gemini and Grok solvers set or match best scores across HotpotQA, MultiChallenge, GDPval, ALFWorld, τ-bench, BFCL and EnterpriseArena, with the largest survival lifts on EnterpriseArena.

The two objections that decide the review

Reliability practitioners have already written the reviewers' arguments. Sylvain Kalache's comprehension debt holds that every routine incident automation resolves is a rep a human did not get, so competence erodes exactly where automation cannot help. Sai Sandeep Koneti's extension of blast radius from deployments to decisions is sharper: before a decision gets automated, define how far a wrong one propagates, and whether its output is absorbed back into the system to influence later decisions.

If the abstention is not logged with what the model saw, reviewers cannot audit it, and the wrong refusals will not surface until a customer finds one.

Both bodies of evidence converge on one point: the reliability work sits in the scaffolding around a model, not inside the next base model. Where they pull apart is what the PRD has to reconcile. The research optimizes for the model producing the correct refusal. The operators care whether a reviewer can see the input the system acted on and the reason it abstained. The forcing function: no abstention lever ships until the log carries both of those fields.

The attach surface nobody has claimed

Uptime Labs publishes on training early-career responders and Gremlin on Kubernetes disruption budgets. The training-under-automation product line is unowned. A team already holding incident data, system topology and resolution history can build a practice mode: replay a real past incident against a synthetic environment, let an engineer work it, then show what the agent did. Second product, same data, and it answers the strongest objection to the first. That is inference from a thin evidence base; five customer interviews will tell you whether the pain is funded or merely felt.

What buyers accept instead of the word

Pulling a load-bearing brand off top products immediately before your largest customer conference is not SKU hygiene. That is the timing you choose when the name has started costing you deals. Wherever you currently write "AI agent," a buyer now silently appends "prove it" — and only three numbers survive procurement: resolution rate, cycle-time reduction, and cost per unit of work. If a claim cannot carry one of those, cut the claim. For the next 60 days, while a category-defining competitor is mid-rename, displacement messaging built on deployed outcomes has an unusually clean run.

The political frame moved the same direction. The Information reports the AI conversation shifting from existential risk — bioterrorism, agent swarms, hacking — to tangible harms voters feel: job losses, erosion of critical thinking, and reduced parental supervision of children. Democrats are favored to take the House and possibly the Senate in elections a couple of months out, Obama has urged party leaders to focus on AI oversight, and Sanders has a bill that would jail people for building superhuman AI. That second list is not a lab problem. It is a spec problem: age assurance, guardian visibility, human-in-the-loop defaults, and designs that build skill instead of replacing it.

The rulebook is already being drafted, privately

Anthropic, OpenAI and Google have been in private discussions about jointly creating a testing and auditing standards body, The Information reports, and those talks predate Amodei's weekend essay calling for industry coordination — which makes the public advocacy an announcement layer on a process already running. At an OpenAI companywide town hall the same week, Altman told staff he supports such an organization and that the major labs will have to build it themselves, without U.S. government support.

Follow that to its delivery mechanism, because it decides your timeline. No federal funding means no federal enforcement, which means the standard arrives as voluntary conformance enforced by procurement: three new questions on a security questionnaire, roughly two to four quarters after any criteria go public, and a deal that sits until you can answer them with stored evidence rather than assurances.

Where the two readings disagree

One reading treats the pacing essay as a commercial non-event that changed nothing about the capability curve you plan against. The other shows the private process underneath it. Both are right, and the reconciliation is the actionable part: ignore the essay, build the artifact. The effort itself is fragile — early-stage, unannounced, no governance structure, no funding model, no membership list — and three direct competitors writing the rules invites antitrust and regulatory-capture scrutiny that could stall the whole thing.

The AI safety standard won't arrive as a law — it'll arrive as three new questions on a security questionnaire, and the team with reproducible eval logs answers them in a day instead of a quarter.

Which is exactly why you build the capability and not the certificate. Versioned evals catch regressions whether or not a coalition ever launches. Audit logs shorten incident response. A governance one-pager shortens enterprise sales cycles this quarter. None of that work is stranded if the talks collapse.

The method matters more than the benchmark

ToolGrad's contribution is an inversion. The conventional pipeline invents a plausible user prompt, then sends search agents hunting for an API chain that satisfies it — often landing on a suboptimal chain with an unverified label. ToolGrad builds the chain, actually executes every call, then back-writes the prompt that would have produced it, so the label is correct by construction. Google reports a 99.8% pass rate across ToolBench's 16,000 APIs against the query-first baseline.

That makes ToolGrad an evals methodology wearing training-data clothing, and it is the cheaper half to steal. Run the same inversion against your own tool surface — build and execute chains, back-write prompts, wire the set into CI as a release gate — and you have the measuring instrument every subsequent model swap needs. The commercial subtext is hard to miss: a lab published the recipe for replacing its own premium function-calling SKU with a 12B open model, which tells you where it expects margin to survive, and it is not in tool orchestration.

Cost-per-capability is now the competitive axis

Cognition's SWE-2 sits just behind Fable 5.1 on raw capability at 64% less cost, achieved by training every reasoning-effort level in one reinforcement-learning run rather than tuning a single operating point. Two consequences for your spec. First, expose reasoning effort as an explicit parameter behind your agent features, and consider mapping it to pricing tiers, because the entire cost curve is now tunable. Second, provenance became a procurement field: SWE-2 is post-trained from Kimi K3, Moonshot's open-weight base, and Cognition's own evaluation puts K3 at 54.5% on a Simplified Chinese censorship test against SWE-2's 95.2%. The risk is mitigable — but only for teams who test it, and that benchmark is the vendor's own, so replicate it before it enters a PRD.

Someone already ran your retrieval cost experiment

Pinterest published the full recall-versus-footprint curve at billion scale on 100 million GraphSage embeddings — the vendor-neutral evidence a serving-cost business case usually lacks.

Per-use-case selection through online A/B tests produced 20-30% serving savings, and the winning pattern is hybrid precision: quantize the on-disk embedding store, keep centroids at full precision. If you own search, recs or RAG, own recall as a product metric with a floor above 90%, or the 93% footprint reduction wins the infrastructure argument and your relevance quietly loses twenty points.

The counterweight: supply, not price

All of that says your cost curve is yours to engineer. The supply curve is not. Nvidia's own disclosures, which Martin Peers and Qianer Liu at The Information built out of SEC footnotes, show three customers each above 10% of sales driving 44% of revenue in the first half of the fiscal year ending July — against two at 36% last year and none above the threshold in fiscal 2023. A third whale ramped while total concentration rose only eight percentage points, so the demand mix is churning, not merely growing. Huang is now funding neoclouds explicitly to manufacture new customers, so a well-capitalized set of providers has an active mandate to win your workload. Meanwhile SpaceX is overhauling its data center build-out with a possible slowdown, and personal AI app Instinct faces a compute crunch severe enough to potentially trigger a round driven by GPU access rather than product milestones.

Capital is abundant and models are abundant; capacity is not — so engineer your margin now and underwrite nothing on tokens getting cheaper.