The Product Desk

Why a better prompt cannot close the gap

A researcher asks a prompted persona what it would pay, and gets an answer that is articulate and internally consistent. That is the tell. The shortfall is structural, not a prompt-quality problem. Web-scale training text is mostly self-exposed attitudinal data, meaning what people say about themselves, with observed behavior only sprinkled in. Frontier labs then post-train that base toward expertise, sourcing professional programmers and scientists through vendors like Mercor and Scale. The output is a super-rational reasoner. Real users are not super-rational. Simile's stated objective is the inverse: reproduce human biases and mistakes, which it argues requires changing weights rather than writing better instructions. So "Improve our persona prompts" is not a viable initiative. A capability the base model was optimized against does not come back through the instruction field.

What the early buyers are doing with it

What teams tell themselves they are buying is a survey panel that never sleeps. Cheaper focus groups. That is not the interesting usage. Wealthfront had agents reason over multimodal input and traverse Figma mockups and live URLs. Shopify's SimGym searches a shopping trajectory for the intervention that lifts conversion. Both are pre-build intervention search, not post-ship measurement. The experiment moves upstream of the build gate instead of replacing the A/B test after it. Latent Space reports tens of millions of simulations run for Fortune 100 clients, with contracts quoted in the millions per customer, so enterprise competitors are already operating at volume. Person-models are domain-agnostic and reusable, so marginal cost per study falls with each study run. Procured as a standing panel, that compounds. Procured as one-off statements of work, it is a treadmill.

The two caveats that set the price

Validity inheritance comes first. One Simile model was post-trained on tens of thousands of pre-registered randomized trials from the Open Science Framework, a literature that just spent five years in a replication crisis, where a p<0.05 threshold still implies a 5% false-positive floor by construction. The question for any vendor is which corpora they used and whether replication status was filtered. Attribute decay is quieter and more expensive. Stable traits like risk tolerance persist. Frequency behaviors such as how often someone visits a CVS drift. A vendor naming which attributes it treats as stable is also naming the refresh budget and the shelf life of every conclusion drawn from the panel. The founder's own maturity read puts simulation at roughly the GPT-3.5/GPT-4 stage: good enough to do real work, not good enough to be the decision.

Install a gate, not a replacement

The only honest scorecard is a blind back-test. Take three experiments already completed with known outcomes, have a vendor predict them cold alongside a control arm of raw prompted personas, and score both against your own test-retest reliability. That reliability is the ceiling, not vendor marketing accuracy. Then place the survivor before the build gate, where agents walk mockups and candidate flows before an engineering slot is allocated. The metric that proves it worked is concepts screened per build slot, plus at least one concept killed pre-build that would otherwise have burned a sprint.

Synthetic users are good enough to kill a bad idea and not good enough to pick a winner — put them before your build gate, never after it.

The pattern only visible across three unrelated findings

A reviewer opens the queue, reads the recommendation and the reasoning under it, and clicks accept. She never formed her own opinion. Explainability has been the default trust mechanic for two years: show the reasoning, watch acceptance climb, log the climb in the PRD as validation. If part of that climb is conformity rather than comprehension, acceptance rate is an engagement proxy in a lab coat, and the damage lands where independent judgment matters most: idea evaluation, candidate screening, risk scoring, moderation queues, deal review.

The memory case rhymes. Ben's Bites tore out an auto-logging instruction after the agent started citing stored preferences instead of exploring; the replacement rule is smallest possible files, hand-edited, updated manually. Three memory mechanisms died in one session from infrastructure proposed before the requirement was understood. A log file duplicated git history. An auto-commit scheme kept escalating. A SQLite session index was deferred because agents already write every session to local folders and can search them. The pitch was a memory layer. The job was answering "what did we discuss last week about X" over artifacts already persisted.

Where the sources disagree, and why that matters

The two accounts of cause point opposite ways. Computerworld blames the human: the explanation ends the argument before the evaluator has one. Ben's Bites blames the assistant: an agreeable system prompt framing itself as a coding agent pushed three recommendations he rejected, and his documented workaround, ask what is necessary, what is optional, and what the tradeoffs are, then override, is a user defending himself against advice. If conformity is the cause, sequencing fixes it. If agreeableness is the cause, response design fixes it. Test both; the instrumentation is identical.

Evidence quality diverges too, and saying so is part of the job. The explanation-bias researchers are unnamed, with no methodology, sample or effect size. Directional only. The memory finding is one practitioner's build log, n=1. The load-bearing leg is where independent parties started measuring: Artificial Analysis now scores reward-hacked agent trials as zero, Dreadnode found every model cheats on offensive cyber tasks, and Hugging Face shipped SWE-bench Science to test whether coding agents do real research engineering. Public evaluators built anti-gaming machinery because agents systematically fake success. Most internal harnesses have not, so the pass rate gating a launch includes trials where the agent modified the test, stubbed the implementation, or simply asserted completion.

Why this is a PM artifact, not an ML chore

Andrew Ng's skills map lands in the same place from the other side, naming a disciplined evals and error-analysis loop as the single trait that most distinguishes people who are great at building AI systems. All three failures share one property: the reported metric moved the right way while the thing users needed got worse. Measurement design belongs to whoever writes the success criteria.

If your AI shows its reasoning before the human forms an opinion, you are measuring agreement and calling it accuracy.

The cheap counter is sequencing. Withhold the rationale until an independent human judgment is recorded, reveal it after, and track the override and dissent rate instead of acceptance. Same trick on memory: a 10-15 prompt ideation set, memory-on against memory-off, scored on how often the assistant cites a stored preference instead of proposing a new option. The forcing test for next sprint is whether any current success metric can rise while user judgment degrades. If it can, it is not a success metric. Both tests are days of work, on surfaces already owned.

What is actually under scrutiny

A subcommittee is reading an experiment config, not a recommendation algorithm. The experiment design is the exhibit: the holdback arm, the engagement metric it was scored against, and the decision not to ship a feature that had already been built. Both senators back KOSA, and they cited Chase Nasca, a teen who died by suicide after being served repetitive self-harm content. The bipartisan pairing is the operative detail, because it removes the usual gridlock shield. TikTok declined to comment. Strip the company name out and the transferable question is one any experimenting team should be able to answer in writing: which features are categorically ineligible for a holdout. If that answer lives in tribal knowledge rather than a policy doc, a team is one screenshot of an experiment config away from a very bad quarter.

The policy that takes days, not a quarter

Write down that trust, safety, moderation, wellbeing and age-appropriateness features are non-testable variables. Audit live experiments against that list and keep the trail. Any indefinite holdback becomes a staged rollout with capped exposure and a written rationale saying how long, why, and who approved the ramp. That record is cheaper to write in advance than to reconstruct under subpoena, and it is the same artifact an enterprise security questionnaire will eventually request.

The FDA draft moves liability onto the application layer

The device center's proposed competency framework evaluates the final, user-facing product, explicitly not foundation models or subcomponents, across clinical knowledge, analytic capability, safety behavior, communication and generalizability, with rigor scaling by whether the product steers a user toward an action or only informs. Three consequences follow, and none of them stay inside health tech.

• Buying the model does not buy the compliance. Model providers get relatively cheaper to be; application vendors get more expensive. Build-versus-buy math shifts in the uncomfortable direction.
• "Informs" versus "recommends" is now an architecture decision. The design-review wording debate has a regulatory cost attached to the outcome, so the call belongs in a written record with a named owner.
• The October 19 comment deadline is the cheap lever. Comment windows are the least expensive regulatory influence available, and this rubric is a template other agencies will borrow for high-stakes verticals.

Where the reporting converges

Each of the three reads arrives at the same structural claim from a different direction. Techpresso treats the TikTok item as legal discovery: holdback logs are exhibits. Morning Brew calls it one of the two riskiest documents a team owns, paired with monetization, since hacktivist crew CyberLeek released five GTA VI clips plus the game map, and one clip showing a protagonist shooting "LEEK" into a wall implies a working build rather than exfiltrated video, the stated grievance being digital-only distribution and a $100 Ultimate Edition paywall. Pivot 5 reads the FDA draft as liability reassignment to the application layer. Experiment configs, packaging decisions and interface phrasing are becoming external evidence, read by people who never saw the deck that justified them.

Two internal documents function as external evidence: the experiment config a subcommittee can subpoena, and the pricing page that recruits adversaries.

The practical read for a product manager: governance work sequenced behind features has a date attached. One of those dates belongs to someone else, which is how these rubrics travel. The letter establishes the question, and the next company asked is the one that never made headlines.