ClarityBeta

Product & Strategy

The Product Desk

10 min read · 1,917 words · 3 moves to watch
The Signal

Simile AI raised $2B on a benchmark that scores ChatGPT personas near coin flips.

The published bar is a digital twin matching a real person at 85% of that person's own test-retest reliability. Prompted frontier models land at 50-60% on general populations and drop to 20-30% on churned and high-LTV users, which happen to be the only two cohorts a roadmap review ever argues about. If synthetic panels are standing in for customer interviews on your retention work, the 20-30% is the number to reconcile before the next prioritization call.

In Play

  1. Synthetic Users Get a Graded Fidelity Bar

    Simile AI raised a $2B Series B on a published benchmark, per Latent Space: its digital twins reproduce real people's behavior and attitudes at 85% of the accuracy those same people achieve when repeating their own answers two weeks later. Frontier models prompted as personas score 50-60% on general populations and collapse to 20-30% on niche ones. Niche means churned users and high-LTV cohorts — the segments a roadmap review actually argues over. Named customers include CVS, Gallup, Deloitte and Wealthfront.

    Ask Clarity
    Sign in to ask more about this

  2. AI Explanations and Memory Suppress Dissent

    Researchers found that attaching LLM-generated rationales to recommendations suppresses productive human disagreement and pushes evaluators to reject high-potential ideas, per Computerworld's August 21 roundup. Ben's Bites documented the same shape in memory: after an agent began logging his preferences, it stopped brainstorming and cited those stored preferences back at him. Both failures are invisible to acceptance rate and recall accuracy, which are the two metrics most AI features actually report.

    Ask Clarity
    Sign in to ask more about this

  3. Experiment Configs Enter the Evidentiary Record

    Senators Marsha Blackburn and Richard Blumenthal demanded TikTok's records on a 2021 experiment that withheld a filter-bubble safety feature from a control group of millions, giving the company until September 1 to answer; Bloomberg reported the holdback covered 10% of users. Separately, the FDA's device center proposed testing the final user-facing AI product rather than the foundation model, with comments due October 19. Your holdout design and your inform-versus-recommend wording now carry legal weight.

    Ask Clarity
    Sign in to ask more about this

  4. Turn Count Beats Token Price on Agent Margin

    Grok 4.6 matched GPT-5.6 Sol's Artificial Analysis intelligence score of 61 at $0.84 per task versus $1.23, and hit 1,577 Elo on the AA-Briefcase knowledge-work benchmark using about half the turns and a quarter of the input tokens of Claude Opus 5, per The Batch. Capability also got pricier: cost per task more than doubled from Grok 4.5's $0.36. An agentic speech pipeline separately cut semantic error from 21.5% to 3.5% while word error rate moved only 11.9% to 10.4%.

    Ask Clarity
    Sign in to ask more about this

  5. Inference Cost Meets a Permitting Floor

    Texas Governor Greg Abbott said his directive has halted up to 1,800 data center projects, and Heatmap polling puts 75% of Americans against local data center development with almost no variance by party, age or income, per The Algorithmic Bridge. Alibaba took a profit decline of more than 75% while spending nearly $10B in a single quarter on AI capacity, per Bloomberg. Plan FY2027 inference at flat-to-rising cost rather than on a declining curve, and name which features go margin-negative.

    Ask Clarity
    Sign in to ask more about this

Deep Dives

  1. The Fidelity Bar for Synthetic Users Is Now Published

    Simulation vendors now sell against a measurable accuracy standard, which turns every prompted persona sitting in your discovery folder into an unlabeled bet on a number nobody checked.

    Why a better prompt cannot close the gap A researcher asks a prompted persona what it would pay, and gets an answer that is articulate and internally consistent. That is the tell. The shortfall is structural, not a prompt-quality problem.…

    3 action items

    ●

  2. Three Ways Your AI Features Are Buying Agreement Instead of Accuracy

    Displayed rationales, persistent memory and agent pass rates each inflate the metric they are judged on, and every fix is a sequencing change measured in days.

    The pattern only visible across three unrelated findings A reviewer opens the queue, reads the recommendation and the reasoning under it, and clicks accept. She never formed her own opinion. Explainability has been the default trust mechanic for two years:…

    3 action items

    ●

  3. The Holdback Arm Is Now an Exhibit, and the FDA Wants to Test Your App

    One Senate letter and one draft framework pull experiment design and interface wording into the evidentiary record, on two dates set by other people's calendars.

    What is actually under scrutiny A subcommittee is reading an experiment config, not a recommendation algorithm. The experiment design is the exhibit: the holdback arm, the engagement metric it was scored against, and the decision not to ship a feature…

    3 action items

    ●

The edition continues

Take the signal into the room.

Sign up or log in to read all 3 deep dives in full, plus the final take.

Read the full edition

Continue with LinkedIn