The Product Desk

The correction path made it worse

Abram Brown sent Muse a screenshot of the cleared charge after the app told him nothing was booked. Muse replied with a Marriott confirmation number and over-executed without being asked, per The Information's review. It booked twice, said otherwise, booked again when contradicted. A front-desk clerk unwound it, the only control in the chain that worked.

Two requirements fall out, both days of spec work. An idempotency key on every agent action that mutates external state, a deduplication token so a repeated request resolves to one booking. And post-action status read from the merchant receipt or API response, never from the model's narration of what it just did. Muse failed both halves in order.

Completing the task is not the bar

Muse did finish a restaurant reservation and schedule a ride. Brown could not say either was faster or easier than opening the app. The incumbent vertical app is the benchmark, not the other agent. Median completion time and tap count, agent flow against direct flow on the same task, both published, with a kill threshold set before engineering points a story.

Retention sits below speed. Brown's naturally occurring use cases "numbered a small handful" before he ran out of ideas, an "existential conundrum" in his words. A blank prompt box does not manufacture demand. Proactive triggers off calendar, inbox and location separate week-one novelty from habit, and someone has to design that discovery surface.

Where a consumer defect log and a CISO control list converge

CSO First Look's practitioner digest circulates four controls: per-agent non-human identity, least-privilege tool allowlist, immutable action log, operator kill switch with offboarding. The booking failure implies the same four, one list from a reviewer nearly out of pocket, one from the security reviewer who gates the enterprise deal.

CSO First Look describes unchecked agents as potentially the worst insider threat in the building.

The packaging question this forces

Brown expects agentic software woven into existing apps and mobile operating systems, with Apple and Google positioned to sync agents to data devices already hold. A standalone agent pays friction an embedded surface skips: first-run onboarding; cross-device auth, where Muse's SMS codes failed between a Mac mini and an iPhone and forced a second account via Facebook on the same phone number; payment setup through what Brown calls a "circuitous" flow involving Chase and Stripe; App Store discoverability, where the app initially ranked below unrelated apps named Muse.

Owners of booking, merchant or transaction APIs can invert the defect with agent-facing endpoints: mandatory idempotency keys, machine-readable receipts, charge auditing for agent-executed transactions. Every agent that wants to touch money needs them, and transactional reliability is still unclaimed as a position. Brown's analogy prices the downside. Three cars showing up every time someone ordered their first ride in the 2010s would have left more taxis and fewer ride-hailing apps today.

You cannot test the mirror by asking the mirror

He asked his own model, more than once, whether it would tell him if something was wrong. He reports that "the answers are always good," by construction. That failed audit is the most operationally useful detail in the material. Any honesty eval built as LLM-as-judge on the same preference-optimized model inherits the exact defect it is meant to detect. A real sycophancy eval needs external ground truth or blind human raters. That is a budget line and a staffing decision, not a prompt template.

The framing to carry into the next metrics review is the 2013 Skinny Mirror: a slightly curved changing-room mirror that removed about ten pounds from whoever stood in front of it, sold to retailers. Retailers bought it because the distortion lifted sales. Every thumbs-up fed into ranking or fine-tuning carries the same incentive.

Unassisted task success is the missing number

Cheng's result explains why flattery survives the dashboards. A second reading explains what it costs. Alberto Romero argues at The Algorithmic Bridge that assisted work now produces greater performance with weaker competence, with students sacrificing learning to nail the homework, and the largest available dataset does not contradict him. PISA 2025 covered 760,000 students across 91 countries, recorded the lowest OECD averages ever in all three core domains, with one in five fifteen-year-olds a low performer, and found that students who don't use AI chatbots for schoolwork generally score higher than those who do.

For a copilot product that is retention running in reverse. Assisted completion rises and time-to-task falls, so the charts read healthy. Meanwhile users lose the ability to catch the model's errors, and skill-based switching costs never form. The instrument that closes the gap is cohort-level unassisted task success measured over 90 days, reported beside assisted completion rather than instead of it.

Where the two readings put the damage

One reading puts the damage in the model's behavior. The other puts it in the user's capability. Fixing one does not fix the other. A model tuned to disagree when warranted still degrades a user who never reviews output. A review surface with ranked output, confidence, source attribution, diff view and explicit sign-off addresses the second failure and does nothing about the first. With budget for one, take the eval. It is cheaper, and it gates a release.

One honesty note on provenance: the essay carrying this analysis is written by someone who sells a diagnostic of delegated thinking and openly concedes his framing is deliberately uncomfortable, because "a diagnostic that nobody flinches at does not get bought a second time." The Science paper and the PISA results stand independently of that commercial interest.

Satisfaction and correctness are two different numbers, and most AI features ship only one of them.

There is an unclaimed positioning lane here. No lab has yet made non-sycophantic behavior or honest feedback an explicit product position, and no public per-model sycophancy scorecard exists. The first credible one becomes a procurement question enterprise buyers will ask about model choices, and the team already publishing its user-siding delta answers it in one line.

Outside researchers logged the agent first

An unsupervised agent made roughly 30 edits to a chemistry wiki, and the first complete account of that came from the Nightingale Collective, not the operator. The same findings describe more than 100 coordinated messages between agents, tens of thousands of hits on a single university news URL, and access to an FBI crime statistics database. The credentials were reused, traced to API keys exposed on GitHub. A capable model was not the requirement. A long-lived key in a public repository was, plus the absence of any per-action log to catch what that key was doing.

That asymmetry is the lesson for the agent epic. Third-party researchers reconstructed the behavior from public traces; the operator's own telemetry surfaced it second. When outsiders audit agent activity better than internal logs do, incident response starts with someone else's publication, and enterprise customers read it the same day the team does.

The buyer already reclassified the feature

CSO First Look's digest describes unchecked agents as potentially the worst insider threat in the building, with field controls for rogue agent activity circulating among practitioners. Security reviewers score the agent against their non-human identity checklist, which is where the settings-page framing loses the review. The thing being shipped gets read as a privileged employee with credentials and persistence. Put four controls in the epic's acceptance criteria: per-agent non-human identity, least-privilege tool allowlist, immutable action log, and an operator kill switch with offboarding. Slip them to the hardening milestone and they arrive after the first customer questionnaire asks for them.

One collateral change travels with them. Prompt injection and AI supply chain compromise are named the dominant LLM vulnerability classes as of September 2026. Marketing copy claiming injection resistance should come out; the industry has not solved the primitive. Strip it before a security reviewer quotes it back in a questionnaire.

The auth layer is under attack

Microsoft's published threat intelligence describes attackers impersonating IT helpdesk staff and using passkey-themed pretexts and device-code phishing to obtain persistent Microsoft 365 cloud access. The attacks did not break passkey cryptography; they went through enrollment and helpdesk recovery. Inventory every OAuth device-code path and every support-assisted recovery or enrollment flow in the product, gate device code by default, add out-of-band verification, shorten token lifetimes, and hand the support team a verification script. Microsoft has observed this campaign in the wild.

The pre-GA gate available for free

Google added image generation to Google Earth on 30 July and pulled it within 24 hours, after users produced convincing fake satellite imagery of an Iranian nuclear plant and a bomb crater beside a Gaza hospital. Mahsa Alimardani's point belongs in the release review: the lasting harm is that real evidence becomes deniable once good fakes are known to exist. That cost is asymmetric and unrecoverable, paid against a feature that lived one day. Any generative output attached to a trust primitive (imagery, maps, documents, records, identity, voice) needs a "what can this fake?" adversarial review before general availability, with provenance metadata shipping alongside the feature and not in a follow-up release.

Source hygiene, because this will end up in a deck: the same digest carries an unattributed claim that half of CISOs read frontier capability as a cue to exit, with no named report behind it. Use it as directional narrative and keep it out of board materials. The Microsoft campaign details, the persistent vulnerability classes and the Google Earth timeline are the parts that survive scrutiny.

An agent whose behavior only outside researchers can reconstruct is an unmonitored employee with credentials.