The Product Desk

Where the tokens actually go

A platform engineer reading the token log for one agent session watches the same observation payload go out again on the next step. Nothing in it is new work for the model. NVIDIA's SoL-Pi work never touched a model either. It ran an auto-research loop that proposed and tested changes to the agent harness, the scaffolding code that assembles context, replays observations and formats tool calls on every step of an agent loop. Most of what a default harness sends back to the model is re-transmitted state rather than new reasoning, which is a measured property of the harness and not an opinion about the model. Four techniques survived the search: Action Fusion, Online Context Compact, ObservationPack and an Evidence-Preserving Reducer.

The caveat travels with the number: the evaluation covers 51 tasks on EdgeBench and is flagged as early-stage. Replicate it on an internal eval set before it enters a pricing model.

The routing lever breaks a tradeoff scoping decks still quote

The second result points at the same layer from a different angle. When2Think, from Sungkyunkwan University and Microsoft, classifies each request by difficulty and routes it between a fast path and a deliberate reasoning path. On AIME24 it raised Pass@3 from 46.0% to 56.0% while cutting tokens from 14,195 to 10,236, or 27.9% fewer. Ten points of accuracy for less money is not the curve most scoping assumes. The claim that the more accurate variant costs proportionally more to serve is a tradeoff two separate teams have now broken.

Microsoft's name on that paper matters for sequencing. Difficulty-aware routing is the kind of optimization hosted providers absorb into their own serving stack within a couple of quarters, so building it in-house buys a temporary margin advantage, and the cost-per-correct-answer model is stale either way. A third result, R4T from Google Research and UIUC, adds a latency axis: a 53.9M-parameter distilled diffusion retriever that cuts query fan-out latency roughly 12–20×. That lever matters most for products that fan out multiple queries per request. Multi-query search is the clearest case; recommendations and set generation follow.

Where the evidence pulls in two directions

One body of reporting says accuracy per token is improving fast, which makes AI features cheaper to serve than most business cases assume. The other says capacity is getting more expensive: credit markets repriced AI data center paper into high-yield territory, hurdle rates for new capacity moved up with it, and some of that capacity has been cancelled. Both stories break the same line item, the inherited per-token price curve underneath the AI features already shipped. Both point at the same defensive move: know the token composition before someone else's cost structure changes it.

Roughly half the tokens in your agent loop are re-transmitted context. That line item usually sits unowned between the platform team and the product team.

The deliverable is unusually cheap: tokens per session, the share consumed by observation and context replay, and the reduction available from compaction and action fusion. That is about a week of instrumentation, and it moves the next margin conversation out of vendor pricing and into a backlog the team controls.

What Netflix actually built

A user opens the page for a title that launched that morning. The artwork she sees was chosen by popularity, because the older asset personalization models treated every image as an opaque ID and a brand-new title has no engagement history to work from. Cold start, and she is looking at a frame nobody picked for her. The fix concatenates a CLIP image embedding with the asset's learned ID embedding, so preference lives in embedding space and transfers across titles. Engage repeatedly with artwork featuring a particular comedian, and the model can favor that comedian's asset on a title it has never shown you.

CLIP embeddings barely move under cropping and resizing, so Netflix also collapsed five per-canvas models into one unified model, pooling signal across surfaces that previously learned in isolation. Two supporting pieces keep it standing in production: inverse propensity scoring on a dedicated exploration slice, and an embedding store that serves the exact same vectors at training and at inference.

The gate that would have killed it twice

Image embeddings alone were flat. The unified model alone was flat. The two together produced a statistically significant lift in a four-week A/B test. A review process that requires each component to move a metric before it survives cuts this project at the embedding experiment, then cuts it again at the model-consolidation project.

The dependency shape generalizes. The components are not additive, they are enabling. The unified model has nothing to pool until the embedding makes preference transferable, and the embedding has nowhere to generalize until the canvases share a model. Both halves fail an ablation gate separately. The lift only exists after both of them ship.

Netflix's cold-start fix produced nothing measurable until both halves shipped together. An ablation gate would have killed it twice.

The same shape, inside a claim the roadmap already makes

Johns Hopkins researchers formalized 100-task continual memorization with no replay buffer and no task IDs, which is the closest published analogue to a product that claims it learns from a user over time. Naive supervised fine-tuning retained 1.2% of the information. The best composed stack reached 34.9%: data, function and weight anchors plus merged LoRA. No single intervention came close, and the best composed result still loses roughly two-thirds of what it was taught.

Status matters more here than the numbers do. Netflix ran a production A/B against a business metric. The memorization work is a formalized benchmark with no deployment behind it. Treat 34.9% as a ceiling on claims, not a product spec. A roadmap line, a sales deck or a PMM slide promising that the product remembers and improves with use is implicitly claiming retention well above the best published composed result at this task count. That is defensible only where the stack that gets there was actually funded.

The embedding store is the prerequisite

The most reusable detail in the Netflix write-up is the one getting the least attention: a single embedding store serving identical vectors during training and serving, with explicit versioning. Without it, training and inference drift apart silently, and quality declines in a way nobody can attribute to a release. The forcing function is a date, not a debate. Make it a platform requirement before the next embedding-dependent feature ships, not after the first unexplainable regression.

Removing the camera opens rooms the camera SKU cannot enter

A nurse who bought Ray-Bans leaves them in her locker for the whole shift, because the ward has a no-cameras rule and the rule does not care what the glasses cost. The Information reports Meta will put a camera-free pair of smart glasses on sale in October, alongside the new AR glasses unveiled at Connect. Cost savings explain part of that decision. The larger effect is on where the product is allowed to sit. Offices with camera policies, gyms, schools, hospitals and privacy-strict jurisdictions are all places a camera SKU cannot enter at any price, and in October Meta will have a SKU for them.

The same exercise runs on any product with a procurement path: the integration that triggers a security review, the data field that forces a residency conversation, the permission IT refuses, the default that makes legal ask for a call. Teams file those under "roadmap debt." Buyers file them under "cannot deploy." For each one, name the specific room the deletion opens and count the accounts sitting in it.

Snap and Meta shipped the same pinch in the same week

Evan Spiegel demoed Snap's Specs the week of September 14 using finger gestures nearly identical to Meta's wristband-plus-pinch model on Ray-Ban Display. Gesture input is now a convention anyone can copy. If input was listed as a differentiator on a spatial, wearable or gesture-driven roadmap, it stopped being one this week, and it also stopped being a way to lose. The contested ground is which contexts the device is permitted into and which workflows it shows up in daily.

"Sold out" measures units built, not demand

Meta said Ray-Ban Display sold out in almost every store. That sentence describes manufacturing volume. The CES announcement, with expansion suspended to fill U.S. orders, looks like capacity rationing, which is equally consistent with a hit and with a stall. Then came a brief April mention, then essentially nothing on earnings calls, with the product still U.S.-only and reviews mixed a year in. Martin Peers puts it bluntly: if it had been a hit, executives would have talked about it a great deal more.

Hold the caveat honestly. Meta has published no usage or retention figure, so this is an inference from disclosure behavior, not a measured failure. Which is the argument for agreeing on the retention number before launch, while it is still cheap to agree on. Boards read quarters of silence the same way this briefing just did.

The surest sign a launch failed is the quarter your executives stop mentioning it.

Practically: cut preorders, waitlist size, sellout rate and day-one units from the headline launch metrics, and put in a D30/D90 active-use threshold plus a geo-expansion readiness gate that has to clear before anyone in the company uses the word success. Ray-Ban Display passed every supply-side test available and then went quiet for months.

Three checks for the keynote

Zuckerberg's keynote lands Wednesday, September 23, the same day as The Information's AI Agenda Live SF, with Google DeepMind, OpenAI and Nvidia participating. Move anything queued for that day; the news cycle is already full. Then treat the keynote as a diagnostic instead of a product launch. Watch for any usage or retention figure on Ray-Ban Display, and for whether international expansion resumes. Then check whether the camera-free SKU prices below the display tier. Those answers say whether the category's first mainstream display device found a real job or only a first wave of buyers.