The Product Desk

The Benchmark Flip That Changes Everything

Two frontier-class open-source models dropped this week that fundamentally alter the AI build-vs-buy equation. Z.AI's GLM-5.1 — a 754-billion parameter MoE model released under MIT License — scored 58.4 on SWE-Bench Pro, the coding benchmark most relevant to production software tasks. That dethroned both OpenAI's GPT-5.4 and Anthropic's Claude Opus 4.6. Simultaneously, Google's Gemma 4 shipped under Apache 2.0 with models ranging from 2B (phone-ready) to 31B (workstation-class), with the 26B MoE variant hitting #6 on Arena AI Leaderboard — outperforming models 20x its size.

If your product charges a premium partly because you're using a 'frontier' proprietary model, that positioning just got weaker. Your competitor can now self-host a benchmark-leading model for the cost of compute alone.

8-Hour Autonomy Changes the Agentic Ceiling

GLM-5.1's most consequential capability isn't raw intelligence — it's endurance. Z.AI explicitly optimized for sustained execution: 8 hours of autonomous operation, 1,700 tool calls per session, with no strategy drift. In testing, it autonomously built a full Linux desktop environment from scratch — file browser, terminal, text editor, games — in a single session. It writes code, compiles it, runs it in Docker, diagnoses bottlenecks, and rewrites its own architecture to fix them.

This intersects directly with cost: if you're currently paying per-token for long-running agent tasks via closed-source APIs, the math may have just changed dramatically. Hours of sustained inference at API pricing versus self-hosted open-source could be the margin that makes or breaks your AI feature economics.

On-Device AI Is No Longer 'Next Year'

Gemma 4's smallest variants (E2B and E4B) process image, video, and audio locally on smartphones and Raspberry Pis. Combined with native agentic support — built-in function calling, structured JSON output, system instructions — this eliminates the server round-trip for a meaningful category of AI features. For mobile and IoT PMs specifically: on-device multimodal AI with agentic capabilities is shippable today under a permissive license.

The Strategic Fork

Four sources this week independently converge on the same conclusion: the AI model market has forked into distinct deployment categories. Security-restricted (Anthropic Mythos, gated access), ambient-consumer (Meta Muse Spark, embedded in 3B+ MAU surfaces), and open-source agentic (GLM-5.1, Gemma 4). The competitive axis is no longer 'smartest model' but deployment geometry. Your roadmap should map each AI feature to the appropriate category — and the open-source category just became viable for your most demanding workloads.

The UBS Data Point That Should Alarm Every SaaS PM

UBS Securities reports that since December 2025, over half of enterprise customer conversations include explicit mentions of 'containing' non-AI software spend to fund AI initiatives. This isn't analyst speculation — it's procurement behavior documented across UBS's enterprise coverage. Your product isn't just competing with direct competitors anymore; it's competing with your customer's AI budget for the same dollar.

The market is classifying every line item as either 'AI spend' or 'spend to cut.' If your product is in the second bucket, no feature improvement saves you — only repositioning does.

The Casualties Are Already Visible

Design and collaboration tools are the most AI-vulnerable categories:

• Figma: down 50% in 2026, enterprise value now $7.9B — versus Adobe's $20B acquisition offer in 2022
• Asana: down 60% YTD
• ServiceNow and Snowflake: each dropped 8% in a single Friday

The new development: the selloff has breached cybersecurity. Palo Alto Networks fell 6.7% and CrowdStrike dropped 4% — categories previously considered AI-insulated. The emerging fear is that AI companies will vertically integrate security capabilities rather than buy from pure-play vendors. Cisco's talks to acquire AI security startup Astrix for $250M+ confirm incumbents are already responding.

The Productivity Paradox Compounds the Problem

Here's the cruelest irony: AI productivity gains are not yet appearing on corporate balance sheets, despite widespread adoption. When your buyer's CFO sees flat margins despite heavy AI investment, skepticism hits every tech line item harder. This creates a doom loop for non-AI software: budgets shift to AI, AI doesn't yet show measurable ROI, and the CFO cuts even deeper on 'traditional' software to fund more AI experiments.

The smart PM response: build business cases around specific, attributable workflow metrics — time-to-first-response, error rates, cycle time — not aggregate productivity claims. And critically, reposition your product as enabling your customer's AI strategy, not competing with it for budget.

The Double Squeeze

This budget pressure arrives simultaneously with the open-source model revolution. Enterprise buyers are cutting non-AI spend while open-source alternatives eliminate the cost advantage of proprietary AI integrations. If you've been justifying premium pricing partly by using frontier proprietary models, that moat is eroding from both sides: your customer wants to pay less, and your cost basis for AI capabilities just dropped.

The Usage Data vs. The Hype Cycle

A large-scale study of millions of ChatGPT conversations delivers a finding that should make every PM pause: users overwhelmingly want decision support and writing help — not autonomous task execution. Coding, despite dominating conference keynotes, is a much smaller share of real-world usage. The dominant work patterns are documenting, interpreting, problem-solving, and advising — all fundamentally copilot patterns where humans make the final call.

If your product strategy bets heavily on 'let the AI do it,' the market is saying 'let the AI help me think about it.' That's a different product, different UX, and different pricing model.

Non-work ChatGPT usage is growing faster than work usage — suggesting the total addressable market for LLM products is broader than enterprise productivity. Consumer and prosumer use cases may be the real growth vector.

Meanwhile, Your Agent Features Are Probably Broken

Independent evaluation data reveals a quality crisis hiding in plain sight. An MCP-powered application tested against DeepEval's MCPUseMetric showed tool calls passing only 1-2 out of 24 test cases — roughly a 4-8% success rate. The fix? Adding structured docstrings to tool descriptions. That single change took pass rates to 24/24 — 100%. This wasn't a model quality issue (Claude Opus was the underlying LLM); it was a metadata quality issue.

The evaluation framework scores two dimensions independently: whether the LLM selects the right tool AND whether it constructs correct arguments, then takes the minimum. This maps to the two user-facing failure modes: 'the AI tried the wrong thing' and 'the AI tried the right thing but botched the parameters.' If your product uses any form of tool calling, assume your descriptions are inadequate until proven otherwise.

The Reliability Gap Widens

LaunchDarkly survey data confirms what the tool-call data implies at a systemic level: AI-generated code ships faster, but production reliability hasn't improved. Deployment velocity is up; stability is flat. This is a measurable, widening gap. If your team closed 30% more tickets this quarter thanks to AI coding tools, ask: did your P1 incident count go up too?

Research from MIT CSAIL and UCSB adds a third data point: agentic skill performance degrades significantly in realistic noisy settings. The gap between demo and production is structural, not incidental. The good news — query-specific skill refinement can substantially recover lost performance — gives you a design pattern: adaptive, context-aware prompt engineering per task, not static system prompts.

The Contradiction That Defines This Moment

Here's the tension: the infrastructure for agents is maturing fast (KAOS v0.4.1 with Kubernetes-native always-on agents, A2A protocol standardization). But actual user behavior and reliability data both say the market isn't ready. The smart move: deploy maturing agent infrastructure for internal ops use cases (monitoring, maintenance, automation) while building customer-facing features in copilot mode. Let the infrastructure catch up with user readiness, not the other way around.