The Board Room

Four studies on AI productivity measurement

This quarter's four independent studies converge rather than contradict. Together they form the clearest indictment yet of how enterprises measure AI impact. The findings cut at the measurement layer, not the technology:

• METR Developer Study: Experienced open-source developers ran 19% slower with AI tools while believing they were 20% faster. That is a 39-percentage-point gap between sentiment and output.
• Brynjolfsson's 5,179-agent study: Junior customer support agents gained 34% productivity. Veterans saw near-zero improvement.
• Harvard/BCG Consultant Experiment: Large gains inside the competence boundary, 19% worse performance outside it.
• MIT NANDA (300 deployments, 150 executive interviews): 95% of enterprise GenAI pilots deliver no measurable P&L impact.

AI compresses rather than amplifies. It raises floors, not ceilings. If engineering leadership reports that AI tools are 'working great' based on team sentiment, the bill arrives twice: once in dollars, once in false confidence.

Where the Money Is vs. Where the Money Went

MIT NANDA locates pilot failure in organizational rigidity, not model quality. Firms bolt AI onto workflows they refuse to change. The specifics matter:

The Expert Pipeline Crisis Nobody Is Pricing In

Stanford's Canaries dashboard, built on ADP payroll data covering 1-in-6 US workers, shows employment for 22-to-25-year-olds falling in AI-exposed roles while rising for less-exposed peers. The chain is causal: AI helps juniors most, so junior roles become the most automatable, and automating them removes the apprenticeship rung that produces future experts. This is a leveraged bet that AI capability improves faster than the expertise reservoir drains. If it doesn't, the capability loss doesn't reverse on any normal timeline.

The Tradeoff Nobody Wants to Name

The board-deck version says deploy AI across the org and capture the gains. The complete version is harder. Capturing gains means moving money away from the function that asked loudest, sales and marketing, toward the function that didn't, back-office. It means retiring self-reported productivity metrics for instrumented A/B measurement. It means 70% of transformation budget going to process change and 30% to technology, the inverse of current allocation.

What Changed on June 22

The executive order on post-quantum cryptography turned an optional engineering project into a dated obligation. We have heard the word urgent attached to crypto transitions before, and four times it did not move budgets. This time the deadline travels with a federal/DOD purchasing gate, which is the part that moves budgets. For anyone selling to the government or sitting on long-lived sensitive data, being late means exclusion from the largest forced technology refresh since cloud migration.

• December 31, 2030: Key establishment must use PQC algorithms
• December 31, 2031: Digital signatures must use PQC algorithms
• Pentagon parallel mandate: Systems that cannot support PQC by 2030 get phased out

The mandate reveals which organizations actually know what their systems encrypt and which only think they do. The billion-dollar figure is the cost of finding out.

Crypto-Agility Is the Real Requirement

DOD is not asking for today's post-quantum algorithms. It is requiring the ability to swap cryptographic primitives as standards evolve. That is a ten-year architectural decision wearing the costume of a feature request. NIST will revise the standards. New vulnerabilities will surface. Firms that hard-coded today's primitives will retrofit under deadline while agile platforms swap and move on. A point solution clears one audit. A platform clears the next three.

The Harvest-Now Threat Is Already Active

Multiple sources confirm nation-state adversaries are already collecting encrypted traffic for future quantum decryption. The breach is not a future event. It is happening now, with the damage deferred. Organizations holding data with multi-year confidentiality requirements — healthcare, financial, defense, enterprise IP — are already on the clock.

Complicating Factor: Standards May Be Compromised

Allegations surfaced this week that intelligence agencies are weakening post-quantum TLS standards, with a July 7 comment deadline. Low confidence, extreme consequence. A reasonable skeptic would call this noise, and most weeks they would be right. If the allegation holds, crypto-agility stops being an engineering nicety and becomes a hedge against standards capture itself.

Market Timing

Nobody migrates cryptographic assets they have not inventoried. Discovery comes first, by logic rather than preference. Whoever owns discovery owns the relationship that sells migration and runs the ongoing management. That is land-and-expand compressed into a four-year government deadline. Expect M&A in cryptographic discovery to accelerate over the next 12-18 months as the larger players decide buying is faster than building.

The Gusto Case Study

Eddie Kim, Gusto co-founder and CTO, shipped a tier-one production product with a five-person team in ten weeks. No PM. No Jira. No Figma. No standups. AI did the primary building. The coordination layer was a permanent Zoom room. This is not a minor acceleration — it's a structural break in how products get built.

The natural skeptic objections: maybe the product was simpler than 'tier-one' implies, maybe the five were unusually senior, maybe the rest of the org absorbed hidden work. All plausible. None explains why a company with the option to staff thirty chose five and still hit ten weeks.

If this model generalizes to greenfield products — and the evidence points that way — most companies are operating org charts designed for 2022. The handoffs, specifications, and coordination rituals that justified 15-person product teams may now be net-negative overhead dressed up as quality control.

The Economics Make It Structural

GLM-5.2 ran a 45-minute autonomous coding session processing 6 million tokens for $3.36. At that cost, an AI engineering-equivalent runs at single-digit dollars per hour. The binding constraint stops being budget. It becomes architecture and organizational readiness to absorb the capability.

Gusto's production agent runs on Cloudflare Workers with the Vercel AI SDK. No LangChain, no CrewAI, no heavy agent framework. Kim's working definition: "an AI SDK running somewhere in the cloud, able to look up files and call tools." Anyone invested in complex orchestration platforms should read this as a leading indicator that the market pays for simplicity.

The Contradiction That Demands Resolution

Sources diverge on an important point. The Gusto data says small AI-native teams can ship at extraordinary velocity. The METR data says experienced developers get slower with AI tools. The reconciliation: the gains come from rebuilding the team around AI, not from adding AI to an existing team. Kim didn't point AI at his existing org and ask it to run faster. He rebuilt from zero with AI as a core contributor. The bolt-on approach produces the expert penalty. The rebuild produces the 6x speed advantage.

China's Talent Arbitrage Compounds This

Chinese AI labs staff with talent averaging 1.6 years experience vs. 5.5 years in the US — a 3.4x gap. One interpretation: quality disadvantage. The more threatening interpretation: experience beyond a threshold is actually drag in a field where paradigms shift faster than institutional knowledge accumulates. If that interpretation holds, US companies spending 3-4x on AI talent may face diminishing returns on experience premiums.