The Scientist

What Google Found — And Why It's Different From Thursday's CoT News

On Thursday, we reported that 97%+ of chain-of-thought reasoning steps are decorative noise — they don't influence the final answer. Today's Google finding is the dangerous complement: when CoT steps do influence answers, hallucinated intermediate facts propagate forward and corrupt the final output.

In controlled experiments, Google showed that reasoning-enabled LLMs act as a computational buffer, generating related facts that help retrieve correct answers for single-hop factual queries. That's the upside. The downside: when the model fabricates an intermediate fact during reasoning, that fabrication becomes a premise for subsequent steps. The hallucinated intermediate looks like valid reasoning, making it harder to catch during human review and more likely to survive quality gates.

This is qualitatively different from direct hallucination. In standard generation, you can fact-check the output. In chain-of-thought, the hallucinated premise is invisible unless you verify every intermediate step.

Methodological caveat: the newsletter describes these as "controlled experiments" but discloses no sample sizes, confidence intervals, or specific models tested. Treat the mechanism as credible but the magnitude as unquantified.

Google's Managed RAG: File Search Tool in Gemini

In the same cycle, Google DeepMind shipped File Search Tool — managed RAG integrated directly into the Gemini API. This isn't a startup's RAG-as-a-service; it's a hyperscaler bundling retrieval infrastructure into its core LLM API, abstracting away embeddings, indexing, and storage. Multimodal retrieval is the stated next phase.

What's conspicuously absent:

• No retrieval quality benchmarks — no recall@k, MRR, or NDCG on any dataset
• No latency numbers — critical for production real-time queries
• No chunking strategy details — fixed-size, semantic, or document-aware?
• No embedding model specification — Gecko? Proprietary Gemini embedding?
• No pricing model — cost-per-query and storage economics unknown

The pattern is clear: hyperscalers are commoditizing retrieval. Google bundling RAG into Gemini follows the same playbook as AWS bundling search into OpenSearch. Your custom pipeline's value proposition is domain-specific quality — if you can't demonstrate measurably better retrieval on your corpus than a managed alternative, your ops cost becomes unjustifiable.

The Combined Implication for Your Stack

These two developments create a fork: you can move to managed RAG (lower ops, unknown quality, vendor lock-in) or maintain custom pipelines (full control, higher ops, measurable quality). But regardless of which path you take, you need to add intermediate CoT verification to any pipeline using reasoning-enabled models. Your final-answer-only evaluation is blind to the error source Google just documented.

McKinsey's RAG Platform Fell to a Textbook Web Vuln

CodeWall's autonomous AI agent found an unauthenticated SQL injection vulnerability in McKinsey's internal RAG platform Lilli — not a sophisticated prompt injection or adversarial ML attack, but a textbook web security flaw. Within two hours, the agent had full read/write access to the production database. The exposed data:

• 46.5 million chat messages — likely containing sensitive client strategy discussions
• 728,000 sensitive files
• McKinsey's entire proprietary RAG knowledge base — the crown jewels of their consulting IP

Details come from the attacker, so some skepticism on exact scope is warranted, though the specificity suggests genuine access.

The lesson isn't about LLM security — it's that AI platforms inherit all the conventional web vulnerabilities of their infrastructure, and internal tools get insufficient security review.

CodeWall's Vulnerability Chaining: A Planning Benchmark

The same CodeWall agent separately demonstrated chaining four individually low-severity bugs on the Jack & Jill hiring platform to achieve admin access — then probed the target's AI defenses. This is architecturally significant: vulnerability chaining is a planning and reasoning problem where the agent enumerates attack surface, evaluates exploitability dependencies, constructs execution sequences, and adapts when steps fail.

For data scientists building risk scoring models, this is a concrete failure case: if your vulnerability prioritization system treats CVSS scores as independent features, you'll systematically miss compound exploit chains where four 'low' bugs combine to 'critical.' Graph-based representations where vulnerabilities are nodes and edges represent chainability would capture this.

The Infrastructure Attack Surface Is Expanding

Beyond these targeted demonstrations, CISA added n8n to its Known Exploited Vulnerabilities catalog — confirming active exploitation of two critical RCE and credential-exposure flaws, with 24,700 instances still exposed. If your team uses n8n for pipeline orchestration (triggering model retraining, moving data between services), the credentials stored in those workflows — database passwords, S3 keys, model registry tokens — make it a high-value lateral movement target.

Meanwhile, cyber insurers are now pricing policies based on AI deployment posture — organizations using AI defensively get lower premiums, while those whose AI introduces new attack surface pay more. Multiple security sources confirm this trend, though no premium differentials or actuarial methodology has been disclosed.

Cross-Source Pattern: The Real Threat Model

Five independent sources this cycle point to the same conclusion: AI system security failures are infrastructure failures, not AI failures. McKinsey fell to SQL injection. n8n fell to unpatched RCE. Perplexity's Comet agent was phished in under 4 minutes. OpenClaw's cottage industry in China deploys agents via untrusted intermediaries with modified configs. OpenAI is reframing prompt injection as social engineering and recommending blast-radius containment over detection.

Your threat model likely covers prompt injection and hallucinated actions. It probably doesn't cover: unauthenticated endpoints on your RAG API layer, compromised deployment chains where third parties modify agent configs, or workflow orchestrators with hardcoded credentials that nobody threat-modeled.

The Cost Gap Is Widening — But the 8x Number Needs Your Workload

Enterprise reports claim self-hosted open-weight models deliver 8x cost savings over API-based inference. Three forces are accelerating this shift: improving open models (Llama 3, Mistral, Qwen closing the quality gap), vanishing VC subsidies (API providers raising prices as free tiers expire), and EU AI Act data sovereignty requirements that may make self-hosting mandatory for certain workloads regardless of cost.

The 8x figure likely holds for high-volume, lower-complexity inference — classification, extraction, summarization — where a fine-tuned 7-13B open model replaces GPT-4-class API calls. For frontier reasoning tasks, the gap narrows or inverts when you factor in GPU procurement, ops overhead, and model update cadence. Run your own numbers — the headline is unreliable without knowing the workload profile.

The question isn't whether to evaluate self-hosted open-weight models — it's whether you can afford not to, given converging cost pressure, regulatory forcing, and disappearing API subsidies.

The SaaS Repricing Signal

Over $1 trillion in software market cap was erased in a single week. ServiceNow dropped 11% despite beating earnings. Microsoft shed $360 billion in a session. The market is pricing in a structural thesis: per-seat pricing, human-centric UIs, and proprietary business logic are being commoditized by agents.

For ML practitioners, this isn't directly a methodology story — it's a market regime change affecting the platforms you build on. Specific implications:

• Per-seat ML platforms (Databricks notebooks, Snowflake credits, analytics tools priced per data scientist) face pricing model disruption. Expect aggressive monetization pivots and M&A turbulence.
• Voice AI vendors are pivoting: ElevenLabs, Deepgram, and Bland AI are abandoning self-serve developer APIs for high-touch enterprise deployments with forward-deployed engineers. If your speech pipeline depends on these APIs, expect pricing and support model changes within 12-18 months.
• The 'data moat' thesis is resurgent: if code moats are collapsing, surviving differentiation is proprietary data, domain-specific eval harnesses, and feedback loops. Your team's ability to rigorously evaluate agent-generated outputs is the moat.

What's Conspicuously Absent: Agent Reliability Metrics

The SaaS repricing narrative assumes agents can replace human SaaS users. What's missing from every source making this claim: any quantitative assessment of agent reliability in production. Demo-grade agent performance and five-nines uptime are separated by an enormous engineering gap. The claim that SaaS applications are "simple CRUD wrappers" that agents can automate dramatically oversimplifies complex state management, compliance, and integration graphs.

The market signal is real (verifiable stock price data). The thesis behind it remains speculative until someone publishes agent task completion rates with confidence intervals across enterprise workflow complexity tiers.