The Scientist

Four Sources, One Conclusion: Build Your Own Evals or Fly Blind

A convergence of signals across multiple sources this week makes one thing unmistakable: the AI evaluation infrastructure the industry relies on is breaking down, and the replacement must come from you.

OpenAI is actively pushing to retire SWE-bench, the dominant benchmark for AI coding agents. Whether this is because they're losing on it or because it's saturating, the practical consequence is identical: if you've been justifying model selection with SWE-bench scores ("we chose Model X at 48% vs. Model Y at 41%"), that comparison framework is disappearing. Separately, AI enthusiasts are turning to personally devised tests involving otters, Minesweeper, and Will Smith eating spaghetti because formal benchmarks can't discriminate between frontier models on real-world tasks. This is Goodhart's Law at industry scale.

The Multi-Step Reliability Problem

Cotool Research benchmarked frontier LLMs on thousands of defensive CTF and investigation tasks and found large reliability gaps across models on multi-step reasoning — gaps that don't appear in single-turn benchmarks. Meanwhile, ARQ's evaluation of instruction-following across Direct (81.5%), CoT (86.1%), and ARQ (90.2%) approaches reveals that even the evaluation of reasoning strategies is methodologically weak — their entire benchmark is only 87 scenarios, with no p-values, no confidence intervals, and no disclosed base model.

Leaderboard scores are marketing. Your production task distribution is the only benchmark that matters.

What To Do About It

The LLMOps evaluation stack is crystallizing as a distinct discipline. DeepEval (open-source) supports task-specific LLM metrics — faithfulness, hallucination, relevance — that go beyond BLEU/ROUGE. But the tool is only as good as your test suite. The minimum viable evaluation infrastructure: 50-100 test cases drawn from your actual production workload, tracking cost-per-correct-completion (not just accuracy), with per-turn instruction compliance monitoring for multi-step agents.

Two Signals, One Architecture Shift

Two independent findings this week point to the same conclusion: unconstrained LLM reasoning is both unreliable and wasteful, and structured constraints are the fix.

ARQ: JSON-Schema Constraints for Instruction Adherence

Attentive Reasoning Queries (ARQ), from the Parlant framework (18k GitHub stars), replaces free-form Chain-of-Thought with domain-specific questions encoded as targeted queries in a JSON schema. These are injected at three agent modules — guideline proposer, tool caller, message generator — forcing the model to explicitly address relevant rules at each step rather than hoping it remembers a 2,000-word system prompt.

The motivation is well-grounded: LLMs demonstrably drift from instructions in multi-turn conversations, forgetting policies as context windows fill up. ARQ's results — 90.2% vs CoT's 86.1% vs Direct's 81.5% — are directionally compelling but statistically inconclusive at n=87. No base model disclosed, no latency measurements, no token cost analysis, no ablation separating JSON constraints from domain-specific query design.

Reasoning Model Overthinking

A separate study found that reasoning models (o1, Claude Thinking) systematically think far past the correct solution, generating 2,000+ tokens of reasoning for problems solvable in 200. The inference cost implications are severe: you may be paying 5-10x more per query than necessary for reasoning-heavy workloads. This aligns with practitioner observations and creates an immediate optimization opportunity.

The Convergence

These findings, combined with Reflection AI's thesis on RL-aware pretraining (co-optimizing data mixtures for downstream RL performance rather than just perplexity), paint a coherent picture:

• Free-form reasoning is insufficient for instruction adherence at scale
• Overthinking wastes inference budget without improving accuracy
• Structured constraints (JSON schemas, process reward models, early-stopping heuristics) are the optimization frontier

The key insight isn't "use ARQ" — it's that your agent architecture needs explicit mechanisms to keep reasoning on-policy and on-budget. Options include: ARQ-style structured checkpoints, confidence-based reasoning truncation, process reward models that reward intermediate steps, and trajectory decomposition for multi-step tasks.

Free-form reasoning is the new unregularized model — it works in demos and overfits in production.

Five Sources Confirm: The Threat Is Inside Your Workflow

Across five independent sources this week, a consistent pattern emerges: the tools data scientists and ML engineers use daily are being actively exploited. This isn't theoretical — these are patched vulnerabilities, confirmed campaigns, and demonstrated attack chains.

Claude Code: RCE Before You Even Click Accept

Check Point disclosed three vulnerabilities in Anthropic's Claude Code, the most severe being CVE-2025-59536 (CVSS 8.7): a malicious project config file could trigger remote code execution before the user consent dialog appeared. A second vulnerability (CVE-2026-21852, CVSS 5.3) enabled plaintext API key exfiltration via config manipulation. The attack chain exploits the Hooks feature and MCP configurations. Fix: update to v2.0.65+.

The supply chain implication is direct: cloning a repo with a poisoned .claude config is sufficient for compromise. ML engineers routinely clone repos to reproduce papers, benchmark models, and evaluate architectures — every git clone on a training node is a potential execution of untrusted code.

Google Sheets as Command-and-Control

Google/Mandiant disrupted GRIDTIDE (UNC2814, PRC-linked), which used Google Sheets API calls as C2 infrastructure across 53 organizations in 42 countries, operating undetected for years. At the network level, a C2 heartbeat to sheets.googleapis.com looks identical to your data pipeline refreshing a config spreadsheet. If your team uses Google Sheets for annotation configs, feature flags, or label collection, you share the same API trust surface.

Agent Credential Exfiltration

Multiple sources confirm that AI agents can be tricked into stealing SSH keys and leaking passwords via prompt injection. Any agent with file-read or shell-execute capabilities is a potential exfiltration vector. The OpenClaw demonstration showed agents freely leaking bank details — this is the adversarial ML problem applied to deployment.

The 82% Malware-Free Baseline

CrowdStrike's data — 82% of 2025 intrusions used zero malware, with 29-minute average breakout time — means attackers are using legitimate credentials and authorized pathways. Your network-level anomaly detection that looks for malicious payloads is blind to the dominant attack class.

The biggest risk to your ML systems isn't a better model from a competitor — it's the poisoned config file in the repo you cloned this morning.

Three Model Ecosystem Shifts to Track, Not Act On

The open-source and open-weight model landscape shifted this week in ways that don't require immediate action but should inform your quarterly planning.

Perplexity Open-Source Embeddings

Perplexity open-sourced embedding models claiming parity with Google and Alibaba's offerings. If this holds on MTEB/BEIR benchmarks, it's meaningful for anyone paying for embedding APIs or running older open-source models (E5-large, BGE-large). The embedding layer is increasingly commoditized, and high-quality open-source options reduce both cost and vendor lock-in for RAG pipelines. Caveat: "rivals" is marketing language — no published benchmark scores were provided.

Mercury 2: Diffusion-Based Reasoning

Inception launched Mercury 2, claiming it's the first diffusion-based reasoning model. Current reasoning models use autoregressive chain-of-thought; diffusion-based reasoning would work via iterative denoising over the entire output, potentially enabling parallel refinement rather than sequential generation. This is architecturally novel — diffusion models excel at continuous domains (images, audio), and applying them to discrete symbolic reasoning is non-trivial. No benchmark numbers, no architecture paper, no ablation studies provided. Pure monitor signal.

The Western Open-Weight Frontier Gap

Reflection AI's $2B+ raise highlights a real structural problem: there is currently no Western, open-weight, frontier-tier model. Llama 4 underperformed. DeepSeek is the strongest open option but comes with geopolitical considerations. Reflection AI (founded by AlphaGo's Ioannis Antonoglou) claims to be building one, but has shipped zero products and published zero benchmarks. Their thesis on RL-aware pretraining — co-optimizing data mixtures for downstream RL — is intellectually interesting but entirely unvalidated.

The open-source model landscape is expanding but unevenly — Perplexity embeddings are worth benchmarking this sprint; everything else is a watch-list item.