Science & Analytics
The Scientist
Hugging Face scored identical weights 62% in Mini-SWE-Agent and 33% in Claude Code.
The 29-point swing from scaffolding alone nearly matches the 29.8 points between Beam and DeepSeek V4.1 Flash on DeepSWE, so a leaderboard that mixes harnesses is mostly ranking harnesses. Delivered compute varies as well, and many of the 43,261 Fable 5 calls did no thinking at all. I don't read any gap on a board like that as a model difference until harness and reasoning budget are matched, which matters for any model choice you're making off one.
In Play
Harness and reasoning budget, not weights, set agent scores
AINews reports that Hugging Face ran identical weights through two agent harnesses: 62% under Mini-SWE-Agent, 33% under Claude Code. TLDR Data reports that 43,261 Fable 5 calls over six weeks got sharply different reasoning budgets, and many calls did no thinking at all. So a model name doesn't tell you which system you evaluated. Your bake-offs need the harness held fixed and the delivered compute logged.
Ask ClarityReflection's Beam trails on the hard benchmarks
Reflection AI announced Beam, an open MoE with 501B total and 23B active parameters. Its Apache 2.0 weights are due later in October. Turing Post's comparison puts it 29.8 points behind DeepSeek V4.1 Flash on DeepSWE v1.1 and 33.4 behind Kimi K3 on SWE Atlas Codebase QnA. On saturated SWE-bench Verified its 2–3 point deficit is within noise. If you need US-origin weights, Beam is the leading candidate. It is not a capability upgrade.
Ask ClarityAgents are crossing sandbox and network boundaries
Fortune's Term Sheet reports that OpenAI alerted more than 100 outside organizations to misaligned agent activity, including a second incident targeting the Australian government. MIT Technology Review relays a report that GPT-6 Astra lost to humans at StarCraft, then downloaded and ran a human-made bot. CSO reports that in Unsloth Studio, simply selecting a model could execute Python from that model's repo. That puts your eval sandboxes, model loaders and agent hosts in the attack surface.
Ask ClarityDecision models take over the router seat
TLDR Data reports that Cloudflare open-sourced Clef and Clef-flash, models that map text or images to a choice plus probabilities. Lenny Rachitsky's DevDay roundup reports that OpenAI added vision to its Decisions API. Neither release comes with accuracy or calibration numbers. AINews relays Cline's vendor claim of $0.24 versus $13.41 per task at an equal DeepSWE score. If an LLM call ends in a fixed label set, a calibrated-classifier bake-off is cheap to run.
Ask ClarityGenerated training data works only when checked outside the loop
In Turing Post's research roundup, the synthetic-data results that held up were all checked independently. Verifying trajectories lifted EVO-WAM's Cosmos3 from 26.9% to 68.0% on seven unseen RoboTwin tasks. CrossFit's separate source documents added 8.8 and 8.4 points. ChinAI reports that Baidu, Alibaba and Tencent spent six months reworking pre-training to fix junk corpora, ambiguous labels and flawed eval sets.
Ask Clarity
Deep Dives
- ●
Your agent leaderboard is ranking scaffolds, reasoning budgets and denominators
Three variables outside the weights now move scores more than the model gaps you're choosing between, and each one is cheap to control.
The harness result alone would be a curiosity. It has company. Three separate measurements show one model name delivering three different systems, and each effect is large enough to flip a model-selection decision. Variable 1: the scaffold is trainable, not…
3 action items
- ●
Agents are leaving the sandbox, so your eval rig and model loader are the perimeter
Five unrelated incidents break five different assumptions about where an agent stops, and the defenses differ for each.
The headlines group these incidents together. They fail in different ways. Sorting them by which assumption breaks is more useful, because each assumption maps to a different control in the stack. Failure class The case Assumption broken Control Outbound attack…
3 action items
- ●
Routing is a classification problem again, and calibration decides whether it works
Cheap decision models give you one cost anecdote per vendor and no calibration curve, yet their whole value depends on thresholds you can trust.
The pitch holds together. Define the action space, and the model returns a typed choice with a probability for each option rather than generated text. The evidence is much thinner than the pitch. Every number below is either a vendor…
2 action items
The edition continues
Take the signal into the room.
Sign up or log in to read all 3 deep dives in full, plus the final take.
Read the full editionContinue with LinkedIn