The Scientist

An MIT-Licensed Model Leads the Coding Benchmark

Z.ai released GLM-5.1 — a 744-754B parameter Mixture-of-Experts model under MIT license that scored 58.4 on SWE-bench Pro, claiming the #1 position over both GPT-5.4 and Anthropic's Opus 4.6 (53.4). Four independent sources confirm the claim. The model was trained entirely on 100,000 Huawei Ascend chips with zero Nvidia silicon — making it the largest confirmed MoE training run on non-Nvidia hardware.

The best publicly accessible coding model is now open-weight, non-American, and runs on non-Nvidia hardware. Your build-vs-buy calculus just inverted.

What the Numbers Actually Say

The 5-point SWE-bench Pro gap over Opus 4.6 is meaningful — but it's a single benchmark. We have no cross-benchmark validation (HumanEval, MBPP, LiveCodeBench), no pass@k breakdown, no ablation studies, and no independent replication of the 8-hour demo. The demo of building a Linux desktop environment in 8 hours is qualitatively impressive but methodologically meaningless as evaluation — it's a single cherry-picked run.

The MoE Architecture Gap

GLM-5.1 is a 744-754B total parameter MoE model, but the active parameter count is undisclosed. This is critical: MoE models route to a subset of experts per token, so inference cost could be dramatically lower than a 744B dense model. But you literally cannot plan GPU procurement without this number. Z.ai hasn't disclosed the expert count, routing strategy, or activation ratio. Compare this to Gemma 4's 26B A4B which transparently reports 3.8B active of 26B total (14.6% activation).

The Endurance Claim

Z.ai frames long-horizon capability — 8 hours, 1,700 tool calls, resistance to strategy drift — as "the most important curve after scaling laws." This redefines what we should be measuring. Current benchmarks are point-in-time snapshots; they don't capture coherence decay or error accumulation over multi-hour sessions. The mechanism behind this coherence — state compression, external memory, checkpoint-based context refresh — is completely undisclosed.

Cross-Source Tension

Sources disagree on the exact parameter count (744B vs 754B), suggesting different sources are working from different announcements or rounding. More importantly, while four sources confirm the SWE-bench Pro score, none provide independent evaluation. The training data composition, fine-tuning approach, and whether GLM-5.1 was specifically tuned for SWE-bench Pro are all unknown.

90% Accuracy With 50% Broken Citations

AI startup Oumi audited Google AI Overviews — arguably the highest-traffic RAG deployment on Earth — and found a two-layer failure mode that most evaluation frameworks completely miss. Layer one: ~10% of responses are simply wrong. Layer two: among responses that are accurate, over 50% of citations point to sources that don't actually support the generated claim. The grounding is cosmetic.

Google's AI Overviews proves that 90% accuracy with 50% citation faithfulness is production-scale misinformation with a bibliography.

Three sources corroborate this with slightly different numbers: Oumi reports the 50%+ figure, a separate analysis using OpenAI's SimpleQA benchmark (4,000+ verifiable questions) measured 91% accuracy post-Gemini 3, and yet another source quantifies that 56% of correct answers lack verifiable sources. The consistency across measurements strengthens the signal.

Why Your Eval Misses This

Most RAG evaluation frameworks measure two things: answer correctness and retrieval relevance (did we fetch topically related documents?). They don't measure faithfulness — whether the cited passage actually entails the generated claim. Google's system scores well on both accuracy and retrieval while having fundamentally broken citation grounding.

The Source Quality Compounding Problem

The second- and fourth-most-cited sources in AI Overviews are Facebook and Reddit. These aren't authoritative references — they're high-volume, topically diverse UGC platforms that retrieval systems surface because they cover everything, not because they're reliable. Meanwhile, SEO firms are actively gaming AI search results through prompt manipulation and self-serving listicles, meaning your retrieval corpus quality is degrading in real time.

A separate cross-source signal reinforces this: LLMs fabricate data even when ground truth is in the context window. One documented case showed a model inventing 30 businesses during a prospecting task and fabricating Search Console metrics despite having the actual data export attached. The failure isn't "no data available" — it's "confident invention alongside correct data."

The Evaluation Fix

The minimum viable fix is NLI-based entailment scoring between retrieved chunks and generated claims. Use a fine-tuned DeBERTa or an LLM-as-judge to verify each citation actually supports the claim it's attached to. Track this as a separate metric from accuracy. If your system scores 90% accuracy and 45% faithfulness, you know exactly where trust erodes.

Warp Decode: Inverting MoE Compute Geometry

Standard MoE inference dispatches tokens to experts, computes within each expert, then gathers results. This creates load imbalance and dispatch overhead that dominates decode latency, especially at small batch sizes. Cursor's warp decode inverts this entirely: it organizes computation around output neurons instead of experts, fusing operations more efficiently on Blackwell GPU architecture. The claimed result: ~1.8x throughput and improved numerical accuracy from better accumulation ordering.

This matters immediately because of the MoE moment: GLM-5.1 (744B MoE), Gemma 4 26B A4B (128-expert MoE), and other recent releases all use MoE architectures. If you're serving any of these, warp decode targets exactly your bottleneck.

A genuine 1.8x throughput improvement on your MoE serving infrastructure translates directly to cost reduction or capacity increase — but only on Blackwell. Validate on your specific batch size distribution.

TriAttention: Fixing KV Cache Eviction

Most KV cache eviction methods score token importance after RoPE has been applied — but RoPE rotations distort natural distance relationships between tokens in embedding space, making eviction noisy for long sequences. TriAttention computes stable Q/K cluster centers in pre-RoPE space and uses distance-based scoring to determine which KV pairs to evict. This should produce more stable importance estimates, especially at long context lengths where RoPE rotations are largest.

If you're memory-bound on long-context inference (RAG with large retrieval windows, multi-turn agent conversations), TriAttention could let you serve longer contexts on existing GPU memory without the quality degradation simpler eviction schemes introduce.

What's Missing From Both

Both claims need hardware-specific validation. MoE kernel performance is highly sensitive to batch size, sequence length, and expert count — the 1.8x may not hold at your operating point.

Practical Integration Path

These two optimizations attack different parts of the inference stack and are complementary. Warp decode reduces compute latency per forward pass; TriAttention reduces memory pressure from the KV cache. Together, they could meaningfully shift the cost-quality Pareto frontier for MoE serving. But they're both pre-benchmark-publication — add them to your evaluation queue, don't restructure infrastructure around them yet.