Science & Analytics

The Scientist

The Signal

Agents scoring 87.9% on rubric partial credit finish only 20.6% of full workflows.

The tasks were multi-stage scientific workflows that had to end in reproducible code, tables and figures. Partial credit measures how far a model got. The thing it doesn't tell you is whether anything at the end actually ran. Rubric-weighted scoring is the default in most internal agent harnesses, so the completion rate you're reporting upward is probably a progress rate wearing a better name.

In Play

  1. Partial Credit Overstates Agent Deployability

    FrontierChallenge scored frontier models at 87.9% on partial progress and 20.6% on full completion of multi-stage scientific workflows that must ship reproducible code, tables and figures. Rubric-weighted partial credit is the default in most internal agent harnesses, so your scorecard overstates deployability about fourfold on compound tasks. RealSWE adds that agents degrade further on short, informal user requests, and the cheapest fix is a required desired-behavior field in task intake.

  2. Coding-Agent Spend Is an I/O Bill

    Spotify's Portal splits coding-agent work into bulk-reader and code-writer modes on Gemini 2.5 Flash, and Dimitri Mazmanov measured roughly a 90% cut in Claude Code context tokens on Java monorepo tests. That lands on a real budget line: 25% of engineering leaders report $200-$500 per developer per month, with outliers past $2,000. The evidence is one engineer's token delta with no task-success rate and no ablation, so treat 90% as a hypothesis with a plausible mechanism behind it.

  3. Tool Selection Is the Injection Surface

    Johann Rehberger's writeup on breaking Claude Code Opus 5 auto mode describes a website that asks the agent to summarize it, nudges it off the constrained WebFetch primitive onto curl, then reuses the model's own generated Python decoder as the exploit primitive. The compromise happens in tool choice, so a guardrail that inspects tool arguments passes it cleanly. Frontier capability is not a robustness proxy, which means a model upgrade should not ship with wider tool scope until your adversarial suite passes at the prior model's rate.

  4. Session Cookies Are the Data-Platform Credential

    JSCeal, documented malware shipped as compiled V8 bytecode, bypasses Google authentication by replaying stolen session cookies rather than cracking credentials. That session lives in the same browser profile as Colab, the Vertex AI console, the BigQuery UI and the GCS browser, so password rotation and MFA are no-ops and only session invalidation plus token revocation work. The compiled-bytecode packaging also strips the lexical surface that source-token, AST and entropy features depend on, so any static JS detector needs per-slice recall reported by packing method.

  5. Serving Frontier MoE Without NVLink

    A community wiki now publishes reproducible Docker builds, runbooks and benchmarks for GLM-5, DeepSeek-V4 and Kimi on Nvidia RTX PRO 6000 Blackwell cards over PCIe alone, with no NVLink in the topology. Sparse mixture-of-experts layouts are why that can work: low active-parameter counts shrink the per-token collective payload that saturates the bus on dense tensor-parallel serving. Bloomberg separately reports DeepSeek plans at least 160,000 Huawei top-end accelerators at one Inner Mongolia site, so accelerator, driver and quantization recipe belong in your eval metadata now.

Deep Dives

  1. The Denominator Every Cost Number Is Missing

    Three unrelated disclosures land on the same replacement metric, and the arithmetic behind it says two models with identical accuracy can differ 40% in serving cost.

    The bias runs one direction and has no bound. That asymmetry is the whole problem: every failure mode inflates the same metric, and nothing in the measurement pushes back the other way. An LLM call can return HTTP 200 and…

    3 action items

  2. Measure the I/O Share Before You Touch the Router

    Routing built before measurement is guesswork, and the same lesson arrives independently from a Postgres load, a Meta proxy and a starving training dataloader.

    A credible mechanism, measured at n=1 The observation underneath Portal is concrete. A coding agent reads five files to answer a question about one method, so most of what it does is I/O rather than reasoning . Two declarative modes,…

    3 action items

  3. Your Engagement Labels Are Mostly Machines

    This is not a traffic-quality complaint; it is a label-noise, covariate-shift and A/B-validity problem that your drift dashboard is structurally unable to see.

    One blogger's server logs, read as a data-quality audit Tracy Durnell's AWStats data is the cleanest public look at what arrives at a small site. 90% of visits last under thirty seconds , and a single IP fetched 423,059 pages…

    3 action items

The edition continues

Take the signal into the room.

Sign up or log in to read all 3 deep dives in full, plus the final take.

Read the full edition

Continue with LinkedIn