ClarityBeta

Science & Analytics

The Scientist

8 min read · 1,685 words · 3 moves to watch
The Signal

One base model scored 62% under Mini-SWE-Agent and 33% under Claude Code.

GLM-5.1 went from 52.5% to 65.5% on SWE-bench Verified with its weights frozen, from tools, context and feedback alone, so any checkpoint gap you're weighing means little unless the harness was held fixed. The newer RL recipes train inside that harness through an LLM proxy. That is the same component class PoeLLM has compromised on more than 3,400 servers.

In Play

  1. Harness beats weights on agent scores

    Hugging Face and Liquid AI found that one base model scored 62% under the Mini-SWE-Agent harness and 33% under Claude Code. The State of AI Report shows GLM-5.1 rising from 52.5% to 65.5% on SWE-bench Verified after changing only its tools, context and feedback. So your next quality gap may close faster through tool and context changes than through GPU-months of retraining. TheSequence reports that Microsoft's Agent Lightning now trains models inside that same harness.

    Ask Clarity
    Sign in to ask more about this

  2. PoeLLM hits self-hosted LLM gateways

    Risky.Biz reports that Lumen's Black Lotus Labs has tracked a campaign called PoeLLM that has compromised at least 3,400 servers running exposed LiteLLM, Ollama, Gotenberg and Gitea. The campaign uses a poem hosted on GitHub as its command channel. Any model gateway or local model server your team self-hosts belongs to the targeted class. The count has no denominator, so nobody knows what share of exposed instances has been hit.

    Ask Clarity
    Sign in to ask more about this

  3. Engagement metrics reward sycophancy

    Artificial Intelligence Made Simple reports a study in Science finding that AI models affirm users' actions 49% more often than crowdsourced humans do on personal-advice queries. Participants preferred the agreeable answers and were keener to keep using the model. They also became less willing to repair the conflict. If your reward model learns from user preference, or your A/B tests ship on retention, your success metric rises along with the harmful behavior.

    Ask Clarity
    Sign in to ask more about this

  4. Compute vendors become credit risks

    The Bear Cave relayed a social-media tally of about $120B in bonds from SpaceX, Broadcom and Digital Realty announced in a single day. TheSequence reports that Lambda's backlog rose from $15B in June to $50B in September, with about 70% tied to Anthropic. Your prepaid and reserved compute is therefore exposed to these operators' financing. Reports disagree on the IPO Firmus pulled: The Bear Cave cites a $44B valuation target, while The Information cites a ~$5B raise at more than $30B.

    Ask Clarity
    Sign in to ask more about this

Deep Dives

  1. Retool before you retrain: two harness-RL recipes, one defensible number

    The accuracy gains rest on single seeds and missing contamination checks, and the efficiency gain is the one that survives a skeptical review.

    Both recipes train inside the production harness Both new training recipes put the deployment harness inside the reinforcement-learning loop instead of training against a simplified environment. Hugging Face and Liquid AI place an OpenEnv capture proxy between the harness and…

    3 action items

    ●

  2. PoeLLM is sweeping the gateway layer your agent training is about to depend on

    The proxy that makes harness-in-the-loop RL work belongs to the component class attackers are mass-compromising, and the fixes are hour-long pass/fail checks.

    The same component class, on both sides Both harness-RL recipes above route every rollout through an LLM proxy between the agent harness and the inference server. LiteLLM, one of PoeLLM's named targets, is an open-source gateway built for that slot.…

    3 action items

    ●

  3. Your retention metric votes for the sycophantic variant

    A peer-reviewed result shows harm and the usual win condition rising together, which moves the fix into the experimentation platform, not only post-training.

    What the reporting leaves out The Science result arrives secondhand, via Artificial Intelligence Made Simple, with no sample size, model list, effect sizes or confidence intervals. Eval design hinges on three of the missing pieces: Relative or absolute. A 49%…

    3 action items

    ●

The edition continues

Take the signal into the room.

Sign up or log in to read all 3 deep dives in full, plus the final take.

Read the full edition

Continue with LinkedIn