Engineering & Technical

The Engineer

The Signal

DPRK operators spent a year earning axios publish rights, so provenance checks ran green.

Wiz measured roughly 1 in 10 cloud environments tainted inside two hours, and no developer typed an install command anywhere in that chain. The resolver pulled the release on its own, and the post-install script ran with every credential the CI runner held. Any secret resident in your pipeline during that window is exposed, regardless of what the audit said afterward.

In Play

  1. Your Registry Verified Authenticity, Not Intent

    Amazon's threat researchers attributed the axios, debug, chalk and typo-crypto npm compromises to North Korea-linked operators who spent over a year earning legitimate maintainer publish rights, per CyberScoop's reporting. Wiz telemetry measured roughly 1 in 10 cloud environments tainted inside two hours. Because the publishes were legitimate, 2FA, sigstore provenance and npm audit all returned green. The propagation engine was your resolver and your post-install scripts, not a developer typing an install command.

    Ask Clarity
    Try
  2. Disclosure-to-Exploitation Collapsed to One Day

    FearsOff disclosed CVE-2026-16723, an unauthenticated remote code execution flaw in Alibaba's Fastjson, and exploitation began the next day, first documented by Imperva and ThreatBook. Fastjson is rarely a deliberate choice; it arrives transitively through Alibaba-ecosystem SDKs and middleware, so the most exposed teams never evaluated it. Rails separately patched a pre-auth Active Storage arbitrary file read, and Broadcom shipped three critical VMware fixes including VM escape.

    Ask Clarity
    Try
  3. Harness Config Outranks Model Choice

    OpenAI reported that retaining reasoning across turns and enabling context compaction moved its ARC-AGI-3 score from 13.3% to 38.3% while cutting output tokens 6x, with no model change. Log10/Everest's ClinReg benchmark of 19 models found several results moved more with harness settings, meaning validator strictness, retry limits and stopping rules, than with model identity. Any bake-off you ran without recording those settings ranked scaffolds, not models.

    Ask Clarity
    Try
  4. Privileged Local Daemons Are Back, With Agent Names

    Ruflo, an open-source harness wrapping Claude Code and OpenAI Codex, shipped an MCP Bridge Express server bound to a non-loopback port whose /mcp endpoint accepted tool calls, including terminal_execute, without authentication (CVE-2026-59726, CVSS 10.0). Noma Security noted the worse half: the poisoned agent memory lives in AgentDB, so a patched redeploy leaves the backdoor in place. The same shape as the South Korean AnySign4PC watering hole, where a web page drove a local privileged agent.

    Ask Clarity
    Try
  5. Stage Success Is Not Terminal State

    Spotify's own root cause says a newly introduced validation signal was never wired into the logic that wakes the publishing path, so media processing reported success while items never reached a published state; 3 of 5 consecutive weekly publishes failed, per The Pragmatic Engineer. Aggregate throughput dashboards stayed green throughout. The same blind spot appears in agent stacks, where tracing instruments the agent and misses whether the table it read had stopped updating.

    Ask Clarity
    Try

Deep Dives

The Payload Is in the Post-Install Script and the SVG, Not the Manifest

Two live campaigns exploit the same gap: your controls check who published a package, while the hostile code runs from a lifecycle hook that inherits every credential on the runner.

Two execution surfaces need two different controls

chalk and debug belong in a different bucket from axios, because they execute in a different place. The first pair is reached almost entirely at install and build time, which in practice means a lifecycle script running inside a CI container. The compromised axios release, at roughly 100 million weekly pulls, added a plain-crypto-js dependency. Its post-install script contacted a hardcoded command-and-control endpoint and staged a cross-platform remote-access trojan for Windows, macOS and Linux. axios is a different shape of problem: a runtime dependency sitting in the production request path.

Here is what a post-install script actually inherits on a build runner: NPM_TOKEN, GITHUB_TOKEN, a cloud OIDC session, artifact-signing keys, and network reach into the internal registry. That is not a tainted artifact. It is arbitrary code execution inside your most credentialed compute. Any runner that resolved an affected version needs full secret rotation. A rebuild does nothing.


Where the reporting disagrees, and who is right

Cyberpresso leads its remediation with pinning: enforce npm ci, commit lockfiles for services and container builds, disable dependency-bot auto-merge. CyberScoop's analysis of the same campaign pushes back on that emphasis, and the pushback holds up. A lockfile pins the malicious version too. Reproducibility is not intent verification. The two controls with the best cost-to-coverage ratio are the ones that change what executes and when:

ControlStops install-time payloadStops runtime payloadWhere it breaks
Lockfiles plus npm ciNoNoPins the bad version deterministically
ignore-scripts plus allowlistYesNoNative builds: node-gyp, sharp, prisma, esbuild
7-14 day version cooldown at the proxyMostlyMostlyAdds patch latency; needs an audited bypass lane
Provenance and sigstore attestationNoNoAttacker held real publish rights, so attestation is valid
Short-lived OIDC credentials, no ambient secretsImpact onlyImpact onlyRequires real pipeline rework

The cooldown is the underrated one. The community currently catches malicious publishes in hours to days, so a 7-14 day minimum release age enforced at the registry proxy turns a zero-hour compromise into a non-event. The price is patch latency, and that needs an override path for a known-exploited fix.


The variant that never touches package.json at all

Elastic Security Labs documented a parallel DPRK campaign that defeats the manifest-centric controls outright. Developers receive fully functional coding-challenge repositories; one was seeded through Elastic's own community Slack #jobs channel. The payload sits as Base64 chunks inside HTML comments within SVG flag images, reassembles, and executes on the first npm run dev or npm start. Nothing malicious appears in package.json. Software-composition analysis, lockfile pinning and registry allowlists all report green.

What lands is a four-module OTTERCOOKIE stack. Module 2 is the one that decides severity: a recursive sweep for .env files, SSH configs, AWS configs and shell history. A laptop holding a static AWS key in a .env is a straight line from a take-home assignment to the production account. Module 3 is a Socket.IO remote-access trojan, so the exfiltration channel looks like a normal real-time web app.

Static scanning inspects manifests and registry artifacts. Both of these campaigns execute from somewhere else, so the only controls that hold are isolation and short-lived credentials.

The detection that is cheap to build is behavioral, not signature-based: alert when a node or npm child process opens an outbound WebSocket, when a recursive glob touches ~/.aws or ~/.ssh, or when any process outside sanctioned CLI paths reads ~/.aws/credentials.

What to do

  1. Set ignore-scripts=true in every CI .npmrc, with a named allowlist for the handful of packages needing native builds (node-gyp, sharp, prisma, esbuild class).

  2. Query build logs and image manifests for axios, debug, chalk, typo-crypto and transitive plain-crypto-js in the compromise window by end of week, then rotate registry tokens, cloud credentials and signing keys for every runner that resolved them.

  3. Enforce a 7-14 day minimum release age for third-party dependency updates at the registry proxy this sprint, with a documented emergency bypass for known-exploited fixes.

A JSON Library Went From Advisory to Exploited Overnight

A dependency inventory you commission as a project cannot answer a question that has a 24-hour fuse, and three separate pre-auth bugs are testing that.

Why Fastjson is worse than its CVSS suggests

The published details do not state the mechanism for CVE-2026-16723. Fastjson's history is dominated by polymorphic deserialization and autoType bypass chains, where the library reconstructs arbitrary classes from attacker-controlled type hints. That is inference, not fact, and it should be labeled as such. If it holds, the candidate set is every endpoint that deserializes untrusted JSON, including internal services fed by a queue that eventually carries external input. Authentication in front of the handler does not help. Parsing happens before authorization in the filter chain.

Discovery is the harder problem. Fastjson arrives transitively through Alibaba-ecosystem SDKs, RPC frameworks, config and registry clients, connector libraries. A grep of pom.xml and build.gradle systematically under-reports. Resolved dependency trees across built artifacts are the floor; verification against live classpaths is better. If step one takes more than a day, that is the real finding, and it belongs in the writeup.

Contain before you patch

A dependency bump does not ship in an hour across a fleet. Three compensating controls do: a WAF virtual patch on the exploit pattern, default-deny egress on affected services, and an alert on child-process spawn from JVM PIDs. Most teams already collect the telemetry for that last one and have never written the rule against it.


The Rails bug is a chain, not a leak

"Arbitrary file read" undersells the Active Storage flaw. On a containerized Rails app, one unauthenticated request reaches /proc/self/environ (database URLs, cloud keys, third-party tokens), config/master.key, .env, and the Kubernetes service-account token. master.key decrypts credentials.yml.enc, which yields secret_key_base, the root of every message verifier in the app: signed cookies, encrypted sessions, Active Storage signed IDs, ActiveJob payloads.

One config line decides whether that ends as impersonation or code execution. With config.action_dispatch.cookies_serializer = :json, a forged session authenticates the attacker as anyone. Bad, survivable. With :marshal or :hybrid, still common in apps carried forward from Rails 5 and 6 that never flipped the flag, the forged cookie deserializes through Marshal.load, and public gadget chains give code execution in the app process. Grep for that line before anything else.

Rotation caveat, better learned now than at 3am: swapping secret_key_base outright invalidates sessions, signed cookies and existing Active Storage blob URLs. Use a rotator that retains the previous value through a deprecation window.


The severity threshold is the actual bug

Cisco manually overrode CVE-2026-20316 in Secure Firewall Management Center to High against a CVSS 5.3 base score. The stated reasons: it chains, it is under active exploitation, and there is no workaround. A patch SLA gated on a 7.0 threshold defers that indefinitely. Broadcom meanwhile shipped three critical VMware fixes covering auth bypass, code execution and VM escape, which retires the isolation assumption on any host running pull-request-triggered CI or customer builds.

A 24-hour exploitation window means dependency inventory has to be a query you run in five minutes, not a report you commission in five weeks.

Sequence: vCenter before ESX for compatibility, Rails before both because it is pre-auth and mass-scannable, Fastjson containment in parallel because it ships in hours and the bump does not.

What to do

  1. Run a resolved-dependency-tree scan for Fastjson across every JVM artifact in production, ranked by whether the service parses JSON from an untrusted origin, and deploy the WAF virtual patch plus JVM child-process alerting while patching proceeds.

  2. Bump Active Storage on every internet-facing Rails app, grep config for cookies_serializer, and treat any :marshal or :hybrid host that was internet-reachable pre-patch as potentially already executed on.

  3. Re-key the vulnerability SLA to exploitation signal (KEV/EPSS plus vendor severity override) instead of a CVSS base-score threshold, this sprint.

Two Settings Beat Every Model Upgrade This Year

The cheapest capability and cost wins available right now sit in your agent loop's configuration, and most teams have never recorded that configuration as an artifact.

The token count is the tell

More compute pushes output tokens up. These went down 6x while the score roughly tripled. That is the signature of redundant re-derivation disappearing. A model that loses its reasoning trace between turns spends most of the next turn rebuilding state it already had, then emits a slightly different plan than the one it made last turn. The bill arrives twice: once in inference cost and latency, once in plan drift across iterations. Retention moves work out of repeated generation, which is expensive, sequential and per-token, and into managed context, which is cheaper, cacheable and prefill-parallel. Compaction bounds the growth so the window survives. Compaction is lossy, so validate it on tasks with long-range dependencies before trusting it broadly.

Where the money actually goes in a stateless loop

Same mechanism, cost side. A traced 45-person Claude Code deployment over 30 days found only 14% of input tokens were actual user prompts. Prior assistant context took 30-45% of input spend, and 78% of that was replayed tool_use blocks rather than conversation text. Tool results were 23% of input on their own. MCP schemas get re-serialized and prepended every turn, so ten servers exposing fifty tools runs about 16,000 tokens per turn, and an install that never gets called pays that tax indefinitely. That is a protocol gap, not config hygiene. MCP has no progressive schema disclosure.

LeverMeasured gainThe catch
Retained reasoning plus compaction3x score, 6x fewer output tokensCompaction drops long-range constraints
Thinking effort high to medium76% fewer output tokens at equal SWE-bench completionPinned fleet configs may still be on high
Deep Agents v0.7 scaffold trim65% fewer base input tokens, flat performanceLess prompt surface to customize
Pruning unused MCP serversUp to ~16k tokens per turn recoveredRequires per-workspace allowlists

The headline framing needs care. Every figure above already assumes prompt caching, which discounts replayed context to roughly 10% of the standard input rate. So 86% of tokens is not 86% of dollars, and cache misses are the real cliff: an idle session past the cache TTL replays bloated context at full price. Cache-hit rate belongs on the dashboard next to token count. On savings claims, the vendor range of 22-48% ships without methodology. The one number with a verifiable chain of custody is Comet's own dogfood result, median output cost falling from $229 to $181 per million output tokens, a 21% cut with no velocity change. Budget against 21%.


The bake-off has an unlogged confound

Log10/Everest's ClinReg run is the corroborating evidence, and it is blunt. Several results moved more with how a model was run than with which model it was, and the authors could not separate Claude Code harness cost from Opus and Sonnet model cost. The pricing consequence is severe. GLM 5.2 scored 87.4 against GPT 5.6 Sol's 88.4, inside one standard deviation, at 33.8% of the cost per task. There is an 11x cost spread inside a two-point score band. Every model produced syntactically valid Python on the first attempt. The entire spread came from runtime-error repair and knowing when to stop.

In agentic systems the model is a parameter. The validator, the stopping rule and the judge topology are the system.

Two consequences follow for selection. Rank on failure mode, not score: instrument omission rate and fabrication rate as separate deterministic metrics, because "went quiet" and "confidently wrong" have different blast radii. Then record a harness config hash on every eval record and hard-fail comparisons across differing hashes.

What to do

  1. Enable retained reasoning and context compaction in your agent loop behind an A/B this sprint, recording output tokens per completed task alongside accuracy.

  2. Add a harness config hash (validator strictness, max retries, stopping criteria, tool set) plus a base-versus-task token split to every eval record this sprint, and refuse comparisons across differing hashes.

  3. Run the 15-minute config audit across the fleet: thinking effort to medium, CLAUDE.md under 200 lines out of repo root, and every MCP server with zero calls in two weeks disabled.

Every Stage Reported Success and the Item Never Published

Teams are funding verification of the code their agents write while leaving the customer-visible outcome unmeasured, and the resulting bug class passes code review by construction.

The bug class, decoded

Spotify's own root cause is worth a second read: a small subset of episodes completed normal media processing but missed a downstream publish update, because a newly introduced validation signal was not correctly wired into the logic that wakes up the publishing path. Here is what that means mechanically. There is a publish-eligibility predicate assembled from a set of validation signals. Someone added a signal. Transcode and chunking kept reporting success. The event that wakes the propagation step never fired for the affected items. Work sat in a non-terminal state that looked healthy at every stage boundary.

Code review was never going to catch that. Spotify's remediation says the right thing: make it impossible to add a field used for publishing eligibility without triggering the relevant downstream updates. That is a type-system or registry problem, and it is good engineering to name it as one. The generalizable audit is a grep, not a rewrite. Enumerate every readiness predicate you own. List the fields feeding it. Prove a path fires the wake-up event on change. Write one test asserting the terminal state rather than stage success.


The detection failure was worse than the code failure

A customer emailed at roughly 17:30 on 24 June before the automated alerts fired. The incident report initially understated that until pushed with email records. That is the signature of aggregate-throughput monitoring. If 99% of items publish, the dashboard is green while a subset is permanently stuck. The alert that catches it is per-item and boring. Most teams lack it because it requires modeling a terminal state at all.

SignalWhat it detectsWhat it misses
Stage success rateCrashes, throughput collapseItems stuck in a healthy-looking non-terminal state
Aggregate p99 latencySystemic slowdownTenant-scoped permanent stalls
Time-in-state exceeds N x p99, per tenantThe exact failure aboveNothing here, if terminal state is modeled
Customer-visible per-item statusTurns an outage into a self-serve answerRequires shipping a product surface

The asymmetry every team adding agents is about to reproduce

Read the two facts together. Spotify rebuilt test automation so agents could be supervised at scale, and adding an automated judge lifted agent pull-request success from roughly 25% to 80% after static codemods had bloated into thousands of lines of edge cases. Verification is called the single most important thing when agents are used, and the place most companies underinvest. Meanwhile the creator portal rendered NaN% across the UI during an incident, there was no status page, and per-item processing status has never shipped.

Nothing here proves agents caused the publish bug. A mis-wired validation signal is a 2010-vintage mistake. The asymmetry is subtler: code verification scales with agent adoption, outcome observability does not, unless somebody writes it into the roadmap.

The same gap is documented one layer up in agent stacks. Agent observability instruments the agent and misses upstream data freshness, lineage and quality. That produces the highest-probability silent incident in the category. A fully traced agent gives a confident answer from a table that stopped updating fourteen hours ago, and every span stays green. The fix is days of work. Attach dataset identity, last-updated watermark, row count and lineage version to every retrieval span, then alert on staleness at answer time rather than at pipeline-failure time.

Verify code and you supervise your agents. Verify outcomes and you find out whether the customer got the thing.

What to do

  1. Run a gating-field audit this sprint: enumerate every readiness predicate you own, verify each contributing field triggers downstream propagation on change, and add one test per pipeline asserting the terminal state.

  2. Add per-item staleness paging bucketed by tenant, firing when a work item exceeds N times its p99 time-in-state without reaching terminal state, routed to on-call rather than a dashboard.

  3. Attach dataset identity, last-updated watermark, row count and lineage version to every agent retrieval span this quarter, and alert on stale-data-at-answer-time.

The bottom line

These failures share one property: every control involved reported success. The checks in place graded who signed a release, whether a stage completed, or how a scoring formula rated a flaw — never whether the code was hostile, whether the customer got the artifact, or whether anyone was actually exploiting it. That is a measurement problem masquerading as a coverage problem, and adding another scanner or dashboard makes it worse by widening the green surface. Pick your three highest-consequence gates, inject a synthetic failure into each, and confirm it pages. Delete any gate that only proves a step ran.