ClarityBeta

Engineering & Technical

The Engineer

5 min read · 1,057 words · 4 moves to watch
The Signal

A Go botnet is scraping cloud keys from exposed Ollama and ComfyUI boxes.

NadMesh scans Shodan for exposed self-hosted AI tooling and already claims 3,811 AWS keys. Shodan finds the box; the operator does the rest. Same day: an OpenSSL HollowByte DoS with a patch already shipped, an unauthenticated WordPress RCE with a live public PoC, and seven RAT-laced npm packages in the Vite chain. Patch the OpenSSL one first. The WordPress PoC is public, so that one won't wait.

In Play

  1. Self-Hosted AI Tooling Is Now an Active Attack Surface

    NadMesh, a Go botnet, Shodan-scans exposed ComfyUI, Ollama, and n8n instances and harvests cloud creds — its dashboard claims 3,811 AWS keys. Also live: OpenSSL's HollowByte DoS (11 bytes → 131KB, patch shipped), an unauthenticated WordPress wp2shell RCE with public PoC, and seven RAT-laced npm packages hitting Vite builds via blockchain C2.

    Ask Clarity

  2. The 2027 Memory Squeeze Meets an Inference-Cost Inversion

    SK Hynix's CEO forecasts the worst-ever memory shortage in 2027, AI demand outrunning supply past 2030 — RAM becomes a first-class capacity variable. Meanwhile Kimi K3 breaks 'open equals cheap': ~half GPT-5.6 Sol's per-token price but 24x DeepSeek V4 Pro. Falling token prices grow total consumption faster than they save.

    Ask Clarity

  3. Rollback Isn't Recovery: The AWS Billing Failure

    AWS's billing computation subsystem broke Thursday night, showing phantom estimates up to ~$2.5B. The engineering signal: rolling back the offending change failed to fix it, because the deploy had recomputed persisted state. Reverting the binary doesn't un-write bad data.

    Ask Clarity

  4. Agent Systems: Cheap Orchestration Wins vs. the Productivity Mirage

    ACP folded into A2A, collapsing the stack to two protocols (MCP for tools, A2A for agents). Two orchestration-layer papers land real wins: early-abort classifiers cut compute up to 60% at 90% success; skill-file auto-tuning adds +28.6%. But an NBER study of 25K workers found ~2.8% task time saved vanishing before P&L.

    Ask Clarity

Deep Dives

Your Ollama Box Is One Shodan Scan From an AWS Takeover

Four live threats hit the build pipeline, the TLS edge, and self-run AI infra — and the OpenSSL bug defeats the mitigation you'd reach for first.

The mechanism that defeats your usual mitigation

OpenSSL's HollowByte hurts because of where the memory lands, not how much of it. Here's what happens: an 11-byte TLS record makes an unpatched server reserve up to 131KB for a body that never arrives. On glibc that reservation sits in an arena the allocator never returns to the OS until the process restarts. Rate-limiting does nothing. A slow trickle of tiny requests starves the worker, and the free cycles that would recover the arena never come. The only clean interim fix is aggressive worker recycling until the patch is in.

The WordPress wp2shell RCE is a different urgency class. Unauthenticated, CVE-assigned, shipping with a working public PoC. Mass scanning is already running. Redis or Memcached-backed WordPress adds a persistent-object-cache condition, which widens the surface past a single-server install.

Why your self-hosted AI is the sleeper

NadMesh targets how these tools actually deploy. The Go botnet keeps a Shodan queue stocked with exposed ComfyUI, Ollama, and n8n instances, then harvests AWS keys and Kubernetes tokens. The operator dashboard claims 3,811 unique AWS keys. These tools ship without auth by design. That frictionless-local default is fine on localhost and a cloud-account takeover the moment the port faces the internet. The seven RAT-laced npm packages in the Vite ecosystem use blockchain-based C2. A domain seizure won't help. The kill switch is on-chain and immutable.

The shared response

These signals all point at patch economics. AI-accelerated disclosure is outrunning calendar-based cadences. The durable answer is risk-based triage on EPSS + CISA KEV and reachability, not raw CVSS. The floor matters more than the ceiling. Default creds and stale protocols are still the actual entry point in government advisories.

What to do

  1. Bind every self-hosted AI service (Ollama, ComfyUI, n8n) to localhost/private networks, front with an authenticating reverse proxy, enforce IMDSv2, and rotate any cloud keys reachable from those hosts.

  2. Patch OpenSSL on all TLS terminators and take/gate every WordPress instance behind a WAF blocking the wp2shell pattern, then hunt for webshells.

  3. Freeze npm additions in the Vite build chain, enable install-time scanning (Socket/npm audit signatures), pin lockfiles with integrity hashes, and rebase patch triage on EPSS+KEV.

The 'Open Means Cheap' Rule Just Broke — and RAM Is About to Get Scarce

Two assumptions baked into your inference cost model — cheap open weights and cheap RAM — are both expiring, and cheaper tokens won't shrink the bill.

The pricing inversion your routing logic didn't expect

Most model-selection layers encode one rule: Chinese open weight equals cheapest. Kimi K3 violates it. Per Artificial Analysis it runs about half of GPT-5.6 Sol per token but roughly 24x DeepSeek V4 Pro. That is a frontier-class open model priced as a premium, not a floor. It decouples four variables your routing conflates: capability, open-vs-closed weights, self-host cost, and per-token API price. Those were never the same axis. Now you have a decision matrix instead of a default branch.

The harder problem is demand elasticity. Per-token prices fall, total consumption rises faster, and spend flows toward the compute layer, not away from it. Jevons paradox for inference. A 40% price drop with 2x volume growth is a cost increase. Model-shopping stops mattering. Compute efficiency is the durable lever.

Where the hardware bites

SK Hynix's CEO forecasts the industry's worst-ever memory shortage in 2027, with AI demand outrunning supply past 2030. Any cheap-RAM line in your capacity plan breaks there. And efficient-to-train is not cheap-to-serve. K3 is a 2.8T-parameter MoE with only 16 of ~900 experts active per token. As one DeepMind researcher put it, a small lab can compress the compute to build a frontier model but still can't afford to serve one. Open weights hand you the checkpoint. They do not hand you the serving bill.

Caveat: several K3 benchmark scores are self-reported, and one Arena chart used a truncated x-axis to exaggerate the lead — treat the pricing facts as firmer than the capability ranking.

What to do

  1. Rebuild inference cost forecasts on an elastic-demand assumption — project total token-volume growth at a 30–50% price drop rather than holding volume fixed — this sprint.

  2. Model a 2x–3x RAM cost scenario against your 2027 capacity plan, right-size your heaviest heaps and caches, and lock reserved commitments before the squeeze.

  3. Stand up an eval harness on your real tasks with a per-token cost column, benchmark K3 when weights drop July 27, and default to a hosted endpoint — don't reflexively self-host 2.8T params.

AWS's Rollback Couldn't Undo a $2.5B Phantom Bill — Here's Why

A hyperscaler just handed you a free post-mortem on the most common gap in incident plans: the rollback that reverts code but not the data the code already mutated.

The diagnosis behind the dollar figures

AWS's billing computation subsystem broke late Thursday. By Friday customers saw estimates in the millions to billions, one Reddit screenshot showing ~$2.5B. Amazon confirmed the numbers were fake. The engineering signal isn't the figure. It's that a rollback of the offending change did not fix the incident. A failed rollback almost always means the change did more than swap code. It mutated or recomputed persisted state. Reverting the binary doesn't un-write the bad computation. That's when teams learn their rollback plan was never a recovery plan for the data layer.

The two things this changes in your stack

First, the direct exposure for AWS customers is your own automation acting on provider-side garbage. Budget kill-switches, cost-based autoscaling caps, and finance pipelines keyed off the Billing API can all misfire on a phantom estimate. Add anomaly detection that alerts without acting destructively.

Second, classify every P0 deploy as reversible-deploy or irreversible-data-mutation. For the latter, a rollback button is theater. You need a forward-fix runbook written before the incident, not improvised at 2am. Every ledger, billing, and event-sourcing system learns this the expensive way.

If your rollback plan doesn't account for state that got recomputed, that isn't a recovery plan. It's a hope.

What to do

  1. Audit critical-path deploys (billing, ledger, anything computing money), classify each as reversible-deploy vs. irreversible-data-mutation, and write forward-fix runbooks where rollback won't recover state.

  2. Add proactive billing anomaly alerting (budget alarms + Cost Anomaly Detection) that notifies but does not trigger destructive automation on provider-side estimate glitches.

Two Cheap Agent Wins — and the Study That Says They Won't Reach Your P&L

The agent-protocol war ended before it started, orchestration-layer wins are getting cheap, and a 25K-worker study explains why latency dashboards look great while business value is zero.

One fewer standard to track

ACP has been merged into A2A. The three-way standards war is now a two-layer stack: MCP for agent-to-tool, A2A for agent-to-agent. A2A absorbed ACP's REST-first patterns. Agent Manifest discovery, direct HTTP, sync-or-SSE responses all carried over, so nothing you built against ACP is gone. If you hedged across three specs, pick one. Commit now.

The wins that live outside the model

Two papers describe techniques that cost no GPU spend. Early failure detection reads a model's first-step internal signals and aborts doomed multi-step tasks. It cuts compute up to 60% while retaining 90% success. That is a lightweight classifier, nothing more, and it is a pure cost play. Skill-file auto-tuning tests skill files against real system-state changes and lifts task success up to +28.6% with zero retraining. That is a pure reliability play. Both sit at the orchestration and prompt layer. Cheap to trial this quarter.

Why the gains may not show up

The counterweight is an NBER study of ~25,000 Danish workers. Self-reported time saved was ~2.8% task time saved. Administrative records showed near-zero change in hours or earnings. Confidence intervals rule out effects above ~2%. Workers redirected the saved time. The value dissipated between the keyboard and the P&L. The fix is architectural. Instrument the downstream KPI with a pre-AI baseline. Close the correction loop. Open-loop automation demos fast and never compounds.

What to do

  1. Collapse any in-flight agent-comms design onto MCP (tool plane) + A2A (agent plane) and drop standalone ACP implementation plans.

  2. Prototype an early-abort classifier on one multi-step agent workflow using first-step internal signals to kill doomed runs.

  3. Before shipping the next AI feature, define the downstream business metric it should move, capture the pre-AI baseline, and verify a closed correction loop exists.

The bottom line

The infrastructure you operate yourself is where both compromise and cost now concentrate — isolate it, instrument it, and write the forward-fix runbook before your own automation outruns your controls.