The Scientist

NadMesh matters because a self-hosted inference box rarely has a contained blast radius. If your Ollama node carries an IAM role or a mounted Kubernetes service-account token, one Shodan hit reads your S3 training buckets, hits the model registry, or talks to the control plane. The GPU is the least valuable thing on the machine.

The method itself is unremarkable, which is the point. A Shodan harvester restocks a scan queue, probes default ports — ComfyUI on 8188, Ollama on 11434, n8n on 5678 — and where it finds an exposed instance scrapes long-lived AWS keys and K8s tokens from environment variables, mounted files, and instance metadata. The operator dashboard claims 3,811 unique AWS keys in roughly two weeks. Treat that as a self-reported bragging ceiling, not verified telemetry. The count is unverified. The mechanism generating it is not.

The same weak link shows up in SaaS token sprawl and shadow AI too: long-lived credentials plus ungoverned egress. Orphaned keys to experiment trackers, managed vector DBs, and model registries sit in notebooks and CI secrets. Separately, about 65% of senior decision-makers report piping prompts and data through unlogged endpoints. Different surfaces, same root cause: credentials and data flows nobody is watching.

The fix is structural, not reactive. Short-lived, scoped tokens (IRSA/OIDC) expire before a harvested credential is useful, which neutralizes NadMesh's business model against a given account without requiring anyone to catch the breach first. For egress, a sanctioned inference gateway — a logged proxy with PII redaction and per-team quotas — converts invisible risk into an instrumented flow. It's the approach that survives the reality that leadership won't self-restrict. Ban-based policy has a compliance ceiling; a better default path doesn't.

One adjacent shift worth tracking: model errors are getting reclassified as security and legal incidents, not just accuracy problems. Drift detection that pages nobody is a metric nobody reads, not incident response. Wiring drift thresholds, confidence-collapse detectors, and output-safety filters to escalation owners and an automated rollback to last-known-good is what turns a failing model into a runbook instead of a dashboard.

The interesting part of the early-abort result is the access requirement, not the 60% figure. A step-1 abort classifier is conceptually cheap: a small model over first-step signals like confidence, entropy, and hidden-state features predicting terminal failure. Those features require model internals, which is trivial on open-weight models you host and impossible behind closed APIs that never expose logits. That makes this efficiency win one more line in the self-host column.

The reported payoff cuts agent compute up to 60% while retaining ~90% task success, framed as an orchestration retrofit rather than a model swap. The more portable sibling is auto-tuned skill files, which optimize an external artifact against real system diffs to lift task success 28.6% with zero retraining, a fit for data agents fighting schema and pipeline drift. Both results sit at news-roundup grade: 0.75 confidence, 'up to' framing, no ablations, no median. That makes the numbers cheap to validate but not yet safe to trust.

What makes them actionable is a companion framework: eval complexity scales with architecture. The recipe stays constant. Pick a task, collect eval data, build a grader. The grading modality shifts every time you add a component:

The uncomfortable implication: if you graduated to a multi-agent orchestrator but your evals still score only the final answer, you're blind to exactly the coordination failures the orchestrator introduced.

Underneath all this, the protocol question finally settled: ACP has been folded into A2A. MCP owns agent-to-tool; A2A owns agent-to-agent, with peers discovered via an Agent Card and an input-required state that loops mid-task. Production stacks run both, and any lingering ACP code path is migration debt worth killing before it compounds.

The study design is what makes the null worth taking seriously. Humlum & Vestergaard linked self-reported survey data to administrative earnings and hours records across roughly 25,000 workers, 7,000 workplaces, and 11 high-exposure occupations. Workers self-reported real savings — 2.8% of total work time, with 64–90% reporting some benefit — but the administrative data puts the estimate near zero, and the confidence interval rules out average effects above roughly 2%. That's a tight bound, not a shrug. The sample size is doing real work here, which is the part a smaller study couldn't have delivered.

Two explanations survive contact with this data, and they don't collapse into one another. There's a conversion gap, where the saved time got redirected into other tasks and never showed up as output. And there's a measurement gap, where the value flowed into channels — revenue, quality, avoided risk — that nobody instrumented. If you're building eval harnesses for production models, telling these two apart is not a side quest. It is the job.

The translation to eval practice is mechanical, which is the good news. For every production model, trace offline metric → workflow change → business KPI, and check that the terminal KPI actually shows up in telemetry. If it doesn't, you can't attribute value to anything, and you will reproduce the same false null the administrative records nearly produced. Pre-register a capture mechanism — throughput, cost, SLA, reallocation — before rollout. Efficiency without a designated sink diffuses. It doesn't concentrate on its own.

Two adjacent findings sharpen this. Multi-step agents compound error at a rate that single-turn accuracy hides entirely: a model at 95% per-step reliability fails roughly 40% of 10-step tasks. The benchmark that reports per-step accuracy is not measuring the bottleneck you'll hit in production — measure task-completion rate and trajectory correctness instead. Separately, the claim that engineering and research have fused is testable against this same logic: whoever owns the versioned, reproducible eval harness is the one whose hypotheses get tested against reality rather than against a leaderboard. Eval infrastructure is the throughput multiplier here, not overhead.

The honest caveat: a two-year post-ChatGPT window is plausibly underpowered against diffusion lag. Solow's paradox took roughly two decades to resolve, and nobody called that a failure of computers. Read this result as "measure the conversion," not "AI doesn't work" — the data supports the narrower claim, not the broader one.