The Engineer

The Convergence

This is not a trend piece. Four distinct supply chain attacks are live against CI/CD pipelines this week. They share one assumption: inputs to the build are trusted because the channel was trusted. The channel is not the artifact.

The build pipeline is running untrusted input as code and calling it a dependency fetch.

Attack 1: GitHub .patch URL Injection

GitHub exposes a .patch view for any commit. The commit message renders inline in that output. GNU patch -p1 cannot distinguish between the real diff and a diff-shaped payload pasted into the commit message. Demo target is .git/hooks/post-applypatch. Silent code execution on the next git am. Reviewers never see it. The payload lives in the commit message, and the diff tab doesn't render that. git cherry-pick is immune. git apply blocks .git/ traversal, then happily applies the injected hunks to working-tree files.

Attack 2: Checkmarx KICS (Lapsus$)

Lapsus$ took Checkmarx's GitHub account and published malicious payloads twice, in March and again last week. KICS runs in CI with source, build artifacts, and cloud credentials in scope. If your pipeline runs KICS, it has been executing attacker code with the runner's full permissions. The downstream is compounding. The Vect ransomware group is now working with TeamPCP to ransom KICS-compromised companies. Vect's encryptor has a flawed algorithm that permanently destroys files larger than 128KB. Paying does not recover the data. The spec destroys it.

Attack 3: ShinyHunters via Anodot

ShinyHunters compromised Anodot, a cloud-cost monitoring SaaS, and pivoted into customer Snowflake data stores. Vimeo, Rockstar Games, Zara, and Payoneer are confirmed. A cost-monitoring tool has no reason to hold data-plane credentials. In practice it accumulates broad read access because nobody scopes the IAM role down. Every SaaS in the stack holding cloud credentials is a potential Anodot.

Attack 4: elementary-data PyPI Trojan

elementary-data, 1.1M monthly downloads, was trojaned for 12 hours through a GitHub Actions script-injection flaw. Malicious version 0.23.3 exfiltrated warehouse credentials, cloud keys, API tokens, and SSH keys from every CI runner and developer machine that pulled it. The vector is the ${{ github.event.*.body }} injection pattern.

The Pattern

Same root cause in all four. The build system treats a mutable pointer as a stable artifact: a .patch URL, a tag reference, a package version, a SaaS API token. GitHub Actions has no lockfiles, no integrity hashes, and no transitive dependency visibility. GitHub acknowledged the gaps and said changing defaults would break existing workflows. That is a permanent condition of the platform now. Plan around it.

The Data Firmed Up

Over 30 engineering teams reported the same failure pattern independently: AI agents produce code that compiles, passes review, and quietly breaks in production. The taxonomy is smaller than expected. Four buckets cover nearly all of it.

The Compounding Trap

Kent Beck calls the downstream result the 'Genie Tarpit'. The mechanism is a feedback loop. Low flexibility makes the next change harder. Harder changes generate more code. More code further reduces flexibility. Eventually even the AI stops being able to make progress on its own output.

A fast junior who writes their own grading rubric needs more review, not less.

Beck's useful observation: flexibility degradation has a delayed feedback signal. Sprint velocity looks fine until the first cross-cutting change lands. Then every generated module is the problem at once. Per-PR metrics will not see it. The bill arrives the first time you swap a data store, change an API contract, or add multi-tenancy.

A New Organizational Failure Mode

A separate pattern should alarm anyone who owns architecture: juniors and PMs are weaponizing AI-generated counterarguments to override senior engineering decisions. An LLM produces a fluent three-paragraph rebuttal to any objection in seconds. The rebuttal reads well. It is not grounded in the system's actual constraints. Seniors stop fighting. Bad ideas reach production. The ADR/RFC process needs to account for the asymmetry explicitly, because the tooling is not going back in the box.

The Defense Pattern That Works

Teams holding the line run the same boring infrastructure. Dual-layer enforcement: Husky pre-commit hooks catching violations locally, plus CI pipeline gates that block merge on code-health regressions. The gates are specific. Complexity thresholds. Coverage minimums. Agent PRs get the scrutiny a new contractor's first PR would get. A complexity budget per PR — cyclomatic complexity, dependency count, API surface delta — flags agent PRs that add complexity without reducing it somewhere else. The model fills in implementation inside human-designed boundaries. Humans own contracts, interfaces, data models, and module decomposition. That division is the whole game.

Chassis-as-Artifact, Not Template

The most useful pattern in Wise's 2025 stack writeup: the microservice chassis ships as a versioned artifact dependency. Not a forkable template. The distinction sounds minor. At scale it decides how you operate. Scaffold from a template and every service starts drifting from it on day one. Six months later a cross-cutting security rollout means a PR against every repo. Wise inverted the dependency direction.

When they needed to roll out SLSA supply-chain security across 700+ Java repos, it was a plugin version bump — not 700 pull requests.

Platform concerns live in a shared library: observability instrumentation, security baselines, config management. Wise extended this with a language-agnostic automation service that can codemod across the entire codebase and auto-generate PRs for teams to review. If you run more than roughly 50 services and still scaffold from templates, this is the shift with the highest payoff available.

The trade-off is honest. The chassis becomes critical infrastructure, and a bad release has blast radius proportional to adoption. That forces exceptional backwards-compatibility discipline and staged rollouts of the chassis itself.

Canary Deployments That Watch the Business

Spinnaker routes 5% of traffic to new versions and watches for 30 minutes. Standard practice. What is not standard: evaluating business metrics alongside technical ones. That catches the deploy that returns 200s while silently computing exchange rates wrong. Catastrophic in fintech, invisible to normal canary analysis. The system auto-blocked hundreds of bad releases in 2024 with zero human intervention. Only 50%+ of services are on Spinnaker, with full migration scheduled mid-2025, which implies an 18-24 month adoption timeline even with organizational commitment.

Observability: Thanos → Mimir at Serious Scale

The migration from Thanos to Grafana Mimir at 6M samples/sec ingestion and 150M active series is a concrete data point for anyone evaluating these systems. Thanos degrades at high cardinality because of query fanout and the store-gateway architecture. Running dedicated observability clusters separated from production workloads keeps the monitoring system from failing during the incident it is supposed to help debug. A preventable failure mode, preventably avoided.

Other Numbers Worth Noting

• iOS zero-change builds: 28s → 2s by migrating 250+ Xcode modules from Xcodegen/CocoaPods to Tuist/SPM
• CI optimization: 15% improvement across 500K monthly builds = 1,000+ hours/month saved
• Data lake: Apache Iceberg on S3 + Trino federated query engine across Iceberg, Snowflake, and Kafka
• ML inference: Ray Serve over SageMaker Endpoints for fraud detection and KYC
• LLM gateway: Multi-provider (Claude, Bedrock, Gemini, OpenAI) with custom LangChain-inspired library — not LangChain itself

Claude Opus 4.7's Stealth Tokenizer Change

Anthropic shipped a new tokenizer with Claude Opus 4.7. The per-token price did not change. Identical inputs now tokenize into 12-27% more tokens. The mechanism is mundane. The new tokenizer has a smaller effective vocabulary for certain byte sequences. Whitespace runs, JSON field names, and repeated punctuation all split into more pieces. A prompt that tokenized to 1,820 tokens on the prior model produced 2,140 on 4.7 in one test harness. That is 17.6% on a single prompt.

The sticker did not move. The meter did.

A dashboard tracking dollars per token will show nothing. Track tokens-per-equivalent-request as a first-class metric. For JSON-heavy prompts, stripping whitespace and shortening field names recovers most of the delta. For code, evaluate whether you need the full file in context or only the diff.

900K Tokens Per Bugfix: The Context Replay Tax

Separately, agentic workloads are exposing a structural cost problem. A single Claude Code bugfix consumes roughly 900,000 tokens, and almost none of it is visible code generation. The majority is context replay. The agent re-reads the repo, tool outputs, prior reasoning, and its own retries on every turn. Pricing is linear in tokens. Token count is quadratic in steps when replay is naive. Double the depth of the plan and the bill more than doubles.

The counter-argument that provider-level prefix caching handles this is sometimes true. Check the response metadata for the cache-hit field. Cache-hit rates drop the moment a tool output changes mid-context, which is every step of an agent loop. The marketing says cached. The billing says otherwise.

The Fix Path

For the tokenizer shift, pull a week of production prompts, re-tokenize with the 4.7 tokenizer locally, and compare counts. Check output tokens too. Completions drift in the same direction. For agent loops, cache the stable prefix, diff tool outputs instead of re-pasting them, and summarize completed subtasks before the next step starts. One research team found that structured reflection summaries between attempts moved Claude-4.5-Opus from 70.9% to 77.6% on SWE-Bench. The scaffolding win likely saves tokens overall by cutting attempt count.