The Unload Path Nobody Tests Still Exits Zero
Cost dashboards, an embedded database, and CDC sinks each broke in a mode their vendor never exercised, and every one of them returned a clean exit code while doing it.
Why the number goes negative
A signed 32-bit integer stops at 2,147,483,647. A 149 GB scan is roughly 160 billion bytes. Write that into an INT32 column and it wraps to a small negative value, which is the -1.5 GB Espresso AI reported. Nothing errored. The unload wrote a file, the job went green, and the corruption is deterministic. It lands on exactly the rows with the largest values.
Trace the propagation before ranking the urgency. Sum bytes scanned per team and the heaviest queries subtract from the total, so chargeback under-bills the biggest consumer. Sort an optimization backlog descending by bytes scanned and the worst offenders sink to the bottom. Join exported telemetry against a job registry on any of the nine wrapped columns and rows silently fail to match. The remedy is two lines of SQL plus one policy: cast the affected columns before export, then add a schema assertion that fails the pipeline on negative bytes, negative durations, or negative row counts. Platform telemetry is data nobody on the team wrote and nobody tests. Validate it at the boundary like any third-party payload.
The pragma you tuned is code upstream never ran
Tailscale spent six months tracing repeated database corruption to a race between SQLite's WAL checkpointing and write transactions, a bug that sat latent for 16 years. It surfaced for them because they checkpoint aggressively and manually. That is the transferable part. The default path collected 16 years of accidental fuzzing from millions of deployments. A configuration hand-tuned for throughput gets whatever coverage the local suite gives it, which is usually none. Good engineering, wrong test population.
So the work is an inventory, not a patch: enumerate every embedded SQLite deployment, write a one-line justification for each non-default pragma, pin a post-fix version, and put PRAGMA integrity_check into a startup or health check so corruption arrives as an alert instead of a support ticket.
CDC gives you ordering, not idempotency
Change data capture buys retryable, ordered, observable downstream writes. It does not make a destination safe to receive the same event twice, and replay after recovery is precisely when it will. Classify every sink into one of three buckets:
- Idempotent by upsert. The write is naturally convergent.
- Idempotent by key. A durable dedupe key, such as source LSN plus primary key, persisted atomically with the side effect.
- Explicitly at-least-once tolerant. Double-counting is documented as acceptable by an owner.
Anything unclassified is a duplicate-write bug waiting for the next replay, when nobody is reading the diff carefully. Metrics double, side effects fire twice, and the incident gets blamed on the replay rather than the sink.
The same discipline applies to numbers you did not produce
There is a tension inside the data reporting. The same material arguing that a valid benchmark must disclose workload shape, cache state, and scaling limits also carries a 3.4x DuckDB-versus-EMR-Serverless claim measured on 1,000 JSON files, with no cache or cold-start disclosure. At that size the ratio is mostly cold-start tax and single-threaded JSON parsing. The direction is probably right: single-node wins the sub-scale tier. The ratio does not transfer.
The security column of this briefing rhymes exactly. CSO First Look describes a defense that never starts rather than one that fails. Same reflex in both columns: assert the invariant on the path you rarely take.
Every correctness failure here came from running a trusted system slightly off its happy path, which is also where the untested code lives.
What to do
Grep every pipeline for QUERY_HISTORY reads today, add explicit casts on the nine overflowing columns before Parquet unload, and invalidate the exports already landed in the warehouse.
Add a schema assertion this sprint to every telemetry pipeline that fails the run on negative bytes, durations, or row counts.
Classify every CDC sink as idempotent-by-upsert, idempotent-by-key, or explicitly at-least-once-tolerant by the end of this sprint, and file the unclassified ones as bugs.