19 instances of database corruption. Six months of “shaky uptime.” Over an hour of downtime per incident in the early stages. And the root cause? A data race so rare that the SQLite developers had never seen it occur organically, despite the bug lurking in their codebase for 16 years.
What makes this story genuinely terrifying isn’t that SQLite failed. It’s why it failed, and what it means for every engineering team that treats “boring technology” as a risk-free default.
Let’s dig into how Tailscale’s deeply buried 16-year-old SQLite bug turned production stability into a forensic investigation, and what it reveals about the dark side of embedded databases at scale.
When “Boring Technology” Bites Back
Tailscale’s control plane runs on a sharded architecture, with each shard powered by a single SQLite database accessed by one Go process. This is exactly how SQLite is meant to be used, the textbook single-writer scenario. They adopted SQLite back in 2022 precisely because it’s “boring technology” in the best sense: well-known, reliable, and battle-tested.
The setup was so standard that it had been running without incident since early 2023. Then, in August of last year, a data pipeline reading their S3 backups reported something alarming: PRAGMA integrity_check detected corruption.
As Alex Chan detailed, SQLite corruption “is possible, but it’s highly unusual and not something you should encounter in normal operation.” They repaired the database, investigated, and moved on. Then it happened again. And again. And again, 19 times over six months.
Here’s the part that should make every infrastructure engineer uncomfortable: the team couldn’t reproduce the bug, couldn’t identify common triggers, and couldn’t find any recent code changes that could explain it. The systems that write this data had been operating unchanged for years.
The Vanishing Write: A Database Mystery in Three Acts
The breakthrough came from an unexpected place: a transaction logging pipeline built for disaster recovery. Since SQLite provides serialisable transactions with a single writer, Tailscale could stream every modifying SQL statement to a log file and replay them against a known-good backup to recover corrupted databases without rolling back data.
But during two incidents, the transaction logs failed to replay cleanly. Data written and committed by one transaction had inexplicably vanished, invisible to later transactions. No error was raised. No lock was held. The write simply… disappeared.
This contradiction, a committed write that never materialized, was the smoking gun. It pointed directly at SQLite’s Write-Ahead Logging (WAL) subsystem, and specifically the checkpoint process.
If you’re not deeply familiar with WAL mechanics, here’s the short version: instead of writing pages directly to the database file, SQLite appends them to a write-ahead log. Periodically, a checkpoint copies those pages from the WAL back into the database file. This process is normally invisible, SQLite decides when to checkpoint automatically.
Tailscale does something different. They take manual control of checkpointing to enable fast, consistent backups. This “tiny” architectural deviation from the default path turned out to be the key that unlocked a 16-year-old data race.
The Debugging Tool That Cracked the Case
The SQLite developers, working under a professional support contract with Tailscale, suspected the checkpoint process early on. But they needed visibility, specifically into the virtual filesystem layer where pages actually hit disk.
SQLite’s layered architecture (parser/code generator → pager → OS interface) made this possible. The team created a wrapper around the virtual filesystem called the tmstmpvfs shim, available in the SQLite public repository. This shim adds detailed tracing information to every filesystem interaction.
Deploying the shim into production meant waiting for the next corruption incident to create a forensic dump. True to form, the bug was unpredictable, after a six-week quiet period, it returned as an “unwelcome Christmas present.”
But when it fired, the logs told the story clearly. The bug, dubbed the WAL-Reset bug, is a data race between a checkpoint and a write transaction:
- A write occurs at a precise moment during a checkpoint
- The checkpointing process gets confused, believing pages have been copied from the WAL into the database file
- Those pages are never written to disk
- Data is permanently lost
- Other pages that reference the missing pages, like indexes, are written
- The database file becomes structurally corrupt
The SQLite team estimates the bug was present since version 3.7.0, released in July 2010. It required exact conditions: WAL mode active, multiple database connections on the same file, and a race between reading and writing at the same memory location. The universe of systems meeting these criteria is small. Among those, the subset where it causes detectable corruption is smaller still.
This is why it survived 16 years. As the WAL-Reset documentation notes, “The developers have never been able to reproduce the bug organically and had to add special testing logic to SQLite that deliberately triggers the circumstances of the bug in order to verify that the issue has been fixed.”
The Fix That Almost Wasn’t
The SQLite team released the fix in 3.52.0, adding an additional check to the checkpointing function that detects when the WAL has been reset by another thread. Tailscale rolled it out carefully, canary shards first, then the full fleet.
Their backup monitor immediately turned red: 13 databases reporting corruption.
The heart-stopping discovery? These weren’t real corruption incidents. SQLite 3.52.0 had introduced a second bug involving stale expression indexes. Tailscale stored high-precision timestamps as text, converted them to floating-point numbers in virtual generated columns, and 3.52.0’s text-to-float conversion optimization subtly changed rounding behavior, making perfectly valid databases look corrupt.
The SQLite team withdrew 3.52.0 entirely and released 3.51.3 with only the WAL-Reset fix. Tailscale fixed their side by reducing timestamp precision to integer seconds. The SQLite team later added a self-healing index feature in 3.53.0 to prevent the stale expression index problem entirely.
Proof of Concept: “SQLite Party Mode”
After deploying the fix, Tailscale still lacked positive proof that the exact bug had occurred in their production environment. An absence of corruption doesn’t prove anything, they’d already survived one deceptive six-week calm.
So they patched their SQLite driver to log a warning whenever the collision conditions for the WAL-Reset bug occurred. If the warning fired without corruption, they’d know the fix was saving them.
For two months: silence. Doubt crept in. Was the warning broken? Was the theory wrong?
Then, finally, the alert fired, nicknamed “SQLite Party Mode” for the celebration it triggered:
The warning proved the collision conditions do occur in Tailscale’s production environment, the fix had genuinely protected them from what would have been another corruption incident.
The Ignominious Speed of Modern Tooling
Here’s where this story gets weird. Antithesis, a company building deterministic testing infrastructure, decided to see if their platform could reproduce the WAL-Reset bug, not by matching Tailscale’s aggressive checkpointing patterns, but by running a completely generic workload with writes and checkpoints running concurrently.
Using Claude with Antithesis agent skills, an engineer instrumented SQLite 3.51.2 with standard assertions like “no lost committed writes” and “database is not corrupt.” The workload was banal, exactly the kind of thing you’d expect in any production SQLite deployment.
The bug was caught in 15 minutes.
The same workload against 3.51.3 came back clean. An engineer on a road trip in British Columbia, using his phone, found and verified a bug that took Tailscale six months of production forensics to identify. The tooling exists, but it’s still not widespread enough to catch these issues before they cause outages.
Lessons for the Rest of Us
This incident is a masterclass in subtle operational risk. Let’s extract the lessons.
1. Boring Technology Has Non-Boring Edges
SQLite’s monolithic design and resilience under resource constraints is genuinely impressive. But “boring” doesn’t mean “risk-free”, it means the risks are concentrated in the corners most people never visit. Every configuration deviation from the standard path is a bet that the community has tested that specific combination.
Tailscale’s manual checkpointing was documented, supported, and publicly recommended for certain use cases. It was also sufficiently unusual that it triggered a bug the SQLite team couldn’t reproduce organically in 16 years.
2. Operational Transparency Has Debugging Value
The transaction logging pipeline Tailscale built wasn’t just a recovery mechanism, it was the diagnostic breakthrough. The ability to see every committed transaction and discover that one had “vanished” was impossible to explain through any other theory. If you’re running databases at scale, consider what observability you’d need to detect a committed-write-was-lost scenario. Real-time data pipelines using SQLite with CDC patterns can expose these issues earlier.
3. The Standard Path Is the Tested Path
The most uncomfortable insight: Tailscale was using SQLite “correctly.” Their architecture, single writer, one process, one shard, was textbook. But their operational choices (manual checkpointing, aggressive pace, backup-driven timing) took them off the “well-trodden path”, as Chan put it.
“It’s worth remembering”, Chan writes, “running boring technology in a non-standard way is a risk.” The common paths are incredibly well-tested because many people walk them. Every step off that path means fewer people have tested that exact combination of features.
4. You Can’t Assume Your Database Is a Black Box
SQLite’s use in durable, distributed workflows with replication is becoming increasingly popular, especially with the rise of local-first architectures that treat the device as the primary data store. But this incident proves that assumptions about embedded databases being simple and safe need serious re-examination when you push them beyond standard usage patterns.
5. Recovery Speed Is a Feature
Tailscale’s incremental improvements, hard-stop on corruption detection, automated backup monitoring, improved runbooks, cut recovery time from over an hour to under an hour. Schema migrations and other live operations can cause similar cascading delays when they go wrong. If you can’t detect and recover from database corruption in minutes, you’re not ready for production at scale.
The Cost of Being First
The WAL-Reset bug affected fewer tailnets than most incidents, thanks to Tailscale’s sharded architecture. The team notes that “the majority of shards and tailnets were never involved in a database corruption incident.”
But trust doesn’t work that way. Every status page event, even for incidents that don’t touch you, erodes confidence. Six months of that, even with rare, isolated incidents, is a tax on customer faith.
The final tally includes some silver linings. The tmstmpvfs shim is now an open-source tool funded by Tailscale, valuable for diagnosing similar bugs in the future. The transaction logging pipeline became a permanent recovery mechanism. The team “fixed dozens of other incidental issues” spotted during the investigation. And the SQLite bug itself is now fixed for everyone.
But the deeper lesson is simpler and more uncomfortable: the most expensive failures are the ones that hide in plain sight. The WAL-Reset bug sat dormant for 16 years, waiting for the right combination of timing and operational choices. There are probably more such bugs out there, in SQLite, in PostgreSQL, in the infrastructure you rely on daily.
The question isn’t whether you’ll encounter them. It’s whether you’ll have the instrumentation, the recovery procedures, and the patience to survive when you do.
