Lost acknowledged writes
Jepsen found NATS JetStream 2.12.1 losing 679,153 of 1,367,069 acknowledged writes after file corruption, and acknowledging messages that had not been fsynced.
jepsen.io/analyses/nats-2.12.1Deterministic simulation testing for Go
seedfall runs your unmodified Go services inside a simulated world, then crashes them mid-write, loses their fsyncs and partitions their network. When something breaks, you get the seed, the exact fault schedule and a trace. Runs on the stock Go toolchain and costs nothing.
seedfall run finds seed 1 losing acknowledged entry 27 in rosedblabs/wal, then seedfall shrink cuts 26 faults to 2. Waits trimmed, shrink at 4× speed.Built on and tested against
What is seedfall
The bugs that lose data come from rare combinations: a crash between write and fsync, a restart that misreads a half-written record, a partition during a leader change. Normal tests almost never hit them, and when production does, you can't reproduce it.
seedfall compiles each node to GOOS=wasip1 and runs it under a scheduler that owns every source of nondeterminism: time, randomness, goroutine interleaving, disk and network. Same seed, same run, byte for byte.
The problem
These aren't hypothetical. They keep turning up in mature Go systems years after release.
Lost acknowledged writes
Jepsen found NATS JetStream 2.12.1 losing 679,153 of 1,367,069 acknowledged writes after file corruption, and acknowledging messages that had not been fsynced.
jepsen.io/analyses/nats-2.12.1Recovery that misreads its own log
A crash mid-append leaves a torn record. Without truncation on recovery, the next write is glued onto garbage and silently lost on the following restart.
WalkthroughDivergence after a crash
etcd 3.5.0–3.5.5 could diverge after a crash during defragmentation, and it went unnoticed for five patch releases.
etcd-dev noticeHow it works
On wasip1, the Go runtime is single-threaded and reaches the outside world through three calls: random_get, clock_time_get and poll_oneoff. seedfall implements them. Map order, select choices, timers and goroutine scheduling all follow the seed. There's no linkname, no source rewriting and no patched runtime, so new Go releases don't break it.
# every seed runs twice, traces compared line by line$ seedfall run -c seedfall.fixed.json -seeds 1-1000 -check property seeds reached seeds true sometimes 1000 406 recovery found a torn tail ok 1000 seeds passed (0 hit the time horizon)
seedfall cause replays a failing seed to dozens of moments and re-randomizes everything after each one: scheduling, latency, disk tearing and every future fault. The share of futures that still fail is the probability of failure from that moment. Bystander faults leave the curve flat. The real cause makes it jump. No snapshots are needed, because replay is exact.
# kvlog, seed 1: 440 alternate futures in 24s0s 5% ██ 292.108ms 5% ██<- partition757.268ms 0% 757.268ms 75% ███████████ <- crash during disk op #56 1.084433s 75% ███████████ 1.7052s 100% ███████████████ <- crash server biggest jump: 0% -> 75% after the crash mid-write
Disk
Your code uses plain os.File. A crash keeps only what was synced, optionally a torn prefix of pending writes. fsync can fail with EIO the way Linux really does it, keeping the data readable but not durable.
Crashes
Nodes die while idle or in the middle of a disk write. Each node is its own wasm instance, so a restart starts from a clean process over the surviving disk.
Network
Partitions, latency and resets for plain net.Listen, net/http and hashicorp/raft's stock TCP transport, or through a one-line sim.Listen.
Correctness
Always, Sometimes and Reachable are counted across the sweep, so you know the dangerous paths ran. Linearizability is checked with Porcupine.
Filesystem
Turn on strict_dirs and a create or rename survives a crash only once its parent directory is fsynced. This is the bug class ALICE found in LevelDB.
Debugging
seedfall shrink drops faults while the failure persists and saves a schedule that replays exactly. 37 faults became 2 on the fsyncgate bug.
Proof
Every result below replays from the seed shown. Each finding has a root cause with file and line, and a patched copy that passes hundreds of seeds.
nats-io/nats-server v2.15.2 · 21k stars · SyncAlways · any filesystem
JetStream writes a stream's config and its checksum as two separate files. A crash between them leaves the config without a checksum, so on restart the stream is skipped. The client re-creates it, but NATS only rewrites the files when the config is missing, so the checksum never comes back. The stream accepts and acknowledges publishes as normal, then vanishes with all of them on the next restart.
fileStore.writeStreamMeta (server/filestore.go:1011) writes meta.inf and meta.sum as two separate atomic writes. Recovery (server/jetstream.go:1453) skips a stream with no meta.sum, and newFileStoreWithCreated (server/filestore.go:636-643) only rewrites the metadata when meta.inf is missing or empty.meta.inf is renamed into place and before meta.sum is, then any later crash while the re-created stream acknowledges publishes. The same schedule with the first crash at disk op 10 passes.$ seedfall replay -c seedfall.always.json -seed 40 \
-faults metasum.faults.json
0.158s crash node#1 during disk op 9
5.000s crash node#2, restart at 5.500s
FAIL stream EVENTS lost after restart,
186 acknowledged messages gone
crash at disk op 10 instead ok
rosedblabs/wal v1.3.8 · used by rosedb · any filesystem
A crash in the middle of a write leaves a torn chunk. On restart, Open seeks to the end of the segment without checking it, so new writes are fsynced and acknowledged after the garbage. The next replay stops at the torn chunk with invalid crc, and everything written since is unreachable. rosedb v2 refuses to open at all.
openSegmentFile seeks to the end of the last segment (segment.go:116, :126) without checking the tail. Replay later hits the torn chunk and returns ErrInvalidCRC (segment.go:453), and rosedb v2.4.0 loadIndexFromWAL (db.go:686-691) returns that error from Open.Open by truncating and fsyncing at the end of the last valid chunk, bounds-check reads in readInternal, and fsync the WAL directory in rotateActiveSegment.rosedb.json and rosedb.strict.json.$ seedfall shrink -c rosedb.json -seed 1
faults before the failure (2):
427.268ms crash server during disk op #56
1.7052s crash server, restart at 1.942589s
FAIL acknowledged entry 27 lost: server now
answers "VAL e141-1111111111111111111111"
tidwall/buntdb v1.3.2 · 4.8k stars · SyncPolicy: Always
buntdb compacts its append-only file automatically. Shrink writes a fresh file, closes it and renames it over the database, but never fsyncs the file or the directory. A crash after a shrink can bring the database back without acknowledged writes, even with the strictest sync policy.
(*DB).Shrink (buntdb.go:636) writes <path>.tmp, closes it (buntdb.go:750) and renames it over the database (buntdb.go:757) without fsyncing the file or the directory. backgroundManager (buntdb.go:547) runs it automatically.f.Sync() before f.Close() in Shrink, and an fsync of the parent directory after the rename.seedfall.shrink.json and seedfall.shrink.strict.json.$ seedfall replay -c seedfall.shrink.json -seed 27 FAIL seed 27 at t=5.559s client#1: acknowledged write lost: k22=v262, server now answers "NONE" patched f.Sync() before rename · 300 seeds ok
tidwall/wal v1.2.1 · 733 stars
A crash during an append leaves a partial entry, and every later Open returns log corrupt. That much is known (issue #21). seedfall also showed that the pending fix, PR #22, is incomplete: it skips the partial entry but never truncates it, so a shorter later write leaves stale bytes behind and the segment corrupts again.
loadNextBinaryEntry (wal.go:599-606) returns ErrCorrupt for the partial tail entry, so Open fails forever. PR #22 trims the entry in memory but never truncates the file, and cycle() (wal.go:355-376) never fsyncs the log directory.ErrCorrupt in the middle of the log.load(), plus an fsync of the log directory in cycle().tidwall.json and tidwall.strict.json. 0.366s crash server#1 during disk op 56
0.730s server#2| open wal: log corrupt
FAIL seed 1 · 1 fault
with PR #22 FAIL seed 117
with PR #22 + fix ok 1500 seeds
hashicorp/raft v1.8 · 9.2k stars · strict POSIX directories
Raft fsyncs the snapshot files and renames the directory into place, but never fsyncs the snapshot directory itself. Under strict POSIX semantics, one crash leaves an empty snapshot after the log was already compacted, and NewRaft panics on every restart. On ext4 and XFS a file fsync usually persists the entry anyway, so the exposure is to strict filesystems.
FileSnapshotSink.Close fsyncs state.bin and meta.json, renames the snapshot directory into place and fsyncs its parent, but never fsyncs the snapshot directory itself. After log compaction, the empty snapshot is skipped and NewRaft panics with log not found (api.go:617).strict.lin.json, seed 7).s.dir and Sync() it before the os.Rename in FileSnapshotSink.Close, skipped on Windows like the existing parent-directory sync.n2#2 | [ERROR] failed to get log: index=1
error="log not found"
n2#2 | panic: log not found
FAIL n2#2 crashed: exit_code(2)
patched seed 7 ok · seeds 1-120 ok
More bugs, in storage and raft libraries
And no false alarms
| hashicorp/raft v1.8 | 3 nodes, 4 concurrent clients, Porcupine | 600 seeds durable · 60 linearizable |
| hashicorp/raft, stock TCP transport | unmodified net.Listen via net_overlay | 20 seeds linearizable |
| etcd-io/raft v3.7 | raftexample-style WAL and snapshots | 100 seeds · 100 strict |
| NATS JetStream, SyncAlways | R1 file stream, torn writes, crashes mid-I/O | 150 seeds |
| NATS JetStream, 3-node cluster | R3 stream on NATS raft, unmodified via net_overlay | 200 seeds |
| tidwall/buntdb, no shrink | torn writes, crashes mid-I/O | 300 seeds |
| examples/kvlog, fixed | crashes, partitions, torn writes | 3,000 seeds |
Each system also has a negative control, such as skipping fsync, and seedfall catches every one, so a pass means something.
All findings and passing systems →For AI coding agents
Agents write more storage and replication code than anyone can review. seedfall mcp gives Claude Code, Cursor and other agents four tools: sweep seeds, replay one exactly, shrink a failure to its faults, and find the moment it became inevitable. The agent reads the fault schedule, fixes the bug, and attaches the passing sweep to its pull request. Your own agent does the reasoning, so no API key or hosted service is involved.
1. Install seedfall
2. Connect your agent
Paste into any MCP client that reads an mcpServers config.
Add it as a step after actions/checkout. Each run uploads an HTML triage report that groups failures and gives a replay command for each.
# the agent callsseedfall_run { "config": "seedfall.json", "seeds": "1-500" } seedfall_shrink { "config": "seedfall.json", "seed": 1 } seedfall_cause { "config": "seedfall.json", "seed": 1 }# and gets back{ "minimal_faults": 2, "failure": { "reason": "client#1: acknowledged entry 27 lost: …", "faults_before_failure": [ "427.268ms crash server during disk op #56", " 1.7052s crash server" ] } }
Compare
Every Go option so far requires you to give something up.
| seedfall | gosim forks | faultline & co. | testing/synctest | Antithesis | |
|---|---|---|---|---|---|
| Stock Go toolchain | yes | needs linkname flags | yes | yes | yes |
| Code changes | none with net_overlay | none | inject net and fs everywhere | fakes required | none |
| Multi-process crash + restart | yes | yes | partial | no | yes |
| Torn writes, lost fsync | yes | lost writes | varies | no | yes |
| Controls goroutine order | yes | yes | varies | no | yes |
| Process pauses, clock jumps | yes | not documented | varies | no | yes |
| fsync errors, clock skew | yes | not documented | varies | no | clock only |
| Runs Antithesis SDK assertions | yes, unchanged | no | no | no | yes |
| Coverage properties | yes | no | no | no | yes |
| Durability mutation testing | yes | no | no | no | agent skill |
| Causality: P(fail) over time | yes | no | no | no | yes |
| MCP server for agents | yes | no | no | no | agent skills |
| Windows | yes | no | yes | yes | SaaS |
| Price | free, Apache-2.0 | free | free | free | from $17,520/yr |
Antithesis price from its AWS Marketplace listing. gosim needs -checklinkname=0 and private runtime contracts.
Quickstart
One line
Create listeners and dialers through sim. Outside the simulator it is plain net.
ln, err := sim.Listen("tcp", ":7000")
conn, err := sim.Dial("tcp", "server:7000")
Describe the world
Nodes, a workload that checks your invariants, and how hostile the world should be.
{
"nodes": [
{ "name": "server", "pkg": "./server" }
],
"workload": {
"name": "client", "pkg": "./client"
},
"faults": {
"crash_per_sec": 0.5,
"torn_writes": true,
"fsync_eio": 0.01
}
}
Sweep, replay, shrink
Failures print the seed. Replays are exact. Shrinking keeps only the faults that matter.
$ seedfall run -seeds 1-5000 $ seedfall replay -seed 9 $ seedfall shrink -seed 9