seedfall

View as Markdown, for AI agents Edit on GitHub

Getting started

This walkthrough takes about five minutes. You will run seedfall on a small example server that loses data after a crash. You will find the bug, cut the failure down to its simplest form, fix it and prove the fix. The example is examples/kvlog, a key-value server that appends every write to a log file.

A few words you will see:

  • A node is one program in your system, such as a server. seedfall runs each node as its own process inside a simulation.
  • A workload is a program that uses your system and checks the answers. It plays the part of a client.
  • A fault is something that goes wrong on purpose: a crash, a lost message, a half-written file.
  • A seed is a number that decides everything random in one run, including when the faults happen. The same seed always gives the same run, so any failure can be replayed.

1. Install

You need Go 1.25 or newer. seedfall runs on Linux, macOS and Windows.

go install github.com/kunalgoesbyken/seedfall/cmd/seedfall@latest
seedfall doctor
git clone https://github.com/kunalgoesbyken/seedfall
cd seedfall/examples/kvlog

seedfall doctor checks your setup and tells you how to fix anything that is missing. Other ways to install (Homebrew, Scoop, a script, Docker) are in install.md. Those need a release newer than v0.2.0.

2. Look at the example

The file seedfall.always.json describes the test: one server, one client and the faults to inject. It extends seedfall.json and changes one setting, the server's -sync flag. Merged, it reads (shortened):

{
  "nodes": [{ "name": "server", "pkg": "./server", "args": ["-sync", "always"] }],
  "workload": { "name": "client", "pkg": "./client", "args": ["-writes", "200"] },
  "faults": { "crash_per_sec": 0.5, "torn_writes": true, "crash_on_io": 0.005 }
}

What this means:

  • The server appends key=value lines to data/kv.log. Before it tells a client "done", it calls fsync. Fsync is the operating system call that forces data onto the disk. Without it, a crash can lose data that is only in memory.
  • The client writes 200 keys. Then it checks that every key the server acknowledged is still there. The check is a sim.Assert call.
  • The faults crash the server about every two seconds and allow torn writes, where a crash leaves a write only half saved.
  • The server's only seedfall-specific code is sim.Listen, which opens its network port.

3. Run a sweep

A sweep runs the test once per seed. Try seeds 1 to 100:

seedfall run -c seedfall.always.json -seeds 1-100

FAIL seed 1 at t=2.891820635s (737 scheduler steps, 445ms real)
  client#1: acknowledged write lost: k96=v96, server now answers "NONE"

Seed 1 fails. The server said it saved a key, and after the crashes the key was gone, even though the server uses fsync before every acknowledgement. That is a real bug.

4. Shrink the failure

Seed 1 had 33 faults, and most do not matter. shrink removes faults and simplifies what is left, while the failure keeps happening:

seedfall shrink -c seedfall.always.json -seed 1

faults before the failure (2):
  427.268ms  crash server during disk op #56, restart at 761.268ms
     802ms  crash server, restart at 802ms
...
shrunk from 33 faults, 1m0s, 200 workload ops
         to 2 faults, 991ms, 63 workload ops

Two faults are enough. Shrink also made them simpler: the first crash restarts 30ms sooner, the second moved from 1.7s to 0.8s, the run ends at 991ms instead of 60s, and the client sends 63 writes instead of 200. (The client can shrink its own size because it uses sim.Budget.)

Here is the bug in plain words:

  1. The first crash lands in the middle of an append. The last line in the log is left half written.
  2. On restart, recovery skips the bad line but keeps appending after it. The next record is glued onto the garbage.
  3. The second crash forces another recovery, which can no longer read that record.

To see the whole run, use seedfall replay -c seedfall.always.json -seed 1. To replay only the two faults that matter, add -faults .seedfall/seed-1.faults.json.

5. Fix it and prove it

The fix is to cut off the half-written tail during recovery. The server already has this behind a flag called -repair, and seedfall.fixed.json turns it on:

seedfall run -c seedfall.fixed.json -seeds 1-1000

property       seeds reached  seeds true
  sometimes            1000         994  a crash interrupted the workload
  sometimes            1000         406  recovery found a torn tail

ok  1000 seeds passed (0 hit the time horizon) in 1m28.94s

All 1000 seeds pass. The table below the result matters too. It shows that the risky paths really ran: in 406 seeds recovery met a torn tail, and each one was handled correctly. A passing sweep that never reached the risky code would prove little.

6. Go further

  • seedfall.fsyncgate.json makes fsync itself fail (fsync_eio). The "fixed" server breaks again, because a failed fsync leaves a hole in the log. seedfall.rewind.json adds the fix.
  • seedfall.strict.json also loses directory changes that were never synced, as POSIX allows.
  • Open .seedfall/failures/seedfall.always-seed-1/timeline.html after the failing sweep for a diagram of every crash and message. Or step through the run with seedfall debug -c seedfall.always.json -seed 1 -at 1.7s (timeline.md).
  • Each of these config files is a few lines that extend seedfall.json. In your own project you can keep such variants as profiles in one file and pick one with -profile, or try a ready-made fault preset with -set faults=resilient (config.md).

Use it on your own project

  1. Make your nodes reachable inside the simulation. Either change net.Listen and net.Dial to sim.Listen and sim.Dial, or set "net_overlay": true to run your code unchanged (networking.md).
  2. Write a workload. It is any Go main package that uses your system and checks the results with sim.Assert, sim.Always and similar calls (properties.md).
  3. Describe the test in seedfall.json (faults.md).
  4. Run sweeps on your machine, then in CI (ci.md), and from your coding agent (agents.md).

seedfall init can write a first config and workload for you.

One requirement: your code must build for GOOS=wasip1. This is Go's target for WebAssembly System Interface, a small sandbox that lets seedfall control time, randomness and the disk. In practice it means no cgo (Go code that calls C). Libraries that need mmap or flock (operating system calls for mapping files and locking them) work when you set "compat_overlay": true; seedfall init turns it on and tells you which dependency is covered (compat.md).