# seedfall

> Find the bugs that only show up when your Go service crashes or the network breaks, and replay each one exactly. seedfall is for people who write Go services that keep data: databases, queues, caches, anything that writes to disk or talks to other nodes. Those services usually pass their tests and then lose data in production, because the bug needs a crash at one particular moment, or a network split at another. Normal tests almost never hit that moment, and when production does, you can't reproduce it.

Install with `go install github.com/kunalgoesbyken/seedfall/cmd/seedfall@latest`. `seedfall init` onboards a Go module, `seedfall mcp` serves every command to coding agents over MCP, and the skills below teach an agent the full loop. Every docs page is an HTML page with a raw Markdown copy beside it. The complete text of every page is in [llms-full.txt](https://seedfall-lemon.vercel.app/llms-full.txt).

## Docs

- [Documentation](https://seedfall-lemon.vercel.app/docs/index.html): seedfall runs your Go system in a simulated world, breaks it on purpose (crashes, lost messages, half-written files), and lets you replay any failure exactly. These pages are grouped by what you want to do. If you are new, read the first one and stop there for now. ([Markdown](https://seedfall-lemon.vercel.app/docs/index.md))
- [Using seedfall from AI coding agents](https://seedfall-lemon.vercel.app/docs/agents.html): This page is for people who let an AI coding agent, such as Claude Code, Cursor or Codex, do the testing work. seedfall can talk to the agent directly through an MCP server (a standard way for an agent to call tools) and ships skills (step-by-step instructions the agent follows). Use it to let the agent find a bug, shrink it, fix it and prove the fix. ([Markdown](https://seedfall-lemon.vercel.app/docs/agents.md))
- [Running Antithesis SDK code](https://seedfall-lemon.vercel.app/docs/antithesis.html): Antithesis is a commercial testing platform with its own Go SDK, a library for writing checks inside your program. If your code already uses that SDK, seedfall runs it unchanged and turns each SDK call into a seedfall check. Read this page if you are coming from Antithesis or share code with a team that uses it. ([Markdown](https://seedfall-lemon.vercel.app/docs/antithesis.md))
- [BUGGIFY: faults inside your code](https://seedfall-lemon.vercel.app/docs/buggify.html): Most seedfall faults hit your program from the outside: a crash, a broken disk, a cut network. BUGGIFY lets you mark spots inside your own code where seedfall may force a rare path on purpose, such as a lost reply, a retry or an error return. Use it when the bug you fear lives in your own error handling, which outside faults rarely reach. ([Markdown](https://seedfall-lemon.vercel.app/docs/buggify.md))
- [Causality analysis](https://seedfall-lemon.vercel.app/docs/cause.html): When a seed fails, you know it failed but not always which event decided it. `seedfall cause` replays the run from many points and shows when the failure became likely and which fault tipped it over. Use it after `shrink` when the short fault list still leaves you asking which fault matters. ([Markdown](https://seedfall-lemon.vercel.app/docs/cause.md))
- [Running seedfall in CI](https://seedfall-lemon.vercel.app/docs/ci.html): This page shows how to run seedfall automatically on every pull request or on a schedule: the GitHub Action, machine-readable output, time limits and sweeps started from `go test`. Use it once seedfall works on your laptop and you want it to guard your main branch. ([Markdown](https://seedfall-lemon.vercel.app/docs/ci.md))
- [Storage engines that need mmap or flock](https://seedfall-lemon.vercel.app/docs/compat.html): Some storage libraries, such as bbolt, pebble and badger, use operating system features called `mmap` and `flock` that seedfall's sandbox does not have. This page explains how seedfall swaps in stand-ins at build time so you can still test the real library. Read it when `seedfall init` or a build tells you a dependency will not compile. ([Markdown](https://seedfall-lemon.vercel.app/docs/compat.md))
- [Config files, profiles and seeded disks](https://seedfall-lemon.vercel.app/docs/config.html): One `seedfall.json` can hold many variations of your test: a base setup, named profiles, ready-made fault presets and one-off command-line overrides. This page also covers starting nodes from an existing disk and restarting them into a new version of your program. Use it when you want to test more than one scenario without copying files around. ([Markdown](https://seedfall-lemon.vercel.app/docs/config.md))
- [Feedback-guided exploration](https://seedfall-lemon.vercel.app/docs/explore.html): A plain sweep tries seeds one after another, which is slow for a bug that needs two rare things to happen in the same run. `seedfall explore` keeps the runs that reached something new and builds on them. Use it when ordinary sweeps pass but you suspect a deeper bug. ([Markdown](https://seedfall-lemon.vercel.app/docs/explore.md))
- [FAQ](https://seedfall-lemon.vercel.app/docs/faq.html): Short answers to the questions people ask most, with links to the pages that go into detail. Start with the first section if you are comparing seedfall to something you already know. ([Markdown](https://seedfall-lemon.vercel.app/docs/faq.md))
- [Configuration and faults](https://seedfall-lemon.vercel.app/docs/faults.html): This is the reference for `seedfall.json`: every field, and every fault seedfall can inject, such as crashes, partly written disk writes, lost network messages and clock jumps. Use it to look up what a setting does or to design the failures you want to test. If you are new, start with [getting-started.md](getting-started.md). ([Markdown](https://seedfall-lemon.vercel.app/docs/faults.md))
- [Getting started](https://seedfall-lemon.vercel.app/docs/getting-started.html): This walkthrough takes about five minutes. You will run seedfall on a small example server that loses data after a crash. You will find the bug, cut the failure down to its simplest form, fix it and prove the fix. The example is `examples/kvlog`, a key-value server that appends every write to a log file. ([Markdown](https://seedfall-lemon.vercel.app/docs/getting-started.md))
- [Use your Gostman collections as workloads](https://seedfall-lemon.vercel.app/docs/gostman.html): Gostman is an API testing tool. Its collections list requests and the checks each reply must pass. This page shows how to reuse a collection as seedfall's workload, so the same checks run while seedfall breaks your service. Use it if you already have a Gostman collection. ([Markdown](https://seedfall-lemon.vercel.app/docs/gostman.md))
- [How seedfall works](https://seedfall-lemon.vercel.app/docs/how-it-works.html): This page explains why the same seed always gives the same run: seedfall controls the clock, the randomness and the scheduling for every node, and simulates the disk and the network. Read it to understand what seedfall can and cannot find, or to decide how far to trust a result. You do not need it to run your first sweep. ([Markdown](https://seedfall-lemon.vercel.app/docs/how-it-works.md))
- [Inspecting the disk at the moment of failure](https://seedfall-lemon.vercel.app/docs/inspect.html): When a test fails after a crash, the cause is often what had and had not reached the disk. `seedfall inspect` shows every node's files at the moment of failure: which bytes were safely saved, which a crash would lose, and where a write was cut short. Use it for durability bugs, such as data that is missing after a restart. ([Markdown](https://seedfall-lemon.vercel.app/docs/inspect.md))
- [Installing seedfall](https://seedfall-lemon.vercel.app/docs/install.html): This page shows every way to install seedfall and how to check that it works. If you have Go, the quickest way is `go install`. Skip the rest unless you want a package manager or a signed release. ([Markdown](https://seedfall-lemon.vercel.app/docs/install.md))
- [Mutation testing your checks](https://seedfall-lemon.vercel.app/docs/mutate.html): A passing sweep only means something if your checks would notice a real problem. `seedfall mutate` weakens the simulated disk, network or clock one guarantee at a time, or changes your own source code, and tells you whether your checks catch it. Use it to find checks that are too weak. ([Markdown](https://seedfall-lemon.vercel.app/docs/mutate.md))
- [Networking](https://seedfall-lemon.vercel.app/docs/networking.html): Nodes in seedfall talk over a simulated network, so seedfall can delay, drop and cut messages between them. This page explains the two ways to connect your code to it: call seedfall's `sim.Listen` and `sim.Dial`, or leave your code alone and turn on `net_overlay`. Read it when nodes cannot reach each other or you are wiring up a new project. ([Markdown](https://seedfall-lemon.vercel.app/docs/networking.md))
- [Properties, linearizability and determinism](https://seedfall-lemon.vercel.app/docs/properties.html): A property is something your system must always (or eventually) get right, such as "an acknowledged write is never lost". This page covers how to write properties with the `sim` package, how to check recovery, and how to check a recorded history of operations for linearizability (concurrent operations behaving as if they happened one at a time, in some order). Use it when you write or improve your workload. ([Markdown](https://seedfall-lemon.vercel.app/docs/properties.md))
- [Triage reports](https://seedfall-lemon.vercel.app/docs/reports.html): A big sweep can find many failing seeds that come from only a few bugs. Reports group them into findings, label each one as new or old, remember history between runs, and can post to Slack, Discord or GitHub. Use this page when you run seedfall at scale or in CI and want to triage instead of reading one failure at a time. ([Markdown](https://seedfall-lemon.vercel.app/docs/reports.md))
- [Testing seedfall itself](https://seedfall-lemon.vercel.app/docs/testing.html): This page is for people who work on seedfall itself, or who want to know why its results can be trusted. It lists the tests that check seedfall's model of crashes, faults and replay, and how to run them. You do not need it to use seedfall. ([Markdown](https://seedfall-lemon.vercel.app/docs/testing.md))
- [Timelines, logs and time-travel debugging](https://seedfall-lemon.vercel.app/docs/timeline.html): After a failure you want to see what happened. This page covers the four views seedfall gives you of one run: a timeline diagram, a sequence diagram, filtered logs and a step-by-step debugger. Use it when the failure message alone does not explain the problem. ([Markdown](https://seedfall-lemon.vercel.app/docs/timeline.md))
- [Troubleshooting](https://seedfall-lemon.vercel.app/docs/troubleshooting.html): This page lists the errors and failure messages seedfall prints, with what causes each one and how to fix it. Find yours by its exact text. Parts in angle brackets, such as `<pkg>`, change from case to case. Commands print errors as `seedfall: <message>` and exit with code 1. ([Markdown](https://seedfall-lemon.vercel.app/docs/troubleshooting.md))
- [Proving a fix with `seedfall verify`](https://seedfall-lemon.vercel.app/docs/verify.html): After you fix a bug you need evidence that the fix works. `seedfall verify` replays the failing run, sweeps many fresh seeds, checks that the run is repeatable and writes a proof for your pull request. Pinned regressions then keep the bug from coming back. Use it as the last step before you open a pull request. ([Markdown](https://seedfall-lemon.vercel.app/docs/verify.md))

## Agent skills

- [seedfall-fix-and-prove](https://raw.githubusercontent.com/kunalgoesbyken/seedfall/main/skills/seedfall-fix-and-prove/SKILL.md): Fix a triaged seedfall failure and prove the fix. Replays the saved minimal fault schedule until it passes, sweeps fresh seeds, then runs seedfall verify for a 95%-confidence proof with determinism and mutant checks, and puts the Markdown proof block into the pull request. Use after seedfall-triage, or whenever a change to storage, recovery or replication code must be shown crash-safe before it is merged.
- [seedfall-properties](https://raw.githubusercontent.com/kunalgoesbyken/seedfall/main/skills/seedfall-properties/SKILL.md): Read a Go service's storage and replication code, turn its durability and consistency promises into seedfall properties (sim.Assert, sim.Always, sim.Sometimes, sim.Reachable, sim.Unreachable, sim.Eventually, linearizability through sim.Put, sim.Get and sim.Invoke, and transactional isolation through sim.Txn), and grade them with seedfall mutate until every exercised mutant is killed. Use after seedfall-setup, when asked what to check, to strengthen a workload, or when a green sweep might be proving nothing.
- [seedfall-setup](https://raw.githubusercontent.com/kunalgoesbyken/seedfall/main/skills/seedfall-setup/SKILL.md): Set up seedfall deterministic simulation testing in a Go module that has no seedfall.json yet. Checks that the services build for wasip1, scaffolds seedfall.json and a workload with seedfall init, wires the workload to the service, and runs the first seed sweep. Use when asked to add seedfall, crash testing, deterministic simulation testing or fault injection to a Go service.
- [seedfall-triage](https://raw.githubusercontent.com/kunalgoesbyken/seedfall/main/skills/seedfall-triage/SKILL.md): Triage a failing seedfall seed down to a root cause before touching code. Classifies the failure, shrinks the fault schedule, finds the fault that made the failure inevitable with seedfall cause, inspects the simulated disk at the failure, then checks the hypothesis with a counterfactual replay of an edited schedule. Use when seedfall run, the seedfall GitHub Action or a CI sweep reports FAIL seed N.

## Optional

- [README](https://raw.githubusercontent.com/kunalgoesbyken/seedfall/main/README.md): bugs found, quickstart, features, examples and limitations
- [Findings](https://seedfall-lemon.vercel.app/proof/)
- [Open creates the database file without fsyncing its directory, so one crash can lose the whole database](https://seedfall-lemon.vercel.app/proof/bbolt-create-dir-fsync.html)
- [AutoRecovery fails on ordinary torn tails, so Open errors out after a crash mid-Put](https://seedfall-lemon.vercel.app/proof/bitcask-autorecovery-torn-tail.html)
- [Open rewrites config.json in place without fsync, so a crash can leave the database permanently unopenable](https://seedfall-lemon.vercel.app/proof/bitcask-config-json-torn.html)
- [AutoRecovery renames an unsynced copy over the last datafile, so a second crash can wipe acknowledged writes](https://seedfall-lemon.vercel.app/proof/bitcask-recovered-datafile-fsync.html)
- [Acknowledged writes can be lost after a datafile rotation (no directory fsync) or after a failed fsync (offset not rewound)](https://seedfall-lemon.vercel.app/proof/bitcask-rotation-dir-fsync-and-eio.html)
- [WriteStream with sync and TempDir doesn't fsync the directory after the rename, so an acknowledged write can revert after a crash](https://seedfall-lemon.vercel.app/proof/diskv-rename-dir-fsync.html)
- [FileSnapshotStore doesn't fsync the snapshot directory before rename, so a crash can leave an empty snapshot after logs were compacted](https://seedfall-lemon.vercel.app/proof/hashicorp-raft-snapshot-dir-fsync.html)
- [A crash in the middle of a log append makes the node unable to restart, now or after its next restart](https://seedfall-lemon.vercel.app/proof/jmsadair-raft-log-torn-tail.html)
- [Snapshots are ordered by wall-clock time, so after the clock steps back a restart loads an old snapshot and the node never starts again](https://seedfall-lemon.vercel.app/proof/jmsadair-raft-snapshot-wall-clock-order.html)
- [The term and vote file is renamed into place without an fsync, and one crash leaves a node that can never restart](https://seedfall-lemon.vercel.app/proof/jmsadair-raft-state-file-fsync.html)
- [A crash while a snapshot is being written leaves a temporary directory that makes every later start fail](https://seedfall-lemon.vercel.app/proof/jmsadair-raft-tmp-snapshot-dir.html)
- [A crash during stream creation leaves meta.inf without meta.sum, and the stream then disappears on every later restart](https://seedfall-lemon.vercel.app/proof/nats-stream-meta-checksum-crash.html)
- [`bucket.Meta` is never fsynced, so a crash can make recovery drop every entry in the bucket](https://seedfall-lemon.vercel.app/proof/nutsdb-bucket-meta-fsync.html)
- [Acknowledged writes can be lost after a data-file rotation (no directory fsync) or after a failed fsync (offset not rewound)](https://seedfall-lemon.vercel.app/proof/nutsdb-rotation-dir-fsync-and-eio.html)
- [A torn write at the end of the active data file makes Open fail with "crc error" on every restart](https://seedfall-lemon.vercel.app/proof/nutsdb-torn-tail-crc.html)
- [rosedblabs/wal appends after a torn chunk, so one crash mid-write makes later fsynced writes unreachable and rosedb can no longer open](https://seedfall-lemon.vercel.app/proof/rosedblabs-wal-torn-tail.html)
- [Shrink renames the rewritten AOF over the database without fsyncing it, so a crash after a shrink can wipe the database](https://seedfall-lemon.vercel.app/proof/tidwall-buntdb-shrink-fsync.html)
- [tidwall/wal can't reopen after a crash mid-append, and the pending fix (#22) leaves stale bytes that corrupt the log after the next segment rotation](https://seedfall-lemon.vercel.app/proof/tidwall-wal-torn-tail.html)