Proving a fix with seedfall verify

After you fix a bug you need evidence that the fix works. seedfall verify replays the failing run, sweeps many fresh seeds, checks that the run is repeatable and writes a proof for your pull request. Pinned regressions then keep the bug from coming back. Use it as the last step before you open a pull request.

seedfall verify is the definition of done for a change that fixes a seedfall failure, whether a person or an AI agent wrote it. It answers one question with evidence: is the failure gone, and how sure are we?

seedfall verify -c seedfall.fixed.json -seed 1 -faults .seedfall/seed-1.faults.json -mutate -md proof.md

verify seedfall.fixed.json (seedfall v1.2.3)
  pass  replay seed 1 with .seedfall/seed-1.faults.json  passes, twice, with identical host calls and traces
  pass  fresh sweep, seeds 2-300                         299 of 299 pass: 95% confidence that a failure with rate >= 0.997% would have shown up
  pass  determinism, seeds 2-11                          every seed ran twice with identical host calls and traces
  pass  mutant fsync_noop                                killed by seed 2
  pass  mutant dirsync_noop                              not exercised
  pass  mutant osync_ignored                             not exercised
  pass  mutant fsync_corrupts                            killed by seed 2
PASS

Against the unfixed config the same command stops at the first check and exits 1:

verify seedfall.always.json (seedfall v1.2.3)
  FAIL  replay seed 1 with .seedfall/seed-1.faults.json  seed 1 FAILS [bug]: client#1: acknowledged write lost: k60=v60, server now answers "NONE"
  skip  fresh sweep                                      an earlier check failed
  skip  determinism                                      an earlier check failed
FAIL

What it checks

In order, stopping at the first stage that fails:

1. Replay. Every failing schedule you name is replayed and must pass now. Each replay runs twice and must produce identical host calls and traces, so a pass cannot be luck.
2. Fresh sweep. New seeds, starting after the highest seed already known, must all pass. By default verify runs as many as needed for 95% confidence that a failure with the historical rate would have shown up: n = ⌈ln(0.05) / ln(1 − p)⌉, the same rule the triage report uses to call a finding resolved. A finding that failed 27.5% of seeds needs 10 clean seeds; when no rate is known, verify assumes a rare failure (1%) and runs 299. -seeds overrides the range.
3. Determinism. The first -check fresh seeds (default 10) run twice and must produce identical host calls and traces, as with run -check.
4. Mutants (with -mutate). The mutants run over the fresh seeds, and every exercised mutant must still be killed. This catches a "fix" that passes by weakening the checks. With -app-mutants N, N source mutants of the node packages run too, each as a row naming its line and change, and each one that builds must be killed. If the node packages cannot be read to list the source mutants, a source mutants row fails the proof.

Any failure is printed with its classification, so a determinism leak is not mistaken for the bug coming back.

What to verify

-seed and -report can be combined. Point -c at the config of the code you changed; the report can come from an earlier run of the same or another config.

Every schedule pinned for the config is replayed too, as a replay pinned seed N with seedfall-regressions/... row, without being named.

$ seedfall verify -c seedfall.fixed.json -report .seedfall/report.html
verify seedfall.fixed.json (seedfall v1.2.3)
  pass  replay assert · client seed 1                                     passes, twice, with identical host calls and traces
  pass  replay assert · client seed 1 with .seedfall/seed-1.faults.json   passes, twice, with identical host calls and traces
  pass  replay assert · client seed 12                                    passes, twice, with identical host calls and traces
  ...
  pass  fresh sweep, seeds 41-50                                          10 of 10 pass: 95% confidence that a failure with rate >= 25.9% would have shown up (historical rate 27.5%)
  pass  determinism, seeds 41-50                                          every seed ran twice with identical host calls and traces
PASS

Pinned regressions

A fixed bug should stay fixed. seedfall pin copies a failing seed's schedule into seedfall-regressions/, next to the config, a directory you commit:

seedfall pin -c seedfall.fixed.json -seed 1 -faults torntail.faults.json

pinned seed 1 (2 faults, from torntail.faults.json) to seedfall-regressions/seedfall.fixed-seed-1.json
it passes now; seedfall run, verify and seedfalltest replay it first and fail if it fails again

Without -faults, pin takes the schedule seedfall shrink saved in .seedfall/seed-N.faults.json, or else the seed's own faults, recorded fault by fault so the pinned faults stay the same if the fault rates change later. The file is a schedule in the usual format with a pinned block: the config it belongs to, the finding and reason if it fails now, where it came from, the seedfall version and the date. It is named <config>-seed-N.json, so two configs in one directory can pin the same seed; <config> carries the profile and a hash of any -set overrides, as artifact folders do (config). Pinning a schedule that still fails is allowed, and the commands below fail until the bug is fixed, the way a test written before its fix does.

A pinned schedule keeps its faults and ops cap but runs for the config's full duration, as -faults does in verify, so a shrunk schedule's short duration cannot end the run before the workload checks anything. The same file replays exactly with seedfall replay -c C -seed N -faults seedfall-regressions/<file>.

Pinned schedules are replayed, in parallel, before anything else by:

A file in seedfall-regressions/ without a pinned.config, for example a schedule copied in by hand, is replayed for every config in that directory. run -pinned=false skips them; over MCP, skip_pinned.

run -seeds-file known-bad.txt replays a list of seeds, each with its own faults, the same way before the sweep. The file holds seeds separated by spaces or lines, with # comments.

The proof block

-md proof.md writes the result as Markdown for the pull request description:

### seedfall verify: PASS

Config `seedfall.fixed.json`, seedfall `v1.2.3`, reproduce with `seedfall verify -c seedfall.fixed.json -seed 1 -faults .seedfall/seed-1.faults.json -mutate`

| | Check | Result |
|---|---|---|
| pass | replay seed 1 with .seedfall/seed-1.faults.json | passes, twice, with identical host calls and traces |
| pass | fresh sweep, seeds 2-300 | 299 of 299 pass: 95% confidence that a failure with rate >= 0.997% would have shown up |
| pass | determinism, seeds 2-11 | every seed ran twice with identical host calls and traces |
| pass | mutant fsync_noop | killed by seed 2 |
| pass | mutant dirsync_noop | not exercised |
| pass | mutant osync_ignored | not exercised |
| pass | mutant fsync_corrupts | killed by seed 2 |

Flags

The exit status is 0 only when every check passes.

From an agent

The MCP tool seedfall_verify takes the same inputs (config, seed, faults, report, seeds, check_seeds, mutate) and returns each check as structured JSON, the overall passed verdict, and the proof block as markdown. See agents.md.