Configuration and faults

This is the reference for seedfall.json: every field, and every fault seedfall can inject, such as crashes, partly written disk writes, lost network messages and clock jumps. Use it to look up what a setting does or to design the failures you want to test. If you are new, start with getting-started.md.

A world is described in a JSON file, seedfall.json by default. Paths inside it are relative to the file. One file can extend another, hold named profiles selected with -profile, take -set path=value overrides and start from a fault preset such as "faults": "resilient"; see config.md.

{
  "nodes": [
    { "name": "n1", "pkg": "./node", "args": ["-id", "n1"] },
    { "name": "n2", "pkg": "./node", "args": ["-id", "n2"] }
  ],
  "workload": { "name": "client", "pkg": "./client" },
  "duration": "60s",
  "latency": ["1ms", "5ms"],
  "supervise": true,
  "faults": {
    "crash_per_sec": 0.5,
    "restart_after": ["10ms", "500ms"],
    "partition_per_sec": 0.2,
    "partition_for": ["50ms", "1s"],
    "torn_writes": true,
    "crash_on_io": 0.005,
    "fsync_eio": 0.01,
    "strict_dirs": true,
    "clock_skew": "200ms",
    "pause_per_sec": 0.2,
    "clock_jump_per_sec": 0.05,
    "buggify": true,
    "swarm": true
  }
}

World

Every node sees the environment variables SEEDFALL=1 and SEEDFALL_NODE=<name>, so sim.Enabled() and sim.Node() work.

Faults

All faults come from the seed. Faults that depend on what the program does, such as which disk operation a crash lands on, are recorded when they happen so that shrinking and replay are exact.

Faults on the workload

By default the workload is the one process that never crashes, so a client's own retry, idempotency and session logic is never tested: what happens when a client restarts halfway through a write and sends it again? Set "include_workload": true to change that:

"workload": { "name": "client", "pkg": "./client" },
"faults": { "include_workload": true, "crash_per_sec": 0.3, "pause_per_sec": 0.3 }

• A crashed workload restarts as a fresh process after restart_after, like a node. It keeps its name, so sim.Node() and any client id derived from it stay the same, and it finds its disk as the crash left it. A client that saves its progress there can resume; one that keeps it in memory starts over.
• A workload crash does not end the run. The run passes only when the last incarnation of every workload exits 0.
• Operations the crashed workload had recorded with sim.Put, sim.Get, sim.Invoke or sim.Txn and not yet completed stay open, and the trace says how many (client#1 died with 2 operations in flight, their outcome is unknown). The history check treats an open write as one that may take effect at any time after it started, and drops open reads, so the check stays sound under every model.
• Per node, "faults": true or false overrides the default: true on one of several workloads faults only that one, and false on a node keeps it out of every fault, network faults included.

workloads[] runs several clients side by side, each its own process, so one can crash while the others keep going:

"workloads": [
  { "name": "writer", "pkg": "./client", "args": ["-mode", "write"], "faults": true },
  { "name": "reader", "pkg": "./client", "args": ["-mode", "read"] }
]

examples/liveness has a client that saves its progress and resends the add in flight after a crash. Without request deduplication on the server, one crash during an add applies it twice.

Network faults

Every network fault is directional underneath: it decides what happens to traffic from one node to another. A connection needs both directions to open, so a stopped direction either way refuses a dial, and a jammed one holds it.

A clog targets one of four shapes, chosen at random: one link in one direction (n1 -> n2), or a node's inbound, outbound, or inbound and outbound traffic. A one-way partition, where n1 can hear n2 but n2 cannot hear n1, is a stop clog on one link or an outbound clog on one node. Faults stack: a slow clog over a partition adds its delay to the held traffic.

Partition topologies follow Jepsen's nemeses. half is the original random split and stays the default, so existing seeds keep their schedules. majority needs at least 3 nodes and majorities_ring at least 4, workload included; the config is rejected otherwise. A ring partition is written as overlapping groups, and two nodes can talk only if some group holds both.

A slowdown by factor F stretches everything the node waits for: its timers and sleeps take F times as long, every message it sends or receives takes F times its latency, and a goroutine that spins is parked F times as long. The node still reads the true time, so it sees its own timeouts fire late, the way an overloaded machine does. CPU work between two waits stays free, as it is for every node in seedfall.

sim.QuietFor and the recovery phase of liveness heal every partition, clog and slowdown at once.

In a fault schedule, the new kinds look like this:

{ "at": "1.2s", "kind": "partition", "groups": [["n1", "n2"], ["n2", "n3"], ["n3", "client"], ["client", "n1"]], "until": "2s" }
{ "at": "1.5s", "kind": "clog", "from": "n1", "to": "n2", "mode": "stop", "until": "3s" }
{ "at": "1.6s", "kind": "clog", "node": "n3", "dir": "out", "mode": "slow", "delay": "80ms", "until": "2.4s" }
{ "at": "2.1s", "kind": "net_reset", "node": "n2", "until": "2.1s" }
{ "at": "2.5s", "kind": "slow", "node": "n1", "factor": 4.5, "until": "6s" }

dir is in, out or both. A swizzle shows up as a run of both clogs in stop mode. Like every other planned fault, each one replays and shrinks on its own, appears in the trace, seedfall cause and the report, and joins swarm testing.

UDP datagrams

TCP traffic is a byte stream, where the kernel retransmits and deduplicates, so a program never sees a packet dropped or repeated on a connection. A UDP program does. udp_loss and udp_dup are probabilities per datagram, drawn from their own stream of the seed, so they take no draws from any other fault:

{ "faults": { "udp_loss": 0.05, "udp_dup": 0.02 } }

A partition or a clog that stops or jams a link also drops a datagram sent across it, rather than holding it for the heal, and a slow clog delays it. Datagrams that a link or a missing socket drops are consequences of the rest of the schedule, so they are not faults of their own: only udp_loss and udp_dup choices are recorded. They count per sender node, from 1, across restarts:

{ "at": "2.1s", "kind": "udp_drop", "node": "n1", "to": "n2", "op": 17, "until": "0s" }
{ "at": "2.2s", "kind": "udp_dup", "node": "n2", "to": "n3", "op": 9, "until": "0s" }

op is the number of the datagram the sender sent, and the fault fires on that datagram whatever its destination; to is a record of where it went. Both replay exactly, shrink like any fault recorded at runtime, can be edited with seedfall faults, and join swarm testing. They do not apply when faults is off for either end of the datagram, or during a quiet period or the recovery phase. Semantics of sockets and addresses are in networking.md.

Disk faults

File system models

reorder_writes turns on out-of-order writeback at a crash. fs_model says which states a crash may then leave behind. A model that is too weak reports bugs your file system cannot produce; one that is too strong misses bugs it can. The models follow Table 1 of ALICE (All File Systems Are Not Created Equal, OSDI '14), which measured these properties on real file systems.

Details that follow from these rules:

• Models apply per file. Different files are independent, so an append to one file can survive while an earlier append to another is lost. ALICE records Append -> Any op failing on ext4 ordered for this reason (delayed allocation).
• Under ext4_ordered a crash that keeps an overwrite past the surviving file size keeps nothing of it, because that page was never part of the file on disk.
• A single directory operation is atomic in every model, as on btrfs. ext2 can also tear one rename so that both names or neither survive; that is not modelled.
• Plain torn_writes without reorder_writes does not use fs_model: it keeps an in-order prefix of all pending writes.

examples/symswap shows the difference. It publishes each generation by renaming a symlink over current, and fsyncs the new generation's file but not its directory entry first. Under ext4_ordered it passes, because a journal keeps directory operations in order. Under dir_reorder the rename can survive without the create, and current points at nothing.

Crashes, torn writes, reorder_writes and fsync_eio test what survives a crash. The faults below test what a storage engine does when the disk itself misbehaves: it fills up, it fails an operation, or it hands back or keeps the wrong bytes. These are the paths that WAL recovery and checksum code exist for, and the ones that are hardest to reach with a real disk.

"faults": {
  "crash_per_sec": 0.5,
  "torn_writes": true,
  "disk_bytes": 1048576,
  "write_eio": 0.01,
  "read_eio": 0.01,
  "read_corrupt": 0.05,
  "lost_write": 0.01,
  "misdirected_write": 0.01,
  "corrupt_on_crash": true,
  "bitrot_per_sec": 0.2,
  "disk_latency": ["1ms", "20ms"]
}

How each one is modelled, and where it stops:

• Disk usage is the sum of every file's size on the node. Holes left by Truncate or by writing past the end count in full, so sparse files use more space than on ext4 or XFS. Directories and metadata use none. A Truncate that grows a file is never refused.
• ENOSPC reaches your code as ENOSPC. wazero's WASI layer has no ENOSPC and reports EIO for it, so seedfall rewrites the result of fd_write and fd_pwrite when the simulated disk was full, and errors.Is(err, syscall.ENOSPC) works in the guest.
• Read and write errors apply to file reads and writes only. Directory listings, stat, open and Truncate never fail.
• Corrupted reads flip one bit in the buffer of one read, as a bad cable or controller would. A checksum plus a retry handles them; a checksum alone turns them into an error.
• Lost and misdirected writes pick one dirty page of the file being fsynced. The page cache keeps the right content, because the drive acknowledged the write, so the program only notices after a crash, or never if the page is rewritten first. A misdirected write stays inside the file; writes landing in another file are not modelled. A file needs at least two pages for a misdirected write.
• Crash corruption damages only bytes that were being written when the node crashed. With plain crashes, unsynced writes are dropped, so it matters only where they survive: with torn_writes, reorder_writes or crash_on_io. If the damaged range overlapped older durable data, that data is what flips.
• Bit rot picks a byte uniformly over all durable data on the node, so large files are hit more often.
• Latency applies to file and directory fsyncs only, not to reads or writes, which go to the page cache. The whole node waits, because a wasip1 guest runs every goroutine on one thread.

Every one of them comes from the seed and replays exactly. The rates are drawn per operation, like crash_on_io, and each firing is recorded with its node and operation number: disk op #N counts the node's writes, truncates and fsyncs, and disk read #N counts its reads. Which page or bit is hit comes from a stream keyed by the seed, the fault kind, the node and that number, so it does not change when shrinking removes other faults. Bit rot is planned in time like a crash. In a fault schedule they look like this:

{ "at": "1.942589003s", "kind": "read_corrupt", "node": "server", "op": 3, "path": "data/kv.log", "offset": 70, "until": "0s" }
{ "at": "912.496ms", "kind": "write_eio", "node": "server", "op": 67, "path": "data/kv.log", "until": "0s" }
{ "at": "2.601158s", "kind": "lost_write", "node": "server", "op": 232, "path": "data/kv.log", "until": "0s" }
{ "at": "5.259168s", "kind": "misdirected_write", "node": "server", "op": 558, "path": "data/kv.log", "offset": 4096, "until": "0s" }
{ "at": "657.015787ms", "kind": "bitrot", "node": "server", "path": "data/kv.log", "offset": 22, "until": "657.015787ms" }
{ "at": "36ms", "kind": "mmap_early_writeback", "node": "server", "op": 3, "path": "data/pages", "offset": 65536, "until": "0s" }

The other kinds are write_enospc and read_eio. A recorded fault fires at the operation its kind, node and op name, and bit rot fires at at. path, offset and target are filled in by the run so that you can read the schedule, and are left out when they are 0. seedfall inspect lists every damaged range of a file under corrupted ranges, and with disk_bytes shows how full each disk is. Disk usage and latency are settings, not faults, so swarm testing leaves them alone and toggles the rest.

examples/kvlog shows each one: a corrupted read changes an acknowledged value, a checksum with a re-read fixes it, and a lost write, a misdirected write or bit rot breaks a store that keeps one copy, with or without checksums.

Editing a schedule

A fault schedule (.seedfall/seed-N.faults.json from shrink, an artifact's faults.json, or a pinned regression) is a list of faults that a replay applies instead of drawing new ones. Changing one fault and replaying is the quickest way to test an explanation of a failure. seedfall faults makes the change and checks it:

seedfall faults -c seedfall.always.json -seed 1 -faults torntail.faults.json -remove 1 -out hypothesis.json
seedfall replay -c seedfall.always.json -seed 1 -faults hypothesis.json

seed 1, 1 faults from torntail.faults.json, wrote hypothesis.json
  1    802ms  crash server, restart at 802ms
duration 991ms
replay: seedfall replay -c seedfall.always.json -seed 1 -faults hypothesis.json

On kvlog's seedfall.always.json, seed 1 fails with torntail.faults.json, still fails with the crash during disk op #56 moved one operation earlier (-shift 1=-1), and passes once that crash is removed. That pass is within the schedule's shrunk 991ms, and the replay prints a note saying so: it shows the crash is necessary for this failure, not that the store is fixed.

Positions in -remove and -shift always refer to the schedule as it was before the edit. An edited schedule drops the shrunk_from and shrink_runs that shrink recorded, since it is no longer what shrink found. The result is sorted by at and checked against the config before anything is written:

Without any edit, seedfall faults only lists and checks the schedule, and writes it only when it came from the seed's own faults, so that there is a file to edit. Over MCP, seedfall_faults_edit takes the same edits as JSON (agents.md).

A schedule file can also hold branches, which seedfall explore writes for a failure it found by branching:

"branches": [{ "at": "921.003003ms", "key": 257366709774643251 }, { "at": "1.2s", "key": 9, "stream": "sched" }]

Each branch re-keys the seed's random streams at the first scheduler step at or after at, in order. Without stream it re-keys every stream; with one of sched, net, disk, io, udp, disklatency, mutant or random it re-keys only that one. Re-keying io or udp changes nothing in a schedule, because those streams only decide faults and a schedule already fixes them; mutant matters only under seedfall mutate. replay, shrink, inspect, debug, logs and pin all honor them, and at values must not decrease.

Choosing rates

Start with crashes and torn writes, then add crash_on_io around 0.005, so a few hundred disk operations per run produce one or two mid-operation crashes. Add fsync_eio, strict_dirs and reorder_writes once the basic durability path passes. For consensus and leader-election code, add partition_topologies beyond half, then clog_per_sec and slow_per_sec: one-way links and slow leaders are the gray failures that break leases and stale-read checks. For a storage engine, add disk_bytes and write_eio to test the error paths of every write, then read_corrupt and corrupt_on_crash to test checksums. lost_write, misdirected_write and bitrot_per_sec lose data that a single copy cannot get back, so use them on replicated systems, or to check that a single node notices and stops instead of serving wrong data. Turn on buggify once you have marked sites in your code that should sometimes take the rare path. Turn on swarm for long CI sweeps. Use property coverage (properties.md) to confirm the faults are actually reaching the code you care about.

Durations

Every duration is a Go duration string, such as "250ms" or "2s". Ranges are two-element arrays.