BUGGIFY: faults inside your code

Most seedfall faults hit your program from the outside: a crash, a broken disk, a cut network. BUGGIFY lets you mark spots inside your own code where seedfall may force a rare path on purpose, such as a lost reply, a retry or an error return. Use it when the bug you fear lives in your own error handling, which outside faults rarely reach.

Every other seedfall fault comes from outside the process: crashes, torn writes, partitions. sim.Buggify lets you mark places in your own code where the simulator may force the rare path, such as a dropped reply, a retry, an error return, a minimal quorum or an extra delay. The idea comes from FoundationDB's BUGGIFY, which FoundationDB credits for many of its deepest bugs.

v := c.apply(req)
if sim.Buggify("reply lost after apply") {
	return
}
reply(conn, v)

Outside the simulator Buggify always returns false, so the site costs one function call in production.

Turning it on

Sites never fire unless the config asks for it:

{ "faults": { "buggify": true } }

With swarm as well, each seed turns buggify on or off along with the other fault types.

How a site decides

A site is identified by its name, so keep names constant. Both decisions use random streams keyed by the seed and the site name (and, for firing, the node), so adding a site or reaching another site more often never changes when an existing site fires. Most seeds run with only some sites enabled, which explores combinations that firing every site everywhere would hide.

Replay and shrink

Each firing is recorded as a fault with the node, the site and which hit it was, counted per node and site:

faults before the failure (1):
  465.631ms  buggify "reply lost after apply" fires on counter at hit #35

{ "at": "465.631252ms", "kind": "buggify", "node": "counter", "site": "reply lost after apply", "until": "0s", "op": 35 }

When a schedule is replayed with -faults, a site fires exactly at the hits the schedule lists and nowhere else, so seedfall shrink removes firings like any other fault, and you can hand-edit a schedule to add or drop one. Replaying a schedule needs "buggify": true in the config, and with swarm it must be on for that seed: otherwise every site returns false. The trace shows when each site is first reached and whether it is enabled for the run, and every firing.

Coverage

seedfall catalogs sim.Buggify and sim.BuggifyWithProb calls whose site is a string literal, and after a sweep with buggify on it prints how often each site was reached, enabled and fired:

buggify  seeds reached  seeds enabled  seeds fired  fired
                    20              6            6     88  reply lost after apply
                    20              5            5    110  slow apply

A site marked never reached (with its location) is code no seed ran; never fired means it ran but never fired: no seed enabled it, or its rate was too low for the hits it got, so sweep more seeds or raise its probability. Neither fails the sweep: pair a site with a sim.Sometimes on the path it opens when you want that path to be required. The same counts are in buggify_sites in run -json and in the MCP seedfall_run result.

Structured randomness

sim.Random() returns a random uint64 and sim.RandomChoice(items) returns one element of a slice (an empty slice fails the run, since there is nothing to return). Inside the simulator both read crypto/rand, which seedfall feeds from a stream keyed by the seed, the node and its restart count, so a choice replays with the seed. RandomChoice rejects out-of-range draws, so every element is equally likely. Outside the simulator they use crypto/rand.

if sim.BuggifyWithProb("slow apply", 0.5) {
	time.Sleep(sim.RandomChoice([]time.Duration{time.Millisecond, 5 * time.Millisecond, 50 * time.Millisecond}))
}

The Antithesis SDK's random.GetRandom and random.RandomChoice read the same source, so Antithesis-instrumented code gets the same replayable choices with no changes. See antithesis.md.

Example

examples/buggify is a counter service whose client retries an increment when the reply does not arrive. Without buggify nothing ever drops a reply and every seed passes; with it, reply lost after apply makes the retry apply twice and the sweep fails, and the fixed server, which remembers applied request ids, passes.

Limits

• A site's name is its identity: two call sites with the same name share one enable decision and one hit count per node.
• Sites named by a non-literal string still fire and are recorded, but cannot be cataloged in advance.
• seedfall cause re-randomizes when sites fire after each branch point, not which sites are enabled, so a seed's enabled set is fixed for that seed.