tidwall/wal can't reopen after a crash mid-append, and the pending fix (#22) leaves stale bytes that corrupt the log after the next segment rotation

Upstream: tidwall/wal#43, following up on #21 and PR #22

Repo: tidwall/wal (tested on v1.2.1, default options, NoSync: false) Status upstream: the first half is known: issue #21, open since June 2023, and PR #22, unmerged. The second and third parts below are not reported.

1. A crash during one append bricks the log (known: #21)

With NoSync: false, every acknowledged write is fsynced, so a crash can only tear the entry that was being written and hadn't been acknowledged. On restart, load() parses the tail segment with loadSegmentEntries (wal.go:293). loadNextBinaryEntry (wal.go:599-606) returns ErrCorrupt for the partial entry, and Open fails. Every later Open fails the same way until someone edits the file by hand.

seedfall finds this on seed 1 and shrinks it to one fault:

cd examples/wals
seedfall replay -c tidwall.json -seed 1
  366ms  crash server during disk op #56
  server#2| open wal: log corrupt
  FAIL server#2 crashed

2. PR #22 recovers in memory but never truncates the file (new)

PR #22 adds RecoverCorruptedTail. When it's set, loadSegmentEntries stops at the first corrupt entry and trims s.ebuf to the valid prefix. load() then seeks the write offset back to len(ebuf), but it doesn't truncate the file.

If the next append is shorter than the torn bytes, the end of the old partial entry stays in the file after the new entry. As long as the segment is the tail, recovery skips those bytes again. Once the segment rotates, it is loaded with loadSegmentEntries(s, false), and Read of any index in that segment returns ErrCorrupt. One recoverable crash becomes permanent corruption in the middle of the log.

This was reproduced with PR #22 applied and RecoverCorruptedTail: true, on seed 117 (4 faults after shrinking). A diagnostic listing of segment 00000000000000000104 showed the following:

server#4 start: size=583 entries=6 valid=509 tail="me131-1111…"     (74-byte torn entry)
server#5 start: size=583 entries=7 valid=552 tail="1111111…"        (new 43-byte entry written at 509, 31 stale bytes left)
server#5| read wal: log corrupt

Fix: truncate the file to len(lseg.ebuf) and fsync it before seeking.

3. cycle() never fsyncs the directory (new; strict POSIX only)

cycle() (wal.go:355-376) creates the next segment with O_CREATE|O_TRUNC. It fsyncs the previous segment and, on later writes, the new one, but it never fsyncs the log directory. POSIX only makes the new file's name durable once the directory is fsynced.

Under seedfall's strict_dirs model, a crash after a rotation can lose the whole new segment, even though every entry in it was fsynced and acknowledged. Seed 2 shows this: after a single crash, segment …041 is missing, and acknowledged entries 41 onward are gone. The server had already fsynced every ancestor directory at startup, so the only gap left is inside the library.

On ext4, XFS and btrfs, fsyncing the file usually persists its directory entry too, so typical Linux deployments are mostly not exposed. This is the same caveat as the hashicorp/raft snapshot finding.

Verified fix

These are applied to a local copy in .prof/tidwall-wal; the full diff is in .prof/tidwall-full-fix.diff:

1. PR #22, with RecoverCorruptedTail: true in the example server
2. in load(): l.sfile.Truncate(int64(len(lseg.ebuf))) and l.sfile.Sync() before the Seek
3. in cycle(): fsync l.path after creating the new segment