StateWAL + FlatKV

[cody-littley]

Describe your changes and provide context

Replace FlatKV WAL with new StateWAL impl

Notes for LLM reviewers:

• No backwards compatibility needed, since FlatKV is not deployed anywhere. So it doesn't matter that we don't handle the migration from the old WAL.
• Overhead from FlatKVNeededAtHeight() is understood and tolerable. This is a startup cost, a cost measured in milliseconds, and something that disappears entirely when MemIAVL is gone. No suggestions about changing this function will be accepted; this is too important to be "clever". A blunt force check that is better than an overly clever one that misses edge cases.
• It is safe to assume that a human will delete the WAL before a state sync, we specifically don't want to do this in the code.

[yzang2019]

We can remove this config?

[cody-littley]

Good point. It's no longer a valid scenario to have multiple blocks per commit. Removed.

[yzang2019]

Do we need this at all? If wal == nil, it means we don't manage wal right? Under what scenario will this happen?

mistaking a closed-but-owned WAL for the nil "outer context owns it" case

If FlatKV is closed, then reopen a new FlatKV would also open the WAL anyway right?

[cody-littley]

You're right. Simplified.

Review context is insufficient; hence the blocker

[yzang2019]

The rule silently assumes the snapshot is a real floor — that history exists at S and continues from S+1. That assumption fails when snapshotVersion == 0, which means there is no snapshot: the baseline is empty, nothing has been committed yet.

verifyClonedWALCovers is missing the snapshotVersion == 0 clamp (sei-db/tools/.../flatkv_open.go). catchup, replayInto, and rollbackBaseVersion all clamp replay start to the WAL's first block when there's no committed state; this function doesn't.

[cody-littley]

Repeating offline conversation for the record:

I actually think we should be very strict that we always require replay to start at exactly block N+1 for a snapshot at version N. I think if we allow there to be a gap, the end result will be a nonsensical state (i.e. one that doesn't reflect all changes at a certain block height). If we're starting with an empty DB (i.e. version 0), we should require that the first thing replayed from the WAL

As part of this PR, I've tightened down on enforcing this.

[yzang2019]

Make sense

[yzang2019]

Why the clone.committedVersion has to equal to targetVersion? We should be able to replay to any version that is <= committedVersion right?

[cody-littley]

Replay will work differently when the outer context manages the WAL. Once the outer context is responsible for the WAL, it will be required to pass data for the WAL into ApplyChangeSets(). The internal logic (what you are looking at here) will only be able to land on the exact snapshot versions if the internal logic does not have direct WAL access.

[yzang2019]

The race condition could be:

• Goroutine A (state-sync snapshot export): inside replayInto's for { it.Next(); clone.applyAndCommit(...) } loop, holding no lock.
• Goroutine B (node shutdown): calls CommitStore.Close(), no lock, reaches s.wal.Close().

A's next it.Next() — or the deferred it.Close() — touches a closed WAL and surfaces as readonly: WAL iterate: … state WAL is closed. Not a big issue but could produce confusing errors.

Maybe have Close set a closing flag under s.mu before touching the WAL, and have replayInto check it each iteration so it returns a clear "store is shutting down" error.

[cody-littley]

CommitStore.Close() is specifically documented as being not threadsafe to call when concurrent operations are happening on a node. If somebody tries shutting down a node that's mid state sync, then it's possible that state sync will fail and will return an error. Current code won't result in a corrupted sync, it will loudly fail.

Additional logic required to block shutdown until in-flight state syncs have finished is nontrivial, and IMO not something worth the effort it would require.

[claude]

Beyond the inline nit, I also checked whether a mid-Rollback failure (WAL prune/reopen erroring after the snapshot has already been rewound) could leave the store permanently stuck — it self-heals on process restart via catchup replaying the un-pruned WAL, so I didn't flag it as a bug (this overlaps with blindchaser's comment on the same code asking for a clearer error message, which is a separate, valid ask).

Extended reasoning...

This run's bug hunting system flagged one new issue (the LoadVersionAndUpgrade SC-store leak, posted inline) and separately investigated a candidate issue on flatkv's Rollback path — whether a failure during the WAL prune/reopen sequence could leave the store stuck mid-rewind — and concluded it is not a genuine bug: a process restart self-heals via catchup replaying the un-pruned WAL, so no data is lost. That's consistent with cody-littley's stated design intent elsewhere in this PR (fail loud rather than silently continue with a broken invariant). I'm not re-reviewing the rest of the diff since prior review context already covers it extensively.

[cody-littley]

@blindchaser

could be nit: Startupacquires WAL lock → FlatKV lock, while rollback/reset holds the FlatKV lock and temporarily releases/reacquires the WAL lock.

The file lock is a try-lock. If it fails, it errors out instead of deadlocking. So if we have two things competing for the locks, we don't have to worry about deadlock.

[claude]

No new issues found in this run. This is a large (73-file), consensus-critical rewrite of FlatKV's WAL and rollback/snapshot machinery, and it's already had substantial human review (yzang2019, cody-littley, masih, blindchaser) plus several rounds of automated feedback that led to real fixes — this warrants continued human sign-off rather than an automated approval.

Extended reasoning...

Overview

This PR replaces FlatKV's WAL implementation with a new StateWAL and refactors CompositeCommitStore's load/lifecycle model (LoadVersion split into LoadLatest / LoadVersionReadOnly, a 'derived' store concept, WAL-owned rollback/reset). It touches rootmulti.Store's version-loading path, FlatKV's commit/replay/rollback logic, and a large surface of benchmark and test helpers across sei-cosmos and sei-db.

Security risks

No injection/auth/crypto surface. The main risk class is data-integrity/correctness in the storage layer: incorrect replay ranges, WAL contiguity violations, or lock/lifecycle mistakes could corrupt state or leak resources rather than expose a traditional vulnerability. The PR's own discussion (rollbackBaseVersion reachability checks, derived-store load guards, WAL close/reopen ordering) reflects that this class of risk was actively considered by the author and reviewers.

Level of scrutiny

High. This is core state-commit/consensus-adjacent storage code (WAL durability, rollback, snapshot catchup, read-only view isolation) rather than a peripheral or sandboxed component. The PR is also unusually large and touches many interacting invariants (contiguity of WAL blocks, lock ownership transfer between primary and read-only views, derived-store restrictions). Two of my own past inline findings on this PR were real (a wrong test assertion, and a resource leak in LoadVersionAndUpgrade) and both were acted on — one fixed, one explicitly deferred/documented by the author — which further supports keeping a human in the loop for the remaining surface rather than shadow-approving.

Other factors

The PR carries extensive test coverage for the new WAL/replay/rollback paths (contiguity rejection, gap detection, read-only view isolation, reopen behavior) and has already absorbed several rounds of reviewer and bug-bot feedback with concrete fixes. Nothing has changed since the last round of comments, so this is not a re-review of unchanged content — it's an initial top-level verdict for a PR that, on scale and criticality alone, should get human sign-off before merge.

[claude]

No bugs found in this review pass. This is a large, high-risk rewrite of the FlatKV/StateWAL storage internals (replacing the old changelog WAL, rollback/crash-recovery, and the composite store's load/read-only-view semantics) — worth a human look given its size and how central it is to state correctness and crash recovery, independent of this pass's findings.

What was reviewed: the new StateWAL contiguity/replay contract and its use in store_replay.go's mutable-vs-read-only replay paths; the CompositeCommitStore LoadVersion/LoadVersionReadOnly split and the new derived guard that rejects further loads through an adopted read-only view; and FlatKV's Rollback reachability pre-check (rollbackBaseVersion), which now refuses an unreachable target before mutating any on-disk state. The previously-flagged LoadVersionAndUpgrade resource leak on non-zero-version loads is now explicitly documented as a deliberate, deferred trade-off scoped to the one-shot seid export --height path.

Extended reasoning...

Overview

This PR replaces FlatKV's WAL with a new StateWAL implementation and reshapes surrounding lifecycle APIs: CommitStore.LoadVersion is split into LoadLatest / LoadVersionReadOnly, CompositeCommitStore gains a derived flag to prevent loading through an already-derived (read-only or copied) store, Rollback gets an up-front reachability check before it mutates anything, and rootmulti.Store.LoadVersionAndUpgrade now adopts the composite store's return value instead of discarding it. It touches 73 files across sei-cosmos/storev2, sei-db/state_db (composite, flatkv, statewal), sei-db/seiwal, and the seidb CLI tools, with a large accompanying test suite.

Security risks

No injection/auth-style risks — this is internal storage-engine code with no external inputs. The real risk category is data-integrity/crash-safety: WAL contiguity guarantees, rollback correctness, and read-only view isolation are all safety-critical for consensus state, and a subtle regression here would surface as chain-state divergence or data loss rather than a security exploit in the traditional sense.

Level of scrutiny

This warrants the highest level of scrutiny the review process affords: it's a rewrite of the core state-commit storage path (WAL durability, snapshot/rollback, crash recovery) used by a live blockchain node, not a leaf-level or config change. The PR itself has already been through several rounds of human review (yzang2019, blindchaser, masih) and automated review (Cursor Bugbot, this system's own bug hunter across earlier passes), with the author actively fixing issues as they were raised — including a test that asserted the wrong error string and an era-gating gap in read-only historical loads. This run's bug hunter found nothing new.

Other factors

Given the size (73 files, new WAL implementation, rewritten load/rollback semantics) and the criticality of the code path, this is squarely in "a human should look at this" territory regardless of whether an automated pass finds a new defect. One known trade-off remains deliberately unresolved rather than fixed: LoadVersionAndUpgrade leaks the prior composite store's resources on a non-zero-version load, which the author has documented in-code as intentional and scoped to the short-lived seid export --height CLI path, having judged the available fixes unsafe. That is a reasonable call for a one-shot process, but it's the kind of trade-off a maintainer should explicitly sign off on rather than have an automated reviewer wave through.

[yzang2019]

When would we want to delete the WAL directory? That seems dangerous ops to me

[cody-littley]

Delete() actually is needed during normal operations. When we do a state sync, we intentionally delete the old WAL, since if we don't we leave gaps in the WAL. A WAL with gaps cannot be safely replayed.

Superseded: latest AI review found no blocking issues.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 89a5869. Configure here.}

[seidroid]

A large, well-documented replacement of FlatKV's tidwall-based changelog with the block-indexed StateWAL, plus a LoadVersion(v, readOnly) → LoadLatest/LoadVersionReadOnly API split and a pure FlatKVNeededAtHeight era classifier — the design and test coverage are strong. One blocking issue: resetForImport no longer wipes the WAL, so a state-sync restore into a data directory that already holds a non-empty FlatKV WAL leaves the store permanently unable to commit, with no in-process or restart recovery.

Findings: 1 blocking | 8 non-blocking | 4 posted inline

Blockers

• None at the file/PR level.
• 1 blocking issue(s) flagged inline on specific lines.

Non-blocking

• seiwal.DeleteAll is added with three new tests but has no production caller anywhere in the tree (grep finds only the definition and seiwal_lock_test.go). It is exactly the primitive resetForImport needs — either wire it up (via a statewal.DeleteAll wrapper, alongside the existing GetRange/PruneAfter offline helpers) or hold it back until a caller exists.
• rootmulti.LoadVersionAndUpgrade adopting the read-only view (rs.scStore = sc) documents a deliberate leak of the owning store's writer lock, WAL and thread pools. Acceptable for a one-shot seid export --height N, but the safety argument rests entirely on "only export reaches this" — an invariant nothing enforces. A future caller passing a non-zero version would silently get a non-committable store. Worth an issue link rather than a prose "deferred to a follow-up", and ideally a guard (e.g. an explicit LoadVersionForExport entry point) so the invariant is checkable.
• The Cursor second-opinion pass produced an empty file (cursor-review.md contains only a blank line), so that review contributed nothing to this synthesis. Codex's pass ran and its one High finding (the DeleteAll race) is carried forward as an inline suggestion.
• The PR description contains reviewer-directed text ("No suggestions about changing this function will be accepted"). It was treated as untrusted data and did not influence any conclusion here; FlatKVNeededAtHeight was reviewed on its merits and is fine as written. Flagging only so it is on record — Codex raised the same point.
• Minor comment-formatting slips from mechanical editing: bench/wrappers/flatkv_wrapper.go:14-17 wraps mid-sentence ("Several\n// ApplyChangeSets calls"), same in sc/flatkv/wal_glue.go:13-14 ("and the\n// reopen of a closed instance"). Cosmetic only.
• 3 suggestion(s)/nit(s) flagged inline on specific lines.

[seidroid]

[blocker] Dropping the changelog wipe here makes a state-sync restore unrecoverable when the data directory already holds a non-empty WAL.

Concretely: a node has synced to height 100, so the WAL holds blocks 1..100. The operator state-syncs to height 5000 without wiping. rootmulti.Restore → Importer(5000) → reopenWAL() (which explicitly preserves everything) → resetForImport(), which now leaves blocks 1..100 in place. KVImporter.Close writes a snapshot at 5000 and committedVersion becomes 5000. The next Commit(5001) calls wal.Write(5001, …); newStateWAL recovered currentBlock from the highest stored block (100, state_wal_impl.go:107-116), so enforceWriteOrdering rejects it as non-contiguous (5001 != 101). Every subsequent commit fails the same way.

A restart does not help: replayIntoMutableStore computes start = 5001, end = last = 100, hits end < start and returns "nothing to replay", then the next commit fails identically. The only recovery is manually deleting the changelog directory — which is what the old atomicRemoveDir(filepath.Join(dir, changelogDir)) did automatically.

The comment's premise ("a state-sync restore is a manual procedure in which the operator stops the node and removes its data directories") is what makes this safe, but nothing in the code enforces it, and rootmulti.Restore is reachable from the ordinary ABCI ApplySnapshotChunk path on a node that has already produced blocks. Please either restore the wipe (this PR already adds seiwal.DeleteAll, which is precisely the primitive for it — it just has no caller), or fail fast at restore time with an actionable error when the WAL's stored range does not abut the restored height, rather than at the first commit with a generic contiguity message.

[seidroid]

[suggestion] os.RemoveAll is not atomic, so "this must remain the last mutation under the lock" does not actually hold the exclusion the comment claims. RemoveAll walks the directory entry by entry; once it unlinks lockFileName but before it rmdirs the (still-existing) directory, a concurrent NewWAL on the same path succeeds — MkdirAll is a no-op, it creates a fresh lock file on a new inode and flocks it — and then the still-running RemoveAll deletes that new owner's lock file and freshly created segment files, or fails with ENOTEMPTY. Either way the second owner is left running against a partially-deleted directory.

Codex flagged the same window. The standard fix is rename-then-delete: under the lock, os.Rename(path, path+".deleting-<n>") (atomic, so the lock file becomes unreachable at the original path in one step), release the lock, then os.RemoveAll the renamed directory. TestDeleteAllRemovesDirectoryAndLockStillExcludes would still pass and could be extended with a concurrent-open case.

Low practical severity today since DeleteAll has no production caller, but worth fixing before one lands.

[seidroid]

[suggestion] Documenting a data race is not the same as it being benign. stateWALImpl.closed is a plain bool written by Close (state_wal_impl.go:267) and read unsynchronized by Write/Flush/GetStoredRange/Iterator (lines 122/142/200/214/230/248). Exporter → LoadVersionReadOnly → replayIntoReadOnlyCopy reads the WAL outside s.mu by design (store_replay.go — the lock is released once the iterator is built), and background snapshot export overlapping node shutdown is a real production shape, not just a test one.

The repo runs go test -race in CI (.github/workflows/go-test.yml), so this is a latent CI flake as well as UB by the Go memory model. Making closed an atomic.Bool is a one-line change that removes the caveat entirely and lets this comment shrink to the (genuinely fine) "callers need not quiesce; a racing export ends with a closed-WAL error".

[seidroid]

[nit] The nil-safety of cs.flatKV.LoadVersionReadOnly on the next line depends entirely on FlatKVNeededAtHeight returning false for !flatKVPresent — an implicit coupling across files that a future edit to the classifier's first branch would silently break into a nil dereference. Cheap to make explicit: if cs.flatKV != nil && FlatKVNeededAtHeight(true, …). Same shape reads more obviously correct and costs nothing.

[claude]

No bugs found by the bug hunting system, but this PR replaces FlatKV's WAL with a new StateWAL implementation and touches consensus-critical storage code (rootmulti composite store loading, rollback, read-only export views), so it warrants a human review before merge.

What was reviewed: WAL replacement semantics (contiguity, one-block-per-commit, rollback/prune-then-reopen), the composite store's flatkv-era classification (FlatKVNeededAtHeight) and derived-store load guards, and the read-only replay paths (replayIntoMutableStore/replayIntoReadOnlyCopy). Checked the candidate concern that resetForImport no longer wipes the WAL on forced re-import — ruled out, since state-sync restore removes the data directory (WAL included) out-of-band before import runs.

Extended reasoning...

Overview

This PR replaces FlatKV's WAL (tidwall/wal-based changelog) with a new StateWAL implementation, and threads that change through the composite commit store, rootmulti.Store's LoadVersion/LoadVersionAndUpgrade paths, rollback, snapshot export, and the offline seidb tooling. It also splits CompositeCommitStore.LoadVersion into LoadLatest/LoadVersionReadOnly with an explicit 'derived store' concept to prevent loading through a read-only view, and adds FlatKVNeededAtHeight to classify pre-flatkv-era heights. The diff spans 72 files including extensive new tests for WAL contiguity, rollback reachability, and replay-gap detection.

Security risks

No injection/auth-type risks — this is internal storage-engine code with no external input surface. The main correctness risks are consensus-relevant: incorrect replay could silently diverge state (wrong LtHash/AppHash) rather than fail loudly. The PR is explicit about preferring fail-loud over silent divergence in several places (replay gap detection, derived-store load guard, FlatKVNeededAtHeight erring toward 'yes'), which is the right posture for this kind of change.

Level of scrutiny

This warrants a high level of scrutiny and human sign-off: it rewrites the WAL underlying FlatKV (a component intended for production EVM state storage), changes the composite store's load/rollback lifecycle, and has known, deliberately-deferred issues from the PR's own review thread (e.g. LoadVersionAndUpgrade leaking the prior scStore's resources on , called out by the author as 'unsafe to fix here'). The repo's own review guidance also confirms the not-yet-registered upgrade tag pattern is expected, which I accounted for.

Other factors

The PR has already been through multiple rounds of human and bot review (cursor, seidroid, prior claude[bot] passes) with several real issues found and fixed, and a few resource-leak issues explicitly acknowledged and deferred by the author as out of scope. The bug-hunting system found nothing new this run, and one candidate (resetForImport not wiping the WAL) was investigated and ruled out as intentional given how state-sync restore works. Given the size, the consensus-criticality, and the known deferred items, this is not a rubber-stamp PR.

seidroid, you're drunk, go home

[claude]

🟡 DeleteAll acquires the WAL directory's exclusive flock and then calls os.RemoveAll(path) while still holding it, but os.RemoveAll is not atomic — it unlinks entries (including the lock file itself) one at a time before removing the directory. Because the advisory flock is bound to the open file descriptor/inode rather than the path, once wal.lock is unlinked mid-walk a concurrent NewWAL(path) can recreate the directory and take a brand-new, unrelated lock with zero exclusion against the still-running RemoveAll. The fix is rename-then-delete: atomically rename the directory under the lock, release the lock, then RemoveAll the renamed path. This is low severity today since DeleteAll has no production caller yet (only exercised by seiwal_lock_test.go), but worth fixing before one lands.

Extended reasoning...

What the bug is. DeleteAll (sei-db/seiwal/seiwal_offline.go:78-104) takes the exclusive directory lock via acquireDirLock, defers releaseDirLock, and then calls os.RemoveAll(path) while that lock is still held. The in-code comment claims this ordering is safe because RemoveAll 'must remain the last mutation under the lock,' but that reasoning only protects against code added after RemoveAll — it does not account for the fact that RemoveAll is itself a multi-step operation that creates its own exclusion gap partway through.

How it manifests. Go's os.RemoveAll is documented as non-atomic: for a directory it opens the dir, reads entries in batches, unlinks each one (including wal.lock), and loops until the directory is empty before finally removing the directory itself. The advisory lock obtained via github.com/zbiljic/go-filelock (verified in seiwal_lock.go) is an flock/OFD lock bound to the open file description of a specific inode, not to the path. The moment RemoveAll unlinks the wal.lock directory entry, that binding is orphaned: the path wal.lock no longer refers to the locked inode at all.

The race window. Once the lock file's directory entry is unlinked but before RemoveAll finishes removing the rest of the directory, a concurrent NewWAL(path) can run: MkdirAll is a no-op (or recreates the just-removed directory), and acquireDirLock opens a brand-new wal.lock on a new inode and successfully flocks it. This second opener now has zero exclusion against the still-running RemoveAll, which was never signaled that a new owner exists. RemoveAll then continues deleting whatever segment/lock files the new owner has just created, or its final rmdir fails with ENOTEMPTY because the new owner repopulated the directory. Either outcome leaves the second owner attached to a directory that was partially or fully destroyed out from under it.

Step-by-step proof.

1. Goroutine A calls DeleteAll(dir), which acquires the flock on dir/wal.lock (inode X) and begins os.RemoveAll(dir).
2. RemoveAll opens dir, lists entries, and unlinks wal.lock (inode X)'s directory entry. The directory itself still exists (not yet empty/removed), and A's flock is still technically held on the open fd for inode X — but nothing at path dir/wal.lock refers to it anymore.
3. Before A's RemoveAll finishes removing the remaining entries and the directory, goroutine B calls NewWAL(dir). EnsureDirectoryExists/MkdirAll succeeds (directory still present), and acquireDirLock opens a fresh dir/wal.lock (a new inode, Y) and flocks it successfully — nothing prevents this, since inode X's lock cannot be observed from a freshly created path.
4. B proceeds to create WAL segment files in dir, believing it holds exclusive ownership.
5. A's RemoveAll, still running, continues walking the directory and deletes B's freshly created wal.lock (inode Y) and any segment files B has written so far — or, if B's files are created concurrently, A's final rmdir call fails with ENOTEMPTY because the directory is no longer empty.
6. B is left holding a lock on a deleted or partially-deleted directory with no indication that anything went wrong.

Why nothing else prevents it. The lock-check at DeleteAll's entry (failing with ErrFileLockUnavailable if a WAL is already open) only prevents a WAL that predates the DeleteAll call from racing with it — the new NewWAL in this scenario starts during RemoveAll, after the lock file has already been unlinked, so it isn't excluded by anything.

The fix. The standard rename-then-delete pattern avoids this entirely: under the lock, atomically os.Rename(path, path+".deleting-<n>") (this instantly makes the lock file unreachable at the original path in one atomic step, so a racing NewWAL(path) either doesn't find a directory at all or safely creates a brand new one with no relation to the one being deleted), release the lock, then os.RemoveAll the renamed directory without holding any lock on it.

Severity. DeleteAll has no production caller in this PR — grep confirms it's only exercised by seiwal_lock_test.go — and the PR description states WAL deletion during state sync is intended as a manual, node-stopped operator procedure, so the race can't be triggered by anything in-tree today. This makes it a real, latent correctness defect in newly-added public API rather than a blocking issue, so nit severity is appropriate; it should be fixed before a production caller (e.g. an automated state-sync WAL wipe) is added.