Skip to content

Fix #99: deterministic multithreaded dump (record name cached during another thread's quick init) - #102

Merged
ejams1 merged 1 commit into
mainfrom
fix-99-dump-race
Oct 19, 2026
Merged

ejams1 merged 1 commit into
mainfrom
fix-99-dump-race

Conversation

@ejams1

@ejams1 ejams1 commented Oct 19, 2026

Copy link
Copy Markdown
Owner

Root cause

TwbMainRecord.mrInvalidateNameCache runs when a localized file's handler generation changes: it clears the cached names and reads the full name again only when mr_names_known (upstream's mrsQuickInitDone) says the names are final. A quick init on another worker has read FULL at that point but sets the flag only after the subrecords are read. A first name read on a second thread inside that window invalidated the cache with the flag still clear, overwrote the full name with '', and the build, already past the FULL subrecord, never wrote it again — so the record kept a name without its FULL for the rest of the dump, in mrName, mrShortName and mrDisplayName (the official file's caches). The invalidation now leaves the full name alone while the record's names are not known, publishes the generation only after the names it read (a reader that sees the new generation sees the name read with it), and keeps the old name when the elements to read it again are not built yet.

A getter that gives up its wait for another thread's init (two builds need each other, threads::init_cycles) now answers upstream's not yet available text and notes the partial read, so GetName, GetShortName and GetDisplayName do not cache a name built from it either.

Before (release at main 6d9646d, --threads 8, two runs of Skyrim.esm)

Both runs differed from the cached 4.1.5q oracle, and from each other. Run 1: char 8161633134, line 211346725, where the port wrote QNAM - Quest: QUST - Quest [0004B869] <WIGamesHideAndSeek> and the oracle has <WIGamesHideAndSeek> "Hide and Seek". Run 2: char 411102402, line 9024066, where the port wrote Alias[001 Arvel] on QUST - Quest [00039645] <MS13> and the oracle has <MS13> "The Golden Claw" (the issue's own evidence was the same class at lines 210,730,161 and 581,277).

After (release with this branch)

  • Three runs at --threads 8 and one at --threads 4: byte-identical to each other and to the cached oracle, 8,322,131,525 bytes (cmp).
  • cargo xtask parity dump --game sse --file Skyrim.esm --jobs 1: equal in three consecutive runs (1 of 1 equal, port peak 2.0 GiB).

Tests and gates

  • crates/xedit-session/tests/threaded_names.rs forces both interleavings through the progress callback of a subrecord the definition does not know: an invalidation between the FULL read and the end of the init, and two quick inits that need each other. Both tests fail before the change (the first leaves CLAS - Class [00000800] <TestClass> and the second caches it; the failing assertions were checked by stashing the core change) and pass after it.
  • crates/xedit-cli/tests/threads.rs hashes three 8-thread dumps of Skyrim.esm and requires them equal; it is ignored in a debug build (an 8.3 GB dump is a release-scale run) and passed in release (217 s).
  • cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings and cargo xtask check: exit 0.
  • cargo test --workspace --no-fail-fast: every target green except the known copies_a_skyrim_npc_as_an_override failure, which fails identically on the unmodified tree.

Closes #99.

The multithreaded `xedit dump` of `Skyrim.esm` was nondeterministic: a
`QUST` name read while another worker built the record was missing its
quoted `FULL`, and the official file's name caches (`mrName`,
`mrShortName`, `mrDisplayName`) kept it for the rest of the dump. Two
8-thread runs of the unfixed release differed from the cached oracle at
line 211,346,725 (`QNAM - Quest: QUST - Quest [0004B869]
<WIGamesHideAndSeek>` against `<WIGamesHideAndSeek> "Hide and Seek"`)
and line 9,024,066 (a VMAD property `Alias[001 Arvel] on QUST - Quest
[00039645] <MS13>` against `<MS13> "The Golden Claw"`), and from each
other.

The race is `TwbMainRecord.mrInvalidateNameCache`: a record of a
localized file clears its cached names when the handler generation
changes, and reads the full name again only when the names are known
(`mr_names_known`, upstream's `mrsQuickInitDone`). A quick init on
another thread has read `FULL` but sets the flag only after the
subrecords are read; a first name read on another thread in that window
invalidated the cache with the flag clear, overwrote the full name with
'', and the build, already past that subrecord, never wrote it again. The
invalidation now leaves the full name alone while the names are not
known, writes the name it reads before it publishes the generation (so a
reader that sees the new generation also sees the name read with it), and
keeps the old name when the elements to read it again are not there yet.

A getter that gives up its wait for another thread's init (two builds
need each other, `threads::init_cycles`) now answers upstream's "not yet
available" text and notes the partial read, so `GetName`, `GetShortName`
and `GetDisplayName` do not cache a name built from it either.

Tests: `crates/xedit-session/tests/threaded_names.rs` forces both
interleavings through the progress callback of a subrecord `Init` does not
know (an invalidation between `FULL` and the end of the init, and two
quick inits that need each other); both fail before the change and pass
after it. `crates/xedit-cli/tests/threads.rs` hashes three 8-thread dumps
of `Skyrim.esm` (ignored in a debug build, which is far slower).

Verification: three 8-thread dumps and one 4-thread dump of `Skyrim.esm`
are byte-identical to each other and to the cached 4.1.5q oracle
(8,322,131,525 bytes, `cmp`); `cargo xtask parity dump --game sse --file
Skyrim.esm --jobs 1` is `equal` three times; `crates/xedit-session`'s
`threaded_names` 2 of 2; `cargo clippy --workspace --all-targets -- -D
warnings` and `cargo xtask check` exit 0; `cargo test --workspace
--no-fail-fast` is green apart from the known
`copies_a_skyrim_npc_as_an_override` failure, which fails identically
without this change.

Co-Authored-By: Claude Code <[email protected]>
@ejams1
ejams1 merged commit 5a10338 into main Oct 19, 2026
2 checks passed
@ejams1
ejams1 deleted the fix-99-dump-race branch October 19, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dump: a record's name read during another thread's quick init is cached partial (Skyrim.esm QUST FULL)

1 participant