Skip to content

packing/shader_split: read magics once, decompress zstd in process - #3002

Open
eric1932 wants to merge 2 commits into
seedds:mainfrom
eric1932:perf/shader-split-reads
Open

eric1932 wants to merge 2 commits into
seedds:mainfrom
eric1932:perf/shader-split-reads

Conversation

@eric1932

Copy link
Copy Markdown
Contributor

What and why

The shader split of Tears of the Kingdom takes 60 to 75 s on an M5 Pro, and under 1% of its workers' time is work on shaders: 1,683 of its 304,243 RomFS files hold a shader container. The time went to the other files:

  • Before the pool had anything to do, the four job finders that know their archives by the magic (CRA, zip, PDAT, RPF6) each opened every file in the main process: 22.9 s of the 26.4 s before the scan. The pool now starts before the jobs are planned and reads every file's magic once; each finder is given the files that start with its own and checks them as before.
  • unzs ran the zstd command for each zstd file that names no dictionary: 32,176 processes, 39% of the workers' time. It decompresses them in the process, every frame, as it did files that name a dictionary.
  • romfs.py opened the image by its path for every file opened inside it, and 18 processes wait on one another there: 133 µs an open against 28 µs in one process alone. The image is opened once per process and each file takes a duplicate of that descriptor.

Two commits, romfs.py first, so it can be taken alone.

The split commit changes DMSH_CHAIN: shader_split.py is in dmsh_chain.py's FILES, so every shader library is built again once. The romfs.py commit leaves the chain as it is.

The outputs are the same. The package and the four tables are byte-identical in three games.

Cold split, wall (CPU), best of three Tears of the Kingdom Breath of the Wild Pokémon Scarlet
main 60.0 s (528 s) 70.1 s (932 s) 10.4 s (64 s)
the romfs.py commit 47.7 s (471 s)
both commits 17.5 s (215 s) 67.1 s (935 s) 10.4 s (62 s)

The saving is 42 s for Tears of the Kingdom, measured against today's split. It does nothing for Scarlet (85 jobs: nearly all of its files are in one pack file, read through its index) and 3 s for Breath of the Wild, whose split is Yaz0 decoding in Python (87% of its workers' time).

An earlier pack on this Mac logged 186 s for this step (Tears of the Kingdom, 2026-11-16) and 155 s (Scarlet). Those times were not reproduced with the same code and files, alone or in a cold pack, and the saving is not measured against them.

Both mechanisms are new entries: The shader split takes a minute on a RomFS of 300,000 files and Processes that scan one RomFS image in parallel wait on its opens.

Checked

M5 Pro (18 cores), macOS 26.7.3, Xcode 27.2, Python 3.13. Based on main c25d4011. Main's chain has changed since, so GitHub shows a conflict: it is only the generated dm/gfx/nvn/dmsh_chain.h, and dmsh_chain.py --header on main's copy resolves it. Applied onto main 0b5fd602 that way here, the new tests, the docs test and the chain test pass.

  • Old against new, the split run as the NSP script runs it (J=18) into a scratch folder: Tears of the Kingdom 1.4.3, Breath of the Wild 1.9.0 with its DLC, Pokémon Scarlet 1.0.0 (28,669, 19,592 and 30,142 shaders). Every file is byte-identical: 13, 11 and 8 package files and the four tables, whole files, in three runs of each revision; no [RED] line in any.

  • Timing: three rounds of main, the split commit and both in turn, each run alone on the Mac. Timed on c22f30a3, four commits of main earlier, where shader_split.py and romfs.py are the same; on this branch each game's split was run once more and is identical again.

  • Inside a cold --platform mac pack stopped when the split ends, Tears of the Kingdom: 70.6 s on main, 21.9 s with both commits, one run each.

  • regress.py chain, main c25d4011 against the branch (c3a6aeb4), because the chain version changes: PASS. Every shader builds on both sides, and no MSL, reflection or pipeline interface differs.

    Game Shaders (base → candidate) Newly failing Newly building Changed MSL Interface (new / known)
    Breath of the Wild 01007EF00011E000 19592/19592 → 19592/19592 0 0 0 0 / 0
    Pokémon Scarlet 0100A3D008C5C000 30142/30142 → 30142/30142 0 0 0 0 / 0
    Tears of the Kingdom 0100F2C0115B6000 28669/28669 → 28669/28669 0 0 0 0 / 0

    These three are the games on this Mac, and the change reaches a game only through its split, whose outputs are identical. No game was run (regress.py check): the host loads the same libraries under a new chain version.

  • Tests:

    • test_split_magics.py (new) runs the whole split on a zip archive under another name and a zstd file of two frames with no zstd command on the path, and checks file_magics. test_romfs_read.py checks that two files of an image open it once.
    • The tool tests as CI runs them (every test_*.py of dm/tools/tests, dm/ios/tests and dm/gfx/tests): 243 of 244 files pass, 2,585 tests, 58 skipped. test_shader_toolchain_cache.py has 3 errors, a NameError in test_mesh_compiles_only_requested_targets, the same on main c25d4011.
    • lint_python.py --against main: no new finding.
  • A red error from a job finder still ends the tool at once with the pool already started (exit 1 in 0.8 s, no process left).

  • Not run: a whole pack to its end, Linux (the pool forks there and inherits the image's descriptor), a RomFS kept as a folder, a zstd file with legacy frames or a damaged one (the module refuses a damaged file as the command did, and it is left out with a red line whose wording differs), any other game.

Left open, in the entry: Breath of the Wild's Yaz0 decoding in Python, the seven searches for container magics over every decompressed file (114 s of the workers' 260 s that are left for Tears of the Kingdom), and why the packs of 2026-11-16 took so much longer.

Docs

  • The findings are two new entries in docs/reference/packing.md, one per commit, with their lines in Tears of the Kingdom's note; docs_symptoms.py was run.
  • A chain tool changed (shader_split.py): dm/gfx/nvn/dmsh_chain.h is regenerated in the split commit (9b791ad68066 to 4f67b99a24f5). The romfs.py commit leaves it as on main.
  • No game files, keys or data extracted from a game in the change.

🤖 Generated with Claude Code

eric1932 and others added 2 commits November 18, 2026 00:12
A file opened inside a RomFS image opened the image by its path each
time. Processes that scan one image in parallel wait on one another
there: in the shader split of Tears of the Kingdom (304,243 files, each
opened twice, 18 processes) an open took 133 us, 12% of the workers'
time, against 28 us in one process alone.

The image is now opened once per process, at its first file, and each
file takes a duplicate of that descriptor. Reads are by offset, and the
mmap wrapper still finds a file's range by the file's own descriptor.

Tears of the Kingdom's split takes 47.7 s instead of 60.0 s (best of
three, M5 Pro), its package and tables byte-identical. romfs.py is not
one of the chain's files: the chain version stays.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
The shader split of Tears of the Kingdom (304,243 RomFS files, 1,683 of
them with a shader container, 18 processes) took 60 s, and its time went
to files that hold no shader:

- Before the pool had anything to do, the four job finders that know
  their archives by the magic (CRA, zip, PDAT, RPF6) each opened every
  file in the main process: 1.2 million opens, 22.9 s of the 26.4 s
  before the scan. The pool now starts before the jobs are planned and
  reads every file's magic once; each of the four is given the files
  that start with its own and checks them itself as before.
- unzs ran the zstd command for each zstd file that names no dictionary:
  32,176 processes, 39% of the workers' time. It decompresses them in
  the process, every frame, as it did files that name a dictionary.

With the one descriptor per image of the commit before, the split takes
17.5 s (best of three, M5 Pro; 47.7 s without this commit). Breath of
the Wild's goes from 70.1 to 67.1 s and Pokemon Scarlet's stays at
10.4 s. The package and the four tables of the three games are
byte-identical.

shader_split.py is one of the chain's files, so the chain version
changes with it (dmsh_chain.h): every shader library is built again once.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers tools Tools and scripts (dm/tools, scripts) docs Documentation shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders game: 0100F2C0115B6000 The Legend of Zelda: Tears of the Kingdom labels Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 0100F2C0115B6000 The Legend of Zelda: Tears of the Kingdom nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant