Repository navigation
Conversation
A file opened inside a RomFS image opened the image by its path each time. Processes that scan one image in parallel wait on one another there: in the shader split of Tears of the Kingdom (304,243 files, each opened twice, 18 processes) an open took 133 us, 12% of the workers' time, against 28 us in one process alone. The image is now opened once per process, at its first file, and each file takes a duplicate of that descriptor. Reads are by offset, and the mmap wrapper still finds a file's range by the file's own descriptor. Tears of the Kingdom's split takes 47.7 s instead of 60.0 s (best of three, M5 Pro), its package and tables byte-identical. romfs.py is not one of the chain's files: the chain version stays. Co-Authored-By: Claude Fable 5.2 <[email protected]>
The shader split of Tears of the Kingdom (304,243 RomFS files, 1,683 of them with a shader container, 18 processes) took 60 s, and its time went to files that hold no shader: - Before the pool had anything to do, the four job finders that know their archives by the magic (CRA, zip, PDAT, RPF6) each opened every file in the main process: 1.2 million opens, 22.9 s of the 26.4 s before the scan. The pool now starts before the jobs are planned and reads every file's magic once; each of the four is given the files that start with its own and checks them itself as before. - unzs ran the zstd command for each zstd file that names no dictionary: 32,176 processes, 39% of the workers' time. It decompresses them in the process, every frame, as it did files that name a dictionary. With the one descriptor per image of the commit before, the split takes 17.5 s (best of three, M5 Pro; 47.7 s without this commit). Breath of the Wild's goes from 70.1 to 67.1 s and Pokemon Scarlet's stays at 10.4 s. The package and the four tables of the three games are byte-identical. shader_split.py is one of the chain's files, so the chain version changes with it (dmsh_chain.h): every shader library is built again once. Co-Authored-By: Claude Fable 5.2 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The shader split of Tears of the Kingdom takes 60 to 75 s on an M5 Pro, and under 1% of its workers' time is work on shaders: 1,683 of its 304,243 RomFS files hold a shader container. The time went to the other files:
unzsran thezstdcommand for each zstd file that names no dictionary: 32,176 processes, 39% of the workers' time. It decompresses them in the process, every frame, as it did files that name a dictionary.romfs.pyopened the image by its path for every file opened inside it, and 18 processes wait on one another there: 133 µs an open against 28 µs in one process alone. The image is opened once per process and each file takes a duplicate of that descriptor.Two commits,
romfs.pyfirst, so it can be taken alone.The split commit changes
DMSH_CHAIN:shader_split.pyis indmsh_chain.py'sFILES, so every shader library is built again once. Theromfs.pycommit leaves the chain as it is.The outputs are the same. The package and the four tables are byte-identical in three games.
romfs.pycommitThe saving is 42 s for Tears of the Kingdom, measured against today's split. It does nothing for Scarlet (85 jobs: nearly all of its files are in one pack file, read through its index) and 3 s for Breath of the Wild, whose split is Yaz0 decoding in Python (87% of its workers' time).
An earlier pack on this Mac logged 186 s for this step (Tears of the Kingdom, 2026-11-16) and 155 s (Scarlet). Those times were not reproduced with the same code and files, alone or in a cold pack, and the saving is not measured against them.
Both mechanisms are new entries: The shader split takes a minute on a RomFS of 300,000 files and Processes that scan one RomFS image in parallel wait on its opens.
Checked
M5 Pro (18 cores), macOS 26.7.3, Xcode 27.2, Python 3.13. Based on main
c25d4011. Main's chain has changed since, so GitHub shows a conflict: it is only the generateddm/gfx/nvn/dmsh_chain.h, anddmsh_chain.py --headeron main's copy resolves it. Applied onto main0b5fd602that way here, the new tests, the docs test and the chain test pass.Old against new, the split run as the NSP script runs it (
J=18) into a scratch folder: Tears of the Kingdom 1.4.3, Breath of the Wild 1.9.0 with its DLC, Pokémon Scarlet 1.0.0 (28,669, 19,592 and 30,142 shaders). Every file is byte-identical: 13, 11 and 8 package files and the four tables, whole files, in three runs of each revision; no[RED]line in any.Timing: three rounds of main, the split commit and both in turn, each run alone on the Mac. Timed on
c22f30a3, four commits of main earlier, whereshader_split.pyandromfs.pyare the same; on this branch each game's split was run once more and is identical again.Inside a cold
--platform macpack stopped when the split ends, Tears of the Kingdom: 70.6 s on main, 21.9 s with both commits, one run each.regress.py chain, mainc25d4011against the branch (c3a6aeb4), because the chain version changes: PASS. Every shader builds on both sides, and no MSL, reflection or pipeline interface differs.01007EF00011E0000100A3D008C5C0000100F2C0115B6000These three are the games on this Mac, and the change reaches a game only through its split, whose outputs are identical. No game was run (
regress.py check): the host loads the same libraries under a new chain version.Tests:
test_split_magics.py(new) runs the whole split on a zip archive under another name and a zstd file of two frames with nozstdcommand on the path, and checksfile_magics.test_romfs_read.pychecks that two files of an image open it once.test_*.pyofdm/tools/tests,dm/ios/testsanddm/gfx/tests): 243 of 244 files pass, 2,585 tests, 58 skipped.test_shader_toolchain_cache.pyhas 3 errors, aNameErrorintest_mesh_compiles_only_requested_targets, the same on mainc25d4011.lint_python.py --againstmain: no new finding.A red error from a job finder still ends the tool at once with the pool already started (exit 1 in 0.8 s, no process left).
Not run: a whole pack to its end, Linux (the pool forks there and inherits the image's descriptor), a RomFS kept as a folder, a zstd file with legacy frames or a damaged one (the module refuses a damaged file as the command did, and it is left out with a red line whose wording differs), any other game.
Left open, in the entry: Breath of the Wild's Yaz0 decoding in Python, the seven searches for container magics over every decompressed file (114 s of the workers' 260 s that are left for Tears of the Kingdom), and why the packs of 2026-11-16 took so much longer.
Docs
docs/reference/packing.md, one per commit, with their lines in Tears of the Kingdom's note;docs_symptoms.pywas run.shader_split.py):dm/gfx/nvn/dmsh_chain.his regenerated in the split commit (9b791ad68066to4f67b99a24f5). Theromfs.pycommit leaves it as on main.🤖 Generated with Claude Code