Skip to content

Shader build speedups: posix_spawn, optional free-threaded Python, parallel c1_guard, xcrun by full path - #366

Open
zorrobyte wants to merge 2 commits into
seedds:mainfrom
zorrobyte:shader-build-speedups
Open

zorrobyte wants to merge 2 commits into
seedds:mainfrom
zorrobyte:shader-build-speedups

Conversation

@zorrobyte

Copy link
Copy Markdown
Contributor

What

Three build-speed changes to the shader step of build_ipa_from_nsp.py. No shader output changes and no cache is invalidated (dmsh_build.py is untouched).

  1. iOS shaders on every core. The step ran tens of thousands of tool runs through fork (Popen falls back to it for a bare program name), held the GIL for the per-shader Python work, and took a libc locale lock per pipe. own_compilers.py now resolves the program and starts it with posix_spawn; the NSP script runs the step on python3.14t when it is on the PATH (uv python install 3.14t, optional) and sets PYTHONUTF8.
  2. c1_guard.py on every core. Each shader is checked on its own, so a process pool (as in cb_extent.py) takes the loop. A game with 73,496 shaders: 9 min on one core before, 54 s now, with the same 30 shaders listed.
  3. own_compilers.py runs xcrun's tools by their full path. One process fewer per Metal compile, about 15% less compile time. 400 of 400 iOS libraries are byte-identical to xcrun's, and compiler_id (the Metal cache key) is unchanged.

docs/development.md (step 6 of the script) mentions the free-threaded Python.

Tested

  • python3 -m py_compile on c1_guard.py, own_compilers.py, build_ipa_from_nsp.py: ok
  • python3 -m unittest on test_docs, test_nsp_chain, test_nsp_mac, test_nsp_incremental, test_nsp_progress: all pass
  • Full iOS shader builds were run on the author's Mac for the numbers above. regress.py was not run on this branch: the change touches no translator, chain or host code.

🤖 Generated with Claude Code

seedds added a commit that referenced this pull request Oct 27, 2026
c1_guard.py read and analysed each shader library's MSL in one loop on
one core, after a full dmsh_build.py run whose compile already used every
core: FINAL FANTASY RESONANCE's 73,496 shaders kept one core busy for
about 7 minutes. Each shader is checked on its own (its MSL and its
reflection's c1 length), so the loop body becomes unproven(), mapped over
the sorted file names with a multiprocessing pool as cb_extent.py already
does; Pool.map keeps the order, so c1_full.tsv lists the same keys in the
same order as before.

Same keys, byte for byte below the header line, on three games:
Silver Pines (56,386 libraries, 13,372 keys) 299 s -> 44 s on an M5;
Pokemon Legends Z-A (28,603, 1,011 keys) 206 s -> 17 s and Pokemon
Scarlet (30,142, 325 keys) 211 s -> 21 s here. c1_guard.py is not a
chain input (dmsh_chain.FILES), so no library or chain version changes.
The PR's second commit (test_nsp_space.py's spec assert) is already on
main as part of 4fc7f51. PR #366 carried the same c1_guard change.

Co-authored-by: Chen Chia Yang <[email protected]>
@seedds

seedds commented Oct 27, 2026

Copy link
Copy Markdown
Owner

The c1_guard.py part of this PR (a process pool over the shaders) landed on main as 75065df, from #381, which carries the same change. Rebasing this branch will drop or conflict on that commit (a5db891).

The other two parts are not on main and stay under review here: posix_spawn with the optional free-threaded Python, and running xcrun's tools by their full path.

@zorrobyte
zorrobyte force-pushed the shader-build-speedups branch from 8318bec to 58ade2a Compare October 27, 2026 15:19
@zorrobyte

Copy link
Copy Markdown
Contributor Author

Rebased onto main and dropped the c1_guard.py commit, which landed as 75065df. The branch now holds only the posix_spawn / optional free-threaded Python commit and the xcrun full-path commit, with the conflicts in own_compilers.py, build_ipa_from_nsp.py and docs/development.md resolved against the new Metal-compiler check. CI is green.

@zorrobyte
zorrobyte force-pushed the shader-build-speedups branch from 58ade2a to 4f458b5 Compare October 28, 2026 03:24
zorrobyte and others added 2 commits October 31, 2026 01:23
…nal free-threaded Python

The shader step ran tens of thousands of tool runs with fork (Popen falls back to it for a bare program name or close_fds on), held the GIL for the per-shader Python work, and took a libc locale lock per pipe. own_compilers.py now resolves the program and spawns it, the NSP script runs the step on python3.14t when it is installed, and sets PYTHONUTF8. dmsh_build.py is unchanged, so no shader cache is invalidated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
One process fewer per Metal compile, about 15% less compile time; 400 of 400 iOS libraries byte-identical to xcrun's, compiler_id (the Metal cache key) unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@zorrobyte
zorrobyte force-pushed the shader-build-speedups branch from 4f458b5 to 37a64dd Compare October 31, 2026 05:25
@github-actions github-actions Bot added packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers tools Tools and scripts (dm/tools, scripts) labels Nov 3, 2026
Know-It-All-Brother pushed a commit to Know-It-All-Brother/nxnative that referenced this pull request Nov 9, 2026
c1_guard.py read and analysed each shader library's MSL in one loop on
one core, after a full dmsh_build.py run whose compile already used every
core: FINAL FANTASY RESONANCE's 73,496 shaders kept one core busy for
about 7 minutes. Each shader is checked on its own (its MSL and its
reflection's c1 length), so the loop body becomes unproven(), mapped over
the sorted file names with a multiprocessing pool as cb_extent.py already
does; Pool.map keeps the order, so c1_full.tsv lists the same keys in the
same order as before.

Same keys, byte for byte below the header line, on three games:
Silver Pines (56,386 libraries, 13,372 keys) 299 s -> 44 s on an M5;
Pokemon Legends Z-A (28,603, 1,011 keys) 206 s -> 17 s and Pokemon
Scarlet (30,142, 325 keys) 211 s -> 21 s here. c1_guard.py is not a
chain input (dmsh_chain.FILES), so no library or chain version changes.
The PR's second commit (test_nsp_space.py's spec assert) is already on
main as part of 7b92332. PR seedds#366 carried the same c1_guard change.

Co-authored-by: Chen Chia Yang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants