Skip to content

c1_guard.py: check the shaders on every core - #381

Closed
Unayung wants to merge 2 commits into
seedds:mainfrom
Unayung:c1-guard-parallel
Closed

Unayung wants to merge 2 commits into
seedds:mainfrom
Unayung:c1-guard-parallel

Conversation

@Unayung

@Unayung Unayung commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

c1_guard.py reads and analyses each shader library's MSL in one loop on one core, right after a full dmsh_build.py run whose compile already used every core. On FINAL FANTASY RESONANCE's 73,496 shaders it kept one core busy for about 7 minutes; with cb_extent.py the two passes after the compile took 11.8 minutes of a 94-minute first build (M5, 10 cores).

Each shader is checked on its own (its MSL and its reflection's c1 length), so the loop body becomes unproven(), mapped over the sorted file names with a multiprocessing pool, as cb_extent.py already does. Pool.map keeps the order, so c1_full.tsv lists the same keys in the same order.

Checked

Output Silver Pines' 56,386 Mac libraries (chain b-98ca73d18869): the same 13,372 keys, byte for byte below the header line
Time 299 s on one core → 44 s with the pool (M5, 10 cores)
Chain c1_guard.py is not a chain input (dmsh_chain.py): no library or chain version changes
Tests python3 -m unittest discover -s dm/tools/tests (334 tests, 16 skipped) passes, including test_nsp_chain.py's c1 guard cases

regress.py has nothing to compare here: the chain does not change, and this changes no host, translator or runtime behaviour.

🤖 Generated with Claude Code

https://claude.ai/code/session_018XsrnTnMEXD4jZMChLCZPK

c1_guard.py read and analysed each shader library's MSL in one loop on one core, after a full dmsh_build.py run whose compile already used every core:
FINAL FANTASY RESONANCE's 73,496 shaders kept one core busy for about 7 minutes (11.8 minutes for it and cb_extent.py together, on an M5 with 10 cores).
Each shader is checked on its own (its MSL and its reflection's c1 length), so the loop body becomes unproven(), mapped over the sorted file names with a
multiprocessing pool as cb_extent.py already does; Pool.map keeps the order, so c1_full.tsv lists the same keys in the same order as before.

Checked on Silver Pines' 56,386 Mac libraries (chain b-98ca73d18869): the same 13,372 keys, byte for byte below the header line; 299 s on one core,
44 s with the pool (10 cores). c1_guard.py is not a chain input (dmsh_chain.py), so no library or chain version changes. The Python tool tests pass.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018XsrnTnMEXD4jZMChLCZPK
@Unayung

Unayung commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor Author

The failing ruff and pyright (ratchet) check is not from this change: main fails it the same way since 78cb26d (its lint run).

The three new findings are all on dm/tools/tests/test_nsp_space.py:11, the importlib.util.spec_from_file_location(...) / .loader.exec_module(...) line from #277:

dm/tools/tests/test_nsp_space.py:11: pyright reportArgumentType: Argument of type "ModuleSpec | None" cannot be assigned to parameter "spec" ...
dm/tools/tests/test_nsp_space.py:11: pyright reportOptionalMemberAccess: "loader" is not a known attribute of "None"
dm/tools/tests/test_nsp_space.py:11: pyright reportOptionalMemberAccess: "exec_module" is not a known attribute of "None"
lint_python: 414 findings, 411 in the baseline, 3 new, 0 fixed but still listed

I ran dm/tools/lint_python.py --against origin/main locally with the pinned tools (.github/requirements/lint.txt). It gives these same three findings on main and on this branch, and none in c1_guard.py. #382 passes because it touches no Python.

The fix is either an assert on the spec and its loader before module_from_spec, or the file in lint_python_baseline.tsv with the other tests that load a script this way. I can add either here if you'd rather not do it on main separately.

🤖 Generated with Claude Code

The ratchet in .github/workflows/lint-python.yml fails on main since 78cb26d with three new pyright findings on this file's line 11
(spec_from_file_location's ModuleSpec | None passed to module_from_spec, and .loader.exec_module on it). An assert that the spec and its loader
exist states what holds for a script path that exists and clears all three; dm/tools/lint_python.py --against origin/main: 411 findings, all in the baseline.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018XsrnTnMEXD4jZMChLCZPK
@Unayung

Unayung commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

Added the lint fix as a separate commit (2/2): an assert spec and spec.loader in dm/tools/tests/test_nsp_space.py. Locally with the pinned tools: lint_python: 411 findings, 411 in the baseline, 0 new; test_nsp_space.py still passes. If main gets its own fix first, that commit can simply be dropped.

🤖 Generated with Claude Code

seedds added a commit that referenced this pull request Oct 27, 2026
c1_guard.py read and analysed each shader library's MSL in one loop on
one core, after a full dmsh_build.py run whose compile already used every
core: FINAL FANTASY RESONANCE's 73,496 shaders kept one core busy for
about 7 minutes. Each shader is checked on its own (its MSL and its
reflection's c1 length), so the loop body becomes unproven(), mapped over
the sorted file names with a multiprocessing pool as cb_extent.py already
does; Pool.map keeps the order, so c1_full.tsv lists the same keys in the
same order as before.

Same keys, byte for byte below the header line, on three games:
Silver Pines (56,386 libraries, 13,372 keys) 299 s -> 44 s on an M5;
Pokemon Legends Z-A (28,603, 1,011 keys) 206 s -> 17 s and Pokemon
Scarlet (30,142, 325 keys) 211 s -> 21 s here. c1_guard.py is not a
chain input (dmsh_chain.FILES), so no library or chain version changes.
The PR's second commit (test_nsp_space.py's spec assert) is already on
main as part of 4fc7f51. PR #366 carried the same c1_guard change.

Co-authored-by: Chen Chia Yang <[email protected]>
@seedds

seedds commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Thanks! The c1_guard.py change landed on main as 75065df, with you as co-author.

I checked it on two more games and got the same keys, byte for byte below the header line:

Library Shaders One core Pool Keys
Pokémon Legends Z-A 28,603 206 s 17 s 1,011
Pokémon Scarlet 30,142 211 s 21 s 325

c1_guard.py is not in dmsh_chain.FILES, so no chain version changes. The tool tests pass (347 tests), and the lint ratchet against main finds nothing new.

I left out the second commit (test_nsp_space.py): main already has the same assert from 4fc7f51, which turned the lint job green again, so the commit now conflicts. Closing this PR.

@seedds seedds closed this Oct 27, 2026
Know-It-All-Brother pushed a commit to Know-It-All-Brother/nxnative that referenced this pull request Nov 9, 2026
c1_guard.py read and analysed each shader library's MSL in one loop on
one core, after a full dmsh_build.py run whose compile already used every
core: FINAL FANTASY RESONANCE's 73,496 shaders kept one core busy for
about 7 minutes. Each shader is checked on its own (its MSL and its
reflection's c1 length), so the loop body becomes unproven(), mapped over
the sorted file names with a multiprocessing pool as cb_extent.py already
does; Pool.map keeps the order, so c1_full.tsv lists the same keys in the
same order as before.

Same keys, byte for byte below the header line, on three games:
Silver Pines (56,386 libraries, 13,372 keys) 299 s -> 44 s on an M5;
Pokemon Legends Z-A (28,603, 1,011 keys) 206 s -> 17 s and Pokemon
Scarlet (30,142, 325 keys) 211 s -> 21 s here. c1_guard.py is not a
chain input (dmsh_chain.FILES), so no library or chain version changes.
The PR's second commit (test_nsp_space.py's spec assert) is already on
main as part of 7b92332. PR seedds#366 carried the same c1_guard change.

Co-authored-by: Chen Chia Yang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants