Skip to content

Split nvn_mtl.m into topic parts with a script that re-cuts it on any main (mechanical, same machine code) - #479

Open
v0lumehi wants to merge 3 commits into
seedds:mainfrom
v0lumehi:split-nvn-mtl-2
Open

v0lumehi wants to merge 3 commits into
seedds:mainfrom
v0lumehi:split-nvn-mtl-2

Conversation

@v0lumehi

@v0lumehi v0lumehi commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Roadmap #313, the first step of "Extract, one module at a time": a mechanical split of dm/gfx/nvn/nvn_mtl.m, an updated #133.

Three commits, so the split can be checked by re-running it:

  1. dm/tools/split_nvn_mtl.py with its plan, split_nvn_mtl_plan.tsv (which top-level block goes into which part), a test, and a parts table in dm/gfx/README.md. It also adapts four tools that read nvn_mtl.m, so they work on the unsplit and the split file (see below).
    • The script cuts at whole top-level blocks only: nothing is renamed, reformatted or reordered.
    • --check proves that putting the parts back gives the original byte for byte, and --join puts them back.
    • It stops on anything it can't cut safely: unbalanced braces, an open comment or #if, a part that would come out in two pieces.
    • A block that is new since the plan goes with the block before it, and every run lists those blocks.
    • dm/tools/impact_map.tsv rules for nvn_mtl.m now match its parts. usage.py and the ABI literal test read the parts as part of nvn_mtl.m.
    • lint_c.py runs clang-tidy on the parts put back together, because the analyzer skips functions that are defined in included files. The baseline is unchanged, and the report is identical to the one for the unsplit file (274 findings).
  2. The split itself, generated by the script from main's nvn_mtl.m at 2aae9f8 (with Capture one NVN frame and replay it against new code (DM_NVN_CAPTURE, nvn_replay), step 1 of #402 #473's capture hooks, which stay in nvn_mtl.m, and dmsh_abi.h: one definition of the shader library ABI for the build and the host #391's dmsh_abi.h).
    • nvn_mtl.m keeps 633 lines: the includes, the residency set, deferred release, mtl_init and the memory pools.
    • It includes 14 parts in the original order, the same parts as Split nvn_mtl.m into topic parts (mechanical, identical machine code) #133: submit 456, ubuf 542, tex 1,193, pass 823, shader 1,014, state 964, pso 1,092, args 603, lookahead 796, merge 855, indirect 405, draw 1,467, shot 342, present 815 (line counts of the files, with their three header lines).
  3. .git-blame-ignore-revs with the split commit, so GitHub's blame shows the real authors. Locally: git config blame.ignoreRevsFile .git-blame-ignore-revs.

You can cut it yourself. If you'd rather land it at a moment of your choosing, or after more changes to nvn_mtl.m, take only the first commit and run python3 dm/tools/split_nvn_mtl.py on your main. Then update the SHA in .git-blame-ignore-revs. After a conflict in a part: --join, resolve in the whole file, and run the script again.

Re-cutting needs no plan change, with one exception: a function renamed in main is treated as a new block (it goes with its neighbour) until --write-plan records it. The cut on 2aae9f8 listed 115 new blocks and the neighbour rule placed all of them, so the plan is the one from the first version.

Checked

  • Round trip: --check gives today's nvn_mtl.m byte for byte. Fed Split nvn_mtl.m into topic parts (mechanical, identical machine code) #133's old file, the script reproduces Split nvn_mtl.m into topic parts (mechanical, identical machine code) #133's 14 parts and its nvn_mtl.m exactly.
  • Preprocessor: clang -E -P of nvn_mtl.m before and after, with dm/host/Makefile's flags, and for iOS with the iPhoneOS 27.1 SDK. The only differences are the line-number arguments of res_add_at (43 changed lines, as __LINE__ now counts from each part's top). They stay distinct: no two sites in different files share a line number, and the script adds a padding comment line where they would, as in Split nvn_mtl.m into topic parts (mechanical, identical machine code) #133. This cut needed none.
  • Machine code: otool -tvV of the nvn_mtl object for Mac and iOS has the same instruction count (125,397 on the Mac, 123,597 on iOS) and addresses. The only differences are 54 mov w1, #imm, the immediates of those res_add_at calls. The compiler warnings on the Mac are the same as before (52).
  • Tests:
    • make -C dm/host/tests GAME=010015100B514000 run and make -C dm/gfx/tests GAME=010015100B514000 run (the GPU tests);
    • CI's CPU-only gfx tests, the host and texture sanitizer jobs, and every gfx test compiled;
    • t_perf_order --cpu (165 checks), perf-order-tsan, check_t_thermal, thermal-tsan and gl-imports;
    • dmhost for the three CI profiles and dm/ios/tools/compile_check.py (184 files);
    • the tool tests, except test_romfs_read, which needs a .NET SDK here and fails the same way on main;
    • the C lint ratchet (274 findings, the report identical to main's, whose one finding above the baseline is in dm/host/tests/t_pollwait.c) and the Python lint ratchet (0 new findings, as on main).
  • Games run: none, as the object code is unchanged apart from the res_add_at line numbers.
  • Devices and OS: M2 Max, macOS 27.2.
  • Regress: none, for the same reason.

Docs

  • Not applicable: no finding. The parts and the script are described in dm/gfx/README.md
  • Not applicable: no chain tool changed
  • No game files, keys or data extracted from a game in the change or the attachments

…cally, with --join and --check

The script reads nvn_mtl.m as top-level blocks (functions, data, types, preprocessor groups, with the comments above them), assigns each to a part by name (split_nvn_mtl_plan.tsv, a new block goes with the one before it), writes the 14 nvn_mtl_<part>.inc files and the include lines, and refuses what it cannot read safely. --check proves the round trip byte for byte, --join puts the parts back for merging. A part whose res_add sites would share a __LINE__ with another file's gets a padding comment line, so the residency statistics keep their sites apart. dm/gfx/README.md lists the parts.

Three tools read nvn_mtl.m and must follow it into the parts. impact_map.tsv matches nvn_mtl.m and its parts. usage.py treats the parts as one file for the call graph, because the statics are shared. The ABI literal test reads the parts with nvn_mtl.m. lint_c.py analyzes the parts put back together, because clang's static analyzer skips functions defined in included files. It keeps the baseline of the unsplit file and moves findings that tests report in a part to their line in the put-back text. All of them work on the unsplit and the split file.

This commit adds the tool and what it needs. The cut is the next commit.
… change)

Generated by dm/tools/split_nvn_mtl.py on upstream/main 2aae9f8. nvn_mtl.m (11,944 lines) keeps its includes, the residency set, deferred release, mtl_init, the memory pools and the capture hooks (633 lines). The rest moved, in whole top-level blocks and in the original order, into 14 files nvn_mtl_<part>.inc that nvn_mtl.m #includes where each part stood. Re-running the script on the unsplit file gives this commit again.

Checked against the same tree unsplit. The parts put back give nvn_mtl.m byte for byte. clang -E -P with the host's flags (Mac) and with arm64-apple-ios26.0 / apple-a18 differ only in the line argument of res_add_at (43 changed lines on the Mac and on iOS, the sites stay distinct). otool -tvV of the object has the same instruction count (125,397 on the Mac, 123,597 on iOS) at the same addresses and differs only in the matching mov w1 immediates (54 on the Mac and on iOS). Same compiler warnings on the Mac (52 on each).
@v0lumehi

Copy link
Copy Markdown
Contributor Author

@seedds a note on how to land this. nvn_mtl.m changes about 16 times a day on main. The cut I pushed this night (1be184a6) conflicted again after a few hours, so any cut I push will be stale before you get to it.

I suggest you take only the first commit, 045da738 (the script, its plan and the tool changes). Then you run the cut yourself on main whenever it suits you:

python3 dm/tools/split_nvn_mtl.py
python3 dm/tools/split_nvn_mtl.py --check

--check confirms the parts join back to the original byte for byte. The script lists every block that is new since the plan, and those go with the block before them. The machine code is the same on Mac and iOS. I checked that on the last cut.

If you prefer, I can drop the second and third commit from this PR so it only holds the script. Tell me which way you want it.

@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) tools Tools and scripts (dm/tools, scripts) docs Documentation labels Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant