Skip to content

An IPA built with a Mac host carries the prewarm lists' pipelines made offline (R rows' bases and M rows' mesh pipelines), and Ocarina of Time's seed list gets an iPhone 17's own rows - #2674

Closed
stan69b wants to merge 3 commits into
seedds:mainfrom
stan69b:pr/oot30-a-base-archive

Conversation

@stan69b

@stan69b stan69b commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Batch OoT-30, session A (pipelines compiled on the phone).

Why

The archive pso_offline.py bundles in an IPA holds the shader pairs' pulling bases and the compute pipelines. It holds no base with a vertex layout (a prewarm list's R row) and no native task / mesh pipeline (an M row). Both depend on what only the host knows: the descriptor pso_desc makes from a row's pso_skey and the shader's reflection. So an iPhone builds every listed one itself, at the prepare step of its first start, and keeps them in its own archive (pso_archive/).

An OS update makes Metal refuse that archive. On an iPhone 17 after iOS 27.2.1, every start logged 121 lines of [RED] pipeline archive file … not usable (deployment target for architecture applegpu_g18p … is not compatible with current OS). The bundled archive, made for the OS's major version, still opened.

In the owner's 30-minute play (pipelines up to frame lines), windows that compiled bases ran at 12-22 fps, for example bases compiled 17 / 12872 ms in one window and 4 / 45122 ms in Castle Town.

PR #2353 made these files for the Mac app (the host's prebuild tool in its bundle form, then metal-tt for other GPUs). The IPA did not get them.

The change

A-1: pso_offline.py --prewarm-host <Mac dmhost>

  • Host stage. Runs the Mac host as the prebuild tool's bundle form (DM_PSO_PREBUILD=tool, DM_PSO_PREBUILD_OUT) on the iOS libraries the app ships. A Mac's Metal loads and compiles a library built for iOS; I checked this with Ocarina of Time's iOS libraries on macOS 27.2. The tool gets the profile's tables less the pair lists (the pairs are the archive's, so the tool would otherwise build them all) and a list of the R and M rows alone (the C rows' compute pipelines are in the archive already). It writes one file per flush period, its index (.keys) and its pipelines script.
  • Compile stage. Compiles each script with metal-tt for every --gpu as <arch>-prewarm-<NN>.mtl4a, beside the period's index. A device computes the same keys. Groups of --batch-size are merged with metal-pack. A pipeline metal-tt refuses is left out by the halving compile_isolating already did, now shared as isolate().
  • Opening. The host opens the files with the archive, through the existing pso_bundle_open_prewarm. No host change.
  • Caching and failure. Both stages are cached in the work folder: the host run by its inputs, each compiled script by its bytes and the compiler's settings. A failure prints a red line and the app goes without these files, as before.
  • build_ipa.sh. Passes DM_PSO_HOST, else dm/host/dmhost when its last build was for this game (made current first). DM_PSO_HOST=none makes none.
  • NSP script. The library search by SHA-256 it had for the Mac app's other GPUs moved into pso_offline.py (script_libraries), and the NSP script calls it.
  • Seed list. Ocarina of Time's pso_manifest.tsv gets 10,415 rows (5,249 S, 4,447 R, 719 C) from the iPhone's own pso_user.tsv.

A-3: the M rows

metal-tt takes the mesh pipeline descriptors the tool writes, so the M rows go in too. This is #2642's open item.

Checked

  • A pilot of 60 R and 20 M rows from Ocarina of Time's list, through both stages for applegpu_g18p (iOS 27.0): 42 bases and 20 mesh pipelines, one file of 6 MB in 8 s, with a complete index of 62 keys.
  • Tests:
    • test_pso_offline.py: 60 tests. New: Prewarm with a fake host and metal-tt (the tool's environment and tables, the files and indexes per GPU, reuse with nothing run, a GPU dropped, new rows rerunning the host alone, groups merged with a pipeline left out, a failed or missing host leaving the archive). Also a real metal-tt compile of a mesh pipeline descriptor for applegpu_g18p, and build_ipa.sh passing DM_PSO_HOST.
    • Also run: test_nsp_mac_app.py (70), test_docs, test_env_switches, test_lint_python. lint_python --against upstream/main finds nothing new in these files.
  • regress.py: not run. The reach is the IPA build tool alone: no host, chain or runtime code changes, and the regress interface check uses pso_offline.plan, which is unchanged.

Phone (iPhone 17, iOS 27.2.1)

The phone build is oot-build-final (the owner's phone tree, 547c5db-based) plus #2353's host part, #2539, 054eea1, cc8f449 and 248bb95, with prewarm files made by this branch's pso_offline.py.

Making the files on a 16 GB M4 MacBook Air (macOS 27.2), from the seed list's 7,138 R and 164 M rows:

  • host stage: 576 s with 4 workers (6,627 bases, 164 mesh pipelines, 0 failed);
  • metal-tt for applegpu_g18p: 54 files, 1,009 MB, about 9.5 minutes, 0 left out.
Start (thermal state 2) Prepare before the first frame Bases compiled / from the archive Mesh pipelines from the archive / built
Device archive off, seed list alone (as on a fresh install or after an OS update) 4.2 s 0 / 6,627 164 / 0
Same, with index keys by file name (a build without 248bb95) 61 s 1,355 / 5,272 102 / 62
Normal start: the device's archive, the phone's own list of 17,577 lines, the 121 refused files removed 9.5 s 77 / 7,471 285 / 8
  • The files open in 1.9 s.
  • 1,043 compute pipelines come from the bundle.
  • No base was compiled in play in these runs (camera turns in one place).
  • The 77 bases of the normal start belong to rows the phone recorded after the copy of its list that the seed list took (945 R and 129 M rows of its current list are not in the seed list).
  • The row with keys by file name is why the index must key libraries by their bytes, as main does since 248bb95. With keys by file name, the Mac's tool and the phone named different twins of byte-identical libraries: 563 of the 6,689 shader keys the R rows name have such a twin.
  • Remove pipeline archive files Metal refuses, so the prebuild runs again #2539: the first start removed the 121 refused files, and none of the eight starts after it (sessions B and D) logged one.
  • Not measured on the phone: a start without the prewarm files.

@github-actions github-actions Bot added packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers ios The iPhone and iPad app (dm/ios) tools Tools and scripts (dm/tools, scripts) docs Documentation game: 0400486024DEC000 A game labels Nov 14, 2026
@stan69b

stan69b commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Batch OoT-30: the combined phone build (iPhone 17, iOS 27.2.1)

Build (DmIOS-all1.app, local branch oot-30-hand 75e9de61, never pushed): the owner's phone tree oot-build-final (be8db251), plus every session's phone version:

Runs: the owner's newest save, the drive script turning the camera only (it cannot walk), 30 fps cap, dynamic resolution, 6 minutes each, back to back. Windows are the 300-frame lines after frame 600.

Run Thermal Prepare before the first frame Frame p50, median of windows (worst) Windows slower than 29 fps GPU p50 Submit+present p50 Submit CPU / instr. per draw Bases compiled in play Residency set peak
Baseline: the installed fix2 (be8db251) 0 → 2 4.5 s (0 compiled) 33.5 ms (44.6) 14 of 61 29.6 ms 4.72 ms 4.71 ms / 33.9 k 0 3,328 MB
Combined, first start 2 61.2 s (687 compiled, once) 33.4 ms (46.6) 9 of 60 30.4 ms 4.76 ms 4.77 ms / 33.9 k 0 3,345 MB
Combined, second start 2 6.0 s (0 compiled) 33.4 ms (49.0) 15 of 58 29.8 ms 4.97 ms 5.00 ms / 32.5 k 0 3,330 MB

Reading it

  • The first start's 61 s is 248bb95's one-time cost. That start removed the phone's 31 archive files indexed by library file names. It compiled again the rows the phone had recorded since the seed list took its copy of the phone's list, which are not in the prewarm files (945 R and 129 M).
  • With the device's archive off and the seed list alone (A's runs on the same build, An IPA built with a Mac host carries the prewarm lists' pipelines made offline (R rows' bases and M rows' mesh pipelines), and Ocarina of Time's seed list gets an iPhone 17's own rows #2674), the prepare step takes 4.2 s and compiles 0 of 6,627 bases and 0 of 164 mesh pipelines.
  • In this scene (indoors, 900-1,100 draws) both builds hold 30 fps most of the time. The slow windows come and go with the thermal state and the GPU clock, as B describes. The two builds do not differ here beyond run-to-run spread.
  • Not reached: Hyrule Field, Kakariko Village and Castle Town (the driver cannot walk). That is where C-3, the prewarm files and D-1 should show.

Owner, a route to validate (the combined build is installed on the phone now):

  • walk from Kakariko Village into Hyrule Field, then to Castle Town and into the castle grounds, by day, then the same at night, about 10 minutes each;
  • then send Documents/log.txt. Its pipelines up to frame lines show bases compiled in play, translation up to frame the submit thread, and the [mem] census lines what held memory at the castle-grounds load (D).

…-row bases made offline, and Ocarina of Time's seed list gets an iPhone 17's own rows

The pipeline archive pso_offline.py makes for an IPA holds the shader pairs' pulling bases and the compute pipelines. A prewarm list's R row is a base with a vertex layout, which depends on the descriptor pso_desc makes from the row's pso_skey and the shader's reflection, so only the host can make it. An iPhone therefore built every listed one itself, at the prepare step of its first start, kept them in its own archive (pso_archive/), and lost that archive after an OS update: an iPhone 17 on iOS 27.2.1 logged 121 "pipeline archive file ... not usable (deployment target ... not compatible with current OS)" lines at every start. The bundled archive, made for the OS's major version, still opened. The Mac app already had the remedy (PR seedds#2353: the host's prebuild tool in its bundle form, then metal-tt for other GPUs); the IPA did not.

pso_offline.py --prewarm-host <Mac dmhost> now makes them for the IPA:
- the host stage runs the given Mac host as the prebuild tool's bundle form (DM_PSO_PREBUILD=tool, DM_PSO_PREBUILD_OUT) on the libraries the app ships (a Mac's Metal loads and compiles a library built for iOS: checked with Ocarina of Time's iOS libraries on macOS 27.2), with the profile's tables less the pair lists (the pairs are the archive's) and a list of the R rows alone, so it writes one archive file per flush period, its index (.keys) and its pipelines script;
- the compile stage builds each script with metal-tt for every --gpu as <arch>-prewarm-<NN>.mtl4a beside the period's index (the keys name libraries by their bytes or by their name in the library folder, which a device computes the same way), in groups of --batch-size merged with metal-pack, leaving out a pipeline metal-tt refuses (the isolation compile_isolating already did, now shared as isolate());
- both are cached (the host run by its inputs, each compiled script by its bytes, the architecture, platform, minimum OS and metal-tt version), and a failure is a red line, never a failed build: the app then has no prewarm files and the device builds the rows as before. Without --prewarm-host there are none (an earlier build's files are removed).
The library search by SHA-256 the NSP script had for the Mac app's other GPUs moved into pso_offline.py (script_libraries); the NSP script calls it.

build_ipa.sh passes DM_PSO_HOST, or dm/host/dmhost when its last build was for this game (made current first); DM_PSO_HOST=none makes none. docs/env_switches.tsv: the new row.

Ocarina of Time's pso_manifest.tsv: 10,415 rows (5,249 S, 4,447 R, 719 C) from the iPhone 17's own pso_user.tsv (Documents, 2026-11-14 06:00), the rows the seed list did not have after both lists' keys were parsed with pso_skey_parse and written again in the current layout (the phone's list held two layouts); every shader key is in shader_map.tsv with its stage. They go before the list's last block, away from PR seedds#2642's two changes to the same file (git merge-file of the two: no conflict). The pair archive's plan is unchanged by them (33,365 bases, 4,125 compute pipelines: the shader map already names every compute shader).

Tests: test_pso_offline.py 59 (new: Prewarm, five cases with a fake host and metal-tt: the tool's environment and tables, the files and indexes per GPU, reuse with nothing run, a GPU dropped, new rows rerunning the host alone, groups merged with a pipeline left out, a failed or missing host leaving the archive; IpaFallback: DM_PSO_HOST reaches pso_offline.py, none and no host make none), test_nsp_mac_app.py 70, test_docs, test_env_switches, test_lint_python (lint_python --against upstream/main: no new finding of these files).
(cherry picked from commit f03c0cb)
… / mesh pipelines, which metal-tt compiles for an iPhone's GPU

PR seedds#2642's open item: pso_offline.py's pair archive leaves Ocarina of Time's 1,284 mesh pairs to the device (a mesh pipeline has no base), so every M row of the prewarm lists was compiled once on the phone, at the prepare step of a first start or, for one no list named yet, at its first draw (1.7-3.8 s each on an iPhone 17).

metal-tt takes a mesh pipeline descriptor as it takes a render one: the host's prebuild tool already writes the M rows' mesh_render_pipeline_descriptors into its pipelines scripts (with their object, mesh and fragment functions, threadgroup sizes, payload length and colour formats), and metal-tt compiled them for applegpu_g18p (iOS 27.0) beside the R rows' bases in the same file, with the same index. So --prewarm-host now hands the tool the M rows too (PREWARM_ROWS). Measured on a pilot of Ocarina of Time's list on the iPhone libraries: 42 bases and 19 mesh pipeline descriptors (20 M rows), one file of 6 MB in 8 s; the tool's own log counted the 20 mesh pipelines as captured.

Tests: test_pso_offline.py 60: the rows and the tool's list hold the M rows; new test_metal_tt_takes_a_mesh_pipeline compiles a real mesh pipeline descriptor (a mesh and a fragment function built with the iOS SDK) with metal-tt for applegpu_g18p through compile_script (skipped without Xcode or metal-tt).
(cherry picked from commit 5d90414)
…mbers

docs/development.md, The pipeline archive made offline: a paragraph on the prewarm lists' pipelines in an IPA (DM_PSO_HOST, pso_offline.py --prewarm-host: the Mac host's prebuild tool on the iOS libraries, then metal-tt per GPU; the caches, DM_PSO_PREBUILD_THREADS, why it matters after an OS update).

docs/reference/rendering.md: new entry "An iPhone builds its prewarm lists' pipelines again after an iOS update" (Symptom, Cause, Fix, Check, Measurement, Found on). It records that the index must key libraries by their bytes: a phone build that still keyed them by file name missed 20% of the bundled bases, those whose shaders have a byte-identical twin library (563 of the 6,689 shader keys the R rows name, in 2,482 of 7,138 rows). Symptom index regenerated (dm/tools/docs_symptoms.py).

Ocarina of Time's note, Found on this game: the line for it.

Measured on the iPhone 17 (iOS 27.2.1, thermal state 2), a phone build of oot-build-final with seedds#2353's host part, seedds#2539, 054eea1, cc8f449 and 248bb95, and the prewarm files made by this branch's pso_offline.py: a start with the device's archive off and the seed list alone prepared in 4.2 s, bases 0 compiled and 6,627 from the archive, 164 of 164 mesh pipelines and 1,043 compute pipelines from the bundle (with keys by file name: 61 s, 1,355 compiled, 62 mesh built). The 121 refused files were removed at the first start, and no start of the eight that followed (sessions B and D) logged one.

Tests: test_docs OK.
(cherry picked from commit eb69e1e)
@stan69b
stan69b force-pushed the pr/oot30-a-base-archive branch from eb69e1e to 978ef95 Compare November 16, 2026 15:24
seedds added a commit that referenced this pull request Nov 17, 2026
The offline IPA archive already holds pulling bases and compute pipelines,
but omitted the prewarm list's vertex-layout bases and native mesh pipelines.
Use the existing Mac host descriptor exporter, then metal-tt for each target
GPU, to supply those pipelines through the existing bundle format.

Keep prewarm validation independent of the pair archive's fast path. Track
library images, directory libraries and reflection in the host cache; resolve
child paths before changing directory and scope launch failures to prewarm.
Do not publish complete indexes for periods where metal-tt omitted pipelines.
Discover the current per-game host target, and share publication that preserves
unchanged file identities with the Mac packer. Retain the original phone seed
rows and record the current validation and coverage limits in the game note.

Validation: 190 tool, Mac packaging, docs and chain tests passed; Python lint
has no new findings; shell syntax, diff and current manifest parser checks pass.
A real M4 Pro export supplied 6,646 bases and 164 mesh pipelines with no failures;
its 21 files total 788 MiB and took 381.6 s to generate. Cached export took
0.3 s without changing published identities. A 280.5 s walking test crossed
from Hyrule Field into Castle Town and exited cleanly with no red lines or GPU
errors. Prepare compiled no bases and used all 164 mesh pipelines from archives;
play subsequently compiled 41 further bases outside the recorded seed coverage.

The focused NieR regression has identical shaders/interfaces, successful boots
and no new red lines. Its raw screenshot flag remains: the same host and shader
artifacts reached different flight animation/tutorial phases. The environment
switch metadata check fails identically on the pinned baseline. Matched phone
starts with and without these files, rebased phone traversal and broader seed
coverage remain open; no comparative startup or fps benefit is claimed.

Co-authored-by: Stan Bono <[email protected]>
@seedds

seedds commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

Taken into main as 422ec2409500da4637294b8b0ab0811755f214d4, rebased onto fresh main. Stan Bono is credited with Co-authored-by: Stan Bono <[email protected]> because the commit retains the PR's exporter design, implementation and seed data.

The landing fixes library-cache invalidation, misleading indexes after partial compilation, relative child paths and launch failures, the pair fast path bypassing prewarm validation, current per-game host discovery, and unnecessary repacking of unchanged prewarm files. Mac and IPA publication share one implementation. Game-specific measurements live in the game note.

Validation: 190 tool/Mac packaging/docs/chain tests passed, including the real iOS mesh descriptor compile; Python lint has no new findings; shell syntax, diff and all 7,138 render/164 mesh manifest-key checks passed. The environment-switch metadata failure was reproduced identically on the pinned baseline; after rebasing onto newer main, that final check also passes (the intervening main changes resolved it).

On a 48 GB M4 Pro (macOS 27.2.1, chain 13b3bdb97e7e), the real R/M export supplied 6,646 bases and 164 mesh pipelines, zero failures or omissions: 21 files, 788 MiB, 381.6 s. Cached export took 0.3 s with all 43 published file identities unchanged. Prepare with an empty device archive took 28.33 s, zero bases compiled, 6,646 archive hits, and all 164 mesh pipelines from the archive. The 280.5 s walking test crossed from Hyrule Field into Castle Town and along its streets, exited 0, and logged no red line, crash or GPU error. It also compiled 41 further bases outside the seed coverage (12,168.5 ms summed compile wall time).

Focused regression against c3efaa9, selected for its smaller shader set and available route:

game raw status shaders new interface failures boot fps frame p99 ms prepare s new RED
NieR Automata REGRESSION (screenshot) 1225/1225 → 1225/1225; identical 0 both successful 59.9 → 59.9 17.37 → 17.23 0.29 → 0.12 0

The 180 s screenshot flag is retained: dHash 14 bits, RGB 9.74. Inspection shows different flight/explosion/tutorial phases; both runs used the same host artifact, rewritten modules, save and shader artifact. This is a qualified visual result, not a clean screenshot-comparator pass. The Mac runtime measurements were made on the pinned test revision; landing onto newer main did not change the patch.

  • Confidence: high in the implementation and archive coverage; comparative phone performance and broader gameplay coverage remain unmeasured.
  • Binding: general tooling at host binding, already the loosest correct binding; the manifest is game data. No build-address special case.
  • Workaround: none in the fixes. Partial archives now accurately avoid publishing an index claiming absent pipelines.
  • From-scratch design: yes: reuse the existing host descriptors and bundle format with independent caches and shared publication.
  • Performance: Ocarina's Mac R/M generation cost was 381.6 s / 788 MiB and cached reuse 0.3 s; preparation with files took 28.33 s. There is no matched no-file start, so no startup or fps improvement is claimed.
  • Regression scope: moderate; one lighter shared-path game (NieR), plus Ocarina specifically for its changed native-mesh manifest and requested actual traversal.

Remaining: matched phone cold starts with/without the files; rebased phone traversal; the 41-base seed-coverage gap; Kakariko, castle grounds and a day/night route. The raw NieR screenshot flag remains documented. Closing this PR because its code was landed as the reworked commit above.

@seedds seedds closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 0400486024DEC000 A game ios The iPhone and iPad app (dm/ios) packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants