Repository navigation
An IPA built with a Mac host carries the prewarm lists' pipelines made offline (R rows' bases and M rows' mesh pipelines), and Ocarina of Time's seed list gets an iPhone 17's own rows - #2674
Conversation
Batch OoT-30: the combined phone build (iPhone 17, iOS 27.2.1)Build (
Runs: the owner's newest save, the drive script turning the camera only (it cannot walk), 30 fps cap, dynamic resolution, 6 minutes each, back to back. Windows are the 300-frame lines after frame 600.
Reading it
Owner, a route to validate (the combined build is installed on the phone now):
|
…-row bases made offline, and Ocarina of Time's seed list gets an iPhone 17's own rows The pipeline archive pso_offline.py makes for an IPA holds the shader pairs' pulling bases and the compute pipelines. A prewarm list's R row is a base with a vertex layout, which depends on the descriptor pso_desc makes from the row's pso_skey and the shader's reflection, so only the host can make it. An iPhone therefore built every listed one itself, at the prepare step of its first start, kept them in its own archive (pso_archive/), and lost that archive after an OS update: an iPhone 17 on iOS 27.2.1 logged 121 "pipeline archive file ... not usable (deployment target ... not compatible with current OS)" lines at every start. The bundled archive, made for the OS's major version, still opened. The Mac app already had the remedy (PR seedds#2353: the host's prebuild tool in its bundle form, then metal-tt for other GPUs); the IPA did not. pso_offline.py --prewarm-host <Mac dmhost> now makes them for the IPA: - the host stage runs the given Mac host as the prebuild tool's bundle form (DM_PSO_PREBUILD=tool, DM_PSO_PREBUILD_OUT) on the libraries the app ships (a Mac's Metal loads and compiles a library built for iOS: checked with Ocarina of Time's iOS libraries on macOS 27.2), with the profile's tables less the pair lists (the pairs are the archive's) and a list of the R rows alone, so it writes one archive file per flush period, its index (.keys) and its pipelines script; - the compile stage builds each script with metal-tt for every --gpu as <arch>-prewarm-<NN>.mtl4a beside the period's index (the keys name libraries by their bytes or by their name in the library folder, which a device computes the same way), in groups of --batch-size merged with metal-pack, leaving out a pipeline metal-tt refuses (the isolation compile_isolating already did, now shared as isolate()); - both are cached (the host run by its inputs, each compiled script by its bytes, the architecture, platform, minimum OS and metal-tt version), and a failure is a red line, never a failed build: the app then has no prewarm files and the device builds the rows as before. Without --prewarm-host there are none (an earlier build's files are removed). The library search by SHA-256 the NSP script had for the Mac app's other GPUs moved into pso_offline.py (script_libraries); the NSP script calls it. build_ipa.sh passes DM_PSO_HOST, or dm/host/dmhost when its last build was for this game (made current first); DM_PSO_HOST=none makes none. docs/env_switches.tsv: the new row. Ocarina of Time's pso_manifest.tsv: 10,415 rows (5,249 S, 4,447 R, 719 C) from the iPhone 17's own pso_user.tsv (Documents, 2026-11-14 06:00), the rows the seed list did not have after both lists' keys were parsed with pso_skey_parse and written again in the current layout (the phone's list held two layouts); every shader key is in shader_map.tsv with its stage. They go before the list's last block, away from PR seedds#2642's two changes to the same file (git merge-file of the two: no conflict). The pair archive's plan is unchanged by them (33,365 bases, 4,125 compute pipelines: the shader map already names every compute shader). Tests: test_pso_offline.py 59 (new: Prewarm, five cases with a fake host and metal-tt: the tool's environment and tables, the files and indexes per GPU, reuse with nothing run, a GPU dropped, new rows rerunning the host alone, groups merged with a pipeline left out, a failed or missing host leaving the archive; IpaFallback: DM_PSO_HOST reaches pso_offline.py, none and no host make none), test_nsp_mac_app.py 70, test_docs, test_env_switches, test_lint_python (lint_python --against upstream/main: no new finding of these files). (cherry picked from commit f03c0cb)
… / mesh pipelines, which metal-tt compiles for an iPhone's GPU PR seedds#2642's open item: pso_offline.py's pair archive leaves Ocarina of Time's 1,284 mesh pairs to the device (a mesh pipeline has no base), so every M row of the prewarm lists was compiled once on the phone, at the prepare step of a first start or, for one no list named yet, at its first draw (1.7-3.8 s each on an iPhone 17). metal-tt takes a mesh pipeline descriptor as it takes a render one: the host's prebuild tool already writes the M rows' mesh_render_pipeline_descriptors into its pipelines scripts (with their object, mesh and fragment functions, threadgroup sizes, payload length and colour formats), and metal-tt compiled them for applegpu_g18p (iOS 27.0) beside the R rows' bases in the same file, with the same index. So --prewarm-host now hands the tool the M rows too (PREWARM_ROWS). Measured on a pilot of Ocarina of Time's list on the iPhone libraries: 42 bases and 19 mesh pipeline descriptors (20 M rows), one file of 6 MB in 8 s; the tool's own log counted the 20 mesh pipelines as captured. Tests: test_pso_offline.py 60: the rows and the tool's list hold the M rows; new test_metal_tt_takes_a_mesh_pipeline compiles a real mesh pipeline descriptor (a mesh and a fragment function built with the iOS SDK) with metal-tt for applegpu_g18p through compile_script (skipped without Xcode or metal-tt). (cherry picked from commit 5d90414)
…mbers docs/development.md, The pipeline archive made offline: a paragraph on the prewarm lists' pipelines in an IPA (DM_PSO_HOST, pso_offline.py --prewarm-host: the Mac host's prebuild tool on the iOS libraries, then metal-tt per GPU; the caches, DM_PSO_PREBUILD_THREADS, why it matters after an OS update). docs/reference/rendering.md: new entry "An iPhone builds its prewarm lists' pipelines again after an iOS update" (Symptom, Cause, Fix, Check, Measurement, Found on). It records that the index must key libraries by their bytes: a phone build that still keyed them by file name missed 20% of the bundled bases, those whose shaders have a byte-identical twin library (563 of the 6,689 shader keys the R rows name, in 2,482 of 7,138 rows). Symptom index regenerated (dm/tools/docs_symptoms.py). Ocarina of Time's note, Found on this game: the line for it. Measured on the iPhone 17 (iOS 27.2.1, thermal state 2), a phone build of oot-build-final with seedds#2353's host part, seedds#2539, 054eea1, cc8f449 and 248bb95, and the prewarm files made by this branch's pso_offline.py: a start with the device's archive off and the seed list alone prepared in 4.2 s, bases 0 compiled and 6,627 from the archive, 164 of 164 mesh pipelines and 1,043 compute pipelines from the bundle (with keys by file name: 61 s, 1,355 compiled, 62 mesh built). The 121 refused files were removed at the first start, and no start of the eight that followed (sessions B and D) logged one. Tests: test_docs OK. (cherry picked from commit eb69e1e)
eb69e1e to
978ef95
Compare
The offline IPA archive already holds pulling bases and compute pipelines, but omitted the prewarm list's vertex-layout bases and native mesh pipelines. Use the existing Mac host descriptor exporter, then metal-tt for each target GPU, to supply those pipelines through the existing bundle format. Keep prewarm validation independent of the pair archive's fast path. Track library images, directory libraries and reflection in the host cache; resolve child paths before changing directory and scope launch failures to prewarm. Do not publish complete indexes for periods where metal-tt omitted pipelines. Discover the current per-game host target, and share publication that preserves unchanged file identities with the Mac packer. Retain the original phone seed rows and record the current validation and coverage limits in the game note. Validation: 190 tool, Mac packaging, docs and chain tests passed; Python lint has no new findings; shell syntax, diff and current manifest parser checks pass. A real M4 Pro export supplied 6,646 bases and 164 mesh pipelines with no failures; its 21 files total 788 MiB and took 381.6 s to generate. Cached export took 0.3 s without changing published identities. A 280.5 s walking test crossed from Hyrule Field into Castle Town and exited cleanly with no red lines or GPU errors. Prepare compiled no bases and used all 164 mesh pipelines from archives; play subsequently compiled 41 further bases outside the recorded seed coverage. The focused NieR regression has identical shaders/interfaces, successful boots and no new red lines. Its raw screenshot flag remains: the same host and shader artifacts reached different flight animation/tutorial phases. The environment switch metadata check fails identically on the pinned baseline. Matched phone starts with and without these files, rebased phone traversal and broader seed coverage remain open; no comparative startup or fps benefit is claimed. Co-authored-by: Stan Bono <[email protected]>
|
Taken into The landing fixes library-cache invalidation, misleading indexes after partial compilation, relative child paths and launch failures, the pair fast path bypassing prewarm validation, current per-game host discovery, and unnecessary repacking of unchanged prewarm files. Mac and IPA publication share one implementation. Game-specific measurements live in the game note. Validation: 190 tool/Mac packaging/docs/chain tests passed, including the real iOS mesh descriptor compile; Python lint has no new findings; shell syntax, diff and all 7,138 render/164 mesh manifest-key checks passed. The environment-switch metadata failure was reproduced identically on the pinned baseline; after rebasing onto newer main, that final check also passes (the intervening main changes resolved it). On a 48 GB M4 Pro (macOS 27.2.1, chain 13b3bdb97e7e), the real R/M export supplied 6,646 bases and 164 mesh pipelines, zero failures or omissions: 21 files, 788 MiB, 381.6 s. Cached export took 0.3 s with all 43 published file identities unchanged. Prepare with an empty device archive took 28.33 s, zero bases compiled, 6,646 archive hits, and all 164 mesh pipelines from the archive. The 280.5 s walking test crossed from Hyrule Field into Castle Town and along its streets, exited 0, and logged no red line, crash or GPU error. It also compiled 41 further bases outside the seed coverage (12,168.5 ms summed compile wall time). Focused regression against c3efaa9, selected for its smaller shader set and available route:
The 180 s screenshot flag is retained: dHash 14 bits, RGB 9.74. Inspection shows different flight/explosion/tutorial phases; both runs used the same host artifact, rewritten modules, save and shader artifact. This is a qualified visual result, not a clean screenshot-comparator pass. The Mac runtime measurements were made on the pinned test revision; landing onto newer main did not change the patch.
Remaining: matched phone cold starts with/without the files; rebased phone traversal; the 41-base seed-coverage gap; Kakariko, castle grounds and a day/night route. The raw NieR screenshot flag remains documented. Closing this PR because its code was landed as the reworked commit above. |
Batch OoT-30, session A (pipelines compiled on the phone).
Why
The archive
pso_offline.pybundles in an IPA holds the shader pairs' pulling bases and the compute pipelines. It holds no base with a vertex layout (a prewarm list'sRrow) and no native task / mesh pipeline (anMrow). Both depend on what only the host knows: the descriptorpso_descmakes from a row'spso_skeyand the shader's reflection. So an iPhone builds every listed one itself, at the prepare step of its first start, and keeps them in its own archive (pso_archive/).An OS update makes Metal refuse that archive. On an iPhone 17 after iOS 27.2.1, every start logged 121 lines of
[RED] pipeline archive file … not usable (deployment target for architecture applegpu_g18p … is not compatible with current OS). The bundled archive, made for the OS's major version, still opened.In the owner's 30-minute play (
pipelines up to framelines), windows that compiled bases ran at 12-22 fps, for examplebases compiled 17 / 12872 msin one window and4 / 45122 msin Castle Town.PR #2353 made these files for the Mac app (the host's prebuild tool in its bundle form, then metal-tt for other GPUs). The IPA did not get them.
The change
A-1:
pso_offline.py --prewarm-host <Mac dmhost>DM_PSO_PREBUILD=tool,DM_PSO_PREBUILD_OUT) on the iOS libraries the app ships. A Mac's Metal loads and compiles a library built for iOS; I checked this with Ocarina of Time's iOS libraries on macOS 27.2. The tool gets the profile's tables less the pair lists (the pairs are the archive's, so the tool would otherwise build them all) and a list of theRandMrows alone (theCrows' compute pipelines are in the archive already). It writes one file per flush period, its index (.keys) and its pipelines script.--gpuas<arch>-prewarm-<NN>.mtl4a, beside the period's index. A device computes the same keys. Groups of--batch-sizeare merged with metal-pack. A pipeline metal-tt refuses is left out by the halvingcompile_isolatingalready did, now shared asisolate().pso_bundle_open_prewarm. No host change.build_ipa.sh. PassesDM_PSO_HOST, elsedm/host/dmhostwhen its last build was for this game (made current first).DM_PSO_HOST=nonemakes none.pso_offline.py(script_libraries), and the NSP script calls it.pso_manifest.tsvgets 10,415 rows (5,249 S, 4,447 R, 719 C) from the iPhone's ownpso_user.tsv.pso_skey_parseand written again in the current layout (the phone's list held two layouts).shader_map.tsvwith its stage.git merge-fileof the two versions gives no conflict.A-3: the
Mrowsmetal-tt takes the mesh pipeline descriptors the tool writes, so the
Mrows go in too. This is #2642's open item.Checked
Rand 20Mrows from Ocarina of Time's list, through both stages forapplegpu_g18p(iOS 27.0): 42 bases and 20 mesh pipelines, one file of 6 MB in 8 s, with a complete index of 62 keys.test_pso_offline.py: 60 tests. New:Prewarmwith a fake host and metal-tt (the tool's environment and tables, the files and indexes per GPU, reuse with nothing run, a GPU dropped, new rows rerunning the host alone, groups merged with a pipeline left out, a failed or missing host leaving the archive). Also a real metal-tt compile of a mesh pipeline descriptor forapplegpu_g18p, andbuild_ipa.shpassingDM_PSO_HOST.test_nsp_mac_app.py(70),test_docs,test_env_switches,test_lint_python.lint_python --against upstream/mainfinds nothing new in these files.regress.py: not run. The reach is the IPA build tool alone: no host, chain or runtime code changes, and the regress interface check usespso_offline.plan, which is unchanged.Phone (iPhone 17, iOS 27.2.1)
The phone build is
oot-build-final(the owner's phone tree, 547c5db-based) plus #2353's host part, #2539, 054eea1, cc8f449 and 248bb95, with prewarm files made by this branch'spso_offline.py.Making the files on a 16 GB M4 MacBook Air (macOS 27.2), from the seed list's 7,138
Rand 164Mrows:applegpu_g18p: 54 files, 1,009 MB, about 9.5 minutes, 0 left out.Rand 129Mrows of its current list are not in the seed list).Rrows name have such a twin.