Skip to content

OoT-30 C: base lookups of predicted keys in the background, a takeover that looks at each entry once, submissions copied without the submission lock - #2677

Open
stan69b wants to merge 3 commits into
seedds:mainfrom
stan69b:pr/oot-30-c-submit
Open

stan69b wants to merge 3 commits into
seedds:mainfrom
stan69b:pr/oot-30-c-submit

Conversation

@stan69b

@stan69b stan69b commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

State after the partial landing (2026-11-18)

Rebased onto main 31b537f. The branch now holds only what is not on main:

  1. C-4: a submission copies its command lists without the submission lock (unchanged from the version reviewed).
  2. The fix for C-4's snapshot overread (its own commit, f2ca39c). A program pinned without the lock fell through to the texture branch, and mtl_tex_pin read a tex_t serial 8 bytes past the 128-byte prog_t copy. The kind is now chosen first. A new target, check_t_queue_lifetime_asan, runs t_queue_lifetime under ASan and UBSan in both encode modes. On the old code it stops at that heap-buffer-overflow; with the fix it passes 43 of 43 in each mode.
  3. DM_PCSAMPLE, split from C-1's commit: off unless set, diagnostics only.

C-3 is on main as 9d3a583 (with edc7eff), and C-1's storage takeover as d664d31. They are no longer in this branch.

Still open for C-4: a CPU-limited comparison, which is what the landing note asked for. Not run yet: the Mac has no Ocarina of Time libraries for main's chain 63853cc4c4e3, and it is short on memory today.

Tests on 31b537f: check_t_queue_lifetime 43 x2, check_t_queue_lifetime_asan 43 x2, t_gfx_suspend 20, t_serial_owner 191, t_mtl_init, test_env_switches, test_docs.


The original description follows; its C-3 and C-1 sections describe what has landed.

Batch OoT-30 (Ocarina of Time at 30 fps on the iPhone 17), session C: the submit thread's CPU in busy scenes. Three commits, one per item, each with its test and its docs; item C-2 has no commit (see below).

C-3: base lookups of predicted keys move to background workers

Problem. In the owner's play, first visits to Hyrule Field and Castle Town spent up to 3.5 s of the submit thread per 300 frames in predictions (predictions 478 / 3533 ms, 527 / 1724 ms), and the game ran at 12-22 fps there.

Cause. With hybrid pipelines, pso_predict_rest_ made a pulling stand-in for every proposed key whose base was not made yet, so it could tell whether that key's draw would wait. Each stand-in was an archive lookup plus a specialization, on the submit thread and outside the two-build budget. Some of those keys are drawn later in the submission; others are never drawn.

Fix (pso_bg_look). A lookup's only lasting result is the base, and a base found (g_pbase) or known to be missing (g_pbase_miss) serves every thread. So the prediction hands each key's base lookup to a background worker:

  • The worker tries the key's own base first, then its pulling base.
  • When neither exists, the draw would wait, so the worker admits the key's build within the budget and has the pulling base built.
  • Each base is handed over once (g_pp_looked).
  • The draw then only specializes a base that is already made.
  • The pipelines line now reports the lookups handed over.

The owner's play ran on a build without upstream 53cbb99 (prediction at a submission's first new miss only); that commit is on main.

C-1: profile of the submit thread, and the storage takeover

Profile. sample of the queue replay thread on the Mac, owner's save, Temple of Time, about 900 draws a frame:

  • The thread was idle 57 % of the time.
  • draw: 34 %. Within it, pipeline compiles 10 %, encode_args 10 % (pool tables 4 %).
  • own_alias_publish → alias_take_over: 3 % (6 % of the busy time).

Takeover fix. alias_take_over looked at each overlapping entry once per 64 KB cell. A spared entry therefore cost a key compare and two hash lookups per cell, hundreds of times for a 16 MB render target the game makes again. It now looks at an entry only in the first cell of the overlap; the result is the same.

Tooling. DM_PCSAMPLE plus dm/tools/pcsample.py sample named host threads on an iPhone, where neither sample nor a debugger may attach.

Target. The target of a third fewer instructions per draw is not reached: the remaining per-draw cost is spread over many places at a few percent each.

C-4: submissions copied without the submission lock (item 4's question)

Question. Does the game's main thread wait on the host? Yes. On the Mac it sat 16.5 % of its time in mutex waits, 12.4 % of that inside nvn_queue_submit.

Cause. The copy of a submission's command lists ran under g_sub_mu, which the queue replay thread holds for the whole translation of each submission. With 22-30 submissions a frame, the game's thread waited for each one the replay was working on.

Fix.

  • The copy now runs without that lock.
  • Program pins get their own lock.
  • A program the replay has already resolved is pinned and copied without g_sub_mu; only one that is not resolved yet takes g_sub_mu to be resolved.

The rest of the main thread's time is the game's own work: guest code, and yields while it waits for its own workers.

C-2: CPU clock in slow windows (measurement, no commit)

Owner's play log. In the slow windows:

  • The frame threads declared the capped computation (30 ms of 33).
  • They ran at 1.3-1.6 GHz, with 2-30 % of process CPU on performance cores.
  • The guest main thread yielded about a million times a second.

Phone runs. Five runs on the iPhone 17, all at thermal state 2 (table below). Each run has a load burst after the save loads:

  • Residency adds about 390 MB.
  • The game's main thread and workers spin on yields.
  • For 1 to 6 windows of 300 frames, the whole process runs on efficiency cores at 1.1-1.4 GHz (0 % on performance cores, 0.4 W), at 26-27 fps.
  • While there, the computation measured as CPU time climbs to the 30 ms cap: CPU time grows as the clock falls.

Experiment. Declaring the computation from cycles at 3.5 GHz (DM_RT_CYC=1, experiment build only) changed two things:

  • The computation stayed at 11 ms through the burst.
  • The burst lasted 1 window, against 2-6 in the other runs.

The process still went to the efficiency cores, so the declaration is not what moves it there. It may prolong the stay, but that is one run. A lower declaration also risks the stalls that nvn_rt.h describes for an under-declared thread. So there is no commit.

Third state. One baseline run stayed for its whole 10 minutes in another state: 70 % on performance cores, yet the submit thread at 1.6 GHz, the declared computations at 20-26 ms, and 15-17 fps. The second baseline run did not repeat it, and its trigger is unknown. The question stays open.

Tests

Test Result
t_pso_bg_look (new) 14 checks
check_t_pso_depth_stencil 298 / 301 checks
t_pso_pending 57
check_t_pso_archive 355
check_t_pso_stall 73
check_t_alias_takeover 48 (new check fails before the fix)
t_serial_owner 176
check_t_queue_lifetime 43, twice (new check fails before the fix)
t_gfx_suspend 20 (updated deliberately: both submissions now return while the prepare step holds the lock)
t_nvn 325
test_docs passes
ruff on pcsample.py passes

Phone and regression check

Setup. iPhone 17, the owner's save, the drive script standing still in the first scene (an indoor hall, 900-1,100 draws), 30 fps cap with dynamic resolution. Thermal state 2 in every window of every run.

  • "Steady" = windows with ≥ 40 % of CPU on performance cores.
  • "E-core windows" = windows with < 10 % on performance cores.
  • Builds: phone base be8db251 ("fix2", the owner's build), and the same base with this PR's three commits plus upstream 53cbb99 and the experiment switches, off ("C").
run (order) build steady fps submit CPU / frame clock instructions / draw E-core windows (fps, clock, declared)
1 C 29.9 5.03 ms 3.33 GHz 32.2 k 5 (27.1, 1.41 GHz, 30 ms)
2 fix2 – (15.8 fps the whole run, submit 6.41 ms at 1.62 GHz, declared 19.6 ms) 33.7 k 0
3 C 29.8 4.73 ms 3.59 GHz 33.0 k 2 (26.6, 1.33 GHz, 30 ms)
4 fix2 29.9 4.84 ms 3.42 GHz 33.6 k 6 (27.1, 1.36 GHz, 30 ms)
5 fix2 + DM_RT_CYC=1 29.9 4.83 ms 3.39 GHz 33.5 k 1 (25.7, 1.12 GHz, 11.4 ms)

What the runs show.

  • In steady windows, C and fix2 both hold the 30 fps cap. C's instructions per draw are 2-4 % lower, and its submit CPU about 0.1 ms lower.
  • The scene is not CPU-bound once the clock is high, so C-3's gain (first visits with new pipelines) does not show here; it needs Hyrule Field or Castle Town.
  • No [RED] lines in any run.
  • The saves were identical before and after every run (save_check.sh). The owner's fix2 build is installed again.

Phone profile (DM_PCSAMPLE, run 1, 1 kHz for 240 s).

  • Queue replay: draw_ 58 % inclusive, encode_args_ 21 %, fb_prepare 9 %, mtl_sub_commit 8.5 %, nvn_hm_get 6 %, pool_gpu 6 %. No single share left above 10 % besides the draw and encode roots.
  • The game's main thread: blocked in its own guest waits (block) 37 %, guest code 17.5 %, and no samples in nvn_queue_submit's lock (C-4).

regress.py check ce370e12 → 4b415669 (before the rebase onto bee218d9; the rebase was clean):

  • Ocarina of Time: PASS, 42848/42848 shaders on both sides, no changed MSL, no interface change.
  • No boot run: the repository has no smoke.tsv route for OoT.
  • Echoes of Wisdom and Super Mario 3D All-Stars (NVN) could not run on this Mac. Their shader packages were split before records carried keys ("holds no record of shader …"), on the baseline too; re-splitting them changes the owner's build folders, so that was left alone.

After the rebase. The targeted tests passed again: t_pso_bg_look 14, t_alias_takeover 48, t_queue_lifetime 43, t_gfx_suspend 20, t_pso_pending 57, t_serial_owner 176, t_nvn 325, test_docs, and test_env_switches (now green on main).

@stan69b

stan69b commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Batch OoT-30: the combined phone build (iPhone 17, iOS 27.2.1)

Build (DmIOS-all1.app, local branch oot-30-hand 75e9de61, never pushed): the owner's phone tree oot-build-final (be8db251), plus every session's phone version:

Runs: the owner's newest save, the drive script turning the camera only (it cannot walk), 30 fps cap, dynamic resolution, 6 minutes each, back to back. Windows are the 300-frame lines after frame 600.

Run Thermal Prepare before the first frame Frame p50, median of windows (worst) Windows slower than 29 fps GPU p50 Submit+present p50 Submit CPU / instr. per draw Bases compiled in play Residency set peak
Baseline: the installed fix2 (be8db251) 0 → 2 4.5 s (0 compiled) 33.5 ms (44.6) 14 of 61 29.6 ms 4.72 ms 4.71 ms / 33.9 k 0 3,328 MB
Combined, first start 2 61.2 s (687 compiled, once) 33.4 ms (46.6) 9 of 60 30.4 ms 4.76 ms 4.77 ms / 33.9 k 0 3,345 MB
Combined, second start 2 6.0 s (0 compiled) 33.4 ms (49.0) 15 of 58 29.8 ms 4.97 ms 5.00 ms / 32.5 k 0 3,330 MB

Reading it

  • The first start's 61 s is 248bb95's one-time cost. That start removed the phone's 31 archive files indexed by library file names. It compiled again the rows the phone had recorded since the seed list took its copy of the phone's list, which are not in the prewarm files (945 R and 129 M).
  • With the device's archive off and the seed list alone (A's runs on the same build, An IPA built with a Mac host carries the prewarm lists' pipelines made offline (R rows' bases and M rows' mesh pipelines), and Ocarina of Time's seed list gets an iPhone 17's own rows #2674), the prepare step takes 4.2 s and compiles 0 of 6,627 bases and 0 of 164 mesh pipelines.
  • In this scene (indoors, 900-1,100 draws) both builds hold 30 fps most of the time. The slow windows come and go with the thermal state and the GPU clock, as B describes. The two builds do not differ here beyond run-to-run spread.
  • Not reached: Hyrule Field, Kakariko Village and Castle Town (the driver cannot walk). That is where C-3, the prewarm files and D-1 should show.

Owner, a route to validate (the combined build is installed on the phone now):

  • walk from Kakariko Village into Hyrule Field, then to Castle Town and into the castle grounds, by day, then the same at night, about 10 minutes each;
  • then send Documents/log.txt. Its pipelines up to frame lines show bases compiled in play, translation up to frame the submit thread, and the [mem] census lines what held memory at the castle-grounds load (D).

stan69b pushed a commit to stan69b/nxnative that referenced this pull request Nov 16, 2026
…'s archive lookup of the same base, not only returns nil, so a draw that comes during a background lookup takes its stand-in instead of compiling its base on the submit thread

pso_build_ex with compile 0 (a draw asking whether its base or its pulling base is made or archived, a prewarm task) returned nil at once when another thread held that base's job, so that it would never wait for the Metal compiler. A job's owner first only looks in the archives, which takes milliseconds, and a non-compiling owner does nothing else. A draw that came during such a lookup of its pulling base found no stand-in; its own base was already known missing, so it compiled that base on the submit thread.

The prediction's background base lookups (pso_bg_look, PR seedds#2677) make the race happen every time: they look up the bases of the draws that come next in the same submission, milliseconds before those draws. On the owner's iPhone 17 play of 2026-11-14 the builds with them had single draws waiting 0.3-1.8 s on first visits ("draws that waited: pulling base not made yet" 1 / 861 ms, 2 / 1,081 ms, 8 / 1,786 ms per 300 frames, while the background lookups found every base: 21 of 21), against 1 / 95 ms on the build without them.

- PbaseBuild.looked / found: the owner leaves looked once it has looked in the archives (or skipped them for a base known missing), with the base it found, before the compaction's capture and before any compile.
- A compile 0 call that finds another thread's job waits on looked and takes found; when the lookup found nothing it returns nil as before, without waiting for the compile.

Mac, M4, Ocarina of Time v131072, an unattended route from a Kakariko save (out to Hyrule Field and back), a first visit (no recorded pipeline list), the archive holding only the pair prebuild's pulling bases (as the phone's bundled archive), hosts on the phone tree (be8db251 + seedds#2677's commits), one run each: draws that waited 214 / 32.0 s without the fix, 216 / 29.5 s with it. On the Mac an archive lookup takes well under a millisecond, so the race's window is small there and the waits left are draws whose pulling base no archive holds; the race shows on the phone, where a lookup takes about 10 ms. Not run: the phone (no phone in this batch).

Checked: t_pso_look_wait 9 checks, new (fails without the fix: the call during a 200 ms lookup returned nil at once); t_pso_pending 66, t_pso_stall 73, t_pso_archive 362, t_pso_depth_stencil 301, t_pso_prewarm 64, t_pso_warm 123, t_pso_bundle 83, t_pso_msaa 34; test_docs; lint_c.py tidy (nothing above the baseline). regress.py: not run. The only game this Mac can run it on is Ocarina of Time, which has no smoke route, so the check is its shader chain alone, which this change does not touch; the route runs above booted the game with the change.

New reference entry: docs/reference/frame-rate.md, "A draw that came during another thread's archive lookup compiled its base"; symptom index updated.

(cherry picked from commit 51a1de4)
@stan69b

stan69b commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main, without the C: done marker commit.

Adapted to main:

  • C-3 follows main's predictor. A draw met in the submission is admitted whatever is running (pso_pend_kick_next); a guess only within the budget. The background lookup now carries that distinction.
  • C-1 keeps main's pool_entries counter beside looks.
  • C-4 keeps main's submitted stamp.

seedds added a commit that referenced this pull request Nov 16, 2026
Prediction synchronously looked up and specialized pulling stand-ins on the
submit thread, including keys that were never drawn. Move the base lookups
to background workers and reuse the process-wide base/miss caches. Take only
C-3 from PR #2677; submission copying, storage takeover and sampling remain
separate.

Give each prediction a retained budget, charged under the pending-table lock
only when a guess is admitted. Reserve a slot while its draw waits so a
background admission cannot replace the completed pipeline before that draw
takes it. Sample admission/completion/steal counts coherently for diagnostics.
The full-table fixture reproduced wrong-pipeline delivery before reservation.

Checked together with PR #2770 on Apple M4 Pro, macOS 27.2.1: lookup 9,
background lookup 17, pending 92, stall 110, depth/stencil 228/231/228/231;
docs 6; clang-tidy LLVM 23.1.5 with no baseline increase; unsigned iOS
production-project compile/link fixture with synthetic inputs.

Full regress.py check: NieR Automata 010056B015FE8000, 220 seconds per side,
1,225/1,225 shaders each, no changed MSL, interface changes or new RED kinds;
PASS. This light NVN title covers the shared archive path. Both runs were
busy, so their timing is not a performance qualification. iPhone testing was
waived by the owner on 2026-11-17; phone benefit remains unverified.

Co-authored-by: Stan Bono <[email protected]>
@seedds

seedds commented Nov 16, 2026

Copy link
Copy Markdown
Owner

The background-lookup portion (C-3) landed on origin/main as 9d3a5836, followed by PR #2770's archive-lookup completion fix edc7eff9. This is a partial landing: this PR stays open.

The landed implementation uses a retained budget per prediction, charged under the pending-table lock only for successful guess admissions. It additionally reserves a completed slot for its waiting draw; a deterministic full-table fixture reproduced wrong-pipeline delivery and loss of the replacement without that correction. Admission/completion/steal diagnostics are sampled coherently under the same lock. The diagnostic underflow was identified by inspection, not reproduced in the old-code stress run.

Submission-copy/program snapshots (including their separate snapshot-dispatch correction), storage takeover and the phone sampler were not included. Stan Bono is co-authored on both landed commits with the verified trailer Co-authored-by: Stan Bono <[email protected]> because meaningful code, design and diagnosis were retained.

Validation: lookup-wait 9, background lookup 17, pending jobs 92, stall 110, depth/stencil 228/231/228/231; documentation 6; generated symptom index current; clang-tidy LLVM 23.1.5 with no baseline increase; unsigned production iOS compile/link fixture with synthetic inputs. After the clean rebase onto 44d67fb, lookup 9, background 17, pending 92, the upstream arena fixture 42 and documentation 6 passed again, as did unsigned iOS compilation. Range-diff confirmed both patches were unchanged by rebase.

The full NieR regression below used the earlier main base 60284e7 and the reviewed combined implementation. NieR was chosen as a light NVN archive title (1,225 shaders), covering the common pipeline path. Both runs were explicitly busy; screenshots at 100/180 seconds were reviewed and show the same flight sequence with formation/timing differences and no obvious new missing geometry. The shared measurement lock serialized game runs.

Full regression report

regress.py check (full): PASS

Baseline 60284e7901cb4acd0e057d99fdf5560e0cd8e2fe 60284e7, candidate worktree 60284e7 (+ uncommitted changes); Apple M4 Pro, macOS 27.2.1, Xcode 27.2; 470.6 s

game status shaders (base → cand) newly failing newly building changed MSL interface (new / known) boot fps p50 frame p99 ms peak MB prepare s new RED
NieR Automata The End of YoRHa Edition 010056B015FE8000 PASS 1225/1225 → 1225/1225 0 0 0 0 / 0 ok 59.9 → 59.9 17.18 → 17.2 2346 → 2332 0.16 → 0.09 0

NieR Automata The End of YoRHa Edition 010056B015FE8000: PASS

  • screenshots (s: dHash bits / RGB diff / baseline noise): 100: 17/5.12/0, 180: 13/5.71/0
  • baseline: exit None, first frame 0.8 s, max frame 13111, reached True, prepare 0.16 s, prebuild 12.0 s (902 built, 0 failed), fps 59.9, p50/p99 16.66/17.18 ms, peak 2346 MB, busy Mac
  • candidate: exit None, first frame 0.8 s, max frame 13087, reached True, prepare 0.09 s, prebuild 13.9 s (902 built, 0 failed), fps 59.9, p50/p99 16.67/17.2 ms, peak 2332 MB, busy Mac
  • time: chain 3.2 s, smoke 467.2 s
  • Should it exist? Yes. An available archive base was reported absent solely because another thread owned its lookup; the lookup fixture reproduces the mechanism. The background admission review also reproduced a full-table handoff returning the wrong pipeline before consumer reservation.
  • Confidence: high in the host synchronization fixes; phone improvement remains unverified.
  • Binding: general host, already the loosest correct binding; no game/build conditions.
  • Workaround: none; fixes lookup publication, budget lifetime and pending-result ownership.
  • From-scratch design: yes: separate lookup/compile completion, per-prediction budgets and reserved consumer ownership.
  • Performance: expected to avoid draw-time compiles when the concurrent lookup finds the archived stand-in; phone benefit remains unmeasured and busy Mac timings establish no gain. Estimated added predictor bookkeeping is about 1–2 KiB per unique predicted base from the 784-byte key and hash-table capacity.
  • Regression level: moderate shared synchronization risk; one light game, NieR, full route PASS. Other games were not run for this change; no heavy OoT Mac regression was needed to cover the shared path.

Remaining: submission-copy, storage takeover and sampler changes are still separate and unlanded by this action; this PR remains open for them. The owner waived iPhone testing for this landing on 2026-11-17, so no phone retest is awaited as a gate and phone performance remains unverified. No gain is claimed from the busy Mac comparison.

seedds added a commit that referenced this pull request Nov 17, 2026
A spared alias appears in every 64 KiB cell its storage reaches. Takeover
repeated its key comparison and ownership lookups in every overlapping
cell, even though the result is unchanged. Inspect it in the first cell
of the overlap alone, retaining the existing eviction and ownership rules.

The held-alias fixture on current main 8d62601, M4 Pro, drops from
4.29/4.71 us to 1.64/1.52 us per takeover in two interleaved processes per
side. Ocarina of Time's settled bridge remains 60 fps on both sides; this
does not establish a game frame-rate gain.

Checked alias and serial ownership tests, focused graphics tests, lint,
and docs. Kirby's full beach regression passes. NieR's timed screenshot
gate differs in flight phase; three fixed-frame replays per side pass,
with no new RED kind, crash, hang or shader/interface failure in either
game. Two Python suites retain the same failures on unchanged main.

Co-authored-by: Stan Bono <[email protected]>
seedds added a commit that referenced this pull request Nov 17, 2026
Record two isolated headless Ocarina of Time bridge runs per variant on
main 8d62601, M4 Pro, against a same-head synchronous control. Both retain
the lookup-wait and archive-admission fixes and start from identical save
and archive copies. The corrected predictor already landed as 9d3a583.

Prediction elapsed time falls from 922.6/448.8 ms to 33.0/33.7 ms across
the logged windows, while worst frames fall from 598.65/373.75 ms to
224.90/233.62 ms. Settled frames remain 60 fps. Exclude the visible runs
where the user moved Link and leave the original M4 Max and iPhone
comparisons open. No instruction-count result qualifies this head.

Checked the documentation suite and the final diff. Submission copying
and the phone sampler remain separate held patches.

Co-authored-by: Stan Bono <[email protected]>
@seedds

seedds commented Nov 17, 2026

Copy link
Copy Markdown
Owner

Pushed the authorized partial landing to origin/main on 2026-11-17:

C-4 submission copying, including its corrected program/texture snapshot dispatch, remains held separately. The phone sampler also remains held. This action does not close this PR or issue #2827.

The tested performance baseline is 8d62601f, with the same libraries (chain 13b3bdb97e7e) on both sides. Fresh main additionally contained GPU scheduling service commit c3efaa90; the rebase preserved it. Range-diff confirms both patches are unchanged. Documentation tests (6), clean diff checks and final review passed after rebasing; game measurements were not repeated on c3efaa90.

The OoT comparison uses v131072, an M4 Pro (14 CPU / 20 GPU cores, 48 GB, macOS 27.2.1), two headless fresh processes per variant, the child Hyrule Field bridge save, identical save/archive copies, frame-based menu input and no movement. Two excluded warmups prime the synchronous and asynchronous paths before freezing the measured archive. Frames 900-4999 are compared; the frame-5000 screenshot is outside the window. Prediction totals use logged 300-frame windows covering frames 901-4800. The synchronous control changes only the background-lookup branch on the same head and retains the lookup-wait and archive-admission fixes. All eight standard-comparison processes have no RED or watchdog stop and matching fixed-frame views. The earlier visible runs where the user moved Link are excluded.

OoT first visit, two runs each Synchronous control Current asynchronous predictor
Prediction elapsed time on submit thread, total ms 922.6 / 448.8 33.0 / 33.7
Worst frame, ms 598.65 / 373.75 224.90 / 233.62
Frames 900-4999, fps 57.5 / 58.8 59.1 / 59.0
Settled frames 3000-4999, fps 60.0 / 60.0 60.0 / 60.0

Alias cost was measured independently: a CPU-only fixture retains a 16 MiB RGBA8 alias and repeatedly takes over the same storage as R32F. Two interleaved processes per side, seven batches of 10,000 calls after warmup, retain the held entry in every process: median elapsed cost 4.29 / 4.71 us before, 1.64 / 1.52 us after. This does not establish a whole-game CPU or FPS gain.

C-4 alone establishes no FPS or CPU gain in the valid bridge comparison (both settled at 60 fps). An attempted 120 fps diagnostic was rejected by the host and is excluded from qualification. The fixed C-4 lifetime/suspend fixtures and ASan/UBSan in both encoding modes pass, but it remains unlanded pending a matched CPU-limited comparison. Its snapshot overread was a defect in the PR branch, not main's locked copying path.

Checks before landing: current-head alias 48, serial 191; combined focused queue/lifetime, suspend, lookup, budget, stall, depth/stencil and RT descriptor/SBT coverage; ASan/UBSan in both queue encoding modes; clang-tidy LLVM 23.1.5 with no baseline increase. A full graphics run passed before integration onto the later main. Of 141 Python suites, 137 initially passed; two runtime-limited suites passed 10 tests on baseline and candidate using the installed .NET11 runtime via process-local roll-forward; ABI and environment catalogue suites retain identical failures on unchanged 8d62601f.

Regression coverage used the reviewed combined implementation, a superset of this partial landing. NieR covers common archive paths and Kirby the program/texture submission paths. NieR's raw timed screenshot gate remains REGRESSION: inspected shots differ in flight phase/position/item count, without obvious new missing geometry; three fixed-frame replays per side PASS within the capture's 2094-pixel tolerance (max 1335 baseline / 897 candidate). Kirby's first configured save was missing; a frozen copy of the existing local beach save then passed the full route, and the source/seed were unchanged. No new RED kind, crash, hang, shader or interface failure in either game. Raw reports follow so the timed gate is not presented as a blanket pass.

Initial full check (including the missing Kirby fixture)

regress.py check (full): ERROR

Baseline 8d62601f375fa8e15bb6d5948b416f63959a0a4d 8d62601, candidate worktree 8d62601 (+ uncommitted changes); Apple M4 Pro, macOS 27.2.1, Xcode 27.2; 1728.7 s

game status shaders (base → cand) newly failing newly building changed MSL interface (new / known) boot fps p50 frame p99 ms peak MB prepare s new RED
NieR Automata The End of YoRHa Edition 010056B015FE8000 REGRESSION 1225/1225 → 1225/1225 0 0 0 0 / 0 ok 59.9 → 59.9 17.24 → 17.27 2302 → 2335 0.11 → 0.1 0
Kirby and the Forgotten Land 01004D300C5AE000 ERROR 27297/27297 → 27297/27297 0 0 0 0 / 0 – – – – – –

NieR Automata The End of YoRHa Edition 010056B015FE8000: REGRESSION

  • REGRESSION: screenshot at 100 s differs (dHash 22 bits, RGB 7.73)
  • REGRESSION: screenshot at 180 s differs (dHash 20 bits, RGB 8.7)
    • screenshots (s: dHash bits / RGB diff / baseline noise): 100: 22/7.73/0, 180: 20/8.7/0
    • baseline: exit None, first frame 0.8 s, max frame 13123, reached True, prepare 0.11 s, prebuild 9.6 s (902 built, 0 failed), fps 59.9, p50/p99 16.66/17.24 ms, peak 2302 MB
    • candidate: exit None, first frame 0.5 s, max frame 13148, reached True, prepare 0.1 s, prebuild 9.8 s (902 built, 0 failed), fps 59.9, p50/p99 16.66/17.27 ms, peak 2335 MB
    • time: chain 50.8 s, smoke 460.6 s

Kirby and the Forgotten Land 01004D300C5AE000: ERROR

  • ERROR: the route's save /saves/beach does not exist (dm/games/01004D300C5AE000/smoke.tsv, save)
    • time: chain 1217.1 s
Kirby full check with frozen beach fixture

regress.py check (full): PASS

Baseline 8d62601f375fa8e15bb6d5948b416f63959a0a4d 8d62601, candidate worktree 8d62601 (+ uncommitted changes); Apple M4 Pro, macOS 27.2.1, Xcode 27.2; 585 s

game status shaders (base → cand) newly failing newly building changed MSL interface (new / known) boot fps p50 frame p99 ms peak MB prepare s new RED
Kirby and the Forgotten Land 01004D300C5AE000 PASS 27297/27297 → 27297/27297 0 0 0 0 / 0 ok 30.0 → 29.9 33.38 → 33.38 3820 → 3819 None → None 0

Kirby and the Forgotten Land 01004D300C5AE000: PASS

  • screenshots (s: dHash bits / RGB diff / baseline noise): 55: 6/1.54/0, 68: 6/1.76/0, 95: 1/0.83/0
  • baseline: exit None, first frame 1.1 s, max frame 2930, reached True, prepare None s, prebuild 176.1 s (21409 built, 0 failed), fps 30.0, p50/p99 33.33/33.38 ms, peak 3820 MB
  • candidate: exit None, first frame 0.8 s, max frame 2947, reached True, prepare None s, prebuild 172.9 s (21409 built, 0 failed), fps 29.9, p50/p99 33.33/33.38 ms, peak 3819 MB
  • time: chain 33.5 s, smoke 551.2 s
NieR fixed-frame replay

regress.py replay (full): PASS

Baseline 8d62601f375fa8e15bb6d5948b416f63959a0a4d 8d62601, candidate worktree 8d62601 (+ uncommitted changes); Apple M4 Pro, macOS 27.2.1, Xcode 27.2; 282.7 s

game status shaders (base → cand) newly failing newly building changed MSL interface (new / known) boot fps p50 frame p99 ms peak MB prepare s new RED
NieR Automata The End of YoRHa Edition 010056B015FE8000 PASS –/– – – – – – – – – – –

NieR Automata The End of YoRHa Edition 010056B015FE8000: PASS

  • replay of frame 10800 (captured with ba9ce21, chain 9eecb8412308, 2026-11-17; noise 1047 pixels, tolerance 2094)
  • baseline: exit 0, 0 of 3 equal the live frame, at most 1335 pixels differ, 5.5 s
  • candidate: exit 0, 0 of 3 equal the live frame, at most 897 pixels differ, 5.5 s
  • time: replay 282.6 s
  • Should it exist? Yes: duplicate alias checks have a measured fixture cost and asynchronous prediction removes measured submit-thread waits. Existing prewarming/cache settings can avoid misses but do not correct repeated overlap work, interleaved admission ownership or the PR snapshot overread; the latter two have deterministic failing reproductions before correction. C-4 remains held because game benefit is unqualified.
  • Confidence: high in the alias and predictor fixes; Standards and Spec reviews have zero remaining actionable findings, and final code/diffs were reviewed. Generalizing performance to other devices remains unverified.
  • Binding: general host → host, already the loosest correct binding; no game, engine, SDK or build conditions were added.
  • Workaround: none; first-overlap ownership, explicit snapshot object-kind dispatch and retained prediction budgets address the causes.
  • From-scratch design: yes: inspect each indexed alias once, retain a budget per prediction and select program versus texture before testing resolution. C-4's separate pin ownership remains in its held patch.
  • Performance: on the M4 Pro / OoT v131072 / 8d62601f comparison, accumulated prediction elapsed time falls from 448.8–922.6 ms to 33.0–33.7 ms and worst frames from 373.75–598.65 ms to 224.90–233.62 ms; the independent alias CPU fixture falls from 4.29–4.71 us to 1.52–1.64 us per takeover. Settled game rate remains 60 fps, and no alias-only or submission-copy-only game FPS gain is established.
  • Regression level: uncertain broad host reach, so two games: NieR for the shared archive path and Kirby for program/texture submission paths. Kirby full route and NieR fixed-frame replay pass; NieR's timed screenshot gate remains flagged, as detailed above; other games were not regressed.
  • Author credit: both landed commits co-author Stan Bono with verified Co-authored-by: Stan Bono <[email protected]> because meaningful PR code, design and diagnosis were retained.

Remaining: corrected submission-copy and sampler patches are held separately. Original M4 Max and iPhone performance retests, C-4 CPU-limited qualification, the timed NieR gate and the two baseline Python-suite failures remain open; no new device/gameplay run was performed during landing. PR #2677 and issue #2827 stay open for their remaining work/retest.

seedds added a commit that referenced this pull request Nov 17, 2026
Record the capped M4 Pro baseline on d6585a2 and the excluded
submission-copy comparison windows for PR #2677. No valid CPU-limited
comparison or candidate performance gain was established.

Keep submission-copy and the sampler held separately. This commit adds
only the measurement note, including the unresolved baseline geometry
diagnostics; it changes no runtime code or shader inputs.

Co-authored-by: Stan Bono <[email protected]>
@seedds

seedds commented Nov 17, 2026

Copy link
Copy Markdown
Owner

Pushed measurement notes only to main: f3e53afe, in Donkey Kong Bananza's game note. Submission-copy and the sampler remain held separately.

  • Should these notes exist? Yes: they record the measured baseline and excluded comparisons without claiming a submission-copy benefit.
  • Confidence: High that the note matches the retained results; the candidate's performance benefit remains unknown.
  • Binding: Game-specific measurement record: the note names its game, device, revision and protocol. No runtime special case was added; the game note is the appropriate scope.
  • Root cause or workaround? Unknown for the recorded baseline geometry diagnostics. No runtime fix or workaround is included in this push.
  • From scratch? Yes: keep dated measurements and their qualification limits in the existing game note.
  • Performance: None expected: documentation only, off all runtime paths. The M4 Pro baseline on d6585a2 held 59.6 fps with 7.75 ms of copy/replay CPU per frame; it was capped. Pipeline creation invalidated the candidate window and the later main repeat, so neither comparison qualifies C4.
  • Regression: Low risk: six documentation tests and diff checks passed after rebase; the final diff is one note addition. No new gameplay regression run was needed for documentation.
  • Author credit: Yes: Stan Bono is credited for the PR's submission-copy design being evaluated.

Remaining: no valid CPU-limited comparison was obtained; submission-copy and the sampler remain separately held, and the baseline geometry diagnostics remain unresolved. The PR and issue stay open.

Aznracer pushed a commit to Aznracer/Mighty-Green-Kokiri-Kid that referenced this pull request Nov 17, 2026
What the 2026-11-16 search tried in in-run A/B and did not keep (open pull
requests seedds#2526, seedds#2677 and seedds#2771 measured on this game, DM_NVN_ARGS_LEAN,
DM_NVN_REC_EAGER, DM_NVN_PASS_FENCE), and what shader_row_bounds measured on
its light loops. Docs only.
atinseau added a commit to atinseau/nxnative that referenced this pull request Nov 17, 2026
What the 2026-11-16 search tried in in-run A/B and did not keep (open pull
requests seedds#2526, seedds#2677 and seedds#2771 measured on this game, DM_NVN_ARGS_LEAN,
DM_NVN_REC_EAGER, DM_NVN_PASS_FENCE), and what shader_row_bounds measured on
its light loops. Docs only.
…ion lock, so the game's thread no longer waits for the queue replay of the one before

Item C-4 asked whether the game's main thread (200-400 ms/s of CPU in the owner's slow windows, 2-4 ms of yields a frame) waits on the host. It does: a `sample` on the Mac (the owner's save, Temple of Time) has the main thread 16.5% in __psynch_mutexwait, 12.4% inside nvn_queue_submit -> pthread_mutex_lock. A submission copied its command lists (nq_flatten) under g_sub_mu, which the queue replay thread holds for the whole translation of each submission (nq_worker), so with 22-30 submissions a frame the game's thread waited for each one the replay had in hand; on an iPhone, with 11-24 ms of replay a frame in busy scenes, the frame was the main thread's work plus the replay's. The rest of the main thread's time is its own: guest code and its waits on the game's own workers (the yields; ModuleSystemWorker).

The copy needed the lock only for the first submission's prepare step and for a program no replay had resolved (prog_get is the replay's), and the program pins it shared with the replay had no lock of their own.
- nvn_queue_submit: no submission lock for the copy; the prepare step runs under it once (mtl_prepared).
- nq_snapshot_object: a program the serial table holds is pinned and copied without it (mtl_prog_snapshot_live, own_prog_pin_live); only one not resolved yet takes the lock to be resolved, as before.
- nvn_owner.m: the program pins have a lock of their own (g_pin_mu), taken before g_smu wherever both are held.

Tests: check_t_queue_lifetime 43 x2 (a submission whose program is resolved returns while the lock is held, and draws; an unresolved program is left to the lock: new, fails before), t_gfx_suspend 20 (updated deliberately: both submissions now return while the prepare step holds the lock instead of the second waiting for it; nothing commits before the resume, as before), t_nvn 325, t_serial_owner 176, check_t_alias_takeover 48, test_docs.
Docs: frame-rate.md "The game's submissions waited for the queue replay's lock", nvn-objects.md (the prepare-step entry's check), symptom index, game note.
(cherry picked from commit f923323)
…ls through to the texture copy (PR seedds#2677 C-4)

C-4's nq_snapshot_object tested `kind == CAP_K_PROG && !mtl_prog_snapshot_live(...)`. When the program was
already resolved, the pin succeeded, the condition was false, and the program took the texture branch: it was
copied again and mtl_tex_pin read a tex_t serial 8 bytes past the 128-byte prog_t copy, then pinned whatever
texture serial those bytes held (a pin never released). The kind is now chosen first, and the resolution test
runs only for programs.

The program pin table's comment in nvn_owner.h now names its own lock (g_pin_mu) instead of the submission lock.

New target check_t_queue_lifetime_asan: t_queue_lifetime under ASan and UBSan, in both encode modes. Before
this fix it stops at "heap-buffer-overflow ... READ of size 8 ... 8 bytes after 128-byte region" in
nq_snapshot_object (the existing case "a submission while the replay holds the submission lock").

Tests: check_t_queue_lifetime 43 x2, check_t_queue_lifetime_asan 43 x2, t_gfx_suspend 20, t_serial_owner 191.
… where neither sample nor a debugger can attach (PR seedds#2677)

DM_PCSAMPLE=<names>[:<us>[:<s>[:<first frame>]]] (hle_nvn.c): a host thread samples the threads whose pthread
name contains one of the names (pc, lr and frame-pointer frames) into pcsample.txt, the app's Documents on iOS,
with the image's load address and the guest modules' ranges. dm/tools/pcsample.py names the samples with atos
and ranks them per thread. Off unless set. It is what found C-1's alias takeover and C-4's submission-lock wait
on the phone.

Split from the PR's C-1 commit, whose storage-takeover half landed on main as d664d31.

Docs: development.md "Profiling a host thread on an iPhone", env_switches.tsv.
Tests: t_mtl_init (builds hle_nvn.c), test_env_switches, test_docs.
@stan69b

stan69b commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the partial landing and the review. The branch is rebased onto 31b537f and holds only C-4, its overread fix and the sampler (summary at the top of the description).

The overread is the one you found: with kind == CAP_K_PROG && !mtl_prog_snapshot_live(...), a program pinned without the lock took the texture branch. It is fixed in its own commit, f2ca39c, and the new check_t_queue_lifetime_asan target reproduces it on the old code.

I haven't made the CPU-limited comparison yet. I'll post it here when it's done. If your held copy of C-4 also changes pin ownership beyond this fix, tell me what to align with.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 0400486024DEC000 A game nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants