Skip to content

Frame generation: a ring slot is free once the present after its own is taken, not once it has run on the GPU - #2582

Open
stan69b wants to merge 2 commits into
seedds:mainfrom
stan69b:pr/framegen-gpu-time
Open

stan69b wants to merge 2 commits into
seedds:mainfrom
stan69b:pr/framegen-gpu-time

Conversation

@stan69b

@stan69b stan69b commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Depends on #2241 (frame generation). This branch is based on its head, so until #2241 merges the diff below includes it; the change itself is the last commit.

What changes

With frame generation on, a snapshot ring slot's ticket is the present after its own (tk + 1), because the slot's picture is shown again after its generated frame. The game's acquire (mtl_snap_acquire_wait) and the copy (snap_take) waited until present tk + 1 had run on the GPU. That present is committed behind the game's next frame on the same queue, so the game waited for the GPU to finish that frame and lost its frame of overlap with the GPU.

Now the slot is free once the display link (or the present thread) has taken present tk + 1 (snap_slot_free). By then the picture owed after the generated frame has been committed or skipped, and the next copy, committed later on the same queue, runs after it. When the link drops presents after a stall, it also drops the picture still owed. Frame generation stays off by default; with it off, nothing changes.

Before and after

Mac (M1 Pro), Ocarina of Time, Hyrule Field, 30 fps cap, the same route each run, DM_NVN_WAITS=1:

Frame time p50 / p95 Snapshot ring blocked
Frame generation off 33.3 / 33.5 ms —
On, before 33.3 / 36-39 ms 68-89 % of frames, 8-16 ms
On, after 33.3 / 33.5-33.8 ms 0 % in most windows (two windows: 69 % for 11.6 ms, 22 % for 3.6 ms)

iPhone 17, Kakariko Village by day, 30 fps cap, 6 minutes each, unattended and back to back:

Build Frame generation fps Frame time p50 / p95 GPU p50 Interpolation p50
Before on 26.2 37.3 / 48.7 ms 25.8 ms 8.4 ms
After on 24.9 38.5 / 50.4 ms 26.7 ms 8.7 ms
After off 27.9 35.6 / 41.3 ms 32.1 ms —

On the phone the difference is within noise. The 4-7 fps seen on the phone with frame generation on comes from the scene, not from frame generation: Kakariko Village at night ran at 8.5 fps (124 ms frames, every present dropped) with frame generation off as well. That is open and outside this change.

Also tried and left out: frame-generation presents on a queue of their own, so that a present's wait for a free drawable does not hold back the game's next frame. With the present waiting for its snapshot on the GPU, the game blocked about 23 ms a frame on its fences (Mac field, 54-64 ms frames). Waiting for the snapshot on the CPU, it matched this change on the Mac and on the phone (26.2 fps), so it brings nothing.

Checks

  • dm/gfx/tests/t_framegen: a new check holds the Metal 4 queue once the link has taken p2, then asks for p1's slot. Before the fix the acquire is still waiting after 200 ms. After the fix it returns and p4 gets its snapshot. 30 of 30 checks pass with Metal validation. The test's alarm goes from 30 s to 60 s, because the test already took 26.6 s before this check.
  • python3 -m unittest dm/tools/tests/test_docs.py passes.
  • scripts/regress.py check was not run. With frame generation off (the default, and how regress runs), the changed code reduces exactly to the old rule (fg_on() is false), so no game in the regress set reaches it. The Mac and phone runs above are the check for the frame-generation path.

The new reference entry is "A frame-generation ring slot waits for the GPU" in docs/reference/frame-rate.md, with its line in Ocarina of Time's note.

@stan69b

stan69b commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

OoT-fps combined phone build, against the baseline (session A, A-2)

iPhone 17, Ocarina of Time, Kakariko Village by day, unattended drive script, 7 minutes each, back to back (2026-11-13, 22:20 and 22:36). Medians over the gameplay windows (more than 1,000 draws a frame).

baseline (512 ring) combined change
fps 28.6 29.4 +0.8
frame time p50 / p95 34.3 / 40.6 ms 33.6 / 37.1 ms p95 −3.5 ms
NVN submit+present p50 14.13 ms 12.25 ms −13 %
GPU p50 31.2 ms 30.4 ms −3 %
submit thread CPU a frame 13.47 ms at 2.05 GHz 12.00 ms at 1.82 GHz −11 % at a lower clock
instructions per draw 26,790 24,614 −8.1 %
draws per frame 2,680 2,770
share of CPU time on performance cores 49 % 51 %
CPU energy 0.99 W 0.84 W −15 %
residency set 3,499-3,561 MB 3,014-3,174 MB −0.4 GB
system wired 5,230-5,397 MB 4,808-4,950 MB −0.4 GB
memory-pressure warns 1 0
thermal state (max) 1 2
  • Both runs show the same screenshots and the same RED line: the stale pipeline archive message, which both builds print.
  • After each run, env.txt was put back byte for byte, and the save was not touched.
  • The game's real-time threads in the combined run were never demoted: 0 of 300 frames on the acquire, queue replay and encode threads.

Against A-1's targets:

  • Submit ≤ 15 ms: reached.
  • A steady 30 fps: nearly; 29.4 median, p95 37 ms.
  • Performance-core share ≥ 60 %: not reached (51 %).

An earlier pair that cannot be compared: the first pair (21:04-21:19) gave 11.4 fps for the baseline and 29.2 fps for the combined build. That gap is the time of day, not the change: the baseline's save loaded Kakariko at night (GPU p50 60 ms) and the combined build's by day, as B found (#2582). Every run's end.png was checked for time of day, and only day runs are compared above.

The combined build is the one installed on the phone.

@stan69b

stan69b commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main. The branch still carries #2241's commit (frame generation), now also rebased onto main:

  • its RED_ONCE keys are 175-178, because main already uses 164-167;
  • the snapshot copy ends through main's cb_end;
  • present_enqueue and t_framegen take main's copy argument.

seedds and others added 2 commits November 18, 2026 11:43
…th the 30 fps frame rate): MetalFX's frame interpolator makes the frame halfway between each two presents and the display link shows it between them, so a game held at 30 fps is shown at 60

Ported from cheemu21's Ocarina of Time fork (kid-with-sword-remake 7979a46, X2-X9), the frame generation part only; its pipeline-script capture (DM_PSO_CAPTURE) is left out. The offline archive's .mtl4x unwrap that came with the same hunks is left out: nothing packs a .mtl4x, and main has since rewritten how the bundled archives are opened (arch_plan, arch_admit).

- After each present's snapshot copy, in the same command buffer, MTL4FXFrameInterpolator makes the frame between the previous snapshot and this one into the slot's own texture; the display link shows it on the present's first tick and the present's own picture half an interval later. Depth and motion inputs are constant (far depth, no motion): the interpolator's optical flow finds the motion, so it needs nothing from a game's profile.
- A picture shown smaller than the game draws it is copied at the shown size, a crop alone 1:1; generation pauses while frames drop until three presents in a row come in time; the picture after a generated frame takes a present allocator of its own (3 to 5); the 30 fps cap keeps the 60 Hz link with frame generation on.
- RED_ONCE keys 175-178 (main uses up to 174).

Rebased onto main 31b537f for PR seedds#2582. Main's display link now holds a frame for whole ticks (tick_time, hold_ticks, tick_period) and
latches the woken acquire's present. Frame generation follows it: a present gets its generated frame when it is held two whole ticks or
more, its own picture is owed at half the hold (g_fg_owe_at on tick_time), and the running tick mean g_fg_tick is gone. The latch never
runs with it, because frame generation turns present snapshots on and the latch is off with them. t_framegen checks the link rate instead
of g_fg_tick.

Checked on the Mac (M1 Pro): t_framegen 29 checks, t_present_snap 14, t_present_alloc 13, t_link_ticks 43, all pass with Metal validation; test_docs, test_launcher_settings, test_settings. Tears of the Kingdom 1.4.3 at the Great Sky Island lookout, 30 fps cap with frame generation: interpolation p50 7.8-8.5 ms, p95 14.6-16.4 ms of GPU at 1920x1080 over 1,800 generated frames, the history never started again, one slow display-link callback (77 ms); the picture itself was not checked (the Mac's shader libraries for this chain were incomplete and every run drew black, frame generation on or off).

Checked after the rebase (Mac): t_framegen 29, check_t_present_snap (all modes), t_present_alloc 13, t_link_ticks 53, t_link_latch_cb 19,
with Metal validation.

Front-ends: new: the launcher setting DM_FRAMEGEN (Frame generation), a toggle in Game drawn while DM_FPS_TARGET is 30 (needs=DM_FPS_TARGET=30), NVN games

Co-authored-by: cheemu21 <[email protected]>
…ter its own is taken, not once that present has run on the GPU

With frame generation a ring slot's ticket is the present after its own,
because its picture is shown again after its generated frame. The acquire
and the copy waited until that present had run on the GPU, and it is
committed behind the game's next frame on the same queue, so the game lost
its frame of overlap with the GPU. Now the slot is free once the display
link has taken that present: the picture owed after the generated frame has
been committed or skipped by then, and the next copy, committed later on
the same queue, runs after it.

Ocarina of Time on the Mac (M1 Pro, Hyrule Field, 30 fps cap): the
snapshot ring blocked 68-89 % of frames for 8-16 ms, frame time p95
36-39 ms; after it 0 % in most windows, 33.5-33.8 ms (33.5 with frame
generation off). On an iPhone 17 in Kakariko Village by day: no
measurable change (24.9 fps against 26.2).

t_framegen holds the queue once the link has taken p2 and asks for p1's
slot. Its alarm goes from 30 to 60 s: the test took 26.6 s before this
check.

(cherry picked from commit 1065b7a)
@stan69b
stan69b force-pushed the pr/framegen-gpu-time branch from 9f398e2 to 2c2bd59 Compare November 18, 2026 10:45
@stan69b

stan69b commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main 31b537f, together with #2241's frame generation commit, which this PR builds on. That commit needed real changes, because main reworked the display-link callback after the last rebase:

  • Hold on counted ticks. Main holds a frame for whole ticks (tick_time, hold_ticks, tick_period) instead of targetPresentationTimestamp + period. A present now gets its generated frame when it is held two whole ticks or more, and its own picture is owed at half the hold (g_fg_owe_at on tick_time). The running tick mean g_fg_tick is gone; t_framegen checks the link rate instead.
  • Late latch. It never runs with frame generation: frame generation turns present snapshots on, and the latch is off with snapshots. The owed-picture check sits after main's early return, which now also clears an owed picture.
  • RED_ONCE keys are 175-178 (main uses up to 174).
  • Left out: the .mtl4x archive unwrap (pso_bundle_unwrap, t_pso_unwrap) that came in the same hunks. Nothing packs a .mtl4x, and main has rewritten how bundled archives are opened (arch_plan, arch_admit).

Checked on the rebased tree with Metal validation: t_framegen 30 checks in both encode modes, check_t_present_snap (all modes), t_present_alloc 13, t_link_ticks 53, t_link_latch_cb 19, plus test_docs, test_env_switches, test_launcher_settings and test_settings. Not run: a game with frame generation on, on the Mac or the iPhone.

stan69b pushed a commit to stan69b/nxnative that referenced this pull request Nov 18, 2026
RED_ONCE(160) is main's. The warning that a texture which left out its largest level could not be made whole
again now has key 179, past main's highest (174). Frame generation (seedds#2241, seedds#2582) takes 175-178.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 0400486024DEC000 A game nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants