Skip to content

nvn/queue: replay completed nodes while the submit thread copies the rest - #2996

Open
atinseau wants to merge 1 commit into
seedds:mainfrom
atinseau:codex/nvn-submission-streaming
Open

atinseau wants to merge 1 commit into
seedds:mainfrom
atinseau:codex/nvn-submission-streaming

Conversation

@atinseau

@atinseau atinseau commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Closes #2987. Extracts the submission streaming change from #2912 onto 0b5fd6020093.

An NVN queue now accepts the submission before the game thread finishes copying it. At each wait or fence, the producer publishes the completed prefix under the queue mutex. A worker that drains that prefix keeps the open job at the head of its queue. It can service other ready queues until the producer publishes the suffix. Program snapshots pin the initialization serial and resolve on replay. Texture snapshots use the existing owner pins. Frame capture and DM_NVN_SUB_STREAM=0 retain whole-submission copying.

Mario Kart World race on M3 Max

Mario Bros. Circuit, Mushroom Cup, v1.0. Both graphics modes ran on the same 36 GB Mac on AC power (macOS 27.0.1, Xcode 27.2). The host, game modules, and shader library were built against the stated revision (chain 03a3ac831bfb). All runs used the same starting save, direct race boot, no player input, and a shared private pipeline archive. A separate warm-up race filled the archive before timing.

Uncapped, the same PR binary alternated streaming every 300 frames. Each mode ran twice with opposite starting states. For frames 1800–5399, the first 75 frames of each phase were discarded to let timing settle. Each row pools 2,700 frames over 12 phases. FPS is 1,000 divided by mean frame time. Lower frame and GPU times are better.

Graphics mode Streaming FPS Frame p50 (ms) Frame p95 (ms) GPU p50 (ms) GPU p95 (ms)
Docked off 76.1 12.61 17.33 8.96 9.91
Docked on 81.7 11.62 16.43 9.13 10.55
Handheld off 97.9 10.00 12.45 6.42 7.20
Handheld on 108.4 8.97 10.98 6.37 7.02

The streamed half had a lower median frame time in 9/12 matched docked phases and 11/12 handheld phases. Each match compares the same frame range across the two runs with the switch reversed.

Frame medians for matched 300-frame phases (ms)
Phase start Docked off Docked on Handheld off Handheld on
1800 12.69 11.95 10.23 9.79
2100 13.52 12.65 10.95 9.75
2400 12.81 11.60 11.11 9.09
2700 13.68 11.10 9.45 9.12
3000 11.42 10.94 9.72 8.75
3300 14.08 9.96 8.86 8.95
3600 11.12 13.38 10.35 8.82
3900 12.75 11.14 9.61 8.89
4200 11.63 14.11 10.45 8.57
4500 13.54 10.94 9.87 8.92
4800 11.27 14.84 10.55 8.46
5100 15.69 10.40 9.12 8.97

Separate fixed-setting docked runs

Run FPS Frame p50 (ms) Frame p95 (ms) Frames over 20 ms
Uncapped, PR streaming on (first pair) 75.50 11.71 16.02 49 / 3600
Uncapped, unmodified main (first pair) 76.79 12.70 17.31 90 / 3600
Uncapped, unmodified main (reverse order) 81.83 11.92 14.40 2 / 3600
Uncapped, PR streaming on (reverse order) 88.04 11.13 13.35 1 / 3600
60 fps target, PR streaming off 57.35 16.67 22.20 232 / 1800
60 fps target, PR streaming on 59.86 16.64 20.20 121 / 1800

In the first fixed docked pair, the PR improved median and p95 frame time, but its mean FPS fell from 76.79 to 75.50. Frame 2212 took 2,917.42 ms wall time. Its recorded GPU time was 15.09 ms and it created no new pipeline or shader library. The unmodified run’s longest frame was 166.39 ms. The cause of the long frame is unresolved. All frames are included in the fixed-run table.
The fixed comparison was repeated in reverse order after a separate archive warm-up. The longest frame was 20.25 ms on unmodified main and 34.67 ms with streaming. At the 60 fps target, the game selected a two-vsync step for 62/1,800 frames with streaming off and 0/1,800 with it on.

The fixed-setting runs use the full race window (3,600 uncapped frames or 1,800 capped frames). FPS in this table comes from elapsed wall time. Other desktop GPU activity was 10–22% before these runs. Desktop indexing also used CPU. Opposite phase orders reduce scene bias, but opponents and items can still vary between races.

Reproduction

  • Boot the same Mushroom Cup race and record DM_NVN_TSV. Use DM_OPMODE=1 for docked or DM_OPMODE=0 for handheld. For an uncapped run set DM_FPS_TARGET=0 DM_PRESENT_HZ=0 DM_PRESENT_LINK=0 DM_PRESENT_SCREEN=0 DM_VSYNC_HZ=240.
  • Run each mode with DM_AB=300 DM_NVN_SUB_STREAM=ab, then DM_AB=300 DM_NVN_SUB_STREAM=ab:0/2. The host logs each active phase as SUB_STREAM or base. For capped runs, use DM_FPS_TARGET=60 and fixed DM_NVN_SUB_STREAM=0 / 1 with the normal presentation settings.

Order, lifetime and regression checks

  • The new t_wait_event case pauses the producer while copying the command after a fence. The GPU completes the preceding copy while the producer is parked. A second submission on the same queue stays pending and reads the first job’s final value after the producer resumes.
  • t_queue_lifetime parks replay, finalizes and reuses the guest program and texture objects, then checks the rendered pixels. Together with t_wait_event, t_queue_origin, t_render_enable_queue, t_sub_overlap, t_serial_owner, and t_nvn_accel, it passed with streaming both enabled and disabled. The docs and switch checks passed.
  • The full graphics suite (git nx qtest --all, repeated in keep-going mode to run the rest) had one failure at the OpenGL test_lod_bias pixel assertion in t_gl_imports.m:427 on this M3 Max. Unmodified 0b5fd602 failed the same assertion with the same Metal validation settings. Every other test target completed successfully.
  • scripts/regress.py check --games 0400C3F00006E000 --quick passed on Mario Kart World, the available local Mac game build and the title from nvn: retest submission streaming in Mario Kart World #2987. Both revisions booted. All 11,559 shaders built on both sides, with no changed MSL, new interface failures, or new RED diagnostics. The harness marked the candidate boot as busy. The existing swizzled-texture CPU fallback also appeared in both race logs.

…rest

Keep an open job at the head of its queue until its producer closes it. Publish sync snapshots at each boundary and retain program serials and texture descriptors until replay is done. Exercise the gap between prefix and suffix with a blocked producer and a second submission.
@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation game: 0400C3F00006E000 A game profile labels Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 0400C3F00006E000 A game profile nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nvn: retest submission streaming in Mario Kart World

2 participants