Skip to content

Record player input as DM_KEYS steps (DM_REC), for bug reports and smoke routes - #448

Closed
v0lumehi wants to merge 1 commit into
seedds:mainfrom
v0lumehi:dm-rec
Closed

v0lumehi wants to merge 1 commit into
seedds:mainfrom
v0lumehi:dm-rec

Conversation

@v0lumehi

Copy link
Copy Markdown
Contributor

Part of #313 ("Reproducible bug reports without sharing game data": DM_REC records a player's inputs with timestamps in DM_KEYS syntax).

Problem

A bug that needs a path through the game ("open the map, turn left, the sky goes white") can only be reproduced by someone who plays it the same way. The reporter can't send that path, and smoke routes for regress.py are written by hand.

Change

DM_REC=<file> writes what the player does as DM_KEYS steps. With the reporter's save, the same build and DM_DETCLOCK, feeding the file back as DM_KEYS replays the path.

  • Where it records: at the three functions every front end calls with physical input: dm_input_set (keyboard, the iOS touch gamepad), dm_pad_set_player (controllers) and dm_touch_set (mouse, iOS touches). It records player 1 as the game reads it, including controllers a one-player game merges into player 1. It lives in hle_hid.c, so iOS has it too. Scripted input (DM_KEYS, DM_CTL, Give scripted input a path of its own, so a connected controller cannot overwrite it #326) is never recorded; dm_touch_script keeps scripted touches out as well.
  • Format: one step per line, f<frame>:<keys>~f<frames>,. Idle stretches are left out.
    • Sticks are written in hundredths, and recorded again on a 0.02 move or a return to centre.
    • Touch is the first finger in thousandths, recorded again on a 0.005 move.
    • Shake is included. A press shorter than one frame is kept as ~f0.
  • Timestamps are frames, not seconds: under DM_DETCLOCK the guest clock follows the presents, so frame N of a replay is frame N of the recording. Seconds since start drift with load.
  • DM_KEYS grows a little so the recording can be replayed. Existing routes behave as before (tool tests, incl. regress.py's and bench.py's, pass):
    • ~f<frames> holds a step for that many frames;
    • a step due when such a hold ends follows it without a release, so buttons stay held and a dragged finger moves instead of lifting;
    • spaces and newlines separate steps like commas;
    • the value is no longer cut at 4,096 bytes, and the step limit is 1,048,576 instead of 1,024, ordered by a stable mergesort.
  • Cost: with DM_REC unset, one branch per input publish. When on, the input thread compares and formats into memory under a mutex. A writer thread appends to the file every 250 ms and at exit (atexit, and SIGTERM through dm_host_shutdown).
  • Docs: docs/development.md "Recording input (DM_REC)": how to pair it with DM_DETCLOCK and a copied save, and how a recording becomes a smoke route. Also 05-debugging.md, 06-dev-loop.md, and docs/env_switches.tsv.

Checked

  • t_rec in t_hid.c feeds a known sequence and checks the recorded text exactly. The sequence: keyboard A, a controller stick with drift, B tapped within one frame, a stick flick within one frame (dropped, as it should be), a dragged touch, ZR plus Shake, scripted presses that must not appear, and Y still held at exit.
  • It then parses the text back with the DM_KEYS parser and replays it on a simulated frame clock: the game's view matches in all 62 frames, and the drag replays as one finger. A second check confirms the merge sort keeps listed order at equal times. hid players: 130 checks, 0 failed, also under ThreadSanitizer.
  • make -C dm/host for Mario and Breath of the Wild; make -C dm/host/tests run; tool tests, test_docs, test_env_switches; lint ratchet 0 new.
  • A game run, partly: Breath of the Wild for 75 s with DM_REC and frame-held DM_KEYS steps. The steps ran in order, SIGTERM took the graceful exit, and the recording stayed empty, as it should with only scripted input. A recording of real keyboard input and its replay in a game is still to do.

dm/host/tests/Makefile and docs/development.md also change in #386; whichever lands second needs a small rebase.

Questions

  1. Is extending DM_KEYS (~f, the hand-over without a release, whitespace, the higher limit) fine, or should DM_REC keep to the old syntax?
  2. Scripted steps start within 5 ms after their frame's present, so a replayed press can land a frame early or late. Applying frame steps inside dm_guest_frame_tick would make it exact. Worth a follow-up?
  3. Players 2–8, a second finger and live motion aren't recorded, because DM_KEYS has no syntax for them yet.
  4. On iOS, a replay needs the recording joined to one line in env.txt. A DM_KEYS=@<file> form would help there.

Unrelated, seen while reading: dm/host/dm_input.h declares dm_pad_hook, DM_HID_RECEIPT_LOG and dm_input_sampled_right, and nothing implements them. Left alone here.

@v0lumehi
v0lumehi force-pushed the dm-rec branch 4 times, most recently from 409dbe7 to 816d610 Compare October 28, 2026 16:41
DM_REC=<file> writes player 1's physical input (keyboard, mouse touches,
player 1's controller and the controllers a one-player game merges into
it) to the file as DM_KEYS steps, so a run plays again from the same save
and build: a bug report's path without game data, or a smoke route
recorded instead of written (roadmap item in seedds#313). It records where the
front ends publish (dm_input_set, dm_pad_set_player, dm_touch_set), so the
Mac and iOS front ends both feed it; DM_KEYS and DM_CTL touch through a
new dm_touch_script and keep their own button path, so scripted input is
never recorded. Off, a publish costs one relaxed load and branch; on, the
publishing thread compares and formats into memory and a writer thread
appends to the file every 250 ms and at exit.

Each step is one state from the presented frame it began to the frame it
ended ("f1524:A+L0.50/-0.25~f12,"), because under DM_DETCLOCK the guest
clock follows the presents while seconds drift with load. DM_KEYS gains
what replaying that needs: "~f<frames>" holds in frames, and a step due
when such a hold ends follows it without a release (buttons stay held, a
dragged finger moves instead of lifting). Commas, spaces and newlines
separate steps, the parser takes the whole value (it cut at 4,096 bytes)
and up to 1,048,576 steps (was 1,024), ordered by a stable merge sort.
Existing routes behave as before. Sticks are kept to hundredths and
re-recorded on a 0.02 move; touch to thousandths and a 0.005 move.

Checked: t_hid's new round trip records a known sequence (keyboard,
controller stick with drift, a sub-frame tap, a dragged touch, shake, a
press held at exit, and scripted presses that must not appear), compares
the exact text, parses it back with the DM_KEYS parser and replays it on
the scripted path with a simulated frame clock: the game's view of every
frame matches the recording (130 checks, also under ThreadSanitizer). Host
unit tests, Mario and Breath of the Wild host builds, the Python tool
tests, docs and env-switch tests pass; lint shows no new findings. On the
Mac host, Breath of the Wild ran 75 s with DM_REC and frame-held DM_KEYS
steps: the steps ran in order, SIGTERM flushed through the graceful exit,
and the recording stayed empty (only scripted input). Recording real
keyboard input on the Mac was not tried.
seedds added a commit that referenced this pull request Oct 30, 2026
DM_REC=<file> writes player 1's physical input (keyboard, mouse touches,
player 1's controller and the controllers a one-player game merges into
it) to the file as DM_KEYS steps, so a run plays again from the same save
and build: a bug report's path without game data, or a smoke route
recorded instead of written (#313). It records where the front ends
publish (dm_input_set, dm_pad_set_player, dm_touch_set), so the Mac and
iOS front ends both feed it; DM_KEYS and DM_CTL touch through a new
dm_touch_script and keep their own button path, so scripted input is never
recorded. Off, a publish costs one relaxed load and branch; on, the
publishing thread compares and formats into memory and a writer thread
appends to the file every 250 ms and at exit.

Each step is one state from the presented frame it began to the frame it
ended ("f1524:A+L0.50/-0.25~f12,"), because under DM_DETCLOCK the guest
clock follows the presents while seconds drift with load. DM_KEYS gains
what replaying that needs: "~f<frames>" holds in frames, and a step that
begins where such a hold ends follows it without a release (buttons stay
held, a dragged finger moves instead of lifting). A gap between steps is
a release: the next frame step presses in a later frame than the release,
so two taps stay two presses when the replay runs late (handing over to
any step already due merged them into one press). Commas, spaces and
newlines separate steps, the parser takes the whole value and up to
1,048,576 steps, ordered by a stable merge sort. DM_KEYS=@<file> reads the
steps from a file: an environment value is capped by ARG_MAX (1 MiB on
macOS, about 40,000 recorded steps) and iOS's env.txt takes one line per
switch. A replayed press can still reach the game a frame early or late
(stamped when input arrives, replayed within 5 ms of the present); the
docs say so.

Checked: t_hid records a known sequence, compares the exact text, parses
it back and replays it with a simulated frame clock (the game's view of
every frame matches), replays two taps late (two presses; the old rule
fails this check) and reads a 50,000-step file of more than 1 MiB: 140
checks, also under ThreadSanitizer. Host unit tests, tool tests, docs and
env-switch tests pass; regress.py not run.

Co-authored-by: v0lumehi <[email protected]>
@seedds

seedds commented Oct 30, 2026

Copy link
Copy Markdown
Owner

Thanks! Landed as 96820ed with you as co-author. Two changes on top:

  1. Hand-over without a release now happens only when the next step begins where the hold ends (next.at <= end), not whenever the next step is already due. Before, a replay that woke late merged two separate taps (f11:A~f2, f14:A~f2) into one long press. A gap between steps is now always a release, and the next frame step presses no earlier than the frame after it, so the game reads the release. t_hid has a late-replay check for this; it fails on the old rule.
  2. DM_KEYS=@<file> (your question 4) reads the steps from a file. An environment value is capped by ARG_MAX (1 MiB on macOS, about 40,000 recorded steps), and iOS's env.txt takes one line per switch. A relative path is in Documents on iOS. The docs' replay recipe uses it now. The test reads a 50,000-step file of more than 1 MiB.

The docs now say a replayed step matches its recorded frame to within a frame, not exactly.

On your questions: (1) the DM_KEYS extensions are fine. (2) Yes, worth a follow-up: applying frame steps at the present and recording at the game's read would make replay frame-exact. It changes the timing of existing frame routes, so it belongs in a PR of its own. (3) Players 2–8, a second finger and motion can wait.

@seedds seedds closed this Oct 30, 2026
@seedds

seedds commented Oct 30, 2026

Copy link
Copy Markdown
Owner

Follow-up on question 2: replay is now frame-exact, in 710559c.

  • Recording: a state is stamped with the frame of the game's reads that saw it (rec_read, called from fill() for player 1), not with the moment the input arrived. A press that arrives just after a frame's read is recorded in the next frame, where the game saw it. Within a frame, buttons are ORed over all its reads, and sticks and touch come from the last read.
  • Replay: a list made only of frame steps held in frames, as DM_REC writes, is applied by keys_sync, which player 1's reads (controller state, touchscreen, gestures) call first. It applies every step due in the read's frame on the reading thread, so each read in frame N sees what frame N's reads saw in the recording. Lists with any step in seconds, including every existing smoke and bench route, still go through keys_main as before.
  • Cost: the publish path no longer calls the recorder. With DM_REC and DM_KEYS unset, a read costs two branches.

t_rec now also covers input that arrives after a frame's read and compares every frame exactly (143 checks, also under ThreadSanitizer). A recording of real play replayed in a game is still to do.

Know-It-All-Brother pushed a commit to Know-It-All-Brother/nxnative that referenced this pull request Nov 9, 2026
…edds#448)

DM_REC=<file> writes player 1's physical input (keyboard, mouse touches,
player 1's controller and the controllers a one-player game merges into
it) to the file as DM_KEYS steps, so a run plays again from the same save
and build: a bug report's path without game data, or a smoke route
recorded instead of written (seedds#313). It records where the front ends
publish (dm_input_set, dm_pad_set_player, dm_touch_set), so the Mac and
iOS front ends both feed it; DM_KEYS and DM_CTL touch through a new
dm_touch_script and keep their own button path, so scripted input is never
recorded. Off, a publish costs one relaxed load and branch; on, the
publishing thread compares and formats into memory and a writer thread
appends to the file every 250 ms and at exit.

Each step is one state from the presented frame it began to the frame it
ended ("f1524:A+L0.50/-0.25~f12,"), because under DM_DETCLOCK the guest
clock follows the presents while seconds drift with load. DM_KEYS gains
what replaying that needs: "~f<frames>" holds in frames, and a step that
begins where such a hold ends follows it without a release (buttons stay
held, a dragged finger moves instead of lifting). A gap between steps is
a release: the next frame step presses in a later frame than the release,
so two taps stay two presses when the replay runs late (handing over to
any step already due merged them into one press). Commas, spaces and
newlines separate steps, the parser takes the whole value and up to
1,048,576 steps, ordered by a stable merge sort. DM_KEYS=@<file> reads the
steps from a file: an environment value is capped by ARG_MAX (1 MiB on
macOS, about 40,000 recorded steps) and iOS's env.txt takes one line per
switch. A replayed press can still reach the game a frame early or late
(stamped when input arrives, replayed within 5 ms of the present); the
docs say so.

Checked: t_hid records a known sequence, compares the exact text, parses
it back and replays it with a simulated frame clock (the game's view of
every frame matches), replays two taps late (two presses; the old rule
fails this check) and reads a 50,000-step file of more than 1 MiB: 140
checks, also under ThreadSanitizer. Host unit tests, tool tests, docs and
env-switch tests pass; regress.py not run.

Co-authored-by: v0lumehi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants