Skip to content

host/input: keyboard, mouse and trackpad on the iPad, Mac mouse camera - #1490

Open
Xipzer wants to merge 1 commit into
seedds:mainfrom
Xipzer:kbm-input
Open

Xipzer wants to merge 1 commit into
seedds:mainfrom
Xipzer:kbm-input

Conversation

@Xipzer

@Xipzer Xipzer commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

A hardware keyboard, mouse and trackpad play on the iPad, and the mouse turns the camera on the Mac.

iPad keyboard: a hardware keyboard (GCKeyboard) plays player 1 with the Mac's key map. WASD the left stick, the arrow keys the D-pad (Shift + arrows the right stick), J K U I = A B X Y, Q E = L R, Z C = ZL ZR, F G = L3 R3, Return +, Tab −, Space shakes the controller. Keys pressed with ⌘ held stay the system's.

iPad mouse and trackpad: with Mouse: Camera (DM_MOUSE, new profile key mouse; default camera with pad_layout full, else touch) the pointer is locked through prefersPointerLocked, GCMouse movement is the right stick, and the buttons press the profile's mouse_buttons (default Y ZL R3). Esc (or `) or opening the in-game menu frees the pointer; a click on the picture locks it again. iPadOS only locks the pointer while the app fills the screen; otherwise, and with Mouse: Touch screen, the pointer stays a finger on the touch screen as before. The touch controls hide while the keyboard or mouse is in use and come back at the next finger touch. DM_MOUSE and DM_MOUSE_SENS are launcher settings; Mouse and trackpad is also in the in-game menu.

Mac: the same mouse camera. A click on the picture captures the pointer (CGAssociateMouseAndMouseCursorPosition), Esc or focus loss frees it. The movement is taken through a local event monitor, because AppKit sends mouse-moved events to the first responder, not to the view under the pointer.

mouse_look.h (shared by both front ends) turns reports into a stick deflection: the speed of each report is measured over at least 4 ms of reports, smoothed over 4 ms on 2 ms ticks, full tilt at 1000 points/s times DM_MOUSE_SENS, plus a 0.12 offset (DM_MOUSE_DEAD) to clear the game's dead zone. A 125 Hz mouse, a 1000 Hz mouse and a trackpad give the same speed for the same hand: with a synthetic report stream, a steady 500 pt/s gives the same deflection at 1, 8 and 16 ms report intervals.

Tested: Breath of the Wild on an iPad Pro 13" M4 with a Magic Keyboard and trackpad. On the Mac, synthetic clicks and movement captured the pointer and turned the right stick.

Checks: make -C dm/host GAME=01007EF00011E000 builds, compile_check.py reports 0 failed, and the env switches, launcher settings, game profile, docs, lint, architecture, settings and player env tests pass.

@Xipzer

Xipzer commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Note: I couldn't run scripts/regress.py check for this one. This Mac has Xcode 27.1 (Metal 32023.921), and the chain lock on main names 27.2's 32023.922, so the local games' shaders don't build against main's chain here. The change is front-end only (input), so shaders and rendering are untouched.

@Xipzer

Xipzer commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main (4b5e515); the controller prompt keeps the keyboard while it is up and frees a captured pointer when it opens, and the mouse settings sit before Press a button to join.

@Xipzer Xipzer changed the title Keyboard, mouse and trackpad on the iPad; mouse camera on the Mac host/input: keyboard, mouse and trackpad on the iPad, Mac mouse camera Nov 13, 2026
@Xipzer

Xipzer commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main (2e6edda), retitled to the area/component form. The only conflict was dmhost's dependency line in dm/host/Makefile.

Checked on this commit (M4 Max): host build, make -C dm/gfx/tests run (every target passes), the settings / docs / switch-table tool tests. The host tests pass except audio-device checks (t_audio's device stall here, t_audio_device on clean main in the same session).

@Xipzer

Xipzer commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Now that this Mac has the chain lock's compiler (Xcode 27.2), scripts/regress.py check on Breath of the Wild 1.9.0, main (2e6edda) against this branch, replacing my earlier note that I could not run it:

regress.py check (full): PASS

Baseline 2e6edda4d334feaf48326407c1718cb96c43e2de 2e6edda, candidate 14ec16f30d128737b4b2704a0d01e1ead382a891 14ec16f; Apple M4 Max, macOS 27.2.1, Xcode 27.2; 446.8 s

game status shaders (base → cand) newly failing newly building changed MSL interface (new / known) boot fps p50 frame p99 ms peak MB prepare s new RED
BOTW 01007EF00011E000 PASS 19592/19592 → 19592/19592 0 0 0 0 / 0 ok 59.7 → 59.8 18.13 → 17.5 3236 → 3185 0.21 → 0.05 0

BOTW 01007EF00011E000: PASS

  • screenshots (s: dHash bits / RGB diff / baseline noise): 60: 12/4.28/0, 80: 12/4.79/0
  • baseline: exit None, first frame 2.4 s, max frame 5928, reached True, prepare 0.21 s, prebuild 203.7 s (15767 built, 0 failed), fps 59.7, p50/p99 16.67/18.13 ms, peak 3236 MB
  • candidate: exit None, first frame 1.9 s, max frame 6090, reached True, prepare 0.05 s, prebuild 170.3 s (15767 built, 0 failed), fps 59.8, p50/p99 16.66/17.5 ms, peak 3185 MB
  • time: chain 165.8 s, smoke 280.8 s

The route's save is my own outdoors save, not the one smoke.tsv was written with. The route sends no keyboard or mouse input beyond its key presses, so it shows only that the change leaves a controller-less boot alone.

@cagatay-io

Copy link
Copy Markdown

A crash on iOS in this branch (head 14ec16f), found while carrying it on top of current main and running the iOS Duo harness (dm/ios/tests, iOS Simulator, Xcode 27.2):

ml_init (dm/host/mouse_look.h:55) creates a new dispatch timer on every call and drops the old one. The timer is created suspended, and releasing a suspended dispatch source traps in libdispatch. -[DmGamepad initWithFrame:] runs kbm_start (DmGamepad.m:244, 657), which calls ml_init, so the second gamepad view in a process ends the app:

EXC_BREAKPOINT (SIGTRAP)
_dispatch_queue_xref_dispose.cold.1
-[OS_dispatch_source _xref_dispose]
ml_init            mouse_look.h:57
kbm_start          DmGamepad.m:659
-[DmGamepad initWithFrame:]  DmGamepad.m:244

The harness makes a gamepad view per test and crashed at the second one. A one-line guard fixes it (a later call keeps the one timer and only takes the new output function and settings):

 static void ml_init(ml_out_fn out) {
     g_ml.out = out; ml_settings();
+    if (g_ml.timer) return;
     g_ml.timer = dispatch_source_create(DISPATCH_SOURCE_TYPE_TIMER, 0, 0, dispatch_get_main_queue());

With it the harness runs all its tests (39, 1 skipped, 0 failures). Not yet checked on a device; a run on an iPhone (A20 Pro, iOS 27.2) with a keyboard will follow.

iPad: a hardware keyboard (GCKeyboard) plays player 1 with the Mac's key map.
With Mouse: Camera (DM_MOUSE, profile key mouse: camera by default in 3D
games) the pointer is locked and a mouse's or trackpad's movement (GCMouse) is
the right stick; its buttons press the profile's mouse_buttons (default Y ZL
R3). Esc or the in-game menu frees the pointer, a click locks it again. The
touch controls hide while the keyboard or mouse is used.

Mac: the same mouse camera; a click on the picture captures the pointer. The
movement comes through a local event monitor, since AppKit sends mouse-moved
events to the first responder rather than the view under the pointer.

mouse_look.h turns report speed into stick deflection: speed measured over at
least 4 ms of reports, smoothed over 4 ms, full tilt at 1000 points a second
times DM_MOUSE_SENS, plus a 0.12 offset past the game's dead zone.
@Xipzer

Xipzer commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current main. The Makefile conflict went away because main now tracks header dependencies itself; on focus loss, input_release also releases main's Joy-Con 1 state next to the mouse capture.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation host The Horizon host: SVCs, services, threads, memory (dm/host) ios The iPhone and iPad app (dm/ios) mac The Mac front end and app (dm/mac, front_mac.m)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants