Skip to content

Feat: AT-SPI2 accessibility for the GraalVM X11 window - #140

Closed
b-re-w wants to merge 40 commits into
developfrom
feat/x11-atspi
Closed

b-re-w wants to merge 40 commits into
developfrom
feat/x11-atspi

Conversation

@b-re-w

@b-re-w b-re-w commented Oct 4, 2026

Copy link
Copy Markdown
Member

Closes #139

Stacked on #134 (and so #128). Ports #131's platform-neutral AT-SPI code (AtspiWire, AtspiModel, AtspiServer, AtspiSemantics, AtspiBridge) and its AtspiTest into renderer/desktop/src/renderer, unchanged. The transport is the only difference: the Kotlin/Native window uses its posix socket, this window the JVM SocketChannel connection (now openable by address). New C names dxc_native_window_origin and dxc_native_window_focused answer on all three desktops.

Merging with #131: both add the same five source files. When #131 lands, its copies under renderer/linux/src should become symlinks to these in renderer/linux/src/shared, like the other shared files; the files are identical, so that is mechanical. I have not touched #131's branch.

The X11 window is still not the Linux default (see #134), so this is reached with DXC_X11_WINDOW until the flip. CI cannot run Orca or libatspi against it; the Docker check and a person can.

b-re-w added 30 commits October 4, 2026 17:33
The native-image renderer opened the AppKit window of its own only when
DXC_APPKIT_WINDOW was set. macOS now always opens it, and the older arrangement of a
second thread for the toolkit is gone from the C entry.

The window takes what the application asks for in its first batch: title, size, minimum
size, resizable, system or transparent title bar with a unified toolbar so the traffic
lights share the bar's line, the icon, and a material behind the page. Files dropped on a
FileDropTarget reach the node under them without the toolkit, with the full path list read
outside the 96 byte event record. Pointer and accessibility coordinates are converted
between points and pixels. The smoke test's auto exit is honoured.

Covers FR-27 (file drop) and NFR-14. fr27_* tests in NativeFileDropTest and
scripts/tests/macos-window-default.test.sh. Korean IME, VoiceOver and resize feel remain
manual checks on a Mac.
… (NFR-14)

An unattended macOS runner refuses TransformProcessType, and the window can still be made
once the activation policy is set to regular. The refusal is reported instead of stopping
the renderer.
A process told it has no display is given no clipboard by Compose desktop, so copy and
paste in a text field went nowhere. The window now supplies the clipboard from its own
pasteboard. The icon is passed as premultiplied RGBA bytes so that every desktop can read
the same record.
Windows: minimum size, resizable, icon, clipboard, OLE file drop with enter and leave,
the caption strip taken into the client area in the window's own procedure (moved from the
shim, which no longer subclasses the toolkit's frame), drawn buttons through window
actions, and the caret position for the IME candidate window.

Linux: CLIPBOARD and PRIMARY selections, text commit events, XIM preedit and commit
(callbacks style, falling back to the plain styles and to the plain lookup), XDND file
drop, _NET_WM_ICON, minimum size and non-resizable hints, undecorated chrome with a drawn
caption and edges that hand the move and resize to the window manager through
_NET_WM_MOVERESIZE, display scale from Xft.dpi or GDK_SCALE, and bring to front from a
notification press without the toolkit.

macOS shares the shell and gains the IME spot. RendererEntryPoints no longer falls back to
the toolkit's window on any desktop. AT-SPI is not included; it is built on the Kotlin/Native
window in #131.

Covers NFR-14 and FR-19 (windows chrome), FR-27 (file drop). scripts/tests/native-window-default.test.sh
replaces the macOS only test.
The toolkit window and everything only it used (chrome, icon, resize edges, file drop,
transparency, the JVM dev entry points) move to renderer/desktop/src/devshell. The shipped
sources get AWT-free replacements for what they still needed: the divider's resize icon,
the dropped path join, the Windows caption height. The input method and accessibility
reachability features go, with the force-loaded toolkit archive and its accessibility roots
on macOS, the look and feel and Metal layer registrations, and the CI step that checked the
toolkit's accessibility classes.

scripts/tests/no-awt-on-shipped-path.test.sh fails for a toolkit import in the shipped
sources and for toolkit classes in the metadata that are not on awt-forced.txt, the list of
what Compose forces (ClipEntry, PointerIcon constants, libawt loading), filed against the
fork as thisisthepy/compose-multiplatform-core-extended#15.

Covers NFR-14.
…debounce (FR-5)

TextChanged was held back 120 ms after the last keystroke and restarted by every key, so a
line echoing the typed text trailed the key by the pause plus a frame or two, and by the
whole burst when typing was fast. Measured on the AppKit window with real key events posted
to it: 152 ms from key to the echo being on screen, and the Host heard nothing during a burst.

The first edit after a quiet period is now sent immediately, later ones are collapsed to at
most one per interval carrying the latest text, and the last one is always sent. Covers the
FR-5 criterion that the Host is notified of edits through a debounced TextChanged: bursts
still collapse. Adds a test that a request for a frame is served by the next turn of the loop.
… not stretched (NFR-14)

AppKit stays in its own tracking loop while an edge is dragged and only calls the view back,
so the renderer's frame loop did not run: the layer's drawable size changed and the last
frame was shown scaled to it. Measured with a scripted resize of 120 sizes: 0 frames
presented and 120 sizes shown stretched.

setFrameSize now sets the drawable size and asks the renderer for a frame at that size through
the draw callback the Windows window already uses, a live resize presents with the window's
transaction, and the layer anchors a stale drawable to its corner instead of scaling it. Same
scripted resize after: 120 presented, 0 stale, 0 stretched, 6.1 ms per size on average.

Also adds the aids used to measure it: DXC_REPORT_LATENCY timestamps from key to pixels,
DXC_REPORT_RESIZE counts, DXC_SYNTH=type,keys,resize to drive the window from inside, and
DXC_LEGACY_RESIZE to see the old behaviour. The counting is in appkit_resize.h and is
tested without a window by scripts/tests/appkit-live-resize.test.sh.
…ed (NFR-14)

The X11 window code stays, reached by DXC_X11_WINDOW. Windows and macOS are unchanged.
The Linux default flips in a separate change with the Docker and hardware check results.
The wire server, object model, semantics reader and bus bridge come from the Kotlin/Native
Linux window (#131) unchanged, now compiled in the desktop module under renderer/, which is
where the other code both Linux renderers use already lives. Only the transport differs: this
module reaches the accessibility bus through its JVM socket connection, now opened by
address. The window reports its origin on screen and whether it has the keyboard, and its
loop publishes, signals and answers once a frame.

AtspiTest is the Kotlin/Native test, run on the JVM. Covers NFR-8 for Linux. Orca needs a
person; the X11 window is still not the Linux default.
…FR-14)

Unmapped keys answered 0, which is the A key, so control with C, V and X arrived as control
with A and selected everything. Letters and digits are mapped to the shared board, the Windows
virtual key table gains the same, XDND drops report where the last position said the files
were, and a shell test reads the X11 table.
A window with a click counter, two text fields echoed on screen (so shortcuts, paste and undo
are visible), the window size class, and a 60 row list with Korean rows. It is its own
workspace root so plain cargo run works on every desktop, and it reads the renderer from
COMPOSE_RUST_RENDERER_DIR. Prints event: lines.
… feat/win-linux-window-default

# Conflicts:
#	renderer/desktop/c/x11_window.c
#	renderer/desktop/src/AppKitWindow.kt
…nto feat/remove-awt-shipped-path

# Conflicts:
#	renderer/desktop/src/AccessibilityReachabilityFeature.kt
…lVM Linux renderers

The five Atspi files, AtspiTest and the two code point helpers now live once under
renderer/desktop and are symlinked into the Linux renderer, like the other shared sources.
… frame is not stretched (NFR-14)

The scaling field was left at its zero value from memset, which is
DXGI_SCALING_STRETCH. An ordinary resize never shows it, because the
buffer is refitted and redrawn at the new size before it is presented,
but a frame the resize handler skips (a refit refused, or dragging
starting before the swapchain exists) leaves DXGI filling the new
client area by stretching the old buffer instead of showing it at its
own size.
… 120 ms throttle (FR-5)

Typing in a field was held back by a 120 ms interval between notifications. An edit is now
told to the Host on the next frame, carrying the text the field holds then, so keys inside
one frame are one event and the last text is always sent. No measured reason for 120 ms
was recorded anywhere in the repository.
b-re-w added 10 commits October 4, 2026 22:29
…nto feat/remove-awt-shipped-path

# Conflicts:
#	.github/workflows/test-native-renderer.yml
…nto feat/x11-atspi

# Conflicts:
#	scripts/tests/linux-input-access.test.sh
…ves to the fork

Restores the macOS arm64, Linux x64 and Linux arm64 native-image build and smoke jobs that #113 removed, in their own path-filtered workflow. Windows stays in test-native-renderer.yml.
Resuming from withFrameNanos goes back through the scene's dispatcher, so the send and the
recomposition after it each cost a frame. Sent in the callback, the Host's batch is applied
before the same frame is composed. Measured on macOS: key to on-screen echo 26 ms to 12.7 ms.
@b-re-w

b-re-w commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

Superseded. The AT-SPI2 accessibility for the GraalVM X11 window landed in the Compose fork, where the window code lives now: thisisthepy/compose-multiplatform-core-extended#127 (protocol moved to extended/window/common/.../atspi, JvmBusConnection and X11Accessibility in graalvm/graalvm-linux, focus and origin in x11_window.c, AtspiTest in common). compose-rust #251 repins it and keeps only the glue in renderer/desktop/src/X11Window.kt, and runs the Korean input and AT-SPI check against the GraalVM X11 window in CI. The rest of this branch (#128 AppKit default, #134 Win32/X11 default) is tracked on those PRs.

@b-re-w b-re-w closed this Oct 17, 2026
b-re-w added a commit that referenced this pull request Oct 17, 2026
… fix

Pins the probe at the fork fix for scripted drags (core-extended #140, cherry-picked onto the
#251 pin) and makes resize_stale_ratio and resize_stretched a gate (PARITY_GATE) on both Linux
paths: a frame at a size other than the window's, or a size without a frame, fails the run.
b-re-w added a commit that referenced this pull request Oct 18, 2026
…R-3)

The newest extended, with core-extended #140 (a scripted drag asks for the next size only once
the last is confirmed and drawn) and #141 (both X11 windows resized through openbox and its
sync counter, stale 0 and stretched 0). The Linux parity runs gate on resize_stale_ratio and
resize_stretched on both paths, and keep the resize detail log for the artifacts.
@b-re-w
b-re-w deleted the feat/x11-atspi branch November 12, 2026 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant