Skip to content

Expose historical touch events - #4696

Open
Murmele wants to merge 31 commits into
rust-windowing:masterfrom
slint-ui:mm/move_history
Open

Murmele wants to merge 31 commits into
rust-windowing:masterfrom
slint-ui:mm/move_history

Conversation

@Murmele

@Murmele Murmele commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor
  • Tested on all platforms changed
  • Added an entry to the changelog module if knowledge of this change could be valuable to users
  • Updated documentation to reflect any user-facing changes, including notes of platform-specific behavior
  • Created or updated an example program if it would help users understand this functionality

Android and Ios do not publish touch events for every single touch movement, but collect them and publish them lower frequent. Using only the lower frequent events, velocity estimation is not appropriate (For a 50ms gesture we would get with 60Hz only 3 events).

Android and Apple expose additional touch event history which can be used.

Implemented for:

  • Android
  • IOs
  • Wayland
  • X11

@kchibisov We could further discuss if we wanna collect all events and send it out whenever the frame event occured? https://wayland-book.com/seat/touch.html (Clients should accumulate all wl_touch events as they're received, then process pending inputs as a single touch event when the "frame" event is received.)

Currently client-tool-kit just sends out the data when receiving the Frame event (https://github.com/Smithay/client-toolkit/blob/master/src/seat/touch.rs#L137-L205)

@Murmele
Murmele marked this pull request as ready for review September 28, 2026 08:49

@ogoffart ogoffart left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(i only looked at the API so far)

Comment thread winit-core/src/event.rs Outdated
/// Should be emitted regardless of window focus.
PointerMoved {
device_id: Option<DeviceId>,
event_time: Option<Duration>,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't that be an Instant?
(and all other event_time)

Also there should be more documentation that explain what these event_time are.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a duration against an unknown clock reference we don't know (On wayland it is not defined, so the idea is to use the first receiving event to calibrate the offset)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, then this needs to be properly documented.

An alternative is to make a newtype struct EventTime(/*priv*/ Duration) with function such as Sub to obtain a duration.

But if that is documented properly, then we can go with that.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used a type EventTime = Duration So all common functionality like wrapped sub and so on must not be implemented manually.

Comment thread winit-core/src/event.rs Resolved
Comment thread winit-core/src/event_loop/mod.rs Resolved
@ogoffart

ogoffart commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Also this adds an event_time to the mouse event, but not to the keyboad event. We might also need them there for completeness. Or?
Maybe the event time could be passed next to the event in the event function of the trait instead in within each event?
These are just some thought.

@Murmele

Murmele commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Also this adds an event_time to the mouse event, but not to the keyboad event. We might also need them there for completeness. Or? Maybe the event time could be passed next to the event in the event function of the trait instead in within each event? These are just some thought.

Android and ios have them also in the event it self, so you can easily pass the event around and don't have to think about to pass also the event time around. I will add to the keyboard as well.
Another idea would be to convert WindowEvent into a struct with a event_time and a event_type. event_type would represent then the enum we currently have?

@Murmele
Murmele requested a review from ogoffart October 7, 2026 10:59
Comment thread winit-core/src/event_loop/mod.rs Resolved
Comment thread winit-core/src/event.rs Resolved
@Murmele
Murmele requested a review from ogoffart October 13, 2026 05:00
Comment thread winit-core/src/event_loop/mod.rs Resolved
@ogoffart

Copy link
Copy Markdown
Contributor

Review from Claude:


Bugs

  1. iOS history has the wrong FingerId. The finger ID comes from the UITouch pointer (Retained::as_ptr(touch)). But coalescedTouchesForTouch returns separate UITouch objects, so every history entry's finger ID differs from the main event's. The history entries should reuse the main event's finger_id.
  2. iOS history duplicates the current sample. As far as I know, the coalesced array ends with the sample of the main touch itself. So the last history entry repeats the PointerMoved position and time. Android's history, by contrast, contains only earlier samples. Drop the last element on iOS, and say in the docs whether the current sample is included.
  3. Win32 keeps one extender per window. TimeStampExtender lives in WindowData. After a 49.7-day tick wrap, a window created later starts from the raw (small) tick value, while older windows hold extended values above 2³². Their times are then no longer comparable, which breaks the doc's promise of "comparable within the same event loop". The extender belongs on the thread or event loop, as X11 and Wayland already do it.
  4. Wayland key-repeat times can go backwards. Repeat times are extrapolated as press + delay + n·gap on the client. The real Released timestamp from the compositor can be earlier than the last synthesized repeat, so the stream isn't monotonic. Either clamp, or document that repeat times are approximate.

API and design feedback

  • Web is inconsistent. Web already splits getCoalescedEvents() into several PointerMoved events, each with an empty history. Android and iOS put the samples into history instead. Users will need different code per platform unless Web switches to history (or the docs explain the difference).

  • HistoricalMoveEvent is in the wrong module. It sits in event_loop but belongs in event, next to PointerSource and EventTime. Also, new() takes its arguments in a different order from the fields. A doc comment on source and position would help too.

  • EventTime = Duration is a weak type. The origin differs on every platform:

    • Wayland: compositor milliseconds
    • X11: server milliseconds
    • Win32: GetTickCount
    • macOS and iOS: system uptime
    • Android: nanoTime (only ms precision on GameActivity)
    • Web: performance.now

    A newtype with duration_since/checked_sub would stop people from treating it as an absolute time or comparing it with Instant. Per-platform notes on origin and resolution belong in the docs either way.

  • The docs should say that event_time can be synthesized. Some times are not hardware timestamps: X11 focus-in synthetic key events reuse the FocusIn time, iOS gestures and keyboard use systemUptime at dispatch, and Wayland repeats are extrapolated.

  • The field is Option everywhere, even where it's always Some. The Win32 message_time() helper and X11 event_time() return Option but never None, so they could return plain values. Only Orbital and a few Wayland events (enter, leave, touch cancel) really lack a time.

  • Gaps that might be worth a TODO:

    • Wayland tablet (the PR already leaves a TODO there)
    • Win32 GetPointerInfoHistory and GetMouseMovePointsEx
    • Android mouse history, which is currently skipped with continue
    • DeviceEvent timestamps, which aren't covered at all

Housekeeping

  • There's no entry in winit/src/changelog/unreleased.md. This is a breaking change: every user who builds these variants, or matches them without .., will fail to compile.
  • In the event.rs test macro, DoubleTapGesture { device_id: None, event_time: None, } has a stray trailing comma. The TouchpadPressure line is over 100 columns and puts the fields in a different order. rustfmt skips macro bodies, so CI won't catch either.
  • The commit history ("WIP", "Update Cargo.toml, …", repeated "fix clippy") should be squashed before merge.
  • The new CI step runs the winit-common tests only on linux-gnu. That's fine, since the extender is pure Rust.

Overall I'd request changes for bugs 1–3, and agree on how history works across platforms (especially Web vs. mobile) before this lands.

Murmele and others added 7 commits October 13, 2026 11:15
coalescedTouchesForTouch returns separate UITouch objects, so deriving the
FingerId from their address gave every history entry a different finger ID
than the PointerMoved event it belongs to.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The array returned by coalescedTouchesForTouch ends with the sample of the
main touch, so the last history entry duplicated the position and time of
the PointerMoved event. Drop it, matching Android where the history only
contains earlier samples, and document that the current sample is not part
of the history.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The extender lived in WindowData, so after the 49.7-day tick wrap a newly
created window started from the raw (small) tick value while older windows
reported extended values above 2^32, making event times from different
windows incomparable. Keep a single extender on the EventLoopRunner, like
the X11 and Wayland backends do.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Key repeats are synthesized on the client with times extrapolated as
press + delay + n * gap, so the real time of the following compositor event
(e.g. the release) can be earlier than the last synthesized repeat. Clamp
key event times to the last emitted one, and document that repeat times on
Wayland are approximate.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
It describes a pointer sample, so it belongs next to PointerSource and
EventTime rather than in event_loop. Order its fields like the arguments of
new() and like WindowEvent::PointerMoved, and document position and source.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Web split getCoalescedEvents() into one PointerMoved event per sample, each
with an empty history, while Android and iOS emit a single PointerMoved and
put the earlier samples into history. Emit one PointerMoved for the
dispatched pointermove event on Web as well and put the coalesced samples,
without the last one (the dispatched event itself), into history.

DeviceEvent::PointerMotion is still emitted once per coalesced sample.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@Murmele

Murmele commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor Author

Review from Claude:

Bugs

1. **iOS history has the wrong `FingerId`.** The finger ID comes from the `UITouch` pointer (`Retained::as_ptr(touch)`). But `coalescedTouchesForTouch` returns separate `UITouch` objects, so every history entry's finger ID differs from the main event's. The history entries should reuse the main event's `finger_id`.

2. **iOS history duplicates the current sample.** As far as I know, the coalesced array ends with the sample of the main touch itself. So the last history entry repeats the `PointerMoved` position and time. Android's history, by contrast, contains only earlier samples. Drop the last element on iOS, and say in the docs whether the current sample is included.

3. **Win32 keeps one extender per window.** `TimeStampExtender` lives in `WindowData`. After a 49.7-day tick wrap, a window created later starts from the raw (small) tick value, while older windows hold extended values above 2³². Their times are then no longer comparable, which breaks the doc's promise of "comparable within the same event loop". The extender belongs on the thread or event loop, as X11 and Wayland already do it.

4. **Wayland key-repeat times can go backwards.** Repeat times are extrapolated as `press + delay + n·gap` on the client. The real `Released` timestamp from the compositor can be earlier than the last synthesized repeat, so the stream isn't monotonic. Either clamp, or document that repeat times are approximate.

API and design feedback

* **Web is inconsistent.** Web already splits `getCoalescedEvents()` into several `PointerMoved` events, each with an empty `history`. Android and iOS put the samples into `history` instead. Users will need different code per platform unless Web switches to `history` (or the docs explain the difference).

* **`HistoricalMoveEvent` is in the wrong module.** It sits in `event_loop` but belongs in `event`, next to `PointerSource` and `EventTime`. Also, `new()` takes its arguments in a different order from the fields. A doc comment on `source` and `position` would help too.

* **`EventTime = Duration` is a weak type.** The origin differs on every platform:
  
  * Wayland: compositor milliseconds
  * X11: server milliseconds
  * Win32: `GetTickCount`
  * macOS and iOS: system uptime
  * Android: `nanoTime` (only ms precision on GameActivity)
  * Web: `performance.now`
  
  A newtype with `duration_since`/`checked_sub` would stop people from treating it as an absolute time or comparing it with `Instant`. Per-platform notes on origin and resolution belong in the docs either way.

* **The docs should say that `event_time` can be synthesized.** Some times are not hardware timestamps: X11 focus-in synthetic key events reuse the FocusIn time, iOS gestures and keyboard use `systemUptime` at dispatch, and Wayland repeats are extrapolated.

* **The field is `Option` everywhere, even where it's always `Some`.** The Win32 `message_time()` helper and X11 `event_time()` return `Option` but never `None`, so they could return plain values. Only Orbital and a few Wayland events (enter, leave, touch cancel) really lack a time.

* **Gaps that might be worth a TODO:**
  
  * Wayland tablet (the PR already leaves a TODO there)
  * Win32 `GetPointerInfoHistory` and `GetMouseMovePointsEx`
  * Android mouse history, which is currently skipped with `continue`
  * `DeviceEvent` timestamps, which aren't covered at all

Housekeeping

* There's no entry in `winit/src/changelog/unreleased.md`. This is a breaking change: every user who builds these variants, or matches them without `..`, will fail to compile.

* In the `event.rs` test macro, `DoubleTapGesture { device_id: None, event_time: None, }` has a stray trailing comma. The `TouchpadPressure` line is over 100 columns and puts the fields in a different order. rustfmt skips macro bodies, so CI won't catch either.

* The commit history ("WIP", "Update Cargo.toml, …", repeated "fix clippy") should be squashed before merge.

* The new CI step runs the winit-common tests only on `linux-gnu`. That's fine, since the extender is pure Rust.

Overall I'd request changes for bugs 1–3, and agree on how history works across platforms (especially Web vs. mobile) before this lands.

Thanks I fixed the issues in the last commits.
Regarding EventTime it is more convinient to have a plain Duration instead of an extra type. The documentation is clear that only the difference is meaningfull and we have the full capability of all Duration functions.

message_time() on windows and event_time() on X11 are just convinience functions and so we don't need to write all the time Some(...). It is an internal function so it is fine

Murmele and others added 3 commits November 5, 2026 08:42
#Conflicts:
#	winit-appkit/src/view.rs
#	winit-common/src/lib.rs
#	winit-core/src/event.rs
#	winit-wayland/src/seat/pointer/mod.rs
#	winit-win32/src/event_loop/runner.rs
#	winit-x11/src/event_processor.rs
#	winit/src/changelog/unreleased.md
@Murmele
Murmele requested a review from ogoffart November 9, 2026 13:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants