fix: correct SGR mouse-wheel button codes (64/65/66/67, not 68/69/70/71)

[smohekey]

Problem

TerminalMouseButton encodes the wheel buttons with the wrong values:

wheelUp(id: 64 + 4),    // = 68
wheelDown(id: 64 + 5),  // = 69
wheelLeft(id: 64 + 6),  // = 70
wheelRight(id: 64 + 7), // = 71

In the X10/SGR mouse protocol the four wheel buttons are 64/65/66/67: bit 6 set (the high-button/"wheel" bit, +64) with the button number carried in the low two bits. The enum adds the raw button number (4/5/6/7) on top of 64 instead of transposing it into the low bits, giving 68/69/70/71.

Those extra bits are the modifier bits of the encoding: 68 = 64 + 4 is wheel-up with Shift, 69 is wheel-down with Shift, and so on. So every wheel report the terminal emits carries a spurious Shift modifier.

Impact

Applications that check the button field reject the report as a modified/invalid wheel event and don't scroll. (More lenient apps mask the modifier bit and still scroll, which is why this often goes unnoticed.) On the alternate screen, TerminalScrollGestureHandler forwards wheel events through mouseInput, so a strict full-screen TUI never scrolls from the wheel.

Fix

Encode the wheel buttons as 64 + (0/1/2/3) — bit 6 plus the button number in the low two bits — matching the SGR spec and xterm.

Verified against an app that enables SGR mouse mode (?1006) with any-event tracking (?1003): a wheel-down previously emitted ESC[<69;…M and did nothing; it now emits ESC[<65;…M and the app scrolls. ESC[M (normal-encoding) reports are corrected the same way.

[benitoborriello]

Independent confirmation of this PR, with measurements — I hit this in
production and derived the same four values before finding the PR.

What happens

TerminalMouseButton gives the wheel buttons the ids 68/69/70/71. MouseReporter
passes button.id straight into the wire format (lib/src/core/mouse/reporter.dart:38
for SGR, :21 for normal, :44 for urxvt), so every wheel report leaves the
terminal with those values.

Verified on xterm 4.0.0 from pub.dev (the current latest release) and on
master at d35ba2cc — lib/src/core/mouse/button.dart lines 8/10/12/14 are
identical in both:

wheelUp     id= 68  ->  ESC [<68;31;13M
wheelDown   id= 69  ->  ESC [<69;31;13M
wheelLeft   id= 70  ->  ESC [<70;31;13M
wheelRight  id= 71  ->  ESC [<71;31;13M

Decoding 68 the way a receiving application does:

wheel(64)=true  SHIFT(4)=true  META(8)=false  CTRL(16)=false  low2=0

So the application is told Shift+wheel-up, not wheel-up.

The legacy ESC[M encoding is wrong in the same way, since it is
32 + button.id: wheel-up emits byte 100 (d) where xterm emits 96
(`).

What should happen

wheelUp     id= 64  ->  ESC [<64;31;13M     normal-mode byte 96
wheelDown   id= 65  ->  ESC [<65;31;13M
wheelLeft   id= 66  ->  ESC [<66;31;13M
wheelRight  id= 67  ->  ESC [<67;31;13M

Impact seen in the field

We ship xterm.dart in a Flutter terminal client. On a full-screen TUI the wheel
simply does nothing. Captured from the wire, 13 consecutive wheel events, all of
the form:

ESC [ < 68 ; 31 ; 13 M

The application received them as Shift+wheel, which it does not bind, and never
scrolled. Nothing is logged, nothing errors — the wheel is just inert, which is
why this is easy to misfile as a gesture-handling problem in the host app.

Why the current values are wrong — re: the comment on line 19

The existing comment is the origin of the bug, so it is worth answering directly:

/// Mouse wheel up / down use button IDs 4 = 0100 (binary) and 5 = 0101 (binary).
/// The bits three and four of the button are transposed by 64 and 128
/// respectively, when reporting the id of the button and have have to be
/// adjusted correspondingly.

The premise is right and the arithmetic contradicts it. "4 and 5" are the X11
button numbers — the name of the physical input. They are not the event code,
and they are not meant to be added to 64. The comment even says the correct
thing — the high bits are transposed, not summed — and then the code sums them.

ctlseqs is explicit that the wheel reuses the low-bit encoding of buttons 1
and 2:

Wheel mice
Wheel mice may return buttons 4 and 5. Those buttons are represented by the
same event codes as buttons 1 and 2 respectively, except that 64 is added to
the event code. Release events for the wheel buttons are not reported.

Buttons 1 and 2 have event codes 0 and 1. Add 64 → 64 and 65. By the same
rule the horizontal wheel (X11 buttons 6 and 7) takes the event codes of the next
two buttons, 2 and 3 → 66 and 67.

Meanwhile bit 6 (value 64) is already the flag that says "this is a wheel event",
and the bits the current code lands in are the modifier field, from the same
document:

The next three bits encode the modifiers which were down when the button was
pressed and are added together:
4=Shift, 8=Meta, and 16=Control.

So 64 + 4 is not "wheel button 4". It is wheel | Shift, i.e. 68 = wheel-up
with Shift held. 65 (wheel-down) and 69 (64+5) differ by exactly the Shift
bit, which is why the mistake is invisible in lenient applications that mask
modifiers off before dispatching, and fatal in strict ones that compare the whole
button field.

Normative cross-check: xterm.js

xterm.dart is a port of xterm.js, and xterm.js gets this right. From
src/common/services/CoreMouseService.ts (v5.3.0; the file is
MouseStateService.ts on master today, with this logic unchanged):

const enum Modifiers { SHIFT = 4, ALT = 8, CTRL = 16 }

function eventCode(e: ICoreMouseEvent, isSGR: boolean): number {
  let code = (e.ctrl ? Modifiers.CTRL : 0) | (e.shift ? Modifiers.SHIFT : 0) | (e.alt ? Modifiers.ALT : 0);
  if (e.button === CoreMouseButton.WHEEL) {
    code |= 64;
    code |= e.action;      // <-- the direction goes in the LOW bits
  } else {
    ...

with (src/common/Types.d.ts):

export const enum CoreMouseAction {
  UP = 0,     // buttons, wheel
  DOWN = 1,   // buttons, wheel
  LEFT = 2,   // wheel only
  RIGHT = 3,  // wheel only
  MOVE = 32   // buttons only
}

64 | 0/1/2/3 → 64, 65, 66, 67. Note that SHIFT = 4 is a modifier in
xterm.js and only ever OR-ed in from e.shift. In xterm.dart it is currently
baked into the button constant unconditionally.

Minimal reproducible example

Pure Dart — no widgets, no Flutter engine. xterm declares a Flutter SDK
dependency so the resolve step needs flutter pub get, but the program itself
runs under plain dart run.

pubspec.yaml:

name: wheelrepro
environment:
  sdk: '>=3.0.0 <4.0.0'
dependencies:
  xterm: 4.0.0

repro.dart:

import 'package:xterm/core.dart';

String vis(String s) => s.replaceAll('\x1b', 'ESC ');

void main() {
  final out = <String>[];
  final term = Terminal()..onOutput = out.add;

  // What any full-screen TUI does: mouse tracking (DECSET 1000)
  // + SGR extended coordinates (DECSET 1006).
  term.write('\x1b[?1000h\x1b[?1006h');

  const pos = CellOffset(30, 12); // reported 1-based as 31;13

  for (final b in [
    TerminalMouseButton.wheelUp,
    TerminalMouseButton.wheelDown,
    TerminalMouseButton.wheelLeft,
    TerminalMouseButton.wheelRight,
  ]) {
    out.clear();
    term.mouseInput(b, TerminalMouseButtonState.down, pos);
    print('${b.name.padRight(11)} id=${b.id.toString().padLeft(3)}  ->  ${vis(out.join())}');
  }

  final id = TerminalMouseButton.wheelUp.id;
  print('\ndecode of $id: wheel(64)=${id & 64 != 0}  SHIFT(4)=${id & 4 != 0}  '
      'META(8)=${id & 8 != 0}  CTRL(16)=${id & 16 != 0}  low2=${id & 3}');
}

flutter pub get && dart run repro.dart

Actual (xterm 4.0.0, and master d35ba2cc):

wheelUp     id= 68  ->  ESC [<68;31;13M
wheelDown   id= 69  ->  ESC [<69;31;13M
wheelLeft   id= 70  ->  ESC [<70;31;13M
wheelRight  id= 71  ->  ESC [<71;31;13M

decode of 68: wheel(64)=true  SHIFT(4)=true  META(8)=false  CTRL(16)=false  low2=0

Expected, and what this PR produces — I applied the diff to a local copy of
4.0.0 and re-ran:

wheelUp     id= 64  ->  ESC [<64;31;13M
wheelDown   id= 65  ->  ESC [<65;31;13M
wheelLeft   id= 66  ->  ESC [<66;31;13M
wheelRight  id= 67  ->  ESC [<67;31;13M

decode of 64: wheel(64)=true  SHIFT(4)=false  META(8)=false  CTRL(16)=false  low2=0

Swap \x1b[?1000h\x1b[?1006h for \x1b[?1000h alone to check the legacy
encoding: the button byte goes from 100 to 96, i.e. 32+64, matching xterm.

The patch

One value per line, exactly as this PR has it:

--- a/lib/src/core/mouse/button.dart
+++ b/lib/src/core/mouse/button.dart
@@
-  wheelUp(id: 64 + 4, isWheel: true),
+  wheelUp(id: 64 + 0, isWheel: true),
 
-  wheelDown(id: 64 + 5, isWheel: true),
+  wheelDown(id: 64 + 1, isWheel: true),
 
-  wheelLeft(id: 64 + 6, isWheel: true),
+  wheelLeft(id: 64 + 2, isWheel: true),
 
-  wheelRight(id: 64 + 7, isWheel: true),
+  wheelRight(id: 64 + 3, isWheel: true),

Because MouseReporter derives all three encodings from button.id, this single
change corrects SGR, normal and urxvt at once.

Regression check

The only other place id is read as a magnitude is
lib/src/core/mouse/handler.dart:71, event.button.id < 3, which
ClickMouseHandler uses to report only the first three buttons. The predicate is
unchanged by the patch:

left        id=  0  id<3=true   isWheel=false
middle      id=  1  id<3=true   isWheel=false
right       id=  2  id<3=true   isWheel=false
wheelUp     id= 64  id<3=false  isWheel=true      (was 68, also false)
wheelDown   id= 65  id<3=false  isWheel=true      (was 69)
wheelLeft   id= 66  id<3=false  isWheel=true      (was 70)
wheelRight  id= 67  id<3=false  isWheel=true      (was 71)

UpDownMouseHandler branches on isWheel, not on the numeric value, and still
suppresses wheel release events — which is what ctlseqs requires ("Release
events for the wheel buttons are not reported") and what xterm.js does by forcing
the final byte to M for wheel.

Environment

• xterm 4.0.0 from pub.dev (latest release) and master d35ba2cc
• Flutter 3.44.0 stable, Dart 3.12.0, macOS arm64

Happy to open a companion issue, add a regression test under test/, or split
the comment rewrite out if that makes review easier.