Skip to content

DEV-1146: take an unsolicited status push off the receive buffer, in any state - #24

Merged
JongChern merged 2 commits into
mainfrom
DEV-1146_unsolicited_status_push
Nov 4, 2026
Merged

JongChern merged 2 commits into
mainfrom
DEV-1146_unsolicited_status_push

Conversation

@marknolan

@marknolan marknolan commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

DEV-1146: every Shimmer3 and Shimmer3R firmware pushes its status unasked whenever its state changes for a reason the host did not cause. This API had no handler for that push, so it hung every later command, or misaligned the stream.

What the firmware sends

[0xFF] 0x8A 0x71 <status…> [crc…], on a dock or undock, a user-button press, a trial-duration expiry or a low-battery stop. The host's own starts and stops do not push. The protocol is in log-and-stream-common docs/SHIMMER3_BT_COMMUNICATION_PROTOCOL.md, §3.2, §5.3 and §5.4.

  • ACK prefix: on by default. SET_INSTREAM_RESPONSE_ACK_PREFIX_STATE (0xA3) turns it off, and the firmware turns it back on at every disconnect.
  • Status width: 1 byte on Shimmer3. On Shimmer3R it is 1 byte before LogAndStream v1.00.024 and 2 bytes from that version.
  • CRC: the session CRC trails the push. It covers the ACK too (ShimBt_instreamStatusRespSend).

§5.4 sets two rules for a host:

  1. Accept a push at any time, including between a command and its reply, whether or not it is streaming.
  2. Never take a push's 0xFF for the ACK of a pending command.

What was happening

startProcessing() broke both rules:

  • Idle: it acted only when receivedBytes.first == 0xFF, then dispatched on commandSent.
    • A prefixed push was read as the pending command's reply, or left at the head of the buffer.
    • An unprefixed push was never consumed.
    • Either way, every later reply queued behind it. There are no command timeouts, so every later await hung.
  • Streaming: it sliced PacketSize+1 bytes without checking the 0x00 header, so one push misaligned every packet after it.

What this changes

  • One check at the head of every pass, in every state. Shimmer3StatusPush.frame looks there before either branch reads the buffer.
    • A complete push is taken off whole, with or without its prefix, CRC included.
    • A partial one is held until the rest arrives, so neither branch reads it.
    • A lone 0xFF is still an ACK.
  • Width. It comes from the hardware and firmware versions (statusPayloadBytes).
    • Before connect() has read them, the byte after the first status byte decides, because a second status byte is only ever 0x00 or 0x01.
    • With a CRC on, the CRC decides. Four one-byte pushes have a CRC whose low byte is 0x00 or 0x01: 0x52 and 0x71 prefixed, and 0x1F and 0x96 unprefixed.
  • The status reaches the app. It goes to deviceStatus and to a new delegate method, shimmerProtocolNewDeviceStatus(message:). The method has a default implementation, so the example apps' delegates compile unchanged. A push whose CRC fails is consumed, to keep framing, but not acted on.
  • A device-side stop ends the stream in the API too. A push that says streaming stopped returns the state to CONNECTED, because the firmware sends no packet after it (ShimSens_saveData sends only while btStreaming is set). Two exceptions:
    • Before the stream's first packet. START_STREAMING is ACKed before TASK_STARTSENSING sets btStreaming, so a push built in that gap says "stopped" and the stream starts anyway.
    • While a stop this API sent is waiting for its ACK.
  • sendStopStreamingCommand now returns true when not streaming. Only the streaming branch reads a stop's ACK, so a stop sent after the device stopped by itself would have blocked every later reply. Both example apps send one from a button without checking state. The check runs after the existing in-flight wait, so a Stop pressed during a Start still goes through.
  • The CRC moved to ShimmerUtilities.shimmerUartCrcCalc, so the framing and the tests share one copy. Shimmer3Protocol.shimmerUartCrcCalc now calls it, and the algorithm is unchanged.

Tests

Shimmer3StatusPush and Shimmer3DeviceStatus live in the new Shimmer3DeviceStatus.swift, because Shimmer3Protocol cannot be constructed in a test.

New Shimmer3DeviceStatusTest:

  • every status bit, and the 1- or 2-byte usbPluggedIn;
  • statusPayloadBytes across hardware, firmware identifier and version;
  • framing with and without the prefix, in each CRC mode, followed by a reply or by nothing;
  • every partial push;
  • what is not a push: an ACK, a NACK, a data packet, an ACK with its reply, the battery reply under 0x8A, and an ACK ahead of a push;
  • a corrupt CRC;
  • an exhaustive check of the width before the versions are known: every first status byte, both widths, both prefix states and every CRC mode.

New in ShimmerUtilitiesTest: the CRC against vectors from the firmware's own Extras/python_scripts/Shimmer_common/shimmer_crc.py. Four framing tests also use those vectors, so neither depends on this API's CRC agreeing with itself.

Each new file has its four pbxproj entries.

Not tested: the wiring in Shimmer3Protocol. That covers the call in startProcessing(), the state change, the delegate call and the stop guard. The class needs a live CBPeripheral.

Verification

Unbuilt and untested. This was written on a machine with no Swift toolchain, so this PR's CI run is the first compile and the first time any of it executes. A green check alone does not prove the tests ran. Confirm with gh run view <id> --log: look for ** TEST SUCCEEDED ** and a passed line for each Shimmer3DeviceStatusTest case.

The framing algorithm was prototyped in Python against the firmware's reference CRC before being written in Swift:

  • exhaustive over every first status byte, both widths, both prefix states and every CRC mode;
  • with a reply following, and with nothing following.

It had no misreads in any case a device can send. With no CRC, a one-byte push directly followed by 0x00 or 0x01 reads as two bytes. That cannot happen before the versions are known, because no stream is running then and no reply starts with either byte. That run is evidence for the algorithm, not for the Swift.

Hardware-verify:

  1. Dock, undock and press the button while idle, mid-connect and mid-stream, on a Shimmer3 and on a Shimmer3R on v1.00.024 or later.
  2. Check that the next command still answers.
  3. Check that a dock mid-stream returns the state to CONNECTED.

Not fixed here

  • Handlers that take or clear the whole buffer. The getInfoMem and getCalibDumpCommand handlers wait 0.5 s, then take the whole buffer as their reply. A push that lands in that window fails the reply's CRC, and connect() then force-unwraps the nil result. Several other handlers end with receivedBytes.removeAll() and drop a push queued behind their reply. Both behaviours predate this change.
  • ACK prefix off with no CRC. An ACK followed immediately by an unprefixed push reads as one prefixed push. This API never turns the prefix off.

Merging with the open work

Checked with git merge-tree against each branch:

Sibling APIs

The same push handling was audited in the other host APIs: Java and C# at master, pyshimmer at main.

API Needs the fix? What breaks
Java (ShimmerBluetooth.java) Yes A push's 0xFF is taken for a pending command's ACK, idle and streaming alike. The idle path drops an unprefixed push unread (clearSerialBuffer()). Mid-stream, a push not directly after a data packet is lost. The push CRC is read but never checked. The width is right.
C# (ShimmerBluetooth.cs, ShimmerLogAndStream.cs) Yes A push's 0xFF is taken as the START_STREAMING ACK. An unprefixed push mid-stream is not recognised. Before the firmware version is read, a push is handled as command bytes. The push CRC is discarded unchecked. The width is right.
pyshimmer Yes A prefixed push kills the read loop: _ack_queue.get_nowait() raises on an empty queue. It disables the prefix by default from LogAndStream 0.15.4, but is exposed before that, and before the 0xA3 is ACKed. The width is fixed in open seemoo-lab/pyshimmer#49. It has no Bluetooth CRC.
TypeScript (shimmer-web-sdk) Reference Shimmer3RClient handles the push. DEV-1142 covers Shimmer3Client.

🤖 Generated with Claude Code

…any state

The firmware pushes [0xFF] 0x8A 0x71 <status> [crc] whenever its state
changes for a reason the host did not cause: a dock or undock, the user
button, a trial-duration expiry or a low-battery stop (log-and-stream-common
docs/SHIMMER3_BT_COMMUNICATION_PROTOCOL.md §5.4). startProcessing() knew
nothing of it:

- Idle, it acted only on a leading 0xFF and dispatched on commandSent. A
  prefixed push was read as the waiting command's reply or left at the head
  of the buffer, and an unprefixed one was never consumed. Every later
  reply queued behind it, and with no command timeouts every later call
  hung.
- Streaming, it sliced PacketSize+1 bytes without looking at the 0x00
  header, so one push shifted every packet after it.

Every pass now asks Shimmer3StatusPush.frame first, in any state and
whatever command is waiting. A push comes off whole, with or without its
ACK prefix and with its session CRC. Its status goes to deviceStatus and to
a new delegate method, shimmerProtocolNewDeviceStatus, which has a default
so existing delegates compile unchanged.

- Width: one status byte, or two on a Shimmer3R from LogAndStream v1.00.024
  (§5.3). Before connect() has read the versions, the byte after the first
  decides, and with a CRC on, the CRC does.
- A push saying streaming stopped, once the stream has delivered a packet,
  returns the state to CONNECTED; the firmware sends no packet after it.
  So sendStopStreamingCommand now returns at once when not streaming,
  because only the streaming branch reads a stop's ACK.
- The CRC moved to ShimmerUtilities so the framing and the tests share one
  copy; Shimmer3Protocol.shimmerUartCrcCalc calls it.

Shimmer3DeviceStatusTest covers the framing, sizing and decoding, and
ShimmerUtilitiesTest checks the CRC against the firmware's Python
reference. The receive loop itself has no test: Shimmer3Protocol needs a
live CBPeripheral.

Unbuilt and untested: written without Xcode, so CI is the first compile.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The one conflict is at the head of the receive loop, where #18's NACK
check and this branch's status-push check both sit ahead of the state
branches. They are chained as if push / else if NACK / else if STREAMING;
the order does not matter, because a NACK starts 0xFE and a push 0xFF
or 0x8A.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@JongChern
JongChern merged commit afe1df9 into main Nov 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants