Skip to content

DEV-1147: consume a bare ACK's link CRC while streaming - #319

Merged
MAzalya merged 1 commit into
masterfrom
DEV-1147_streaming_bare_ack_crc
Oct 5, 2026
Merged

MAzalya merged 1 commit into
masterfrom
DEV-1147_streaming_bare_ack_crc

Conversation

@marknolan

@marknolan marknolan commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

Fixes DEV-1147.

Defect

  • With a Bluetooth link CRC on, the firmware appends one to every reply, a bare ACK included (ShimBt_sendRsp, log-and-stream-common Comms/shimmer_bt_uart.c). The reply to a SET command is therefore FF F4 65, or FF F4 with a 1-byte CRC: F4 65 is the CRC of the single byte 0xFF.
  • While streaming, processPacket() expected the byte after an ACK to be the next packet's 0x00 or an in-stream response's 0x8A. It took F4 for a parse error ("Unknown parsing error while streaming") and resynced to the next 0x00 or 0xFF, usually one inside the packet's own payload:
    • the data packet before the ACK was lost;
    • the ACK was thrown away too, so processAckFromSetCommand() never ran;
    • 2 s later checkForAckOrRespTask, seeing a stream, cleared the whole instruction queue.
  • The worst case is "start logging" on a streaming device. startSDLogging() queues SET_RWC_COMMAND and then START_LOGGING_ONLY_COMMAND. The RTC write's ACK is lost, the timeout clears the queue, and START_LOGGING is never sent, so the device never starts logging. Consensys's Live Data panel calls this on a streaming device. Reproduced against the real ACK timer: the queue went from 2 to 0 and isSDLogging() stayed false.
  • stopSDLogging() lost a packet and finished only on the timeout, whose STOP_LOGGING_ONLY special case reports success anyway.
  • In-stream responses (FF 8A …) were fine, because their CRC comes at the end. So was a bare ACK with the CRC off, which is followed by 0x00.

A second defect on the same path. checkCrc() discarded bytes itself when a 2-byte CRC failed on its second byte, and processPacket() then discarded again. The second discard skipped past the start of the next packet, so that packet was lost as well.

Fix

  • processPacket(), data packet followed by an ACK. When the bytes after the ACK are the bare-ACK CRC (isBareAckCrcSoFar(), checked against BARE_ACK_CRC), it waits for the whole CRC and then:
    • passes the data packet on if the packet's own CRC checks out. If it does not, only the packet is dropped, because the ACK's CRC has already checked out;
    • consumes the packet, the ACK and its CRC exactly once, keeping any later bytes with their PC timestamps (clearBytesFromStartOfBuffers());
    • processes the ACK (processAckWhileStreaming(), shared with the no-CRC path, which is otherwise unchanged).
  • Nothing else can be taken for a bare ACK. No response opcode is 0xF4 in any Shimmer3 or Shimmer3R firmware, and an in-stream response starts 0x8A.
  • checkCrc() only checks. Its caller decides what to discard.

Testing

New API_00027_StreamingBareAckCrcTest (10 tests, in ShimmerDriverPC). It feeds the streaming parser one byte at a time from an in-memory queue, as the IO thread does.

  • startSDLogging() while streaming, with a 1-byte CRC, a 2-byte CRC and none. Both ACKs are processed, START_LOGGING_ONLY is still queued after the RTC write, every packet is parsed once and in order, and the ACK timer is cancelled. The payloads include 0x00 and 0xFF, the bytes a resync stops on.
  • An in-stream VBATT response with a 1-byte and a 2-byte CRC. This already worked.
  • A corrupt packet before a bare ACK, including a 2-byte CRC wrong only in its second byte. Only the packet is dropped.
  • A bare ACK with the next frame already in the buffer. Only the packet, the ACK and its CRC are consumed, and each kept byte keeps its timestamp.
  • A 2-byte CRC failing on its second byte. Only that packet is dropped.

Proof the tests catch the bug.

  • On unfixed master, 7 of the 10 fail, and the 3 covering paths that already worked pass.
  • With this fix but checkCrc()'s own discard put back, only the 2-byte second-byte test fails.

Full suites (JDK 11, Gradle 8.14.3):

Command Result
cd ShimmerDriver && ./gradlew test 154 tests, 0 failures
cd ShimmerDriverPC && ./gradlew build -i BUILD SUCCESSFUL, 39 tests, 0 failures

The ShimmerDriverPC run includes the byte-level simulator suites that stream through this parser: _CRC_Streaming, _Throw_Bytes, _Throw_Bytes_2 and the contiguity checks.

Not verified on hardware

No device has been run against this. To check on one:

  1. In Consensys, keep the default CRC mode (1 byte) and start streaming.
  2. Start SD logging from the Live Data panel.
  3. Confirm the device reports logging within a second, and that no "Unknown parsing error while streaming" appears in the log.

Alongside the other open PRs

This branch trial-merges cleanly with #293 (DEV-982), #316 (DEV-976), #318 (DEV-1141) and #279 (DEV-895).

After merge: consumers

  • ShimmerAndroidAPI does not get this from a driver bump. Its Shimmer.java overrides processPacket() and processWhileStreaming() with its own copy of the parser, which has the same defect: on F4 it throws the packet away a byte at a time and never processes the ACK. It needs its own fix there.
    • It does call checkCrc() and discards again itself, so the double discard goes once it is on a driver with this. It pins shimmerdriver 0.11.7_beta.
  • Shimmer-MATLAB-ID vendors the driver as libs/ShimmerJavaClass.jar, so the jar needs rebuilding and re-vendoring.
  • Consensys builds the driver from source through sibling projects, so it gets this when built against a checkout that contains it.
  • ASM_PC needs no bump, because nothing there uses the Bluetooth driver.

Siblings (CLAUDE.md §4)

  • C# (Shimmer-C-API): stays in sync, but silently loses one packet per bare ACK.
    • The streaming case ACK_COMMAND in ReadData reads the byte after the ACK. Because it is not 0x8A, it drops the packet it is holding (KeepObjectCluster = null, ShimmerBluetooth.cs:1366-1369 on master).
    • It is back in frame at the next 0x00, the drop is not counted as packet loss, and nothing there waits for a streaming ACK.
    • It happens in normal use: with VBATT enabled, BuildMsg sends SET_BLINK_LED (0x30) as the battery crosses 3400 or 3000 mV, and firmware up to Shimmer3R v1.01.012 and Shimmer3 v1.01.003 ACKs 0x30.
    • Optional fix: if the byte after the ACK is F4, read the rest of the CRC and keep the held packet. DEV-1145: drop a bare ACK's link CRC while connected, so a stream started after a SET is seen Shimmer-C-API#213 (DEV-1145) does the same in the connected state and does not touch the streaming branch.
  • Web SDK Shimmer3RClient: needs a fix.
    • While streaming, every byte goes to the schema parser. It drops the packet before the ACK and then FF F4 (65) as junk, and recovers at the next packet.
    • The ACK never reaches _waitForAck, so the command rejects with "ACK timeout" after 1.5 s although the device ran it.
    • setRtcTime() (SET_RWC) has no streaming guard, and the clearAnchor() after its await then never runs.
  • Web SDK Shimmer3Client: has no link CRC, so it cannot meet this case.
  • Swift (SwiftAPI): cannot meet this case through its API, but has a broader defect.
    • The only SET commands it can send while streaming are on the firmware's blocked list, so the device NACKs them. It has no START/STOP_LOGGING, SET_RWC or LED write.
    • Its streaming loop (Shimmer3Protocol.swift:606-625) neither checks a packet's header nor resyncs, so any non-data frame (a NACK, an unsolicited status push, a bare ACK) shifts every later packet for the rest of the session.
    • A command awaiting its reply has no deadline: timeoutInSeconds is unused.
    • DEV-982: handle NACK from a Shimmer3 instead of leaving the caller suspended SwiftAPI#18 (DEV-982) handles the NACK only.
  • pyshimmer: its Bluetooth layer has no link CRC. Its only CRC code is in the dock UART API.
  • Within this repo:
    • LiteProtocol has its own processPacket() but no CRC support.
    • ShimmerGRPC and ShimmerPCBTBCove use ShimmerBluetooth's parser, so they get this fix.

🤖 Generated with Claude Code

With a Bluetooth link CRC on, the firmware appends one to a bare ACK too,
so a SET command sent while streaming is answered FF F4 65 (FF F4 in
1-byte mode). processPacket() expected the next packet's 0x00 or an
in-stream response's 0x8A after the ACK. It took F4 for a parse error and
resynced, throwing away the data packet before the ACK and the ACK itself.
2 s later the ACK timeout cleared the instruction queue, so
startSDLogging() on a streaming device never sent
START_LOGGING_ONLY_COMMAND, queued behind the SET_RWC_COMMAND whose ACK
was lost.

processPacket() now recognises the bare ACK's CRC, waits for all of it,
consumes the packet, the ACK and its CRC exactly once, and processes the
ACK. The packet is passed on if its own CRC checks out.

checkCrc() no longer discards bytes itself. Its caller already does, so a
2-byte CRC that failed on its second byte was discarded twice, past the
start of the next packet.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
marknolan added a commit to ShimmerResearch/ShimmerAndroidAPI that referenced this pull request Oct 4, 2026
AGENTS.md said this repo has no parsing logic of its own, so a protocol
fix never needs porting here. That is wrong for the Bluetooth stream
parser. Shimmer overrides processPacket(), processWhileStreaming(),
processInstreamResponse() and clearSingleDataPacketFromBuffers() from the
driver's ShimmerBluetooth, so a streaming-parse fix in the driver never
reaches it, even after a republish and pin bump. The bare-ACK link-CRC
fix in ShimmerResearch/Shimmer-Java-Android-API#319 is one that needs its
own port.

Decoding (processDataPacket() -> buildMsg()) and checkCrc() are still the
driver's, and Shimmer3BLEAndroid uses the driver's parser. The section
now says both.

Verified on master 44b9cbc against the pinned driver, API_v0.11.7_beta.
Docs only.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

@MAzalya MAzalya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok tested, looks good

@MAzalya
MAzalya merged commit adb09e9 into master Oct 5, 2026
2 checks passed
@MAzalya
MAzalya deleted the DEV-1147_streaming_bare_ack_crc branch October 5, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants