Repository navigation
Handle NACK from a Shimmer3/3R instead of dropping the connection - #293
Conversation
…ction The Shimmer3/Shimmer3R protocol has had a NACK_COMMAND_PROCESSED (0xFE) reply for a long time, but the driver never learned about it. There was no NACK constant anywhere on the Shimmer3 side, and both protocol implementations tested only for ACK with no else branch, so a NACK was read, silently discarded, and the transaction stalled with mWaitForAck still true and the instruction stack still locked. Two seconds later the ACK timer expired and checkForAckOrRespTask called connectionLost(): a refused command was indistinguishable from a dead link, and there was no retry to soften it because NUMBER_OF_TX_RETRIES_LIMIT is 0 and reading the NACK byte itself resets mNumberofTXRetriesCount, making the soft-recovery branch dead code. This is already reachable on shipping firmware - any SET command while the device is sensing, a sync-mode mismatch, a failed SET_DAUGHTER_CARD_MEM, and the BMP180/BMP280 calibration reads on a Shimmer3R all NACK. The existing coping strategy was avoidance rather than handling; see the comment on GET_PRESSURE_CALIBRATION_COEFFICIENTS_COMMAND, which is skipped for a BMP581 specifically because the firmware NACKs it. A refusal now unwinds the transaction and keeps the connection: - ShimmerObject: add NACK_COMMAND_PROCESSED = (byte) 0xFE. - ShimmerBluetooth.processNackFromCommand(): stop the ACK timer, clear mWaitForAck/mWaitForResponse, drop the refused instruction so the stack advances (as both ACK paths do for their own command), complete the transaction and release the stack lock. - Handled in three places: waiting for an ACK while not streaming (the main path), waiting for a response while not streaming, and a data packet followed by a NACK while streaming, which is the SET-while-sensing case. - eventCommandRefused() is the application hook. It is deliberately concrete with an empty default rather than abstract, and not a new ProtocolListener callback, so every existing subclass and listener implementation keeps compiling. Override it to surface the refusal. Two supporting changes: - The resync helper accepted only 0x00 and 0xFF as a packet boundary, so a NACK in the buffer could not be skipped past cleanly. It now accepts 0xFE too, and is renamed from findOffsetOfNextZeroOrFF to findOffsetOfNextPacketBoundary to match what it does. - NACK is registered in mBtCommandMapOther so btCommandToString names it in the logs. Deliberately not in mBtResponseMap: isKnownResponse() must stay false for it, or processResponseCommand() would try to parse it as a response body. A stray NACK arriving with no command in flight is logged and ignored rather than turned into a refusal, in processBytesAvailableAndInstreamSupported() which runs only when !mWaitForAck && !mWaitForResponse. The TypeScript SDK gates its own NACK framing the same way, so a leaked stream byte cannot fabricate a refusal. LiteProtocol gets the same treatment. Its NACK value is declared locally because the generated LiteProtocolInstructionSet has no NACK entry; adding it to LiteProtocolInstructionSet.proto and regenerating would be tidier but rewrites a large generated file committed in two places for one constant. Compile-verified only - there is no test harness for the BT receive path (ShimmerBluetooth is abstract with many abstract I/O methods and no test instantiates it), and no hardware was involved. Co-Authored-By: Claude Opus 5 <[email protected]>
There was a problem hiding this comment.
🟡 Changes recommended
There is at least one confirmed transaction-stack correctness bug (and an uncovered NACK path in LiteProtocol) that can cause queued commands to be dropped or time out incorrectly.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates the Shimmer driver to recognize and handle NACK_COMMAND_PROCESSED (0xFE) so a refused command unwinds the in-flight transaction and the connection is not dropped due to an ACK/response timeout.
Changes:
- Adds a protocol constant for NACK (
0xFE) and registers it for logging/command-to-string mapping. - Handles NACK in
ShimmerBluetoothacross ACK-wait, response-wait, and streaming (data packet followed by NACK) paths. - Adds analogous NACK handling and an overridable
eventCommandRefused(...)hook inLiteProtocol.
File summaries
| File | Description |
|---|---|
| ShimmerDriver/src/main/java/com/shimmerresearch/driver/ShimmerObject.java | Adds NACK_COMMAND_PROCESSED constant (0xFE). |
| ShimmerDriver/src/main/java/com/shimmerresearch/comms/radioProtocol/LiteProtocol.java | Introduces NACK constant and transaction-unwind handling plus eventCommandRefused(...). |
| ShimmerDriver/src/main/java/com/shimmerresearch/bluetooth/ShimmerBluetooth.java | Adds NACK support in multiple receive paths, updates packet-boundary resync, and adds eventCommandRefused(...). |
Review details
Suppressed comments (1)
ShimmerDriver/src/main/java/com/shimmerresearch/bluetooth/ShimmerBluetooth.java:1379
- findOffsetOfNextPacketBoundary() uses a raw
0instead of the existing DATA_PACKET constant; now that this method was touched, using the named constant would avoid a magic number and match the surrounding code style (e.g., bufferTemp[0]==DATA_PACKET).
byte b = buffer[i];
if (b == 0 || b == ACK_COMMAND_PROCESSED || b == NACK_COMMAND_PROCESSED)
return i;
- Files reviewed: 3/3 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…pilot review) processNackFromCommand() removed instruction 0 unconditionally, but a GET's instruction is already removed when its ACK arrives. A NACK arriving in the response-wait state therefore dropped whatever was queued behind it and silently skipped that command. It is now dropped only when mWaitForAck was still set, which is exactly the state that still has it queued. Same fix in LiteProtocol, and the javadoc no longer claims the drop is unconditional. Also from the review: - LiteProtocol was missing the response-wait NACK branch that ShimmerBluetooth has, so a NACK there would still have fallen through to the timeout. Added, checked before isKnownResponseByte() to match. - The resync helper's inline comment still said it looked for DATA_PACKET or ACK only. - The helper used a raw 0 rather than the DATA_PACKET constant it now sits alongside (suppressed review comment). Co-Authored-By: Claude Opus 5 <[email protected]>
|
All four review items are addressed in Suppressed comment ( Summary of the round:
The PR description has been updated rather than just corrected here, since the first version described the instruction drop as unconditional and a future reader could otherwise "fix" the code back to that. Still compile-verified only, and still no automated coverage of the receive path — the transaction-stack bug is a good argument for the fake-transport harness noted in the description, since a test would have caught it and review only just did. |
There was a problem hiding this comment.
🔵 Needs a closer look
It changes low-level transport state-machine behavior (including streaming parsing) without an automated/hardware-backed test to validate the end-to-end receive/unwind paths.
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 0 new
- Review effort level: Lite
processNackFromCommand() unwound the transaction but sent no progress report, so an operation containing a refused command never reached its end value. BluetoothProgressReportPerDevice.updateProgress() derives the counter from the remaining instruction stack size, and ShimmerDeviceCallbackAdapter only calls finishOperation() when that counter equals mProgressEndValue - so a single NACK left the operation running indefinitely, with mOperationUnderway still true and the application sitting in CONFIGURING/CONNECTING. This matters most at connect time. With SD sync enabled the firmware NACKs every command except SET_SD_SYNC_COMMAND and ACK (ShimBt_isCmdAllowedWhileSdSyncing), which covers the whole driver initialisation sequence. The report is sent before the refused instruction is removed, and only while it is still queued, to match the arithmetic on the ACK path: the response-wait path already reported when its ACK arrived, so reporting again there would skip a count. The three timer-driven commands the ACK path excludes are excluded here too. Known limitation: a refused command still counts as progress and the operation completes as SUCCESS. Reporting a partially-refused operation as FAIL_WITH_WARNING would mean changing the shared OperationState handling in finishOperation(), which is a wider change than this fix; eventCommandRefused() remains the hook for an application that needs to know. Co-Authored-By: Claude Opus 5 <[email protected]>
The two earlier harnesses both refused a command while sensing, so both took the mIsStreaming branch of checkForAckOrRespTask - which clears the instruction stack but leaves the connection up. The ~2 s teardown the ticket describes is in the other branch, reached only by a refusal while not streaming, and nothing exercised it. Configuring mid-stream is not a supported operation either, so those runs were testing a path the product does not use. This drives the supported case. With SD sync enabled the firmware refuses every command except SET_SD_SYNC_COMMAND and ACK (ShimBt_isCmdAllowedWhileSdSyncing), so the driver's whole initialisation sequence is NACKed at connect time. On master that ends in connectionLost() about two seconds in; on this branch the refusals are recognised and the link stays up. Nothing is written to the device, so the run is non-destructive - SD sync is set once in Consensys beforehand and cleared afterwards. The test reports INCONCLUSIVE rather than a pass if FULLY_INITIALIZED arrives, which means sync was not actually enabled. Compiled against ShimmerPCBasicExamples with Gradle 8.10, the version CI uses. The checked-in wrapper (6.1) cannot configure ShimmerDriverPC at all - its Shadow plugin requires Gradle 7+ - which is more evidence the stale wrapper deserves its own ticket. Co-Authored-By: Claude Opus 5 <[email protected]>
|
I reviewed this against current Checked and fine
SD sync can't trigger the not-streaming NACKEnabling SD sync turns the Bluetooth radio off while idle. In We ran So:
Suggested replacement for the bench testUse a command the firmware always refuses but that changes nothing, during a normal connection with SD sync off:
The driver has no public writer for these, so the test can queue the bytes the way // after FULLY_INITIALIZED
device.getListofInstructions().add(new byte[] {0x30, 0x00}); // SET_CHARGE_STATUS_LED: always NACKed
device.getListofInstructions().add(new byte[] {ShimmerObject.GET_SAMPLING_RATE_COMMAND});Follow-up (non-blocking)On the streaming branch (data packet followed by NACK), Once the description is corrected and the reworked test passes on hardware, I'm happy to approve. |
Review found the bench test could not work: with SD sync enabled the firmware switches Bluetooth off while idle, so the port never opened, and the test still reported "master behaviour" because it took any DISCONNECTED for a teardown. It now refuses a command the firmware always refuses and that changes nothing: 0x30 with one argument byte (the driver's SET_BLINK_LED, the firmware's SET_CHARGE_STATUS_LED_COMMAND, dispatched but never implemented), with GET_SAMPLING_RATE queued straight behind it. It follows those two instructions through the stack and gives a verdict only once the Shimmer has finished connecting, is idle, and has firmware that refuses 0x30 (Shimmer3R LogAndStream v1.01.013+, Shimmer3 v1.01.004+): - PASS: 0x30 retired, the GET answered, the link up 8 s later; - FAIL: the link torn down, or either instruction never completed; - INCONCLUSIVE: otherwise. The window is 8 s because when not streaming the ACK timer gives a SET ACK_TIMER_DURATION + 3 = 5 s, not 2 s. It moves to com.shimmerresearch.bluetooth, since the instruction list and the transaction state are protected, and takes the COM port as its argument. On a Shimmer3R, LogAndStream v1.01.025, over classic Bluetooth, 3 runs each: this branch PASS 3/3 (0x30 retired after 137-224 ms, the GET answered after 241-272 ms); master FAIL 3/3 (the link lost 5.0 s after the refusal, no NACK logged). Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
Thanks. All of it is addressed, and the description is updated.
A side note: 0x30 is the driver's |
JongChern
left a comment
There was a problem hiding this comment.
Approving. The idle path is right: a refusal unwinds the transaction, drops the instruction only while it is still queued, reports progress so an operation still completes, and keeps the link up. The hardware run (3/3 here, 3/3 failing on master) covers the check Mas asked for.
Merge order:
- Merge this and #345 back to back, before any release. This PR's streaming NACK branch is the first version that #345 replaces (it doesn't wait for the NACK's CRC, clears the rest of the buffer, and fails a command with none in flight).
- Retarget #345 to
masterbefore deleting this branch, or GitHub closes #345. - Then the Java HIL at
master, the Java release, the Android driver bump with ShimmerResearch/ShimmerAndroidAPI#153, and only then ShimmerResearch/log-and-stream-common#228 out of draft.
Not blocking: a NACK that arrives while a stream has stalled (no data packet before it) still lands at the start of the buffer and isn't handled. Master behaves the same.
The protocol has had a
NACK_COMMAND_PROCESSED(0xFE) reply for a long time, butthe driver never learned about it. Grepping the whole driver for
NACK— excludingthe generated instruction set and the unrelated Verisense protocol — returned two
comments and nothing else.
Since the review (10 Nov)
mastermerged in (8a8a771f, no conflicts); ShimmerDriver 178/178, ShimmerDriverPC 103/103, CI green.ACK_TIMER_DURATION+ 3 s before the ACK timer runs out.TestNackNotStreamingis reworked as suggested in the review: it refuses 0x30 (always NACKed, changes nothing) with a GET queued behind it, and gives a verdict only on a connected, idle unit whose firmware refuses 0x30. It was run on hardware: this branch PASS 3/3,masterFAIL 3/3 (see Verification).What was happening
Both protocol implementations tested only for ACK, with no
else:ShimmerBluetooth.java:833LiteProtocol.java:569So a NACK was read, silently discarded, and the transaction stalled with
mWaitForAckstill true and the instruction stack still locked. About five secondslater the ACK timer expired (when not streaming, a SET gets
ACK_TIMER_DURATION+ 3 s)and
checkForAckOrRespTaskcalledconnectionLost().There was no retry to soften it.
NUMBER_OF_TX_RETRIES_LIMITis0, and readingthe NACK byte itself resets
mNumberofTXRetriesCountto0, so0 >= 0alwaysholds, one of the two
connectionLost()branches always won, and the soft-recoveryelsewas dead code.A refused command was indistinguishable from a dead link, and cost the
connection.
Where this actually bites
checkForAckOrRespTaskbranches on whether the device is streaming. TheconnectionLost()teardown above is the not-streaming branch; while streamingthe same timeout instead calls
clearAllInstructions(), which destroys the queuedinstructions but leaves the link up.
Configuring a device mid-stream is not a supported operation — the only commands
sent while sensing are start/stop onboard recording and stop streaming, and
START_LOGGING_ONLY_COMMAND/STOP_LOGGING_ONLY_COMMANDalready have their ownspecial-cased timeout handling in
checkForAckOrRespTaskthat avoidsconnectionLost(). So the not-streaming branch is the one that matters, and anearlier revision of this description had that backwards.
Counting the firmware's refusals in
log-and-stream-common/Comms/shimmer_bt_uart.c,exactly one of ~14
sendNacksites is the sensing gate. The rest are reachablewhile idle:
SET_SD_SYNC_COMMANDandACK, but only while a sync recording has the radio on (see below)839, viaShimBt_isCmdAllowedWhileSdSyncing839args[]852SET_SAMPLING_RATE_COMMANDwith a zero clock divider11561195SET_DAUGHTER_CARD_MEMEEPROM write failure1397SET_CHARGE_STATUS_LED_COMMAND— dispatched but never implemented1408SET_BT_COMMS_BAUD_RATE— not supported1417SET_INFOMEM_COMMANDoffset out of range1502RESET_BT_ERROR_COUNTS1586,1589ShimBt_isCmdBlockedWhileSensing)845— the only streaming oneCorrected after review: an earlier version of this description said a
sync-enabled Shimmer cannot be connected to on master, because the sync gate refuses
the driver's whole initialisation sequence. That is wrong. With SD sync enabled the
firmware switches Bluetooth off while idle (
ShimConfig_checkBtModeFromConfig),so a host connection never starts, on either branch; the gate only applies while the
sync code has the radio on during a sync recording. The other idle refusals in the
table stand.
The plainest one is
SET_CHARGE_STATUS_LED_COMMAND(0x30), which the firmware hasrefused unconditionally since Shimmer3R LogAndStream v1.01.013 and Shimmer3
v1.01.004. The driver knows 0x30 as
SET_BLINK_LEDand still sends it from thepublic
writeLEDCommand(), which its check-alive task calls as a ping while a streamhas stalled. That is the case the bench test now uses.
The existing coping strategy was avoidance rather than handling, which is decent
evidence the failure mode is real —
readPressureCalibrationCoefficients()skipsthe BMP581 calibration read with the comment "its firmware NACKs … so don't send
it." Avoiding each NACK individually does not scale, and it makes every new
firmware NACK a latent connection-drop in every released copy of the driver.
What this changes
A refusal now unwinds the transaction and keeps the connection up.
ShimmerObjectNACK_COMMAND_PROCESSED = (byte) 0xFEprocessNackFromCommand()mWaitForAck/mWaitForResponse, reports progress, drops the refused instruction so the stack advances, completes the transaction, releases the stack lockeventCommandRefused()The refused instruction is dropped only when it is still queued, which
processNackFromCommand()determines frommWaitForAckon entry. A GET'sinstruction is removed as soon as its ACK arrives, so an unconditional removal
would drop whatever was queued behind it and silently skip that command —
Copilot caught that in review, and it is fixed in
648897ba. TheACK-wait and streaming paths both still have the instruction queued; the
response-wait path does not.
Handled in three places:
one that matters.
firmware only discovers it cannot serve a GET while building the response.
Checked before
isKnownResponse().LiteProtocolgained the matching branchin
648897baafter review; it only had the ACK-wait one at first.but still strictly better than master destroying the whole queue.
Completing the operation (
8bd5b940)Unwinding the transaction was not enough on its own. Operation completion is driven
off the ACK path:
sendProgressReport(...)advances a counter, andShimmerDeviceCallbackAdapteronly callsfinishOperation()oncemProgressCounter == mProgressEndValue.processNackFromCommand()sent noprogress report, so a single refused command inside an operation meant the
operation never completed —
mOperationUnderwaystayed true and the applicationsat in CONFIGURING/CONNECTING indefinitely.
That matters most at connect time, where every command is part of an operation.
Master never completed the operation either, but it killed the connection first, so
the hang was never visible; making the link survive is what exposes it.
The report is sent before the refused instruction is removed, and only while it is
still queued, because
BluetoothProgressReportPerDevice.updateProgress()derivesthe counter from the remaining stack size — the response-wait path already reported
when its ACK arrived, and reporting twice there would skip a count. The same three
timer-driven commands the ACK path excludes are excluded here.
Known limitation: a refused command still counts as progress, so the operation
completes as
SUCCESS. Reporting a partially-refused operation asFAIL_WITH_WARNINGwould mean changing the sharedOperationStatehandling infinishOperation(), which is wider than this fix;eventCommandRefused()remainsthe hook for an application that needs to know.
Two deliberate non-obvious choices
eventCommandRefused()is concrete with an empty default, not abstract — andnot a new
ProtocolListenercallback either. Every other notification hook here(
sendProgressReport,connectionLost,eventLogAndStreamStatusChanged) isprotected abstract, so adding one would break every downstream subclass andlistener implementation at compile time. This way nothing outside the driver has to
change, and anyone who wants the refusal surfaced overrides it.
A stray NACK with no command in flight is logged and ignored, not treated as a
refusal.
processBytesAvailableAndInstreamSupported()runs only when!mWaitForAck && !mWaitForResponse, so there is no transaction to fail there. TheTypeScript SDK gates its own NACK framing the same way (on a command genuinely
awaiting a reply) so that a leaked stream byte cannot fabricate a refusal; same
reasoning applied here.
Two supporting changes
0x00and0xFFas a packet boundary, so aNACK sitting in the buffer could not be skipped past cleanly. It now accepts
0xFE, and is renamedfindOffsetOfNextZeroOrFF→findOffsetOfNextPacketBoundaryto match what it actually does (private, one call site).
mBtCommandMapOthersobtCommandToStringnames it inthe logs. Deliberately not in
mBtResponseMap—isKnownResponse()must stayfalse for it, or
processResponseCommand()would try to parse it as a responsebody.
LiteProtocolgets the same treatment (it is live —BasicShimmerBluetoothManagerPcinstantiates it). Its NACK value is declared locally because the generated
LiteProtocolInstructionSethas no NACK entry; adding it toLiteProtocolInstructionSet.protoand regenerating would be tidier, but thatrewrites a large generated file committed in two places for the sake of one
constant. Happy to do it that way instead if preferred.
Verification
Bench-tested for both cases.
Mas confirmed on real hardware that master never recognises a device NACK (it times
out silently) while this branch detects it, unwinds and advances. That run used the
SET-while-sensing case, so it exercised the streaming branch — which is why the
idle connection drop did not reproduce in it, and why a command queued behind the
refused one did not survive on master: the streaming timeout calls
clearAllInstructions()by design.TestNackNotStreaming(ShimmerPCBasicExamples, reworked in7351b7e4) drives thenot-streaming path. Run it with the Shimmer's Bluetooth COM port as its argument, on
an idle Shimmer with SD sync off:
FULLY_INITIALIZED(20 s). Without it, the verdict is INCONCLUSIVE.0x30 0x00andGET_SAMPLING_RATE_COMMANDbehind it, and follows both through the instruction stack for 8 s.It now lives in
com.shimmerresearch.bluetooth, because the instruction list and thetransaction state are protected. Nothing is written to the device.
On a Shimmer3R, LogAndStream v1.01.025, over classic Bluetooth, 3 runs each with the
same test source:
NACK Received for Command:loggedShimmerDriverandShimmerPCBasicExamplesboth compile clean with no newwarnings under Gradle 8.10, the version CI uses.
The note that used to be here about the Gradle 6.1 wrapper no longer applies:
masternow has the 8.14.3 wrapper (DEV-1013), and this branch has
mastermerged in.Automated tests come with #345, stacked on this PR.
ShimmerDriverPChas sincegained an in-memory radio for the receive path (
API_00027onwards), and #345 uses itto drive both NACK paths: idle through the real IOThread, and streaming through the
parser one byte at a time.
Sibling implementations
The same wire format is implemented independently in several other places, so this was
checked against all of them rather than assumed to be Java-only. Three carried the same
defect and now have their own PRs:
Shimmer-C-API0xFEfalls to the switch default, reported as "Misaligned ByteStream Detected" while streaming, then disconnects after ~10 s of read timeoutsSwiftAPICheckedContinuationis never resumed, andtimeoutInSecondsis declared but never used, so the caller'sawaitnever returns at allpyshimmer_run_readloop, which catches onlyReadAbort, killing the reader thread and blocking every waiter for goodshimmer-web-sdkdevices/shimmer3/protocol.tsdefinesNACK = 0xFE, andShimmer3Clientgates the framing on a command genuinely being in flightShimmer-MATLAB-ID,Shimmer-Labview-API,Shimmer-Advance-APINothing links these repos and nothing tests them together, so the timing differs wildly
by accident: about five seconds here when idle, ten in C#, never in Swift and Python.
Not in scope
The related
writeMemdefect from the tracker ticket is untouched here: theover-range branch in
writeMem()detects a buffer running past the device's memoryrange and does nothing, its body being entirely commented out, and
MAX_CALIB_DUMP_MAX(4096) disagrees with the firmware'sSHIMMER_CALIB_RAM_MAX(1024 on Shimmer3) so the guard would be wrong even if enabled. Deciding what should
happen on an over-range write is a separate question from teaching the driver to
read a NACK, so I have left it for its own change.
🤖 Generated with Claude Code