Repository navigation
DEV-971: Java event markers - event files, live records, status, and a test window - #348
Merged
Merged
Conversation
…ction The Shimmer3/Shimmer3R protocol has had a NACK_COMMAND_PROCESSED (0xFE) reply for a long time, but the driver never learned about it. There was no NACK constant anywhere on the Shimmer3 side, and both protocol implementations tested only for ACK with no else branch, so a NACK was read, silently discarded, and the transaction stalled with mWaitForAck still true and the instruction stack still locked. Two seconds later the ACK timer expired and checkForAckOrRespTask called connectionLost(): a refused command was indistinguishable from a dead link, and there was no retry to soften it because NUMBER_OF_TX_RETRIES_LIMIT is 0 and reading the NACK byte itself resets mNumberofTXRetriesCount, making the soft-recovery branch dead code. This is already reachable on shipping firmware - any SET command while the device is sensing, a sync-mode mismatch, a failed SET_DAUGHTER_CARD_MEM, and the BMP180/BMP280 calibration reads on a Shimmer3R all NACK. The existing coping strategy was avoidance rather than handling; see the comment on GET_PRESSURE_CALIBRATION_COEFFICIENTS_COMMAND, which is skipped for a BMP581 specifically because the firmware NACKs it. A refusal now unwinds the transaction and keeps the connection: - ShimmerObject: add NACK_COMMAND_PROCESSED = (byte) 0xFE. - ShimmerBluetooth.processNackFromCommand(): stop the ACK timer, clear mWaitForAck/mWaitForResponse, drop the refused instruction so the stack advances (as both ACK paths do for their own command), complete the transaction and release the stack lock. - Handled in three places: waiting for an ACK while not streaming (the main path), waiting for a response while not streaming, and a data packet followed by a NACK while streaming, which is the SET-while-sensing case. - eventCommandRefused() is the application hook. It is deliberately concrete with an empty default rather than abstract, and not a new ProtocolListener callback, so every existing subclass and listener implementation keeps compiling. Override it to surface the refusal. Two supporting changes: - The resync helper accepted only 0x00 and 0xFF as a packet boundary, so a NACK in the buffer could not be skipped past cleanly. It now accepts 0xFE too, and is renamed from findOffsetOfNextZeroOrFF to findOffsetOfNextPacketBoundary to match what it does. - NACK is registered in mBtCommandMapOther so btCommandToString names it in the logs. Deliberately not in mBtResponseMap: isKnownResponse() must stay false for it, or processResponseCommand() would try to parse it as a response body. A stray NACK arriving with no command in flight is logged and ignored rather than turned into a refusal, in processBytesAvailableAndInstreamSupported() which runs only when !mWaitForAck && !mWaitForResponse. The TypeScript SDK gates its own NACK framing the same way, so a leaked stream byte cannot fabricate a refusal. LiteProtocol gets the same treatment. Its NACK value is declared locally because the generated LiteProtocolInstructionSet has no NACK entry; adding it to LiteProtocolInstructionSet.proto and regenerating would be tidier but rewrites a large generated file committed in two places for one constant. Compile-verified only - there is no test harness for the BT receive path (ShimmerBluetooth is abstract with many abstract I/O methods and no test instantiates it), and no hardware was involved. Co-Authored-By: Claude Opus 5 <[email protected]>
…pilot review) processNackFromCommand() removed instruction 0 unconditionally, but a GET's instruction is already removed when its ACK arrives. A NACK arriving in the response-wait state therefore dropped whatever was queued behind it and silently skipped that command. It is now dropped only when mWaitForAck was still set, which is exactly the state that still has it queued. Same fix in LiteProtocol, and the javadoc no longer claims the drop is unconditional. Also from the review: - LiteProtocol was missing the response-wait NACK branch that ShimmerBluetooth has, so a NACK there would still have fallen through to the timeout. Added, checked before isKnownResponseByte() to match. - The resync helper's inline comment still said it looked for DATA_PACKET or ACK only. - The helper used a raw 0 rather than the DATA_PACKET constant it now sits alongside (suppressed review comment). Co-Authored-By: Claude Opus 5 <[email protected]>
processNackFromCommand() unwound the transaction but sent no progress report, so an operation containing a refused command never reached its end value. BluetoothProgressReportPerDevice.updateProgress() derives the counter from the remaining instruction stack size, and ShimmerDeviceCallbackAdapter only calls finishOperation() when that counter equals mProgressEndValue - so a single NACK left the operation running indefinitely, with mOperationUnderway still true and the application sitting in CONFIGURING/CONNECTING. This matters most at connect time. With SD sync enabled the firmware NACKs every command except SET_SD_SYNC_COMMAND and ACK (ShimBt_isCmdAllowedWhileSdSyncing), which covers the whole driver initialisation sequence. The report is sent before the refused instruction is removed, and only while it is still queued, to match the arithmetic on the ACK path: the response-wait path already reported when its ACK arrived, so reporting again there would skip a count. The three timer-driven commands the ACK path excludes are excluded here too. Known limitation: a refused command still counts as progress and the operation completes as SUCCESS. Reporting a partially-refused operation as FAIL_WITH_WARNING would mean changing the shared OperationState handling in finishOperation(), which is a wider change than this fix; eventCommandRefused() remains the hook for an application that needs to know. Co-Authored-By: Claude Opus 5 <[email protected]>
The two earlier harnesses both refused a command while sensing, so both took the mIsStreaming branch of checkForAckOrRespTask - which clears the instruction stack but leaves the connection up. The ~2 s teardown the ticket describes is in the other branch, reached only by a refusal while not streaming, and nothing exercised it. Configuring mid-stream is not a supported operation either, so those runs were testing a path the product does not use. This drives the supported case. With SD sync enabled the firmware refuses every command except SET_SD_SYNC_COMMAND and ACK (ShimBt_isCmdAllowedWhileSdSyncing), so the driver's whole initialisation sequence is NACKed at connect time. On master that ends in connectionLost() about two seconds in; on this branch the refusals are recognised and the link stays up. Nothing is written to the device, so the run is non-destructive - SD sync is set once in Consensys beforehand and cleared afterwards. The test reports INCONCLUSIVE rather than a pass if FULLY_INITIALIZED arrives, which means sync was not actually enabled. Compiled against ShimmerPCBasicExamples with Gradle 8.10, the version CI uses. The checked-in wrapper (6.1) cannot configure ShimmerDriverPC at all - its Shadow plugin requires Gradle 7+ - which is more evidence the stale wrapper deserves its own ticket. Co-Authored-By: Claude Opus 5 <[email protected]>
… mid-stream Firmware that cannot start a log (no SD card or a bad one, docked, on USB power, logging turned off in its configuration) is to NACK START_LOGGING_ONLY_COMMAND (0x92) and START_SDBT_COMMAND (0x70) and start nothing, including when 0x92 is sent in the middle of a stream. On master the driver dropped the 0xFE: when idle the ACK timer took START_LOGGING_ONLY for started and dropped the connection for START_SDBT; mid-stream the NACK went with the packet in front of it and the timer again took the device for logging. Built on DEV-982's NACK handling (processNackFromCommand), which unwinds the refused command and keeps the link. On top of it: - A refused log start is taken for a log that did not start (processLogStartRefused): the device is left not logging, the status reads startDataLogAndStreaming() stopped start again after START_SDBT, and eventLogStartRefused(command), a new protected hook, is called before eventCommandRefused. By default it passes LOG_START_REFUSED_MESSAGE to sendStatusMSGtoUI, which Android shows as a toast; ShimmerPC's is empty. - While streaming, a NACK after a data packet waits for its own link CRC (C5, or C5 56) and checks it, as a bare ACK's is: exactly the packet, the NACK and its CRC are consumed, and what follows stays as the next frame. DEV-982 cleared the whole buffer at the NACK, so the CRC landed where the next header should be and bytes already buffered were lost. A packet whose CRC fails is dropped without losing the NACK; a 0xFE whose CRC does not match is not a NACK; a NACK with no command in flight is ignored, as it is when not streaming. - processNackFromCommand() is protected, for ShimmerAndroidAPI's Shimmer, whose own streaming parser has to fail the command the same way. Tests: - API_00036_LogStartRefusedTest (idle, IOThread over an in-memory radio): refused log-only and combined starts, no CRC, 1- and 2-byte CRC, each handled at once with the next command answered; an ACKed start still logs. The radio now appends real CRCs. - API_00037_StreamingLogStartRefusedTest (the streaming parser): a 0x92 refused between two packets with no CRC, 1- and 2-byte CRC; still not logging after the ACK timer would have run; a corrupt packet before the NACK; the next frame already buffered; a 0xFE with the wrong CRC; a NACK with no command in flight. Against master's parser 9 of the 10 fail; against DEV-982's, 6. ShimmerDriver 178/178, ShimmerDriverPC 118/118. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review found the bench test could not work: with SD sync enabled the firmware switches Bluetooth off while idle, so the port never opened, and the test still reported "master behaviour" because it took any DISCONNECTED for a teardown. It now refuses a command the firmware always refuses and that changes nothing: 0x30 with one argument byte (the driver's SET_BLINK_LED, the firmware's SET_CHARGE_STATUS_LED_COMMAND, dispatched but never implemented), with GET_SAMPLING_RATE queued straight behind it. It follows those two instructions through the stack and gives a verdict only once the Shimmer has finished connecting, is idle, and has firmware that refuses 0x30 (Shimmer3R LogAndStream v1.01.013+, Shimmer3 v1.01.004+): - PASS: 0x30 retired, the GET answered, the link up 8 s later; - FAIL: the link torn down, or either instruction never completed; - INCONCLUSIVE: otherwise. The window is 8 s because when not streaming the ACK timer gives a SET ACK_TIMER_DURATION + 3 = 5 s, not 2 s. It moves to com.shimmerresearch.bluetooth, since the instruction list and the transaction state are protected, and takes the COM port as its argument. On a Shimmer3R, LogAndStream v1.01.025, over classic Bluetooth, 3 runs each: this branch PASS 3/3 (0x30 retired after 137-224 ms, the GET answered after 241-272 ms); master FAIL 3/3 (the link lost 5.0 s after the refusal, no NACK logged). Co-Authored-By: Claude Opus 5.5 <[email protected]>
…a full write Decode the Shimmer3R sound level channel: ID 0x38, a signed 16-bit little-endian count of 0.01 dB(A), LAeq over the microphone's most recent 128 ms block. 0x8000 means no level and calibrates to NaN. The decoder takes the channel's position from the channel list (inquiry response or SD header), like any other channel. A Shimmer3 never sends 0x38 and keeps the unknown-channel handling. The channel is enabled through configuration byte 233 bit 4, not a sensor bitmap bit. A new SensorSoundLevel (Shimmer3R only) owns the byte: its config hooks take the enable from bit 4 and give back bits 0-3 (the audio mode and gain preset) as read. A full configuration write builds its image from zeros, so until now it wrote 0 to byte 233 and also turned off a Shimmer3R's audio recording. Bit 7 set (erased flash) reads as 0, as the firmware reads it. A Shimmer3's byte 233 stays 0. Tests: API_00040 (ShimmerDriver) covers decoding at the spec position and elsewhere, the no-level value, SD, Shimmer3, and every value of byte 233. API_00041 (ShimmerDriverPC) covers the byte surviving deepClone() and decoding with the arrays data structure. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…evel SensorSoundLevel's javadoc names LogAndStream_Shimmer3R v1.01.026, and the firmware without byte 233 as the releases before it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Handle NACK from a Shimmer3/3R instead of dropping the connection
…g_start DEV-1431: a log start the device refuses (NACK) ends at once, idle or mid-stream
…1.026 Releases before v1.01.026 store configuration byte 233 untouched, so on them bit 4 read back set while no channel was sent: the Sound Level tile showed on, and nothing arrived. The bit also stayed in the unit, so a later firmware update would turn the channel on unasked. The sensor's compatibility list is now Shimmer3R LogAndStream v1.01.026 and later, so the tile is offered only there. On older firmware a bit 4 already set reads as off and the next configuration write clears it. Bits 0-3, the audio mode the full write used to wipe, are kept on every Shimmer3R firmware as before. API_00040 runs at v1.01.026 and adds three tests at v1.01.025: the tile is not offered, a stored bit 4 reads off and is cleared on write, and enabling sets nothing. Without the gate all three fail. ShimmerDriver 195/195, ShimmerDriverPC 105/105, on JDK 11. Co-Authored-By: Claude Opus 5.5 <[email protected]>
DEV-1413: Shimmer3R sound level channel (0x38), and keep configuration byte 233 on a full write
Resolves the conflict #343 predicted with #293 (DEV-982) in processNotStreamingWaitForAck(): the event channel probe's NACK is taken first, then any other NACK as a refusal. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…nnel status The Java driver now does the host side of the event channel that the web SDK does (log-and-stream-common docs/SHIMMER3_EVENT_CHANNEL.md and SHIMMER3_BT_COMMUNICATION_PROTOCOL.md section 7.15): - EventFile, EventRecord: the SD card's events/ files, a 64-byte header and 16-byte records, with pairing to the data file by its initial timestamp. - writeEventStream(boolean): SET_FEATURE / FEATURE_EVENT_STREAM. Each record frame, [0x8A][0xBB][0x10][record][CRC-16], reaches EventChannelListener. Frames are taken between data packets (any link CRC mode), straight after the opt-in's ACK, at the start of the buffer, and when idle, including before a command's ACK or reply. They are only parsed after an opt-in, and only once their own CRC checks out. - readEventChannelStatus(): GET_EVENT_CHANNEL_STATUS_COMMAND (0xB9). Not streaming only: its reply is not wrapped in 0x8A, so the streaming parser could not tell it from a packet. Tests port the web SDK's tests/events/record.test.ts inputs and results, and run the IO thread over an in-memory radio for the streaming and idle paths. Android's Shimmer.java overrides the streaming parser and does not know the frame, so it must not opt in until ported. Co-Authored-By: Claude Opus 5.5 <[email protected]>
… event channel Tools > Event markers (Shimmer3R), or the EVENT MARKERS button, opens a window that does with the Java driver what Shimmer Capture web does with the web SDK: - whether the unit has the event channel, and asking again; - configuration byte 232 (button mode, what starts logging, TTL input), written with the whole configuration and read back; - opt in / opt out, the live records with sequence gaps flagged, and the frame and bad-CRC counters; - the event channel status (not while streaming); - an event file from a card: its header, completeness, the data file it pairs with, and its records timed from the file open. Each live edge is also marked on the main plot as a red vertical bar, placed by the latest data packet's Shimmer clock against the plot's PC-time axis. CI does not build ShimmerPCBasicExamples; built locally. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Contributor
Author
|
Bench, 13 Nov 2026: live event records on a Shimmer3R over Bluetooth Classic. Run from the DEV-1473 Java Shimmer Capture, which is stacked on this PR's driver and opens its Event markers window. Unit on LogAndStream v1.1.26, over a Windows Classic COM port; the record times below are from the driver's console log.
Not covered by this run:
🤖 Generated with Claude Code |
Contributor
Author
|
Four parser defects found in this PR's review, each reproduced by a test that failed on this code, are fixed in the follow-up #349 (same base, 🤖 Generated with Claude Code |
JongChern
added a commit
that referenced
this pull request
Nov 13, 2026
…ehind a bare ACK's CRC From the review of #349: - The idle tests' radios now take each burst whole and let reads wait for their bytes, as a serial port does (WaitingTestRadio), so the IOThread can no longer race the test's writes. API_00043's idle tests use it too. - The status-push, frame-after-ACK and status-reply tests now run with no link CRC, a 1-byte and a 2-byte one. - The misframing test checks that the good packets after the damaged one still arrive; the status-reply test that the command is retired, the lock released, the timer cancelled and the packets either side parsed. The CRC runs reproduced one more defect, which is fixed here: with a link CRC on (the driver turns it on at connect where the firmware has it), the records sent in the same burst as the opt-in's ACK were lost, 0 of 2. The bare ACK's link CRC is left after the ACK is taken; the idle path read it as an unknown byte and cleared the whole serial buffer, records included. With the event channel on, the idle path now drops only the bytes it read. All 7 tests in API_00044 fail on #348's code and pass here. ShimmerDriver 219/219, ShimmerDriverPC 155/155; the Java 8 MATLAB jar builds. Co-Authored-By: Claude Opus 5.5 <[email protected]>
JongChern
added a commit
that referenced
this pull request
Nov 13, 2026
DEV-971: fix five event-channel parser defects found in #348's review
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DEV-971. The Java driver's host side of the event channel, stacked on #343, plus a window in SensorMapsExample to bench it. It does what shimmer-web-sdk#218 and shimmer-capture-web#11 do, following log-and-stream-common
docs/SHIMMER3_EVENT_CHANNEL.mdanddocs/SHIMMER3_BT_COMMUNICATION_PROTOCOL.md§7.15.Not run on a unit yet. Draft until it has been benched.
Reviewing
The first commit merges
masterinto #343's branch. #343 conflicts with #293 inprocessNotStreamingWaitForAck(), and the merge resolves it as #343's description suggests: the event channel probe's NACK is taken first, then any other NACK as a refusal. Until #343 takes the same merge, this PR's diff also shows #346, #345 and #293. Review the last two commits.Driver (
ShimmerDriver)Event files.
EventFileandEventRecordreadevents/<experiment>/<session>/<NNN>: the 64-byte header, then 16-byte records.isComplete()says whether the file ends with a final record.belongsWithDataFile()pairs a file with its data file by the data file's initial timestamp, andgetCompanionFile()finds it by the card layout.Live records.
writeEventStream(boolean)sendsSET_FEATURE/FEATURE_EVENT_STREAM. Each record frame,[0x8A][0xBB][0x10][record][CRC-16], goes to everyEventChannelListener. Frames are taken:They are parsed only after an opt-in, as §7.15 says, and only once their own CRC checks out. Frames with a bad CRC are counted and dropped.
Status.
readEventChannelStatus()sendsGET_EVENT_CHANNEL_STATUS_COMMAND(0xB9).[ACK][0xBA][9]has no 0x8A wrapper, so the streaming parser could not tell it from a packet. The web SDK can, because its framing is different.Per connection. Connecting resets the opt-in, the counters and the last status, as the firmware does.
Test window (
ShimmerPCBasicExamples)Open it with Tools › Event markers (Shimmer3R) or the EVENT MARKERS button. It shows:
Each live edge is also drawn on the main plot as a red vertical bar. The plot's x axis is PC time, so the record's time is placed against the latest data packet's Shimmer clock.
Tests
API_00042_EventRecordFileTest(ShimmerDriver, 16 tests). It uses the same inputs and results as the web SDK'stests/events/record.test.ts: the worked event file and frame from the docs, a file cut short, longer records, bad magic and version, a frame with a bad CRC or cut short, the status bytes, and placing a record on a stream's unwrapped timeline.API_00043_EventStreamTest(ShimmerDriverPC, 9 tests). The streaming cases feed the parser one byte at a time; the idle cases run the IOThread over a simulated radio. They cover:build, as CI runs it).shadowJar -PmatlabJava8) builds.Still to do
ShimmerAndroidAPI'sShimmer.javaoverrides the streaming parser and doesn't know the frame, so an Android app must not opt in until it is ported. Reading event files and byte 232 reach Android through the driver pin as usual.🤖 Generated with Claude Code