Skip to content

DEV-971: Java event markers - event files, live records, status, and a test window - #348

Merged
JongChern merged 17 commits into
DEV-971_event_channelfrom
DEV-971_java_event_markers
Nov 13, 2026
Merged

JongChern merged 17 commits into
DEV-971_event_channelfrom
DEV-971_java_event_markers

Conversation

@JongChern

Copy link
Copy Markdown
Contributor

DEV-971. The Java driver's host side of the event channel, stacked on #343, plus a window in SensorMapsExample to bench it. It does what shimmer-web-sdk#218 and shimmer-capture-web#11 do, following log-and-stream-common docs/SHIMMER3_EVENT_CHANNEL.md and docs/SHIMMER3_BT_COMMUNICATION_PROTOCOL.md §7.15.

Not run on a unit yet. Draft until it has been benched.

Reviewing

The first commit merges master into #343's branch. #343 conflicts with #293 in processNotStreamingWaitForAck(), and the merge resolves it as #343's description suggests: the event channel probe's NACK is taken first, then any other NACK as a refusal. Until #343 takes the same merge, this PR's diff also shows #346, #345 and #293. Review the last two commits.

Driver (ShimmerDriver)

  • Event files. EventFile and EventRecord read events/<experiment>/<session>/<NNN>: the 64-byte header, then 16-byte records.

    • Longer headers and records from a later format are skipped over, not refused.
    • isComplete() says whether the file ends with a final record.
    • belongsWithDataFile() pairs a file with its data file by the data file's initial timestamp, and getCompanionFile() finds it by the card layout.
  • Live records. writeEventStream(boolean) sends SET_FEATURE / FEATURE_EVENT_STREAM. Each record frame, [0x8A][0xBB][0x10][record][CRC-16], goes to every EventChannelListener. Frames are taken:

    • between data packets, with no link CRC or a 1- or 2-byte one;
    • straight after the opt-in's ACK, whether a data packet comes first or not;
    • at the start of the buffer;
    • when idle, including before a command's ACK or between the ACK and the reply.

    They are parsed only after an opt-in, as §7.15 says, and only once their own CRC checks out. Frames with a bad CRC are counted and dropped.

  • Status. readEventChannelStatus() sends GET_EVENT_CHANNEL_STATUS_COMMAND (0xB9).

    • It is sent only to a unit known to have the event channel, since firmware without it doesn't answer.
    • It is not sent while streaming: the reply [ACK][0xBA][9] has no 0x8A wrapper, so the streaming parser could not tell it from a packet. The web SDK can, because its framing is different.
  • Per connection. Connecting resets the opt-in, the counters and the last status, as the firmware does.

Test window (ShimmerPCBasicExamples)

Open it with Tools › Event markers (Shimmer3R) or the EVENT MARKERS button. It shows:

  • whether the unit has the event channel, with a way to ask again;
  • byte 232: the button mode, log triggers and TTL input, written with the whole configuration and read back;
  • opt in / opt out, and the live records with sequence gaps flagged, plus the frame and bad-CRC counters;
  • the status;
  • an event file from a card: its header, whether it is complete, the data file it pairs with, and its records timed from the file open.

Each live edge is also drawn on the main plot as a red vertical bar. The plot's x axis is PC time, so the record's time is placed against the latest data packet's Shimmer clock.

Tests

  • API_00042_EventRecordFileTest (ShimmerDriver, 16 tests). It uses the same inputs and results as the web SDK's tests/events/record.test.ts: the worked event file and frame from the docs, a file cut short, longer records, bad magic and version, a frame with a bad CRC or cut short, the status bytes, and placing a record on a stream's unwrapped timeline.
  • API_00043_EventStreamTest (ShimmerDriverPC, 9 tests). The streaming cases feed the parser one byte at a time; the idle cases run the IOThread over a simulated radio. They cover:
    • frames between packets with each CRC mode;
    • the opt-in's ACK followed at once by its record;
    • a frame with a bad CRC, after which the stream goes on;
    • no parsing without an opt-in;
    • frames before an ACK and before a reply.
  • Results.
    • ShimmerDriver: 219/219.
    • ShimmerDriverPC: 148/148 (build, as CI runs it).
    • The Java 8 jar for MATLAB (shadowJar -PmatlabJava8) builds.
    • ShimmerPCBasicExamples compiles. CI doesn't build it, and the window hasn't been opened yet.

Still to do

  • The bench test. Needs a Shimmer3R on LogAndStream v1.01.026 and someone to press the button: opt in idle and while streaming at a high rate, presses and TTL edges, the status, a byte 232 write and read-back, and an event file from the card.
  • Android. ShimmerAndroidAPI's Shimmer.java overrides the streaming parser and doesn't know the frame, so an Android app must not opt in until it is ported. Reading event files and byte 232 reach Android through the driver pin as usual.
  • Other host APIs. C# still needs the same. shimmer-host-core has it (Consensys_v1.6.3 - CON-648 changed Euler source from quaternion to angle axis - DONT MERGE! #94).
  • Before a release. The Java HIL, since this touches the streaming parser.
  • Consensys. Importing event files is DEV-1457, in the Advance API.

🤖 Generated with Claude Code

marknolan and others added 17 commits September 3, 2026 14:01
…ction

The Shimmer3/Shimmer3R protocol has had a NACK_COMMAND_PROCESSED (0xFE)
reply for a long time, but the driver never learned about it. There was no
NACK constant anywhere on the Shimmer3 side, and both protocol
implementations tested only for ACK with no else branch, so a NACK was read,
silently discarded, and the transaction stalled with mWaitForAck still true
and the instruction stack still locked. Two seconds later the ACK timer
expired and checkForAckOrRespTask called connectionLost(): a refused command
was indistinguishable from a dead link, and there was no retry to soften it
because NUMBER_OF_TX_RETRIES_LIMIT is 0 and reading the NACK byte itself
resets mNumberofTXRetriesCount, making the soft-recovery branch dead code.

This is already reachable on shipping firmware - any SET command while the
device is sensing, a sync-mode mismatch, a failed SET_DAUGHTER_CARD_MEM, and
the BMP180/BMP280 calibration reads on a Shimmer3R all NACK. The existing
coping strategy was avoidance rather than handling; see the comment on
GET_PRESSURE_CALIBRATION_COEFFICIENTS_COMMAND, which is skipped for a BMP581
specifically because the firmware NACKs it.

A refusal now unwinds the transaction and keeps the connection:

- ShimmerObject: add NACK_COMMAND_PROCESSED = (byte) 0xFE.
- ShimmerBluetooth.processNackFromCommand(): stop the ACK timer, clear
  mWaitForAck/mWaitForResponse, drop the refused instruction so the stack
  advances (as both ACK paths do for their own command), complete the
  transaction and release the stack lock.
- Handled in three places: waiting for an ACK while not streaming (the main
  path), waiting for a response while not streaming, and a data packet
  followed by a NACK while streaming, which is the SET-while-sensing case.
- eventCommandRefused() is the application hook. It is deliberately concrete
  with an empty default rather than abstract, and not a new ProtocolListener
  callback, so every existing subclass and listener implementation keeps
  compiling. Override it to surface the refusal.

Two supporting changes:

- The resync helper accepted only 0x00 and 0xFF as a packet boundary, so a
  NACK in the buffer could not be skipped past cleanly. It now accepts 0xFE
  too, and is renamed from findOffsetOfNextZeroOrFF to
  findOffsetOfNextPacketBoundary to match what it does.
- NACK is registered in mBtCommandMapOther so btCommandToString names it in
  the logs. Deliberately not in mBtResponseMap: isKnownResponse() must stay
  false for it, or processResponseCommand() would try to parse it as a
  response body.

A stray NACK arriving with no command in flight is logged and ignored rather
than turned into a refusal, in processBytesAvailableAndInstreamSupported()
which runs only when !mWaitForAck && !mWaitForResponse. The TypeScript SDK
gates its own NACK framing the same way, so a leaked stream byte cannot
fabricate a refusal.

LiteProtocol gets the same treatment. Its NACK value is declared locally
because the generated LiteProtocolInstructionSet has no NACK entry; adding it
to LiteProtocolInstructionSet.proto and regenerating would be tidier but
rewrites a large generated file committed in two places for one constant.

Compile-verified only - there is no test harness for the BT receive path
(ShimmerBluetooth is abstract with many abstract I/O methods and no test
instantiates it), and no hardware was involved.

Co-Authored-By: Claude Opus 5 <[email protected]>
…pilot review)

processNackFromCommand() removed instruction 0 unconditionally, but a GET's
instruction is already removed when its ACK arrives. A NACK arriving in the
response-wait state therefore dropped whatever was queued behind it and
silently skipped that command. It is now dropped only when mWaitForAck was
still set, which is exactly the state that still has it queued. Same fix in
LiteProtocol, and the javadoc no longer claims the drop is unconditional.

Also from the review:
- LiteProtocol was missing the response-wait NACK branch that ShimmerBluetooth
  has, so a NACK there would still have fallen through to the timeout. Added,
  checked before isKnownResponseByte() to match.
- The resync helper's inline comment still said it looked for DATA_PACKET or
  ACK only.
- The helper used a raw 0 rather than the DATA_PACKET constant it now sits
  alongside (suppressed review comment).

Co-Authored-By: Claude Opus 5 <[email protected]>
processNackFromCommand() unwound the transaction but sent no progress
report, so an operation containing a refused command never reached its
end value. BluetoothProgressReportPerDevice.updateProgress() derives the
counter from the remaining instruction stack size, and
ShimmerDeviceCallbackAdapter only calls finishOperation() when that
counter equals mProgressEndValue - so a single NACK left the operation
running indefinitely, with mOperationUnderway still true and the
application sitting in CONFIGURING/CONNECTING.

This matters most at connect time. With SD sync enabled the firmware
NACKs every command except SET_SD_SYNC_COMMAND and ACK
(ShimBt_isCmdAllowedWhileSdSyncing), which covers the whole driver
initialisation sequence.

The report is sent before the refused instruction is removed, and only
while it is still queued, to match the arithmetic on the ACK path: the
response-wait path already reported when its ACK arrived, so reporting
again there would skip a count. The three timer-driven commands the ACK
path excludes are excluded here too.

Known limitation: a refused command still counts as progress and the
operation completes as SUCCESS. Reporting a partially-refused operation
as FAIL_WITH_WARNING would mean changing the shared OperationState
handling in finishOperation(), which is a wider change than this fix;
eventCommandRefused() remains the hook for an application that needs to
know.

Co-Authored-By: Claude Opus 5 <[email protected]>
The two earlier harnesses both refused a command while sensing, so both
took the mIsStreaming branch of checkForAckOrRespTask - which clears the
instruction stack but leaves the connection up. The ~2 s teardown the
ticket describes is in the other branch, reached only by a refusal while
not streaming, and nothing exercised it. Configuring mid-stream is not a
supported operation either, so those runs were testing a path the
product does not use.

This drives the supported case. With SD sync enabled the firmware
refuses every command except SET_SD_SYNC_COMMAND and ACK
(ShimBt_isCmdAllowedWhileSdSyncing), so the driver's whole
initialisation sequence is NACKed at connect time. On master that ends
in connectionLost() about two seconds in; on this branch the refusals
are recognised and the link stays up.

Nothing is written to the device, so the run is non-destructive - SD
sync is set once in Consensys beforehand and cleared afterwards. The
test reports INCONCLUSIVE rather than a pass if FULLY_INITIALIZED
arrives, which means sync was not actually enabled.

Compiled against ShimmerPCBasicExamples with Gradle 8.10, the version CI
uses. The checked-in wrapper (6.1) cannot configure ShimmerDriverPC at
all - its Shadow plugin requires Gradle 7+ - which is more evidence the
stale wrapper deserves its own ticket.

Co-Authored-By: Claude Opus 5 <[email protected]>
… mid-stream

Firmware that cannot start a log (no SD card or a bad one, docked, on
USB power, logging turned off in its configuration) is to NACK
START_LOGGING_ONLY_COMMAND (0x92) and START_SDBT_COMMAND (0x70) and
start nothing, including when 0x92 is sent in the middle of a stream.
On master the driver dropped the 0xFE: when idle the ACK timer took
START_LOGGING_ONLY for started and dropped the connection for
START_SDBT; mid-stream the NACK went with the packet in front of it and
the timer again took the device for logging.

Built on DEV-982's NACK handling (processNackFromCommand), which unwinds
the refused command and keeps the link. On top of it:
- A refused log start is taken for a log that did not start
  (processLogStartRefused): the device is left not logging, the status
  reads startDataLogAndStreaming() stopped start again after START_SDBT,
  and eventLogStartRefused(command), a new protected hook, is called
  before eventCommandRefused. By default it passes
  LOG_START_REFUSED_MESSAGE to sendStatusMSGtoUI, which Android shows as
  a toast; ShimmerPC's is empty.
- While streaming, a NACK after a data packet waits for its own link CRC
  (C5, or C5 56) and checks it, as a bare ACK's is: exactly the packet,
  the NACK and its CRC are consumed, and what follows stays as the next
  frame. DEV-982 cleared the whole buffer at the NACK, so the CRC landed
  where the next header should be and bytes already buffered were lost.
  A packet whose CRC fails is dropped without losing the NACK; a 0xFE
  whose CRC does not match is not a NACK; a NACK with no command in
  flight is ignored, as it is when not streaming.
- processNackFromCommand() is protected, for ShimmerAndroidAPI's Shimmer,
  whose own streaming parser has to fail the command the same way.

Tests:
- API_00036_LogStartRefusedTest (idle, IOThread over an in-memory radio):
  refused log-only and combined starts, no CRC, 1- and 2-byte CRC, each
  handled at once with the next command answered; an ACKed start still
  logs. The radio now appends real CRCs.
- API_00037_StreamingLogStartRefusedTest (the streaming parser): a 0x92
  refused between two packets with no CRC, 1- and 2-byte CRC; still not
  logging after the ACK timer would have run; a corrupt packet before the
  NACK; the next frame already buffered; a 0xFE with the wrong CRC; a
  NACK with no command in flight. Against master's parser 9 of the 10
  fail; against DEV-982's, 6.
ShimmerDriver 178/178, ShimmerDriverPC 118/118.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review found the bench test could not work: with SD sync enabled the
firmware switches Bluetooth off while idle, so the port never opened, and
the test still reported "master behaviour" because it took any
DISCONNECTED for a teardown.

It now refuses a command the firmware always refuses and that changes
nothing: 0x30 with one argument byte (the driver's SET_BLINK_LED, the
firmware's SET_CHARGE_STATUS_LED_COMMAND, dispatched but never
implemented), with GET_SAMPLING_RATE queued straight behind it. It follows
those two instructions through the stack and gives a verdict only once
the Shimmer has finished connecting, is idle, and has firmware that
refuses 0x30 (Shimmer3R LogAndStream v1.01.013+, Shimmer3 v1.01.004+):
- PASS: 0x30 retired, the GET answered, the link up 8 s later;
- FAIL: the link torn down, or either instruction never completed;
- INCONCLUSIVE: otherwise.
The window is 8 s because when not streaming the ACK timer gives a SET
ACK_TIMER_DURATION + 3 = 5 s, not 2 s.

It moves to com.shimmerresearch.bluetooth, since the instruction list and
the transaction state are protected, and takes the COM port as its
argument.

On a Shimmer3R, LogAndStream v1.01.025, over classic Bluetooth, 3 runs
each: this branch PASS 3/3 (0x30 retired after 137-224 ms, the GET
answered after 241-272 ms); master FAIL 3/3 (the link lost 5.0 s after
the refusal, no NACK logged).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…a full write

Decode the Shimmer3R sound level channel: ID 0x38, a signed 16-bit
little-endian count of 0.01 dB(A), LAeq over the microphone's most recent
128 ms block. 0x8000 means no level and calibrates to NaN. The decoder takes
the channel's position from the channel list (inquiry response or SD header),
like any other channel. A Shimmer3 never sends 0x38 and keeps the
unknown-channel handling.

The channel is enabled through configuration byte 233 bit 4, not a sensor
bitmap bit. A new SensorSoundLevel (Shimmer3R only) owns the byte: its
config hooks take the enable from bit 4 and give back bits 0-3 (the audio
mode and gain preset) as read. A full configuration write builds its image
from zeros, so until now it wrote 0 to byte 233 and also turned off a
Shimmer3R's audio recording. Bit 7 set (erased flash) reads as 0, as the
firmware reads it. A Shimmer3's byte 233 stays 0.

Tests: API_00040 (ShimmerDriver) covers decoding at the spec position and
elsewhere, the no-level value, SD, Shimmer3, and every value of byte 233.
API_00041 (ShimmerDriverPC) covers the byte surviving deepClone() and
decoding with the arrays data structure.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…evel

SensorSoundLevel's javadoc names LogAndStream_Shimmer3R v1.01.026, and the
firmware without byte 233 as the releases before it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Handle NACK from a Shimmer3/3R instead of dropping the connection
…g_start

DEV-1431: a log start the device refuses (NACK) ends at once, idle or mid-stream
…1.026

Releases before v1.01.026 store configuration byte 233 untouched, so on
them bit 4 read back set while no channel was sent: the Sound Level tile
showed on, and nothing arrived. The bit also stayed in the unit, so a
later firmware update would turn the channel on unasked.

The sensor's compatibility list is now Shimmer3R LogAndStream v1.01.026
and later, so the tile is offered only there. On older firmware a bit 4
already set reads as off and the next configuration write clears it.
Bits 0-3, the audio mode the full write used to wipe, are kept on every
Shimmer3R firmware as before.

API_00040 runs at v1.01.026 and adds three tests at v1.01.025: the tile
is not offered, a stored bit 4 reads off and is cleared on write, and
enabling sets nothing. Without the gate all three fail. ShimmerDriver
195/195, ShimmerDriverPC 105/105, on JDK 11.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
DEV-1413: Shimmer3R sound level channel (0x38), and keep configuration byte 233 on a full write
Resolves the conflict #343 predicted with #293 (DEV-982) in
processNotStreamingWaitForAck(): the event channel probe's NACK is taken
first, then any other NACK as a refusal.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…nnel status

The Java driver now does the host side of the event channel that the web SDK
does (log-and-stream-common docs/SHIMMER3_EVENT_CHANNEL.md and
SHIMMER3_BT_COMMUNICATION_PROTOCOL.md section 7.15):

- EventFile, EventRecord: the SD card's events/ files, a 64-byte header and
  16-byte records, with pairing to the data file by its initial timestamp.
- writeEventStream(boolean): SET_FEATURE / FEATURE_EVENT_STREAM. Each record
  frame, [0x8A][0xBB][0x10][record][CRC-16], reaches EventChannelListener.
  Frames are taken between data packets (any link CRC mode), straight after the
  opt-in's ACK, at the start of the buffer, and when idle, including before a
  command's ACK or reply. They are only parsed after an opt-in, and only once
  their own CRC checks out.
- readEventChannelStatus(): GET_EVENT_CHANNEL_STATUS_COMMAND (0xB9). Not
  streaming only: its reply is not wrapped in 0x8A, so the streaming parser
  could not tell it from a packet.

Tests port the web SDK's tests/events/record.test.ts inputs and results, and
run the IO thread over an in-memory radio for the streaming and idle paths.

Android's Shimmer.java overrides the streaming parser and does not know the
frame, so it must not opt in until ported.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
… event channel

Tools > Event markers (Shimmer3R), or the EVENT MARKERS button, opens a window
that does with the Java driver what Shimmer Capture web does with the web SDK:

- whether the unit has the event channel, and asking again;
- configuration byte 232 (button mode, what starts logging, TTL input),
  written with the whole configuration and read back;
- opt in / opt out, the live records with sequence gaps flagged, and the
  frame and bad-CRC counters;
- the event channel status (not while streaming);
- an event file from a card: its header, completeness, the data file it
  pairs with, and its records timed from the file open.

Each live edge is also marked on the main plot as a red vertical bar, placed
by the latest data packet's Shimmer clock against the plot's PC-time axis.

CI does not build ShimmerPCBasicExamples; built locally.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@JongChern

Copy link
Copy Markdown
Contributor Author

Bench, 13 Nov 2026: live event records on a Shimmer3R over Bluetooth Classic.

Run from the DEV-1473 Java Shimmer Capture, which is stacked on this PR's driver and opens its Event markers window. Unit on LogAndStream v1.1.26, over a Windows Classic COM port; the record times below are from the driver's console log.

What Result
Probe on connect (SET_FEATURE 3 0) ACKed: event channel present
Opt-in (SET_FEATURE 3 1) ACKed, then the initial-state record (seq 0)
Idle, not streaming 3 presses: seq 1-6, press and release each
While streaming 7 presses: seq 7-20. Frames taken between data packets, and a battery read's in-stream reply mid-stream was still answered
Records 21, seq 0-20, no gaps, every dropped 0
Parser No unexpected headers, no bytes thrown, no resyncs, no bad-CRC frames, no NACKs, the link stayed up
Arrival 4-64 ms after each edge's own time, by the unit's clock against this PC's
Stop Stream stopped, back to Ready

Not covered by this run:

  • Byte 232 was already set to record presses, so the write and read-back path is untried on hardware.
  • GET_EVENT_CHANNEL_STATUS (0xB9) was not asked.
  • TTL edges, and an event file read from the card, were not tried.
  • The markers on the plot were not checked against the press times.

🤖 Generated with Claude Code

@JongChern
JongChern marked this pull request as ready for review November 13, 2026 06:40
@JongChern
JongChern merged commit 997f874 into DEV-971_event_channel Nov 13, 2026
@JongChern
JongChern deleted the DEV-971_java_event_markers branch November 13, 2026 06:45
@JongChern

Copy link
Copy Markdown
Contributor Author

Four parser defects found in this PR's review, each reproduced by a test that failed on this code, are fixed in the follow-up #349 (same base, DEV-971_event_channel).

🤖 Generated with Claude Code

JongChern added a commit that referenced this pull request Nov 13, 2026
…ehind a bare ACK's CRC

From the review of #349:

- The idle tests' radios now take each burst whole and let reads wait for
  their bytes, as a serial port does (WaitingTestRadio), so the IOThread can
  no longer race the test's writes. API_00043's idle tests use it too.
- The status-push, frame-after-ACK and status-reply tests now run with no
  link CRC, a 1-byte and a 2-byte one.
- The misframing test checks that the good packets after the damaged one
  still arrive; the status-reply test that the command is retired, the lock
  released, the timer cancelled and the packets either side parsed.

The CRC runs reproduced one more defect, which is fixed here: with a link CRC
on (the driver turns it on at connect where the firmware has it), the
records sent in the same burst as the opt-in's ACK were lost, 0 of 2. The
bare ACK's link CRC is left after the ACK is taken; the idle path read it as
an unknown byte and cleared the whole serial buffer, records included. With
the event channel on, the idle path now drops only the bytes it read.

All 7 tests in API_00044 fail on #348's code and pass here. ShimmerDriver
219/219, ShimmerDriverPC 155/155; the Java 8 MATLAB jar builds.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
JongChern added a commit that referenced this pull request Nov 13, 2026
DEV-971: fix five event-channel parser defects found in #348's review
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants