Skip to content

DEV-971: keep configuration byte 232 on a full write, expose its fields, probe for the event channel - #343

Merged
JongChern merged 14 commits into
masterfrom
DEV-971_event_channel
Nov 13, 2026
Merged

JongChern merged 14 commits into
masterfrom
DEV-971_event_channel

Conversation

@marknolan

@marknolan marknolan commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

DEV-971 adds an event channel to Shimmer3 and Shimmer3R LogAndStream. It also adds configuration byte 232, which holds the button mode, the log triggers and the TTL input enable. The byte is specified in log-and-stream-common docs/SHIMMER3_CONFIGURATION_INFOMEM.md §8.3.

The driver's full configuration write builds its image from zeros, so any configure from Consensys or the Android config dialog put 0 in byte 232. That silently turned the button back to legacy, which undid a setting such as "button disabled" (0x01) or "button disabled, logging off" (0x11). This PR closes that gap and adds the byte to the config model. The driver can also tell whether the firmware has the byte.

It also carries the driver's host side of the event channel, merged in from #348 and #349: event files, live event records and the status (section 4).

The firmware and web sides merged on 11 Nov 2026: ShimmerResearch/log-and-stream-common#216, ShimmerResearch/shimmer3-firmware#138, ShimmerResearch/shimmer3r-firmware#319, ShimmerResearch/shimmer-web-sdk#218 and ShimmerResearch/shimmer-capture-web#11. ShimmerResearch/shimmer-host-core#94 has it in the Rust library.

What changes

1. A full configuration write keeps byte 232

  • Parse. ShimmerObject.configBytesParse() keeps the byte as read, 0xFF included.
  • Generate. configBytesGenerate() writes the byte back unless the caller set one. Where the byte can be trusted (see 3), it writes the value the firmware would correct it to.
    • This covers every path that goes through the generator: the Bluetooth writeConfigBytes(), the dock write, the reload after flashing, and the clone-settings copy.
  • Before anything is read. A configuration can be written on connect before the read, for example with isSetupDeviceWhileConnecting() or a fixed config. writeConfigBytes() then leaves out the bytes the driver does not have from this Shimmer, so the Shimmer keeps its own:
    • byte 232, if it was neither read nor set;
    • on a Shimmer3R, byte 233 (DEV-99 audio mode and gain, DEV-1413 sound level), if it was not read and the sound level is not turned on. The host never sets bits 0-3, so writing them unread put 0 there and turned the audio mode off. With the sound level turned on before a read, the byte is still written with bits 0-3 at 0, since one bit cannot be written alone.
    • The chunk at 128 goes as the runs between the bytes left out, for example [128..231] [234..255]. No other chunk moves, which matters because the firmware treats the chunks at 0 and 128 specially (calibration dump, MAC address).
    • This only happens on firmware that can have byte 232 (see 3). Older firmware is written exactly as before, byte 233 included.

2. The three fields in the config model

  • ButtonLogMode ports the byte's fields and the firmware's correction rules (shimmer_button_log_mode.c, through the web SDK's buttonLogMode.ts), rule for rule. That includes the SR48-6-0 rule: press recording and the TTL input are taken out on that board.

  • Accessors on ShimmerObject:

    • getButtonMode() / setButtonMode()
    • getLogTriggers() / setLogTriggers()
    • isTtlInputEnabled() / setTtlInputEnabled()
    • setButtonLogMode(mode, triggers, ttl)
    • getButtonLogMode() (the raw byte) and getButtonLogModeCorrected()
  • Fields can be set in any order. Log triggers set while the mode is still legacy are kept until the mode is set.

  • The firmware's side effects are applied to the model, in the setters and again before a write, on firmware that has the byte (see 3):

    • userButtonEnable (byte 217 bit 5) is held at the button mode's nearest equivalent.
    • singleTouchStart (byte 218 bit 7) is cleared. No LogAndStream release is built with single touch (IS_SUPPORTED_SINGLE_TOUCH is 0 in log-and-stream-common), so each clears the bit on every configuration it takes, whatever byte 232 holds.

    Because this happens in the model, writeTrial() (SET_TRIAL_CONFIG) sends the same bits. Other firmware is left alone, so a setter called for an SDLog Shimmer, the one firmware the driver offers Single Touch Start for, changes neither bit.

  • Config labels. Configuration.Shimmer3.GuiLabelConfig.BUTTON_MODE, LOG_TRIGGERS and TTL_INPUT go through set/getConfigValueUsingConfigLabel(), so the fields can also be set from a USER fixed config map. They are not added to the Consensys option maps, so Consensys does not show them yet.

3. Detection

LogAndStream_Shimmer3R v1.01.026, a pre-release of 11 Nov 2026, is the first Shimmer3R release with the event channel. No Shimmer3 release has it yet: the latest, v1.01.009, came out before it. The builds before a release report the version of the release before them, so the version cannot say whether a build has it. The driver asks instead:

  • The probe. ShimmerBluetooth.readEventChannelSupport() sends SET_FEATURE (0xB7) with FEATURE_EVENT_STREAM (3) and 0. That changes nothing, since a connection starts opted out. It is sent on connect, and isEventChannelSupported() returns the answer.
    • ACK. The firmware has the event channel.
    • NACK. It does not. processNackFromCommand(), the NACK handling from Handle NACK from a Shimmer3/3R instead of dropping the connection #293, unwinds it like any refused command: the link stays up, the next command goes out, and it counts in the operation's progress. Because the NACK is an answer, it sets isEventChannelSupported() to false and is not passed to eventCommandRefused().
  • Which firmware is asked. SET_FEATURE arrived with log-and-stream-common (df58877 and e5ea88c, 2026-01-29), first released in LogAndStream_Shimmer3R v1.00.052 and LogAndStream_Shimmer3 v1.00.017. That is ShimmerVerObject.isSupportedSetFeature(), and v1.00.017 is also the release isSupportedRn4678ErrorDetection() already gates on.
    • The first version of this PR gated Shimmer3 at v1.00.020, because the check had looked only at the path where v1.00.020 moved the module. In v1.00.017-019 it sat at LogAndStream/log-and-stream-common. Fixed in 955084c.
    • v1.00.017-019 have SET_FEATURE but not the event channel, so they NACK the probe.
    • Older firmware drops the 0xB7 byte unanswered and runs the two bytes after it as commands. 03 is GET_SAMPLING_RATE_COMMAND, which the firmware ACKs, so a probe there would read as "supported". Those versions are never asked, and are treated as having no event channel.
    • The layout places byte 232 (ConfigByteLayoutShimmer3.idxButtonLogMode) only from those releases on.
  • When the byte is trusted (isButtonLogModeMeaningful()): only when the probe was ACKed, or when the value read back is one the firmware keeps.
    • Firmware without the byte leaves 0xFF, which reads as legacy and is written back as read.
    • The dock cannot probe, so there only the value read counts.

4. The event channel's host side (#348, #349)

Following log-and-stream-common docs/SHIMMER3_EVENT_CHANNEL.md and docs/SHIMMER3_BT_COMMUNICATION_PROTOCOL.md §7.15, as shimmer-web-sdk#218 does:

  • Event files. EventFile and EventRecord read events/<experiment>/<session>/<NNN> ("SEVT", a 64-byte header, then 16-byte records). belongsWithDataFile() and getCompanionFile() pair a file with its data file.
  • Live records. writeEventStream(boolean) opts the connection in or out. Each record frame, [0x8A][0xBB][0x10][record][CRC-16], goes to every EventChannelListener, idle or streaming, with any link CRC mode, and only once its own CRC checks out.
  • Status. readEventChannelStatus() sends GET_EVENT_CHANNEL_STATUS_COMMAND (0xB9). It is refused while streaming, or while a start of streaming is queued, since its reply has no 0x8A wrapper.
  • Test window. EventMarkersFrame in ShimmerPCBasicExamples (SensorMapsExample, Tools › Event markers).
  • Android must not opt in yet. ShimmerAndroidAPI's Shimmer.java overrides the streaming parser and does not know the record frame. Reading event files and byte 232 reach Android through the driver pin as usual.

The same full-write problem in the sibling APIs

API Full configuration write Needs the fix?
C# (Shimmer-C-API) WriteShimmer3Infomem() is [Obsolete] (unfinished) and writes the caller's 384 bytes unchanged. Its only callers in ShimmerCapture are commented out. SET_TRIAL_CONFIG sends back the bytes it last read. No
Python (pyshimmer) Writes no configuration at all: the dock API only reads InfoMem. No
Swift (SwiftAPI) writeShimmer3InfoMem() writes the caller's image unchanged, so the library keeps whatever it is given. The ShimmerExamplePlot app mostly writes the image it read, modified. One hard-coded 384-byte preset (accel) has 0 at byte 232 and would turn a Shimmer back to legacy. No for the library. The example preset is a note only.
Web SDK Has its own DEV-971 change, ShimmerResearch/shimmer-web-sdk#218. Covered there

The live event channel is in the web SDK (#218) and shimmer-host-core; C# still needs it.

Who needs a republish

Nothing reaches a consumer until a driver release is cut after this merges (the last is API_v0.11.12_beta):

  • ShimmerAndroidAPI pins shimmerdriver/shimmerbluetoothmanager 0.11.12_beta. The release workflow opens the pin-bump PR. The probe's NACK is handled in the IO thread's not-streaming path, which Shimmer.java does not override, so it arrives with the pin. The live frame does not (section 4).
  • Shimmer-MATLAB-ID: ShimmerJavaClass.jar, also re-vendored by a PR the release workflow opens.
  • ASM_PC builds this repo from a pinned submodule commit. It needs a bump to a master commit after the merge.
  • Consensys picks it up with its next driver update. Importing event files there is DEV-1457.

Tests

  • API_00045_ButtonLogModeTest (ShimmerDriver):
    • the §8.3 table row by row, the 16 values with and without the TTL bit: what a press does, what starts logging, the userButtonEnable held and single touch;
    • that those 32 are exactly the values the firmware keeps;
    • all 256 values against an independent restatement of the correction;
    • encode, setting fields in either order, and SR48-6-0.
  • API_00046_ButtonLogModeConfigWriteTest (ShimmerDriverPC):
    • every kept value survives a full write;
    • userButtonEnable is held, and single touch is cleared in every mode;
    • setters and config labels work;
    • 0xFF is kept as read (single touch still cleared), and a byte the firmware would correct is trusted only after the probe;
    • SR48-6-0;
    • older firmware is written as before, and keeps bytes 217 and 218 when a byte 232 setter is called (LogAndStream before the byte, and SDLog);
    • the Bluetooth write's chunks leave out 232 only while the byte is unknown, and carry the right bytes.
  • API_00047_AudioConfigWriteTest (ShimmerDriverPC): before a read, a Shimmer3R's byte 233 is left out with 232; once read it goes back as read; the sound level turned on before a read is written; 232 set before a read is written without 233.
  • API_00038_EventChannelProbeTest (ShimmerDriverPC), driving the real IO thread over an in-memory radio:
    • ACK → supported;
    • NACK → not supported, the link stays up and the next command goes out, the NACK still counts in the operation's progress, and it is not reported as a refused command;
    • firmware older than SET_FEATURE, and a streaming Shimmer, are not asked.
  • API_00042_EventRecordFileTest (ShimmerDriver), API_00043_EventStreamTest and API_00044_EventStreamReviewReproTest (ShimmerDriverPC): the event channel, from DEV-971: Java event markers - event files, live records, status, and a test window #348 and DEV-971: fix five event-channel parser defects found in #348's review #349.
  • Mutation check. At 3606c951: without the byte-232 write, 9 of the 232 tests fail; without the NACK branch, 2 fail; with the old single-touch rule, 3 fail. At 24d985e5, before DEV-971: #343 review fixes - leave byte 233 out before a read, one NACK handler, renumber tests #351: 3 of API_00047's 5 fail without the byte-233 skip.
  • CI (build and JUnit) passes at a7dde8da, the merge of DEV-971: #343 review fixes - leave byte 233 out before a read, one NACK handler, renumber tests #351.
  • Full suites pass locally at d7d7fd80 (DEV-971: #343 review fixes - leave byte 233 out before a read, one NACK handler, renumber tests #351, the same tree): ShimmerDriver test 219/219, ShimmerDriverPC 161/161 (clean build, as CI runs it), the Java 8 jar (shadowJar -PmatlabJava8) builds, and ShimmerPCBasicExamples compiles.

On a unit

Results posted on #348, #349 and #351, over Bluetooth Classic with the 1-byte link CRC the driver sets at connect.

A Shimmer3R on v1.1.26, with the event channel:

  • the probe ACKed on connect;
  • byte 232: button mode 4 written, and read back as 0x04;
  • live records idle and while streaming, between data packets: 21 in one run, 13 in each of two more, with no sequence gaps and no bad CRC;
  • a double press started logging and another stopped it, and nothing behind the unit's status pushes was lost;
  • the status answered idle and after a stream, and refused straight after startStreaming();
  • a 20 s stream at 51.2 Hz, opted in: 0 samples missing.

A Shimmer3R on v1.1.25, the release before the event channel (at d7d7fd80):

  • The probe was NACKed on connect and again when asked later. isEventChannelSupported() was false, the opt-in and the status read were refused without being sent, the link stayed up, and a stream ran normally afterwards.
  • A configuration written before the read (configure-on-connect): the chunk at 128 went out as [128, 104] and [234, 22], and the firmware ACKed both pieces. The unit kept its own bytes 232 and 233. Both pieces were stored, and the MAC read back as the unit's own.

Not yet on a unit:

  • a Shimmer3: its latest release, v1.01.009, will NACK the probe on every connect;
  • a configuration written before the read on v1.1.26;
  • TTL edges, and an event file read from a card.

Review fix (3606c951)

The rules kept singleTouchStart wherever byte 232 left a press that can start logging, following log-and-stream-common §8.3. That rule is only built with IS_SUPPORTED_SINGLE_TOUCH, which no LogAndStream release sets, so a full write sent a bit the Shimmer drops. The rules now clear it on firmware with the byte and change nothing on other firmware (section 2). §8.3 is corrected in ShimmerResearch/log-and-stream-common#216 (a531896), and the web SDK made the same fix in ShimmerResearch/shimmer-web-sdk#218.

Review fixes (#351)

From the review comment of 2026-11-13: byte 233 is left out before a read as byte 232 is, the probe's NACK goes through the one NACK handler, and the button-mode tests are renumbered API_00045 and API_00046, since API_00036 and API_00037 were taken on master by the log-start-refused tests.

🤖 Generated with Claude Code

marknolan and others added 3 commits November 2, 2026 15:00
Byte 232 holds the button mode, the log triggers and the TTL input enable
on LogAndStream firmware with the event channel. The full configuration
write built its image from zeros, so every configure from Consensys or the
Android config dialog put 0 there and turned the button back to legacy.

- ButtonLogMode: the byte's fields and the firmware's correction rules
  (log-and-stream-common shimmer_button_log_mode.c, via the web SDK's
  buttonLogMode.ts), including press recording and the TTL input taken
  out on SR48-6-0.
- ShimmerObject: the parse keeps the byte as read, 0xFF included, and
  the generator writes it back (corrected where it can be trusted).
  Before packing bytes 217 and 218 it holds userButtonEnable at the
  button mode's nearest equivalent and clears singleTouchStart where no
  press can start logging, as the firmware does. Getters and setters for
  the three fields, and config labels for them.
- The byte is trusted only when the Shimmer ACKed the probe, or the
  value read is one the firmware keeps: firmware without it leaves 0xFF,
  and the dock cannot probe.
- ShimmerVerObject.isSupportedSetFeature(): SET_FEATURE arrived with
  log-and-stream-common, in LogAndStream_Shimmer3R v1.0.52 and
  LogAndStream_Shimmer3 v1.0.20. Older firmware discards 0xB7 and runs
  the two bytes after it as commands (03 is GET_SAMPLING_RATE, which is
  ACKed), so it is never sent there. The layout places byte 232 from
  those releases on; the event channel's builds report the release
  before them, so the version cannot say more.
- ShimmerBluetooth.readEventChannelSupport(): SET_FEATURE with
  FEATURE_EVENT_STREAM and 0, sent on connect. ACK means the event
  channel is there; the NACK is handled without dropping the link, and
  still counts towards the operation's progress.
- writeConfigBytes(): when the byte was never read nor set, as when a
  configuration is written on connect before the read, the chunk at 128
  is sent either side of byte 232, so the Shimmer keeps its own.

Not benched.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
LogAndStream_Shimmer3 v1.00.017 is the first release whose
log-and-stream-common pin has SET_FEATURE (df58877, with the two-byte
argument fix e5ea88c). From v1.00.017 to v1.00.019 the module sat at
LogAndStream/log-and-stream-common, and v1.00.017 builds its Bluetooth
handler from there. The earlier check looked only at the path v1.00.020
moved it to.

The gate moves to v1.0.17, the same release isSupportedRn4678ErrorDetection()
already uses. Those releases NACK FEATURE_EVENT_STREAM, so they read as
having no event channel, and their SET_INFOMEM handler is the same one
that takes the split write around byte 232.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
applyButtonLogModeRules() kept singleTouchStart wherever byte 232 left a
press that can start logging. That rule exists only in a firmware build
with IS_SUPPORTED_SINGLE_TOUCH, and no LogAndStream release has it: every
one clears the bit on each configuration it takes, whatever byte 232
holds. So a full write sent a bit the Shimmer drops, and the driver showed
single touch on where it never acts.

- The rules now clear singleTouchStart on firmware that has byte 232
  (LogAndStream from the release that brought SET_FEATURE, the same gate
  the layout uses), and keep userButtonEnable at the button mode's nearest
  equivalent there as before.
- On other firmware they change nothing. A byte 232 setter called for an
  SDLog Shimmer, or LogAndStream before the byte, used to move
  userButtonEnable and clear single touch on firmware that has neither
  rule, and SDLog is the one firmware the driver offers Single Touch
  Start for.
- API_00037: single touch is cleared in every mode, 0xFF included, and
  firmware without the byte keeps bytes 217 and 218. All three tests fail
  on the old rule.

Not benched.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Brings in DEV-1408 (the callback consumer thread survives a throwing handler).
No conflicts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…hannel

readEventChannelSupport()'s javadoc: no Shimmer3 release has it yet, and the
builds before a release report the release before them, so the driver still
asks rather than going by the version.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@marknolan marknolan self-assigned this Nov 12, 2026
JongChern and others added 3 commits November 13, 2026 12:41
Resolves the conflict #343 predicted with #293 (DEV-982) in
processNotStreamingWaitForAck(): the event channel probe's NACK is taken
first, then any other NACK as a refusal.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…nnel status

The Java driver now does the host side of the event channel that the web SDK
does (log-and-stream-common docs/SHIMMER3_EVENT_CHANNEL.md and
SHIMMER3_BT_COMMUNICATION_PROTOCOL.md section 7.15):

- EventFile, EventRecord: the SD card's events/ files, a 64-byte header and
  16-byte records, with pairing to the data file by its initial timestamp.
- writeEventStream(boolean): SET_FEATURE / FEATURE_EVENT_STREAM. Each record
  frame, [0x8A][0xBB][0x10][record][CRC-16], reaches EventChannelListener.
  Frames are taken between data packets (any link CRC mode), straight after the
  opt-in's ACK, at the start of the buffer, and when idle, including before a
  command's ACK or reply. They are only parsed after an opt-in, and only once
  their own CRC checks out.
- readEventChannelStatus(): GET_EVENT_CHANNEL_STATUS_COMMAND (0xB9). Not
  streaming only: its reply is not wrapped in 0x8A, so the streaming parser
  could not tell it from a packet.

Tests port the web SDK's tests/events/record.test.ts inputs and results, and
run the IO thread over an in-memory radio for the streaming and idle paths.

Android's Shimmer.java overrides the streaming parser and does not know the
frame, so it must not opt in until ported.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
… event channel

Tools > Event markers (Shimmer3R), or the EVENT MARKERS button, opens a window
that does with the Java driver what Shimmer Capture web does with the web SDK:

- whether the unit has the event channel, and asking again;
- configuration byte 232 (button mode, what starts logging, TTL input),
  written with the whole configuration and read back;
- opt in / opt out, the live records with sequence gaps flagged, and the
  frame and bad-CRC counters;
- the event channel status (not while streaming);
- an event file from a card: its header, completeness, the data file it
  pairs with, and its records timed from the file open.

Each live edge is also marked on the main plot as a red vertical bar, placed
by the latest data packet's Shimmer clock against the plot's PC-time axis.

CI does not build ShimmerPCBasicExamples; built locally.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
JongChern and others added 4 commits November 13, 2026 14:44
DEV-971: Java event markers - event files, live records, status, and a test window
Each was first reproduced by a test that failed on the unchanged code
(API_00044_EventStreamReviewReproTest), then fixed:

1. Idle: event records right behind an in-stream reply were lost. After an
   unsolicited status push ([ACK][0x8A][0x71]), sent on a dock, an undock or a
   press that starts logging, clearSerialBuffer() wiped the records queued
   behind it (0 of 2 arrived). With the event channel on, the bytes after the
   reply are now left for the next pass. A record frame after a stray ACK no
   longer has link-CRC bytes cleared after it either: it carries its own CRC.
2. Streaming: an ACK nobody waited for, before a record frame, re-ran the ACK
   handling of the SET that went last. It is now taken only while an ACK is
   awaited.
3. Streaming with no link CRC: the packet before a 0x8A was taken as a packet
   even when the 0x8A was a data byte after a lost byte, so misframed bytes
   reached the app as a sample. It is now taken only once the frame after it
   passes its own CRC; otherwise the bytes are discarded as before.
4. readEventChannelStatus() could be queued behind a start of streaming not yet
   acknowledged, and its unwrapped reply, [ACK][0xBA][9], was then not taken
   between data packets and the command timed out. It is now refused while a
   start of streaming is queued, and a reply that still arrives between packets
   is taken.

ShimmerDriver 219/219 and ShimmerDriverPC 154/154 pass; the Java 8 MATLAB jar
builds.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ehind a bare ACK's CRC

From the review of #349:

- The idle tests' radios now take each burst whole and let reads wait for
  their bytes, as a serial port does (WaitingTestRadio), so the IOThread can
  no longer race the test's writes. API_00043's idle tests use it too.
- The status-push, frame-after-ACK and status-reply tests now run with no
  link CRC, a 1-byte and a 2-byte one.
- The misframing test checks that the good packets after the damaged one
  still arrive; the status-reply test that the command is retired, the lock
  released, the timer cancelled and the packets either side parsed.

The CRC runs reproduced one more defect, which is fixed here: with a link CRC
on (the driver turns it on at connect where the firmware has it), the
records sent in the same burst as the opt-in's ACK were lost, 0 of 2. The
bare ACK's link CRC is left after the ACK is taken; the idle path read it as
an unknown byte and cleared the whole serial buffer, records included. With
the event channel on, the idle path now drops only the bytes it read.

All 7 tests in API_00044 fail on #348's code and pass here. ShimmerDriver
219/219, ShimmerDriverPC 155/155; the Java 8 MATLAB jar builds.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
DEV-971: fix five event-channel parser defects found in #348's review
@JongChern

Copy link
Copy Markdown
Contributor

Reviewed at 24d985e by reading; I did not run the suites, and CI was still running. I read the probe and its NACK handling, the firmware gate, the configuration write around byte 232, and (in #349's review) the streaming and idle parser. I did not go line by line through ButtonLogMode, EventFile or the example window.

Checks out

Needs attention

  • The description is out of date. It lists event files and the live event frame under "Not in this PR", but both are in it now, through DEV-971: Java event markers - event files, live records, status, and a test window #348 and DEV-971: fix five event-channel parser defects found in #348's review #349. The title covers only the byte 232 work.
  • The probe now goes to every unit with SET_FEATURE on every connect. Released firmware should NACK it, but the bench runs so far were on v1.1.26, which ACKs. The NACK path has in-memory tests only. If any release drops the command unanswered, the ACK timer closes the connection. The Java HIL on the bench units' released firmware would settle it.
  • Android must not opt in yet: frames between data packets need a port to Shimmer.java's streaming parser first. The probe itself arrives through the driver's IO thread and is fine.

Smaller

  • Two test numbers are used twice on the branch: API_00036 (ButtonLogMode, LogStartRefused) and API_00037 (ButtonLogModeConfigWrite, StreamingLogStartRefused).
  • processEventChannelProbeNack() duplicates processNackFromCommand(); it could be a few lines inside the general handler.
  • Before a read, byte 232 is skipped and kept, while byte 233 (the audio mode, DEV-1413: Shimmer3R sound level channel (0x38), and keep configuration byte 233 on a full write #346) is written as 0. Worth aligning in a follow-up.

… tests

From the review of #343:

- A Bluetooth configuration write before any read sent a Shimmer3R's
  byte 233 (DEV-99 audio mode and gain, DEV-1413 sound level) as 0, which
  turned the unit's audio mode off. It is now left out as byte 232 is,
  unless it was read or the sound level is turned on, on the same
  firmware. writeMem() takes a list of bytes to leave out. 3 of the 5
  tests in API_00047_AudioConfigWriteTest fail without it.
- The event channel probe's NACK now goes through processNackFromCommand(),
  which marks the event channel unsupported and does not report the NACK
  as a refused command. processEventChannelProbeNack() is gone.
- API_00036_ButtonLogModeTest and API_00037_ButtonLogModeConfigWriteTest
  are now API_00045 and API_00046: master's log-start-refused tests have
  those numbers.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@JongChern

Copy link
Copy Markdown
Contributor

Proposed description for this PR, once #351 is merged. The current description is out of date, as the review comment says. This version adds:

Mark, it's your PR, so take this in if it reads right. The source is below, ready to paste.

Markdown source
DEV-971 adds an event channel to Shimmer3 and Shimmer3R LogAndStream. It also adds configuration byte 232, which holds the button mode, the log triggers and the TTL input enable. The byte is specified in log-and-stream-common `docs/SHIMMER3_CONFIGURATION_INFOMEM.md` §8.3.

**The driver's full configuration write builds its image from zeros, so any configure from Consensys or the Android config dialog put 0 in byte 232.** That silently turned the button back to legacy, which undid a setting such as "button disabled" (`0x01`) or "button disabled, logging off" (`0x11`). This PR closes that gap and adds the byte to the config model. The driver can also tell whether the firmware has the byte.

It also carries the driver's host side of the event channel, merged in from #348 and #349: event files, live event records and the status (section 4).

Firmware and web support are in draft on the same branch name: ShimmerResearch/log-and-stream-common#216, ShimmerResearch/shimmer3-firmware#138, ShimmerResearch/shimmer3r-firmware#319, ShimmerResearch/shimmer-web-sdk#218 and ShimmerResearch/shimmer-capture-web#11.

## What changes

### 1. A full configuration write keeps byte 232
- **Parse.** `ShimmerObject.configBytesParse()` keeps the byte as read, `0xFF` included.
- **Generate.** `configBytesGenerate()` writes the byte back unless the caller set one. Where the byte can be trusted (see 3), it writes the value the firmware would correct it to.
  - This covers every path that goes through the generator: the Bluetooth `writeConfigBytes()`, the dock write, the reload after flashing, and the clone-settings copy.
- **Before anything is read.** A configuration can be written on connect before the read, for example with `isSetupDeviceWhileConnecting()` or a fixed config. `writeConfigBytes()` then leaves out the bytes the driver does not have from this Shimmer, so the Shimmer keeps its own:
  - byte 232, if it was neither read nor set;
  - on a Shimmer3R, byte 233 (DEV-99 audio mode and gain, DEV-1413 sound level), if it was not read and the sound level is not turned on. The host never sets bits 0-3, so writing them unread put 0 there and turned the audio mode off. With the sound level turned on before a read, the byte is still written with bits 0-3 at 0, since one bit cannot be written alone.
  - The chunk at 128 goes as the runs between the bytes left out, for example `[128..231] [234..255]`. No other chunk moves, which matters because the firmware treats the chunks at 0 and 128 specially (calibration dump, MAC address).
  - This only happens on firmware that can have byte 232 (see 3). Older firmware is written exactly as before, byte 233 included.

### 2. The three fields in the config model
- **`ButtonLogMode`** ports the byte's fields and the firmware's correction rules (`shimmer_button_log_mode.c`, through the web SDK's `buttonLogMode.ts`), rule for rule. That includes the SR48-6-0 rule: press recording and the TTL input are taken out on that board.
- **Accessors on `ShimmerObject`:**
  - `getButtonMode()` / `setButtonMode()`
  - `getLogTriggers()` / `setLogTriggers()`
  - `isTtlInputEnabled()` / `setTtlInputEnabled()`
  - `setButtonLogMode(mode, triggers, ttl)`
  - `getButtonLogMode()` (the raw byte) and `getButtonLogModeCorrected()`
- **Fields can be set in any order.** Log triggers set while the mode is still legacy are kept until the mode is set.
- **The firmware's side effects are applied to the model, in the setters and again before a write, on firmware that has the byte (see 3):**
  - `userButtonEnable` (byte 217 bit 5) is held at the button mode's nearest equivalent.
  - `singleTouchStart` (byte 218 bit 7) is cleared. No LogAndStream release is built with single touch (`IS_SUPPORTED_SINGLE_TOUCH` is 0 in log-and-stream-common), so each clears the bit on every configuration it takes, whatever byte 232 holds.

  Because this happens in the model, `writeTrial()` (SET_TRIAL_CONFIG) sends the same bits. Other firmware is left alone, so a setter called for an SDLog Shimmer, the one firmware the driver offers Single Touch Start for, changes neither bit.
- **Config labels.** `Configuration.Shimmer3.GuiLabelConfig.BUTTON_MODE`, `LOG_TRIGGERS` and `TTL_INPUT` go through `set/getConfigValueUsingConfigLabel()`, so the fields can also be set from a `USER` fixed config map. They are not added to the Consensys option maps, so Consensys does not show them yet.

### 3. Detection
LogAndStream_Shimmer3R **v1.01.026** will be the first Shimmer3R release with the event channel, and no Shimmer3 release has it yet. The builds before a release report the version of the release before them, so the version cannot say whether a build has it. The driver asks instead:

- **The probe.** `ShimmerBluetooth.readEventChannelSupport()` sends `SET_FEATURE` (0xB7) with `FEATURE_EVENT_STREAM` (3) and 0. That changes nothing, since a connection starts opted out. It is sent on connect, and `isEventChannelSupported()` returns the answer.
  - **ACK.** The firmware has the event channel.
  - **NACK.** It does not. `processNackFromCommand()`, the NACK handling from #293, unwinds it like any refused command: the link stays up, the next command goes out, and it counts in the operation's progress. Because the NACK is an answer, it sets `isEventChannelSupported()` to false and is not passed to `eventCommandRefused()`.
- **Which firmware is asked.** `SET_FEATURE` arrived with log-and-stream-common (df58877 and e5ea88c, 2026-01-29), first released in LogAndStream_Shimmer3R **v1.00.052** and LogAndStream_Shimmer3 **v1.00.017**. That is `ShimmerVerObject.isSupportedSetFeature()`, and v1.00.017 is also the release `isSupportedRn4678ErrorDetection()` already gates on.
  - The first version of this PR gated Shimmer3 at v1.00.020, because the check had looked only at the path where v1.00.020 moved the module. In v1.00.017-019 it sat at `LogAndStream/log-and-stream-common`. Fixed in 955084ca.
  - v1.00.017-019 have `SET_FEATURE` but not the event channel, so they NACK the probe.
  - Older firmware drops the 0xB7 byte unanswered and runs the two bytes after it as commands. `03` is `GET_SAMPLING_RATE_COMMAND`, which the firmware ACKs, so a probe there would read as "supported". Those versions are never asked, and are treated as having no event channel.
  - The layout places byte 232 (`ConfigByteLayoutShimmer3.idxButtonLogMode`) only from those releases on.
- **When the byte is trusted** (`isButtonLogModeMeaningful()`): only when the probe was ACKed, or when the value read back is one the firmware keeps.
  - Firmware without the byte leaves `0xFF`, which reads as legacy and is written back as read.
  - The dock cannot probe, so there only the value read counts.

### 4. The event channel's host side (#348, #349)
Following log-and-stream-common `docs/SHIMMER3_EVENT_CHANNEL.md` and `docs/SHIMMER3_BT_COMMUNICATION_PROTOCOL.md` §7.15, as shimmer-web-sdk#218 does:
- **Event files.** `EventFile` and `EventRecord` read `events/<experiment>/<session>/<NNN>` ("SEVT", a 64-byte header, then 16-byte records). `belongsWithDataFile()` and `getCompanionFile()` pair a file with its data file.
- **Live records.** `writeEventStream(boolean)` opts the connection in or out. Each record frame, `[0x8A][0xBB][0x10][record][CRC-16]`, goes to every `EventChannelListener`, idle or streaming, with any link CRC mode, and only once its own CRC checks out.
- **Status.** `readEventChannelStatus()` sends `GET_EVENT_CHANNEL_STATUS_COMMAND` (0xB9). It is refused while streaming, or while a start of streaming is queued, since its reply has no 0x8A wrapper.
- **Test window.** `EventMarkersFrame` in ShimmerPCBasicExamples (SensorMapsExample, Tools › Event markers).
- **Android must not opt in yet.** `ShimmerAndroidAPI`'s `Shimmer.java` overrides the streaming parser and does not know the record frame. Reading event files and byte 232 reach Android through the driver pin as usual.

## The same full-write problem in the sibling APIs
| API | Full configuration write | Needs the fix? |
|---|---|---|
| C# (`Shimmer-C-API`) | `WriteShimmer3Infomem()` is `[Obsolete]` (unfinished) and writes the caller's 384 bytes unchanged. Its only callers in ShimmerCapture are commented out. SET_TRIAL_CONFIG sends back the bytes it last read. | **No** |
| Python (`pyshimmer`) | Writes no configuration at all: the dock API only reads InfoMem. | **No** |
| Swift (`SwiftAPI`) | `writeShimmer3InfoMem()` writes the caller's image unchanged, so the library keeps whatever it is given. The `ShimmerExamplePlot` app mostly writes the image it read, modified. One hard-coded 384-byte preset (accel) has 0 at byte 232 and would turn a Shimmer back to legacy. | **No** for the library. The example preset is a note only. |
| Web SDK | Has its own DEV-971 change, ShimmerResearch/shimmer-web-sdk#218. | Covered there |

The live event channel is in the web SDK (#218) and shimmer-host-core; C# still needs it.

## Who needs a republish
Nothing reaches a consumer until a driver release is cut after this merges (the last is `API_v0.11.12_beta`):
- **ShimmerAndroidAPI** pins `shimmerdriver`/`shimmerbluetoothmanager` `0.11.12_beta`. The release workflow opens the pin-bump PR. The probe's NACK is handled in the IO thread's not-streaming path, which `Shimmer.java` does not override, so it arrives with the pin. The live frame does not (section 4).
- **Shimmer-MATLAB-ID**: `ShimmerJavaClass.jar`, also re-vendored by a PR the release workflow opens.
- **ASM_PC** builds this repo from a pinned submodule commit. It needs a bump to a master commit after the merge.
- **Consensys** picks it up with its next driver update. Importing event files there is DEV-1457.

## Tests
- `API_00045_ButtonLogModeTest` (ShimmerDriver):
  - the §8.3 table row by row, the 16 values with and without the TTL bit: what a press does, what starts logging, the `userButtonEnable` held and single touch;
  - that those 32 are exactly the values the firmware keeps;
  - all 256 values against an independent restatement of the correction;
  - `encode`, setting fields in either order, and SR48-6-0.
- `API_00046_ButtonLogModeConfigWriteTest` (ShimmerDriverPC):
  - every kept value survives a full write;
  - `userButtonEnable` is held, and single touch is cleared in every mode;
  - setters and config labels work;
  - `0xFF` is kept as read (single touch still cleared), and a byte the firmware would correct is trusted only after the probe;
  - SR48-6-0;
  - older firmware is written as before, and keeps bytes 217 and 218 when a byte 232 setter is called (LogAndStream before the byte, and SDLog);
  - the Bluetooth write's chunks leave out 232 only while the byte is unknown, and carry the right bytes.
- `API_00047_AudioConfigWriteTest` (ShimmerDriverPC): before a read, a Shimmer3R's byte 233 is left out with 232; once read it goes back as read; the sound level turned on before a read is written; 232 set before a read is written without 233.
- `API_00038_EventChannelProbeTest` (ShimmerDriverPC), driving the real IO thread over an in-memory radio:
  - ACK → supported;
  - NACK → not supported, the link stays up and the next command goes out, the NACK still counts in the operation's progress, and it is not reported as a refused command;
  - firmware older than `SET_FEATURE`, and a streaming Shimmer, are not asked.
- `API_00042_EventRecordFileTest` (ShimmerDriver), `API_00043_EventStreamTest` and `API_00044_EventStreamReviewReproTest` (ShimmerDriverPC): the event channel, from #348 and #349.
- **Mutation check.** At `3606c951`: without the byte-232 write, 9 of the 232 tests fail; without the NACK branch, 2 fail; with the old single-touch rule, 3 fail. At `24d985e5`, before #351: 3 of `API_00047`'s 5 fail without the byte-233 skip.
- **Full suites** pass locally at `d7d7fd80` (#351): ShimmerDriver `test` 219/219, ShimmerDriverPC 161/161 (`clean build`, as CI runs it), the Java 8 jar (`shadowJar -PmatlabJava8`) builds, and ShimmerPCBasicExamples compiles.

## On a unit
Results posted on #348 and #349, from a Shimmer3R on LogAndStream v1.1.26 over Bluetooth Classic, with the 1-byte link CRC the driver sets at connect:
- the probe ACKed on connect;
- byte 232: button mode 4 written, and read back as `0x04`;
- live records idle and while streaming, between data packets: 21 in one run, 13 in each of two more, with no sequence gaps and no bad CRC;
- a double press started logging and another stopped it, and nothing behind the unit's status pushes was lost;
- the status answered idle and after a stream, and refused straight after `startStreaming()`;
- a 20 s stream at 51.2 Hz, opted in: 0 samples missing.

**Not yet on a unit:**
- **The probe on released firmware that NACKs it:** a Shimmer3R on v1.00.052 to v1.01.025, or a Shimmer3 on v1.00.017 or later. Only `API_00038`'s in-memory radio covers it.
- The byte-233 change, TTL edges, and an event file read from a card.

## Review fix (`3606c951`)
The rules kept `singleTouchStart` wherever byte 232 left a press that can start logging, following log-and-stream-common §8.3. That rule is only built with `IS_SUPPORTED_SINGLE_TOUCH`, which no LogAndStream release sets, so a full write sent a bit the Shimmer drops. The rules now clear it on firmware with the byte and change nothing on other firmware (section 2). §8.3 is corrected in ShimmerResearch/log-and-stream-common#216 (`a531896`), and the web SDK made the same fix in ShimmerResearch/shimmer-web-sdk#218.

## Review fixes (#351)
From the review comment of 2026-11-13: byte 233 is left out before a read as byte 232 is, the probe's NACK goes through the one NACK handler, and the button-mode tests are renumbered `API_00045` and `API_00046`, since `API_00036` and `API_00037` were taken on master by the log-start-refused tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

🤖 Generated with Claude Code

@JongChern

Copy link
Copy Markdown
Contributor

Bench, 13 Nov 2026: the event channel probe on released firmware that NACKs it.

A Shimmer3R on LogAndStream v1.1.25, the release without the event channel, over Bluetooth Classic with the 1-byte link CRC the driver sets at connect. Code: d7d7fd80 (#351). A headless check: connect, read what the driver learnt, ask again, then stream for 5 s. Nothing in the unit's configuration, clock or logging was written.

Check Result
Connect, with the probe in the sequence Finished in 6.8 s, link kept
The probe on connect NACKed. isEventChannelSupported() false, logged as NACK Received for Command: [0xB7] SET_FEATURE_COMMAND, no event channel
writeEventStream(true) Refused, nothing sent
readEventChannelStatus() Refused, nothing sent
The probe asked again while idle NACKed again: still false, link kept
A stream after the NACKs Started; 245 packets in 5 s at 51.2 Hz; stopped, back to Ready
Link Never lost; no ACK timeouts
Byte 232 Read as 0x00 (legacy), a value the firmware keeps
Byte 233 Read; audio mode and gain 0

The same check just before, with the unit still on v1.1.26: the probe ACKed, connect took 4.1 s, and a 5 s stream gave 247 packets with the link kept.

Not covered:

  • A Shimmer3 on firmware with SET_FEATURE (v1.00.017 or later).
  • The configuration write before a read, which leaves bytes 232 and 233 out. This run did not write a configuration.

🤖 Generated with Claude Code

@JongChern

Copy link
Copy Markdown
Contributor

Bench, 13 Nov 2026: a configuration written on connect, before the read.

A Shimmer3R on LogAndStream v1.1.25 over Bluetooth Classic, with the 1-byte link CRC. Code: d7d7fd80 (#351). The configure-on-connect path (setSetupDeviceWhileConnecting(true)) queues writeConfigBytes(false) ahead of the configuration read. So the driver writes its image, named BENCH351, without having read bytes 232 and 233, and leaves them out.

The run:

  1. Read the unit's configuration and save it.
  2. Write it back with bytes 232 and 233 set to 0x04, and the unused bytes 234 and 255 set to 0x5A, so the write would show over them.
  3. Reconnect with configure-on-connect.
  4. Write the saved configuration back.
Check Result
SET_INFOMEM sent [0, 128] [128, 104] [234, 22] [256, 128] [384, 128]: the chunk at 128 in two pieces, 232 and 233 in neither
Firmware's answer Every piece ACKed: no NACK, no timeout, link kept
Bytes 232 and 233 Still 0x04: the unit kept its own
Piece [128..231] Landed: the name reads back as BENCH351. The MAC (224-229, written as 0xFF) reads back as the unit's own, so the firmware's handling of the chunk at 128 still applies when it is 104 bytes long
Piece [234..255] Landed: bytes 234 and 255 went from 0x5A to the 0x00 written
Every other byte Read back as written
Saved configuration written back Read back as saved, except byte 230, the SD file write flags (0x00 to 0x01), which the firmware sets after a write and later clears. The next run's first read matched the original byte for byte

Run three times. The first attempt never reached the write: COM32 was still busy 3 s after the previous disconnect. The second and third passed; the third added the 0x5A check.

Not covered:

🤖 Generated with Claude Code

DEV-971: #343 review fixes - leave byte 233 out before a read, one NACK handler, renumber tests

@JongChern JongChern left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at a7dde8d. With #349 and #351 merged in, the points from my review are fixed, the branch is up to date with master, and the bench runs on v1.1.25 (probe NACKed, a write before the read) and v1.1.26 (the live event channel) cover the risks I raised. I also read ButtonLogMode against the firmware's correction rule and EventFile's bounds checks this time. Reviewed by reading; CI on this head was still running.

Before merge: take up the proposed description (it's in a comment above), and let CI finish. After merge: the Java HIL on master's commit before the next release, and the Android opt-in guard with the driver pin bump. No Shimmer3 has been on the bench for this yet.

@JongChern
JongChern marked this pull request as ready for review November 13, 2026 09:15
@JongChern
JongChern merged commit 1e60708 into master Nov 13, 2026
2 checks passed
JongChern added a commit that referenced this pull request Nov 13, 2026
Brings in #343 as merged, with #351's event channel fixes and renumbered tests.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
JongChern added a commit that referenced this pull request Nov 13, 2026
JongChern added a commit that referenced this pull request Nov 16, 2026
DEV-1459: merge master (#343) into DEV-1459, and write an audio mode set before a read
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants