Skip to content

DEV-982: handle NACK from a Shimmer3/3R instead of timing out the connection - #204

Merged
JongChern merged 3 commits into
masterfrom
DEV-982_nack_handling
Oct 20, 2026
Merged

JongChern merged 3 commits into
masterfrom
DEV-982_nack_handling

Conversation

@marknolan

@marknolan marknolan commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

The protocol has had a NACK_COMMAND_PROCESSED (0xFE) reply for a long time. This
API never learned about it, so 0xFE fell through to the switch default in both
branches of ReadData().

What was happening

  • While streaming — the default logs "Misaligned ByteStream Detected" and nulls
    the packet in progress. A clean refusal was reported as stream corruption.
  • Otherwise — the byte was consumed and dropped silently.

Either way the reply the caller was waiting for never arrived, ReadByte() carried on
raising TimeoutException, and once StreamTimeOutCount passed 10 the API raised
"Connection lost" and disconnected — roughly ten seconds after the device had
already told us, cleanly, that it would not do the thing.

A refused command was indistinguishable from a dead link, and the log actively
misdirected.

Why this is being raised now

This is the same defect as
Shimmer-Java-Android-API#293,
found by checking the sibling implementations of the wire format afterwards. The
mechanism and the timing differ — Java tore the link down after two seconds via its ACK
timer, this API times out and disconnects after about ten — but the cause is identical:
only ACK was ever tested for. SwiftAPI has the same gap and is being fixed alongside;
the TypeScript SDK already handles it.

Where it bites

Of the ~14 sendNack sites in the firmware's shimmer_bt_uart.c, only one is the
blocked-while-sensing gate, and sending commands mid-stream is not a supported
operation anyway. The rest are reachable while idle:

Condition
SD sync enabled refuses everything except SET_SD_SYNC_COMMAND and ACK — ShimBt_isCmdAllowedWhileSdSyncing
a sync command while sync is disabled same XOR gate
payload truncated past args[]
SET_SAMPLING_RATE_COMMAND with a zero clock divider
calibration RAM write out of range
SET_INFOMEM_COMMAND offset out of range
SET_DAUGHTER_CARD_MEM EEPROM write failure
SET_CHARGE_STATUS_LED_COMMAND, SET_BT_COMMS_BAUD_RATE dispatched but never implemented

The SD sync one is the sharpest: with sync enabled the firmware refuses the whole
connect sequence, so a sync-enabled Shimmer cannot be connected to at all.

What this changes

ProcessNackFromCommand() is protected virtual, so an application can override it to
surface the refusal; by default it logs and raises a notification event on the existing
MSG_IDENTIFIER_NOTIFICATION_MESSAGE channel. It also:

  • clears mWaitingForStartStreamingACK — a refused START_STREAMING would
    otherwise leave that flag set for the rest of the session, so the next stray ACK
    would flip the state machine to STREAMING.
  • consumes the CRC bytes the firmware appends to the NACK packet when CRC mode is
    on. The firmware writes the refusal as a complete response packet and appends CRC to
    it exactly as it would to a response, so without this those bytes would be read as
    the next packet header.

None of the three SDBT_switch overrides looked at 0xFE, so adding an explicit case
ahead of the default changes nothing else.

The Xamarin duplicate is deliberately left alone

ShimmerCaptureXamarin carries a second, diverged copy of ShimmerBluetooth.cs with
the same defect. I patched it, then took it back out: nothing has touched that tree
since July 2017, neither CI workflow builds it, and Xamarin is end of life, so
nothing could verify the change — not CI, not a local build. I would rather flag it here
than ship unverified edits to dead code. Worth doing if that sample is ever revived.

Verification

ShimmerAPI builds clean — 0 errors. The ten warnings are pre-existing and in files
this does not touch (TestRadio.cs, SerialPortRadio.cs, ShimmerSDLog.cs,
ShimmerLogAndStream.cs) plus NU1701 package-restore noise.

No hardware test. There is no fake transport for the receive path in this repo, so
exercising a NACK end-to-end would mean building one; that is worth its own piece of
work rather than being half-built inside this change. The equivalent change in
pyshimmer does carry tests, because that repo already has the fixtures for it.

Worth a bench check alongside the Java one: enable SD sync on a Shimmer, connect, and
confirm the connection now survives and the refusal is reported, where previously it
dropped with "Connection lost".

🤖 Generated with Claude Code

The protocol has had a NACK (0xFE) reply for a long time; this API never
learned about it. A 0xFE fell through to the switch default in both
branches of ReadData(): while streaming that logged "Misaligned
ByteStream Detected" and discarded the packet in progress, and otherwise
it was dropped silently.

Either way the reply the caller was waiting for never arrived, ReadByte()
carried on raising TimeoutException, and once StreamTimeOutCount passed
10 the API raised "Connection lost" and disconnected - about ten seconds
after a refusal the device had already reported cleanly. A refused
command was indistinguishable from a dead link, and the log actively
misdirected.

This is the same defect as Shimmer-Java-Android-API#293, found by
checking the sibling implementations of the wire format after that one.
The timing and the mechanism differ - Java tore the link down after two
seconds via its ACK timer - but the cause is identical: only ACK was
tested for.

The firmware refuses more than it used to. Every command except
SET_SD_SYNC_COMMAND and ACK is refused while SD sync is enabled
(ShimBt_isCmdAllowedWhileSdSyncing), which includes the whole connect
sequence, so a sync-enabled Shimmer cannot be connected to at all. Also
any SET while sensing, a sync-mode mismatch, an out-of-range InfoMem or
calibration write, and several commands the dispatcher accepts but that
were never implemented.

ProcessNackFromCommand() is protected virtual so an application can
surface the refusal; by default it logs and raises a notification event.
It clears mWaitingForStartStreamingACK, which a refused START_STREAMING
would otherwise leave set for the rest of the session, and consumes the
CRC bytes the firmware appends to the NACK packet when CRC mode is on -
those would otherwise be read as the next packet header.

None of the three SDBT_switch overrides looked at 0xFE, so adding an
explicit case ahead of the default changes nothing else.

ShimmerCaptureXamarin carries a second, diverged copy of
ShimmerBluetooth.cs with the same defect. It is deliberately left alone:
nothing has touched it since July 2017, neither CI workflow builds it,
and Xamarin is end of life, so a change there could not be verified by
anything. Worth fixing if that sample is ever revived.

ShimmerAPI builds clean; the ten warnings are pre-existing and in files
this does not touch.

Co-Authored-By: Claude Opus 5 <[email protected]>
@marknolan

marknolan commented Oct 4, 2026 •

Copy link
Copy Markdown
Member Author

Two things found while working on #212 (DEV-976), which changes how WriteCRCMode and ReadData handle the ACK to SET_CRC_COMMAND. #212 merges cleanly with this branch. Both were checked in a scratch merge of the two branches, using #212's test double CrcLinkDevice, which answers as the firmware does.

1. When not streaming, a NACK's CRC is dropped twice.

  • ProcessNackFromCommand drops it (ShimmerBluetooth.cs:1268-1275 at 69442ca).
  • In the connected-state branch (:1606), the loop's trailing drop (:1903-1913) then drops as many bytes again. It runs for every b other than 0xFF, and here b is 0xFE.
  • The streaming branch has no trailing drop, so only the connected state is affected.
  • The extra bytes come out of whatever arrives next within the 1 s read timeout:
    • With a one-byte CRC that is the next reply's leading ACK. That is harmless, except when the reply is the ACK to START_STREAMING: the API then stays Connected and delivers no packets. The comment below measured that in both CRC modes.
    • With a two-byte CRC it is the ACK and the response code, so that reply is lost.
  • In the test double, a NACKed WriteAccelRange was followed by ReadSamplingRate. With two bytes the rate was never read; with one byte it was.

Suggested fix, tried in the scratch merge: leave the NACK out of the trailing drop.

if (b != 0xff && b != (byte)PacketTypeShimmer2.NACK_COMMAND)

With that, the rate is read in both modes.

2. A refused SET_CRC_COMMAND still changes the API's mode. This part needs #212.

  • The firmware skips the switch when it NACKs, so the NACK comes in the old mode. In log-and-stream-common Comms/shimmer_bt_uart.c at the v1.01.017 pin:
    • the NACK gates (:839-855) come before the switch whose SET_CRC_COMMAND case (:942) sets the mode;
    • the NACK is framed with the unchanged mode (:2415-2426).
  • With DEV-976: switch the link CRC at the SET_CRC ACK, which comes in the new mode #212, ReadData switches only on an ACK, so ProcessNackFromCommand drops the right number of bytes.
  • But WriteCRCMode still applies the requested mode when its 300 ms wait ends, as master always has.
  • In the test double, an off → two-byte switch was refused. The API ended at TWO_BYTE and the device stayed at OFF. The first read after that arrived. The second, 200 ms later, was lost.
  • When it happens: with SD sync enabled and a CRC asked for. Sync refuses every command except SET_SD_SYNC_COMMAND, including the connect-time CRC write.

Suggested fix, tried in the scratch merge on top of #212:

  • In ProcessNackFromCommand, if mWaitingForCRCModeACK is set, clear it and set a new mCRCModeRefused flag.
  • WriteCRCMode clears that flag before it writes, and skips BluetoothCRCMode = mode when the flag is set.

With both changes the API stays at OFF, and both reads arrive. #212's 11 tests still pass with them. The change fits in whichever of #204 and #212 merges second.

🤖 Generated with Claude Code

@marknolan

Copy link
Copy Markdown
Member Author

This drops a NACK's CRC twice while connected. With a link CRC on, ProcessNackFromCommand() reads the CRC bytes. Then the connected branch of ReadData() drops them again after the switch:

if (BluetoothCRCMode != BTCRCMode.OFF)
{
    if (b != 0xff)   // 0xFE passes, so the NACK's CRC is dropped a second time

Before this PR, that trailing drop alone took a NACK's CRC correctly: 0xFE fell through to the default, then to this drop. Now the second drop reads N more bytes, and those are the start of whatever arrives within ReadTimeout:

After a NACK, with a CRC on Result
a GET reply, one-byte CRC its ACK is eaten, which is harmless
a GET reply, two-byte CRC FF and the response code are eaten. The reply is lost, and its payload is parsed as responses
StartStreaming(), either CRC mode its ACK is eaten. The API stays Connected and delivers no data (0 of ~65 packets)

The streaming branch has no trailing drop, so there the drop in ProcessNackFromCommand() is the only one, and it is correct.

How I reproduced it. I merged this branch onto #212, with and without #213, and ran the API's read loop against #212's CrcLinkDevice test double. I made the double NACK SET_GSR_RANGE_COMMAND. Results:

  • a NACK followed by a sampling-rate read fails with a two-byte CRC;
  • a NACK followed by StartStreaming() stays Connected in both CRC modes;
  • everything passes with the CRC off.

Suggested fix. Drop the CRC in the streaming branch's NACK_COMMAND case, and leave the connected branch to its trailing drop:

// streaming branch only
case (byte)PacketTypeShimmer2.NACK_COMMAND:
    for (int k = 0; k < (int)BluetoothCRCMode; k++)
    {
        ReadByte();
    }
    ProcessNackFromCommand();
    break;

Then remove the CRC loop from ProcessNackFromCommand(). With that change, all of the cases above pass, and so does a NACK in the middle of a stream, in both modes. It also keeps framing out of the method this PR invites applications to override. An override that doesn't call base can then no longer leave the CRC in the stream.

I found this while fixing DEV-1145 in #213, which deals with the same trailing drop for a bare ACK.

JongChern and others added 2 commits October 20, 2026 14:00
While connected, ProcessNackFromCommand dropped the NACK's link CRC and
the drop after every reply that does not start with an ACK dropped it
again, taking the next reply's first bytes: with a two-byte CRC, the ACK
and response code of the next reply, or the start command's ACK. The
drop now happens in ReadData only: after the switch while connected, and
in the NACK case while streaming, where nothing else drops it.

The firmware refuses SET_CRC_COMMAND before it switches, and frames the
NACK in the old mode. WriteCRCMode applied the new mode anyway when its
wait ended; it now keeps the old one when the command is refused.

ShimmerNackTest runs the read loop against CrcLinkDevice, which can now
refuse chosen commands. Five of its seven tests fail without these
fixes; the mid-stream one fails if the streaming drop is removed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copilot AI lite review requested due to automatic review settings October 20, 2026 06:15

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Moderate CRC-handling and protocol-state issues remain unresolved.

1 open issue
What changed in this PR

Adds Shimmer3/3R NACK handling so refused commands are reported instead of causing stream corruption or connection loss.

Changes:

  • Handles NACK responses and CRC-aware state handling.
  • Preserves streaming and CRC state after refusals.
  • Adds simulated-device regression tests.
File Reviewed changes
ShimmerAPI/​ShimmerUnitTests/​ShimmerUnitTests.csproj Includes the new NACK test file.
ShimmerAPI/​ShimmerUnitTests/​ShimmerNackTest.cs Adds NACK behavior regression tests.
ShimmerAPI/​ShimmerUnitTests/​ShimmerCrcSwitchTest.cs Extends the test device to emit NACK responses.
ShimmerAPI/​ShimmerAPI/​ShimmerBluetooth.cs Implements NACK parsing, notifications, CRC handling, and state updates.

Review effort: Lite

/// or after calling this, so an override that does not call it cannot
/// leave the CRC in the stream.
/// </remarks>
protected virtual void ProcessNackFromCommand()

@JongChern JongChern left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Both of Mark's findings are fixed in 9357f83:

  • A NACK's link CRC is now dropped once. While connected, the drop after every reply that does not start with an ACK takes it; while streaming, the NACK case does. ProcessNackFromCommand no longer drops it.
  • A NACKed SET_CRC_COMMAND keeps the old CRC mode, since the firmware refuses before it switches.

ShimmerNackTest adds 7 tests against CrcLinkDevice, which can now refuse chosen commands. Five fail without the fixes, and the mid-stream test fails if the streaming drop is removed. 77/77 pass locally and in CI, with master merged in.

@JongChern
JongChern merged commit 8dcb20b into master Oct 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants