Skip to content

DEV-1148: consume a bare ACK's link CRC while streaming - #121

Merged
JongChern merged 2 commits into
masterfrom
DEV-1148_streaming_bare_ack_crc
Oct 21, 2026
Merged

JongChern merged 2 commits into
masterfrom
DEV-1148_streaming_bare_ack_crc

Conversation

@marknolan

@marknolan marknolan commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

Fixes DEV-1148. This is the Android half of DEV-1147 (ShimmerResearch/Shimmer-Java-Android-API#319).

Defect

  • DEV-1147's driver fix never reaches Android. Shimmer.java overrides the driver's streaming parser (processPacket(), processWhileStreaming(), processInstreamResponse() and clearSingleDataPacketFromBuffers()), so a driver bump alone does not help.
  • The bare ACK's CRC was read as a parse error. With a link CRC on, the reply to a SET command while streaming is FF F4 65, or FF F4 with a 1-byte CRC. The override took the F4 for a parse error ("Unknown parsing error while streaming") and then discarded one byte at a time ("Packet syncing problem") until it reached the next packet:
    • the data packet before the ACK was lost;
    • the ACK was never processed;
    • 2 s later the driver's ACK timer, seeing a stream, cleared the instruction queue.
  • Start logging on a streaming device does nothing.
    • ShimmerBluetoothManagerAndroid.startLogging() accepts a STREAMING device.
    • startSDLogging() queues SET_RWC and then START_LOGGING_ONLY. The RTC write's ACK is lost, and the timeout clears the queue.
    • Reproduced against the real 2 s timer: the queue went from [SET_RWC, START_LOGGING_ONLY] to [], and isSDLogging() stayed false.
  • The CRC is on by default. The driver's initializeShimmer3or3R() sets a 1-byte CRC on firmware version code 8 or later.
  • A second defect on the same path, as in DEV-1147. On the pinned driver (0.11.7_beta), checkCrc() skips to the next frame itself when a 2-byte CRC fails on its second byte. The override then discarded one more byte, the start of the next packet, so that packet was lost as well.

Fix

  • processPacket(), data packet followed by an ACK. When the bytes after the ACK are the bare ACK's CRC (isBareAckCrcSoFar(), checked against BARE_ACK_CRC), it waits for the whole CRC and then:
    • passes the packet on if the packet's own CRC checks out, and otherwise drops only the packet;
    • consumes the packet, the ACK and its CRC exactly once, keeping any later bytes with their timestamps (the override's clearSingleDataPacketFromBuffers() already works this way);
    • processes the ACK (processAckWhileStreaming(), shared with the no-CRC path, which is otherwise unchanged).
  • checkCrc() only checks. It is overridden as DEV-1147 changes the driver's version. Once the driver pin includes DEV-1147, the override can go.
  • The bulk-read loop in processWhileStreaming() is unchanged.
  • These mirror the driver's private BARE_ACK_CRC, isBareAckCrcSoFar() and processAckWhileStreaming().

Testing

First commit: a local unit-test setup. src/test had no JUnit, so nothing in the library could run on the JVM. src/androidTest holds only a Shimmer2R device test, which calls a Shimmer(Handler) constructor that no longer exists.

New AAPI_00002_StreamingBareAckCrcTest (10 tests). It ports the driver's API_00027_StreamingBareAckCrcTest. It drives the real Shimmer through a subclass that replaces only the socket.

  • processWhileStreaming() reads whatever has arrived, so each bare-ACK scenario is replayed with the bytes arriving in reads of every size from 1 to 48 bytes, the last being all at once.
  • It counts the bytes the parser skips to resync. A CRC left behind as junk therefore fails the test, as does any byte lost.

Proof the tests catch the bug.

  • On unfixed master, 7 of the 10 fail. The 3 covering paths that already worked pass: no CRC, and an in-stream VBATT response with a 1-byte and a 2-byte CRC.
  • With the fix but the checkCrc() override removed, exactly the 2 tests with a 2-byte CRC failing on its second byte fail.
  • F4 65 was checked against an independent implementation of the CRC.

Full runs (Gradle 8.5 wrapper, JDK 21):

Command Result
./gradlew -Pandroid.useAndroidX=true :ShimmerAndroidInstrumentDriver:test 10 tests, 0 failures, in debug and in release
:ShimmerAndroidInstrumentDriver:assembleDebug assembleRelease BUILD SUCCESSFUL

Running the tests. The build root's gradle.properties is gitignored. It needs android.useAndroidX=true, as it did when it was tracked. If yours also sets android.enableJetifier=true, add android.jetifier.ignorelist=jackson-core: Jetifier cannot read jackson-core 2.18.2's Java 21 classes on the unit-test classpath.

Not fixed here: in-stream defects found while porting

In-stream responses (FF 8A …) have defects of their own in this override. This PR leaves them alone. Each was reproduced with throwaway tests, not committed.

  1. A response whose CRC arrives in a later read is processed twice.
    • processInstreamResponse() checks for the response without its CRC.
    • clearSingleDataPacketFromBuffers() then refuses to clear past the end of the buffer.
    • The next pass parses the response again, so the data packet before it is passed on twice.
  2. The reply to a directly written command removes the head of the instruction queue.
    • While streaming, readBattery() and readStatusLogAndStream() write straight to the socket instead of queueing. The in-stream branch still runs removeInstruction(0), cancels the ACK timer and unlocks the queue.
    • If such a reply lands while a queued SET command awaits its ACK, that command is removed, and its ACK is then processed against the next instruction. For SET_RWC with START_LOGGING_ONLY behind it, processAckFromSetCommand() throws ArrayIndexOutOfBoundsException on the IO thread.
    • This already happens with the CRC off. With this PR, the CRC path processes ACKs too, so it now shares that exposure where it used to lose the ACK.
    • While streaming, only the 60 s battery timer (the 5 s status timer stops) or an app call can trigger it. The window is roughly 100 ms per 60 s for each SET command.
  3. While STREAMING_AND_SDLOGGING, an in-stream response falls through to the driver's processInstreamResponse(). That version reads the response from the socket instead of from the bytes already buffered.

These need a ticket of their own.

Siblings and docs (CLAUDE.md §4)

  • Siblings. DEV-1147's PR covers the sibling APIs (C#, Web SDK, Swift, pyshimmer), and nothing here changes that assessment.
  • Within this repo. Shimmer.java is the only parser: neither Shimmer4Android nor VerisenseDeviceAndroid overrides it.
  • Driver pin. Unchanged (DEV-1115). This PR does not depend on #319.
  • Docs need correcting. Add AGENTS.md for coding agents #120's AGENTS.md says this repo has no parsing logic and that a protocol fix never needs re-writing here. The shared CLAUDE.md §4 says the same. Shimmer.java contradicts both, and that is how DEV-1147 missed Android.

Not verified on hardware

No device has been run against this. To check on one:

  1. Stream from a Shimmer3R with the default 1-byte CRC.
  2. Call startLogging(), for example from bluetoothManagerExample.
  3. Confirm the state reaches STREAMING_AND_SDLOGGING within a second, and that logcat shows no "Unknown parsing error while streaming".

🤖 Generated with Claude Code

marknolan and others added 2 commits October 4, 2026 21:49
src/test had no JUnit dependency, so nothing in the library could be
tested on the JVM. Add JUnit 4 and have the stub android.jar return
defaults: Shimmer's constructor and its logging call into it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
With a Bluetooth link CRC on, the firmware sends a CRC after a bare ACK
too, so a SET command's reply while streaming is FF F4 65, or FF F4 with
a 1-byte CRC. Shimmer.processPacket() overrides the driver's parser and
took the F4 for a parse error: it threw away the data packet before the
ACK a byte at a time, never processed the ACK, and 2 s later the ACK
timer cleared the instruction queue. startSDLogging() on a streaming
device therefore never sent START_LOGGING_ONLY_COMMAND.

Port DEV-1147's driver fix to the override. When the bytes after the ACK
are the bare ACK's CRC, wait for all of it, pass the packet on if its own
CRC checks out, consume the packet, the ACK and its CRC exactly once
(keeping later bytes and their timestamps), and process the ACK. The
bulk-read loop in processWhileStreaming() is unchanged.

Also override checkCrc() so it only checks. On the pinned driver
(0.11.7_beta) it skips to the next frame itself when a 2-byte CRC fails
on its second byte, and processPacket() then discarded a byte more: the
start of the next packet.

AAPI_00002_StreamingBareAckCrcTest ports the driver's
API_00027_StreamingBareAckCrcTest and replays each scenario with the
bytes arriving in reads of every size from 1 to 48.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
marknolan added a commit that referenced this pull request Oct 4, 2026
ad9aa5c corrected the wrapper claim as of master. This brings it up to
DEV-1148 (#121) and fills gaps in the build notes:

- checkCrc() is overridden too since DEV-1148, as a stopgap until the
  driver pin includes DEV-1147. The section said it was the driver's.
- readBattery() and readStatusLogAndStream() write straight to the
  socket while streaming instead of queueing, and processWhileStreaming()
  reads in bulk where the driver reads a byte at a time.
- DEV-1147's driver fix had to be ported here by hand as DEV-1148. That
  is the consequence a reader needs, so it is now said outright.
- The build root's gradle.properties is gitignored, and AGP refuses to
  build without android.useAndroidX=true. Jetifier has to stay off: it
  fails on a transitive jackson-core 2.18.2 (class file version 65).
- DEV-1148 adds JVM unit tests (JUnit 4, returnDefaultValues). The
  command that runs them is documented.
- "Java 11" was the source and target level. AGP 8 needs Gradle to run
  on JDK 17 or later.

The unit-test setup and the checkCrc() override exist only on #121's
branch, so this should merge after #121.

Docs only.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@JongChern

Copy link
Copy Markdown
Contributor

Hardware check: Classic HIL, with and without this PR

The Android HIL (#128) ran this PR on a Shimmer3 (LogAndStream v1.1.5) over Bluetooth Classic. Phone: Redmi M2101K6G, Android 11. Driver 0.11.9_beta.

  • With this PR: merged onto current master (65a33ae).
  • Baseline: the same tree with master's Shimmer.java.
  • Same unit, same session, straight after a power cycle.
Code Link CRC streamRest midStreamSet
this PR 2-byte FAIL: 7.6% lost (580/7587) SD logging started and stopped 3/3, 0 parser errors. FAIL on packet loss: 5.3% away from the SD transitions
this PR 1-byte (default) PASS, 0/7651 lost SD logging started and stopped 3/3, 0 parser errors, 0.42% lost (nearly all at the SD transitions). FAIL on 1 packet lost 86 ms after one battery read
master 2-byte FAIL: 9.6% lost (730/7618) SD logging never started, 15 "Packet syncing problem", stopSDLogging() blocked, stream went silent
master 1-byte (default) PASS, 0/7667 lost SD logging never started, 13 "Packet syncing problem", stopSDLogging() blocked, stream went silent

This fixes DEV-1148 on hardware, with both CRC modes, including the default 1-byte. Master reproduces the defect with both.

The failures that remain don't come from this PR:

  • 2-byte CRC packet loss (DEV-1248), a separate defect. Data packets that fail checkCrc are dropped without a log line, in bursts of 1–9 packets. It's the same with and without this PR (7.6% vs 9.6%), and the 1-byte CRC loses nothing either way. The 147 packets midStreamSet counts as lost near battery reads with the 2-byte CRC are these same bursts.
  • One packet lost after a battery read, 1-byte CRC. The battery reply is an in-stream response, which is DEV-1149's area (DEV-1149: handle in-stream responses once, whole, and by opcode #122). Master never reached the battery reads, so there's no baseline for it.

Off the bench: merged onto today's master, the 10 new unit tests pass and the library builds against driver 0.11.7_beta and 0.11.9_beta. The checkCrc() override matches the signature in both.

Before merging:

🤖 Generated with Claude Code

@JongChern
JongChern merged commit 15050ce into master Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants