Skip to content

DEV-1061: read a NeuroLynQ node's logged files, and download them over its dock link - #306

Open
marknolan wants to merge 6 commits into
DEV-1047_neurolynq_host_supportfrom
DEV-1061_node_storage
Open

marknolan wants to merge 6 commits into
DEV-1047_neurolynq_host_supportfrom
DEV-1061_node_storage

Conversation

@marknolan

Copy link
Copy Markdown
Member

Summary

A NeuroLynQ node (a Verisense running the GQ's firmware ID, DEV-1047) now logs sessions as GQ-format files under its own hardware ID, 61 to 68, and keeps them in NAND behind a new $ component, STORAGE (0x0C). The format and protocol are specified in verisense-firmware docs/VERISENSE_NEUROLYNQ_STORAGE.md. This PR is the driver's side: read such a file, and download a node's sessions over its dock link.

Stacked on #304, whose branch it targets.

Reading a node's files

  • ShimmerVerObject:
    • a node takes firmware version code 6, which gives its records a GQ's 3-byte timestamps;
    • isSupportedSdDataImport(hwVer, fwId): a node's files are importable;
    • isSupportedNodeStorageDownload();
    • isSupportedSdCardAccess() stays false for a node, which has no card.
  • ShimmerObject:
    • header sensor bit 16, NEUROLYNQ_PPG, marks 1 to 3 24-bit PPG counts per record, one per channel enabled in header byte 15;
    • buildMsg's hardware gate admits a node;
    • a node's GQ sensor map gains the new SensorPPGNeuroLynQ. A GQ's is unchanged.
  • ConfigByteLayoutShimmerGq802154: the PPG channel mask at InfoMem 263, a byte no GQ reads.

GSR needs nothing here: the node re-encodes its GSR into a GQ's ADC domain before writing it.

Downloading over the dock link

  • UartPacketDetails: the STORAGE component and its properties.
  • ErrorCodesWiredProtocol: codes 7040 to 7045.
  • AbstractCommsProtocolWired.readBurst(): a READ is answered by a burst of data frames, a NAND page at a time, then an END carrying the count and CRC-32. While a burst is read, those frames go on a queue of their own. Other requests are unchanged.
  • comms/wiredProtocol/neurolynq/:
    • NeuroLynQStorageCodec: a port of neurolynq-web-sdk's codec, held to the same vectors as it and the firmware's codec test;
    • NeuroLynQStorageProtocol: the web SDK's read rule, with gap re-request and CRC-32; cancel() stops readFile() before its next window;
    • SimulatedNeuroLynQNodeSerialPort: a node behind an AbstractSerialPortHal, able to drop, corrupt, cut short or NACK what it sends. It's in main because Shimmer-Advance-API's dock tests drive it too.
  • MsgDock: MSG_ID_DOCK_NODE_DOWNLOAD_PROGRESS (60) with a byte count and total.

Two driver fixes the new tests found

Both affect every docked device.

  • processRxBuf() recursed once per packet. A read holding a whole burst, about 1,700 frames, ran the reader's thread out of stack. StackOverflowError escaped serialPortRxEvent()'s catch and killed the thread, and the port then went silent for good. It is now a loop; each packet is parsed exactly as before.
  • shimmerUartCommandTxRx() cleared mThrownException after txPacket(). A NACK parsed before the write returned was lost, and the command failed 500 ms later with TIMEOUT. It now clears first, as the three NoWait commands already do. It matters to a host: a node whose firmware predates storage answers INFO with BAD_CMD, and a TIMEOUT there would read as a failed scan.

The other hosts of the $ protocol were checked for both. shimmer-web-sdk drains in a loop, neurolynq-web-sdk walks a cursor, and pyshimmer reads by length. The C# and Swift APIs share the CRC but not the dock protocol. None of them needs either fix.

Verification

  • New tests: API_00016 (11), API_00017 (8) and API_00018 (19), against the simulated node. The driver suite passes, 163/163.
  • Faults caught: five put into the reader, each caught — no gap tracking, the END's CRC ignored, a CRC error made fatal, the burst not routed, absence never reported.
  • Tests that fail on the old code: the recursion test (a 512 KiB window as one read) and the NACK-race test (aNackFasterThanTheWriteIsNotLost) both fail before their fixes and pass after.
  • File parsing: the Shimmer-Advance-API PR tests it against golden node files written by the firmware's own writer.
  • On hardware: the protocol has run against a real node from the web host (neurolynq-web-sdk, same rules): whole-file downloads byte-identical, CRC-32 checked. This driver's path has not yet driven a real node. That comes with NeuroLynQ's import on the bench.

Test numbers: API_00015 is taken by DEV-1067 (SensorGSRVerisense). Whichever lands second should check its neighbour's.

🤖 Generated with Claude Code

marknolan and others added 6 commits September 25, 2026 12:54
A NeuroLynQ node, a Verisense running the GQ's firmware ID (DEV-1047),
now logs sessions to NAND as GQ-format files under its own hardware ID,
61 to 68 (verisense-firmware docs/VERISENSE_NEUROLYNQ_STORAGE.md). The
driver keyed everything on the file's hardware ID, so it read those
files wrongly, or not at all.

ShimmerVerObject
- A node takes firmware version code 6. That code is what gives a GQ's
  records 3-byte timestamps (updateTimestampByteLength). A node had -1,
  which read them as 2 and misaligned every record.
- isSupportedSdDataImport(hwVer, fwId): a node's files are importable.
  The one-argument form sees only the hardware ID, which a node shares
  with a stock Verisense, so it is left as it was.
- isSupportedNodeStorageDownload(): the host may try to download the
  device's sessions over its dock link. A node has no SD card, and
  isSupportedSdCardAccess() stays false for it.

ShimmerObject
- SDLogHeader.NEUROLYNQ_PPG is bit 16 of the header's sensor bitmap,
  unused until now. It marks 1 to 3 24-bit PPG counts per record,
  between GSR and heart rate, one for each channel that header byte 15
  enables: bit 0 green, 1 IR, 2 red. No GQ or Shimmer3 sets the bit.
- buildMsg's hardware gate admits a node, so its timestamps parse as a
  GQ's do.
- The PPG counts are parsed raw, before ECG_TO_HR_FW, in record order.
- A node's GQ sensor map gains SensorPPGNeuroLynQ. A GQ's is unchanged.

SensorPPGNeuroLynQ (new) holds the channels, as UINT24 little-endian raw
counts, and the mask rules: 0 and 0xFF, an unwritten byte, both mean
green only. It exists only on the file path. The node streams no PPG,
and ShimmerGQ_802154 must keep a GQ's sensor map (ADV_API_00072), so
there the mask is a setting and not a sensor.

The rest:
- ConfigByteLayoutShimmerGq802154: the mask sits at InfoMem 263, the
  byte after the radio settings, which no GQ reads.
- Configuration: SENSOR_ID.NEUROLYNQ_PPG = 154, and a compatibility
  list for the node alone.
- AbstractSensor: SENSORS.PPG_NEUROLYNQ.

GSR needs nothing here. The node re-encodes its GSR into the GQ's ADC
domain before writing it, and the SD path's GSR maths is the GQ's
whatever the hardware ID.

API_00016 covers the driver's side (11 tests). The file-level tests are
in Shimmer-Advance-API, where ShimmerSDLog is. They use golden files that
the firmware's own writer produced on a PC: no node has logged a file
yet.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A node keeps its sessions in NAND behind a new $ component, STORAGE
(0x0C), with properties INFO, SESSION, FILES, READ, ERASE and FORMAT
(verisense-firmware docs/VERISENSE_NEUROLYNQ_STORAGE.md section 7).
Five of them are one request and one response, as the driver already
handles. READ answers with a burst instead: data frames sent a NAND page
at a time, then one END that carries the byte count and CRC-32.

- UartPacketDetails: the STORAGE component and its properties, for a
  node only.
- ErrorCodesWiredProtocol: an error code for each request, 7040 to 7045.
- AbstractCommsProtocolWired: readBurst(). While a burst is read, the
  parser puts that component's data and END frames on a queue, not on
  the list other requests poll every 100 ms. A frame whose CRC fails is
  counted and lost, and the reader asks for it again. A NACK throws, as
  it does for any request. Requests that are not bursts are unchanged.

New in comms/wiredProtocol/neurolynq/:
- NeuroLynQStorageCodec: the bytes. It is a port of neurolynq-web-sdk's
  storage.ts, held to the same vectors as that and the firmware's codec
  test.
- NeuroLynQStorageProtocol: the requests, with the web SDK's read rule.
  - A window keeps the bytes that arrived in order, and asks again after
    a gap.
  - A window whose END contradicts what arrived is thrown away.
  - A file is whole once every byte has arrived, END or not.
  - Six empty windows in a row fail the read rather than hang.
  - isStorageAbsent() means INFO answered BAD_CMD: a GQ, or a node whose
    firmware predates storage.
- SimulatedNeuroLynQNodeSerialPort: a node behind an
  AbstractSerialPortHal. It sends as the firmware does, a page's frames
  at a time from its own thread, and on request drops, corrupts, cuts
  short or NACKs what it sends. It sits in main, not test, because
  Shimmer-Advance-API's dock tests drive it too.

API_00017 tests the codec (8 tests). API_00018 tests the protocol
against the simulated node (13 tests). Five faults put into the reader
were each caught: no gap tracking, the END's CRC ignored, a CRC error
made fatal, the burst not routed, and absence never reported.

Tested against the simulated node only; no real node has been downloaded
yet.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
processRxBuf() handled the first packet of a read, then called itself on
the rest. Each packet cost one level of stack, and so did each byte of
noise it dropped. No response the dock protocol had sent before came
close to the limit.

A NeuroLynQ node's READ burst does. It sends 4 KB a page, and a read
taken after the reader was held up holds everything that arrived in the
meantime. At the node's 329 kB/s that is some 1,300 frames a second, and
in a test a read of about 1,700 frames ran the reader's thread out of
stack. The StackOverflowError is an Error, not an Exception, so
serialPortRxEvent()'s catch did not stop it and the thread died. The
port then went quiet for good: no later response was parsed.

processRxBuf() is now a loop that takes one packet a pass.
processFirstPacket() is the old body, less its final self-call, and it
parses a packet exactly as before. Nothing else changes, for any dock.

The other hosts of the $ protocol were checked. shimmer-web-sdk drains
in a loop (drainByteStream), neurolynq-web-sdk walks a cursor, and
pyshimmer reads by length. The C# and Swift APIs share the CRC but not
the dock protocol. None of them need this.

API_00018's aReadHoldingAWholeBurstParses sends a 512 KiB window, about
2,100 frames, as one read (the simulated node's new oneReadPerBurst).
With the recursive parser, the read fails and the stack overflow is in
the log. With the loop, the file arrives whole and the port answers
INFO afterwards.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
An eight-hour SR68 session is 68 MB, about three and a half minutes over
the dock link. The import's CANCEL has to be able to stop that, and its
progress panel needs something to show while it runs.

NeuroLynQStorageProtocol
- cancel() stops readFile() before its next window, which is under two
  seconds away at the link's rate. readFile() then throws
  ReadCancelledException, a StorageReadException.
- A window is never cut short, so the burst it asked for is over before
  anything else goes out.
- An instance stays cancelled; a new one reads again.

MsgDock: MSG_ID_DOCK_NODE_DOWNLOAD_PROGRESS (60, free) carries a node's
mBytesDone of mBytesTotal during an SD scan. A node has no card, so the
scan is where its sessions come off it. Shimmer-Advance-API's BasicDock
sends it and DockManager passes it on.

The simulated node now ends a burst when a request arrives, with END
ABORTED, before it answers that request. This is what the firmware does
(gq_dock_link.c, storage spec section 7). It can also pause between
pages, as a node does while it reads the next page from NAND, and it
records each END it sends.

API_00018 (3 new tests):
- A cancel lands between windows. The link stays usable after it: the
  next INFO and READ are answered.
- A read cancelled before it starts sends nothing.
- A request made while a burst is still running is answered within
  INFO's 500 ms, and the burst ends ABORTED. Unended, that paced burst
  would outlast the request.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A BasicDock reads a docked device's details again after a CLEAR, and
starts with its MAC: a device whose MAC it cannot read counts as
undocked. A real node answers the read (DEV-1047). The simulated node
now does too, once it is given a MAC, so Shimmer-Advance-API's dock
tests can run a whole CLEAR. It still answers anything else outside the
storage component with BAD_CMD.

API_00018 checks the answer through readMacId().

Co-Authored-By: Claude Opus 5.5 <[email protected]>
shimmerUartCommandTxRx() cleared mThrownException after txPacket(). A
NACK answered and parsed before txPacket() returned was stored there,
then wiped. The command waited out SERIAL_PORT_TIMEOUT and failed with
TIMEOUT 500 ms later, instead of at once with the NACK the device sent.
The three NoWait commands already clear it before sending. This one now
does the same.

It showed up in the NeuroLynQ node's dock tests, which failed from time
to time against a simulated node that answers within microseconds. It
matters beyond the tests too: a node whose firmware predates storage
answers INFO with BAD_CMD, which is how the host knows it has nothing to
import. Reported as a timeout, it would be taken for a failed scan.

The simulated node's new answerBeforeTheWriteReturns answers a request,
and has the answer parsed, before the write returns. That is the window
the NACK was lost in. API_00018's aNackFasterThanTheWriteIsNotLost fails
with the old order and passes with the new. API_00018's link also takes
a dock-style unique ID now ("Verisense.01.01"): DockException takes the
ID apart, and "node" made a timeout throw ArrayIndexOutOfBounds instead.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@marknolan marknolan self-assigned this Sep 28, 2026
@marknolan
marknolan changed the base branch from DEV-1047_neurolynq_node_as_gq to DEV-1047_neurolynq_host_support September 28, 2026 19:34
@marknolan

Copy link
Copy Markdown
Member Author

Retargeted from #304's branch onto DEV-1047_neurolynq_host_support, the branch of #308. #304 was closed without merging and is superseded by #308, which carries its commits unchanged. This PR's commits are unchanged too, so it now stacks on #308.

@jyong15 jyong15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review: Shimmer-Java-Android-API #306 (driver: node files and dock-link download)

Verdict: blocked. 163/163 tests pass and both "fails before the fix" claims were reproduced, but the burst reader fails every window after any stall or on any link under about 51 kB/s. Reproduced with the PR's own simulator; a small fix is described below and passes the suite.

One of six coordinated reviews of the DEV-1047 / DEV-1061 stack (#308, #306; ShimmerResearch/Shimmer-Advance-API#483, #482; ShimmerResearch/ConsensysApps#494, #493), with builds and tests run locally against the sibling branches.

Findings

Verified: ShimmerDriver 163/163, ShimmerDriverPC build OK, merged with master 237/237. Both "fails before the fix" claims reproduced (StackOverflowError with the old recursive processRxBuf; the NACK race returns TIMEOUT instead of BAD_CMD).

  • B1 Blocker, reproduced. AbstractCommsProtocolWired.java:900 (readBurst) ends the burst at the first END frame whoever it belongs to, including the END ABORTED the node sends for the previous given-up READ. NeuroLynQStorageProtocol.readWindow (:177-184) tries to skip "an earlier READ's END" but the burst has already returned and this window's frames are lost; readFile (:226-251) re-requests the same offset, so the next ABORTED END matches and is taken as the answer. Every later window gets nothing and the file fails with StorageReadException, which the app shows as DOCK_NODE_DOWNLOAD_FAILED. Triggers: any gap of 1 s or more between frames, or any window that reaches its 10.2 s budget, i.e. a sustained link under about 51 kB/s. Reproduced with the PR's own simulator: one 400 ms pause then normal streaming fails with "nothing at 4080 in 6 tries"; a steady 40 kB/s link fails at 11 s. Fix: readBurst takes a predicate for "this request's END" and never ends on status ABORTED; readWindow skips ABORTED. With that patch both scenarios pass and API_00018 still passes 19/19. The repro tests can be supplied on request.
  • M1 Major, test gap. API_00018 aRequestEndsABurstInProgress (:218-228) gives up a burst and only checks INFO still works. No test sends a second READ after a given-up window, and none covers a slow link against the budget. That is why B1 passed the suite.
  • m1 AbstractCommsProtocolWired.java:909: after a burst is given up, late frames pile into mListOfUartRxPacketObjects (up to about 2,100 objects) until the next command, and a CRC failure among them sets mThrownException, so the next command fails with CRC_ERR.
  • m2 processRxBuf (:654): the loop is correct but still copies the remaining buffer on every packet, O(n²), roughly 0.5 GB of memcpy for one 512 KiB read. Walk an index instead.
  • m3 Hardware range 61 to 68 vs the code's 61, 62, 63, 64, 68 (cross-stack item 6).
  • m4 Configuration.java:1434 and UartPacketDetails.java:99-100: the "NeuroLynQ node alone" compatibility lists match any hardware running GQ_802154 firmware, i.e. real GQs too.
  • m5 NeuroLynQStorageProtocol.readFile:236 ignores END status, so a NAND_ERR window is kept if its bytes, count and CRC agree. The simulator never sends NAND_ERR. Check the firmware's rule.
  • m6 cancel() javadoc (:91-96) claims "under two seconds" and "the link is left as a finished read leaves it". The first needs 262 kB/s; the second is false after any given-up window (B1).
  • m7 SimulatedNeuroLynQNodeSerialPort is in src/main, ships in the production jar, and each instance starts a daemon thread and an executor. Use test fixtures.
  • n1 readBurst sets no errorCode on a DockException from txPacket; ReadBurst.ended, ReadWindow.complete, duplicates, pastGap are never read.
  • n2 SensorPPGNeuroLynQ.java:56 object-cluster name "PPG_Green" collides with SensorMAX86916.MAX86916_PPG_GREEN with a different meaning (raw counts).
  • n3 No driver test exercises the new ShimmerObject.buildMsg PPG block (:2343) or the node timestamp gate (:941); only Advance-API's ADV_API_00078 does.

PPG parsing (u24 little-endian, 0 to 3 channels in mask bit order, GQ skipped) and the InfoMem 263 placement (radio settings end at 262, image is 384 bytes) check out. Unverified: "five faults put into the reader", codec vectors against the firmware, byte-identical hardware runs.

Cross-stack issues (shared by all six reviews)

  1. Stale heads in the DEV-1061 stack. Advance-API #482's head lacks the last two commits of its own base (the Span crash fix and the channel rating that adds eventSpanManagerSpectrumAnalyzerScanFailed). ConsensysApps #493's head lacks its base's last commit. GitHub merges both cleanly, and the merged three-repo stack was compiled (main and test sources, all three repos), but the trees the author tested are not the trees that will land.
  2. Master has moved. Driver master is 34 commits past #308's branch point, Advance-API master 13 past #483's. Both merge cleanly into their master-targeted PRs and the merged trees build and pass. But #482 textually conflicts with Advance-API master in ShimmerSDLog.processSDLogHeader, where DEV-1123's pressure-ID warning and this PR's parseNeuroLynQPpgChannels land on the same lines. Resolve by keeping both. Also, Advance-API master now needs the driver's SdHeaderPressureSensorId, so the final landing needs driver master plus #306 together.
  3. DEV-1095 (SD timestamp anchor) now applies to node files via the parseTimestampShimmer3 gate #306 opens. It was verified on Shimmer3 only. Re-run ADV_API_00078 (golden node files) against a master-merged driver before merging.
  4. CI never ran on #306, #482 or #493. The workflows trigger only on PRs into master. When #482 does reach master, ADV_API_00081 will go red on ubuntu (see #482 B1).
  5. CI clone URLs in ConsensysApps and Shimmer-Advance-API gradle.yml still point at github.com/ShimmerEngineering/… and work only through GitHub's rename redirect. Advance-API master already fixed its own (DEV-1126); the #483 branch predates that fix.
  6. Hardware ID 61 to 68. The PR text and javadocs say a node is "hardware ID 61 to 68", but isShimmerGenVerisense lists only 61, 62, 63, 64 and 68. A node on 65 to 67 would be a plain GQ (SD access true, no PPG, no storage download). Either fix the text everywhere or the range.
  7. Windows only. The nRF52840 Span and the NeuroLynQ node are detected only in DeviceInfoWin. DeviceInfoLinux and DeviceInfoMac are untouched, although the javadoc claims "every platform's detection calls this" and the release workflow builds macOS installers.
  8. No JDK 11 here. Every build and test in this review ran on JDK 21 with source/target 11. One pre-existing Mockito test fails on 21 and was ignored.

Notes

All builds and tests in this review ran on JDK 21 with source/target 11; no JDK 11 was available. Nothing here was run on hardware. Full report with the verification table is held by the reviewer; findings are ordered most severe first. Verdict is given in the text; the approve/request-changes decision is left to the human reviewer.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants