Repository navigation
Conversation
A NeuroLynQ node, a Verisense running the GQ's firmware ID (DEV-1047), now logs sessions to NAND as GQ-format files under its own hardware ID, 61 to 68 (verisense-firmware docs/VERISENSE_NEUROLYNQ_STORAGE.md). The driver keyed everything on the file's hardware ID, so it read those files wrongly, or not at all. ShimmerVerObject - A node takes firmware version code 6. That code is what gives a GQ's records 3-byte timestamps (updateTimestampByteLength). A node had -1, which read them as 2 and misaligned every record. - isSupportedSdDataImport(hwVer, fwId): a node's files are importable. The one-argument form sees only the hardware ID, which a node shares with a stock Verisense, so it is left as it was. - isSupportedNodeStorageDownload(): the host may try to download the device's sessions over its dock link. A node has no SD card, and isSupportedSdCardAccess() stays false for it. ShimmerObject - SDLogHeader.NEUROLYNQ_PPG is bit 16 of the header's sensor bitmap, unused until now. It marks 1 to 3 24-bit PPG counts per record, between GSR and heart rate, one for each channel that header byte 15 enables: bit 0 green, 1 IR, 2 red. No GQ or Shimmer3 sets the bit. - buildMsg's hardware gate admits a node, so its timestamps parse as a GQ's do. - The PPG counts are parsed raw, before ECG_TO_HR_FW, in record order. - A node's GQ sensor map gains SensorPPGNeuroLynQ. A GQ's is unchanged. SensorPPGNeuroLynQ (new) holds the channels, as UINT24 little-endian raw counts, and the mask rules: 0 and 0xFF, an unwritten byte, both mean green only. It exists only on the file path. The node streams no PPG, and ShimmerGQ_802154 must keep a GQ's sensor map (ADV_API_00072), so there the mask is a setting and not a sensor. The rest: - ConfigByteLayoutShimmerGq802154: the mask sits at InfoMem 263, the byte after the radio settings, which no GQ reads. - Configuration: SENSOR_ID.NEUROLYNQ_PPG = 154, and a compatibility list for the node alone. - AbstractSensor: SENSORS.PPG_NEUROLYNQ. GSR needs nothing here. The node re-encodes its GSR into the GQ's ADC domain before writing it, and the SD path's GSR maths is the GQ's whatever the hardware ID. API_00016 covers the driver's side (11 tests). The file-level tests are in Shimmer-Advance-API, where ShimmerSDLog is. They use golden files that the firmware's own writer produced on a PC: no node has logged a file yet. Co-Authored-By: Claude Opus 5.5 <[email protected]>
A node keeps its sessions in NAND behind a new $ component, STORAGE
(0x0C), with properties INFO, SESSION, FILES, READ, ERASE and FORMAT
(verisense-firmware docs/VERISENSE_NEUROLYNQ_STORAGE.md section 7).
Five of them are one request and one response, as the driver already
handles. READ answers with a burst instead: data frames sent a NAND page
at a time, then one END that carries the byte count and CRC-32.
- UartPacketDetails: the STORAGE component and its properties, for a
node only.
- ErrorCodesWiredProtocol: an error code for each request, 7040 to 7045.
- AbstractCommsProtocolWired: readBurst(). While a burst is read, the
parser puts that component's data and END frames on a queue, not on
the list other requests poll every 100 ms. A frame whose CRC fails is
counted and lost, and the reader asks for it again. A NACK throws, as
it does for any request. Requests that are not bursts are unchanged.
New in comms/wiredProtocol/neurolynq/:
- NeuroLynQStorageCodec: the bytes. It is a port of neurolynq-web-sdk's
storage.ts, held to the same vectors as that and the firmware's codec
test.
- NeuroLynQStorageProtocol: the requests, with the web SDK's read rule.
- A window keeps the bytes that arrived in order, and asks again after
a gap.
- A window whose END contradicts what arrived is thrown away.
- A file is whole once every byte has arrived, END or not.
- Six empty windows in a row fail the read rather than hang.
- isStorageAbsent() means INFO answered BAD_CMD: a GQ, or a node whose
firmware predates storage.
- SimulatedNeuroLynQNodeSerialPort: a node behind an
AbstractSerialPortHal. It sends as the firmware does, a page's frames
at a time from its own thread, and on request drops, corrupts, cuts
short or NACKs what it sends. It sits in main, not test, because
Shimmer-Advance-API's dock tests drive it too.
API_00017 tests the codec (8 tests). API_00018 tests the protocol
against the simulated node (13 tests). Five faults put into the reader
were each caught: no gap tracking, the END's CRC ignored, a CRC error
made fatal, the burst not routed, and absence never reported.
Tested against the simulated node only; no real node has been downloaded
yet.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
processRxBuf() handled the first packet of a read, then called itself on the rest. Each packet cost one level of stack, and so did each byte of noise it dropped. No response the dock protocol had sent before came close to the limit. A NeuroLynQ node's READ burst does. It sends 4 KB a page, and a read taken after the reader was held up holds everything that arrived in the meantime. At the node's 329 kB/s that is some 1,300 frames a second, and in a test a read of about 1,700 frames ran the reader's thread out of stack. The StackOverflowError is an Error, not an Exception, so serialPortRxEvent()'s catch did not stop it and the thread died. The port then went quiet for good: no later response was parsed. processRxBuf() is now a loop that takes one packet a pass. processFirstPacket() is the old body, less its final self-call, and it parses a packet exactly as before. Nothing else changes, for any dock. The other hosts of the $ protocol were checked. shimmer-web-sdk drains in a loop (drainByteStream), neurolynq-web-sdk walks a cursor, and pyshimmer reads by length. The C# and Swift APIs share the CRC but not the dock protocol. None of them need this. API_00018's aReadHoldingAWholeBurstParses sends a 512 KiB window, about 2,100 frames, as one read (the simulated node's new oneReadPerBurst). With the recursive parser, the read fails and the stack overflow is in the log. With the loop, the file arrives whole and the port answers INFO afterwards. Co-Authored-By: Claude Opus 5.5 <[email protected]>
An eight-hour SR68 session is 68 MB, about three and a half minutes over the dock link. The import's CANCEL has to be able to stop that, and its progress panel needs something to show while it runs. NeuroLynQStorageProtocol - cancel() stops readFile() before its next window, which is under two seconds away at the link's rate. readFile() then throws ReadCancelledException, a StorageReadException. - A window is never cut short, so the burst it asked for is over before anything else goes out. - An instance stays cancelled; a new one reads again. MsgDock: MSG_ID_DOCK_NODE_DOWNLOAD_PROGRESS (60, free) carries a node's mBytesDone of mBytesTotal during an SD scan. A node has no card, so the scan is where its sessions come off it. Shimmer-Advance-API's BasicDock sends it and DockManager passes it on. The simulated node now ends a burst when a request arrives, with END ABORTED, before it answers that request. This is what the firmware does (gq_dock_link.c, storage spec section 7). It can also pause between pages, as a node does while it reads the next page from NAND, and it records each END it sends. API_00018 (3 new tests): - A cancel lands between windows. The link stays usable after it: the next INFO and READ are answered. - A read cancelled before it starts sends nothing. - A request made while a burst is still running is answered within INFO's 500 ms, and the burst ends ABORTED. Unended, that paced burst would outlast the request. Co-Authored-By: Claude Opus 5.5 <[email protected]>
A BasicDock reads a docked device's details again after a CLEAR, and starts with its MAC: a device whose MAC it cannot read counts as undocked. A real node answers the read (DEV-1047). The simulated node now does too, once it is given a MAC, so Shimmer-Advance-API's dock tests can run a whole CLEAR. It still answers anything else outside the storage component with BAD_CMD. API_00018 checks the answer through readMacId(). Co-Authored-By: Claude Opus 5.5 <[email protected]>
shimmerUartCommandTxRx() cleared mThrownException after txPacket(). A
NACK answered and parsed before txPacket() returned was stored there,
then wiped. The command waited out SERIAL_PORT_TIMEOUT and failed with
TIMEOUT 500 ms later, instead of at once with the NACK the device sent.
The three NoWait commands already clear it before sending. This one now
does the same.
It showed up in the NeuroLynQ node's dock tests, which failed from time
to time against a simulated node that answers within microseconds. It
matters beyond the tests too: a node whose firmware predates storage
answers INFO with BAD_CMD, which is how the host knows it has nothing to
import. Reported as a timeout, it would be taken for a failed scan.
The simulated node's new answerBeforeTheWriteReturns answers a request,
and has the answer parsed, before the write returns. That is the window
the NACK was lost in. API_00018's aNackFasterThanTheWriteIsNotLost fails
with the old order and passes with the new. API_00018's link also takes
a dock-style unique ID now ("Verisense.01.01"): DockException takes the
ID apart, and "node" made a timeout throw ArrayIndexOutOfBounds instead.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
jyong15
left a comment
There was a problem hiding this comment.
Adversarial review: Shimmer-Java-Android-API #306 (driver: node files and dock-link download)
Verdict: blocked. 163/163 tests pass and both "fails before the fix" claims were reproduced, but the burst reader fails every window after any stall or on any link under about 51 kB/s. Reproduced with the PR's own simulator; a small fix is described below and passes the suite.
One of six coordinated reviews of the DEV-1047 / DEV-1061 stack (#308, #306; ShimmerResearch/Shimmer-Advance-API#483, #482; ShimmerResearch/ConsensysApps#494, #493), with builds and tests run locally against the sibling branches.
Findings
Verified: ShimmerDriver 163/163, ShimmerDriverPC build OK, merged with master 237/237. Both "fails before the fix" claims reproduced (StackOverflowError with the old recursive processRxBuf; the NACK race returns TIMEOUT instead of BAD_CMD).
- B1 Blocker, reproduced.
AbstractCommsProtocolWired.java:900(readBurst) ends the burst at the first END frame whoever it belongs to, including the END ABORTED the node sends for the previous given-up READ.NeuroLynQStorageProtocol.readWindow(:177-184) tries to skip "an earlier READ's END" but the burst has already returned and this window's frames are lost;readFile(:226-251) re-requests the same offset, so the next ABORTED END matches and is taken as the answer. Every later window gets nothing and the file fails withStorageReadException, which the app shows as DOCK_NODE_DOWNLOAD_FAILED. Triggers: any gap of 1 s or more between frames, or any window that reaches its 10.2 s budget, i.e. a sustained link under about 51 kB/s. Reproduced with the PR's own simulator: one 400 ms pause then normal streaming fails with "nothing at 4080 in 6 tries"; a steady 40 kB/s link fails at 11 s. Fix:readBursttakes a predicate for "this request's END" and never ends on status ABORTED;readWindowskips ABORTED. With that patch both scenarios pass and API_00018 still passes 19/19. The repro tests can be supplied on request. - M1 Major, test gap. API_00018
aRequestEndsABurstInProgress(:218-228) gives up a burst and only checks INFO still works. No test sends a second READ after a given-up window, and none covers a slow link against the budget. That is why B1 passed the suite. - m1
AbstractCommsProtocolWired.java:909: after a burst is given up, late frames pile intomListOfUartRxPacketObjects(up to about 2,100 objects) until the next command, and a CRC failure among them setsmThrownException, so the next command fails with CRC_ERR. - m2
processRxBuf(:654): the loop is correct but still copies the remaining buffer on every packet, O(n²), roughly 0.5 GB of memcpy for one 512 KiB read. Walk an index instead. - m3 Hardware range 61 to 68 vs the code's 61, 62, 63, 64, 68 (cross-stack item 6).
- m4
Configuration.java:1434andUartPacketDetails.java:99-100: the "NeuroLynQ node alone" compatibility lists match any hardware running GQ_802154 firmware, i.e. real GQs too. - m5
NeuroLynQStorageProtocol.readFile:236ignores END status, so a NAND_ERR window is kept if its bytes, count and CRC agree. The simulator never sends NAND_ERR. Check the firmware's rule. - m6
cancel()javadoc (:91-96) claims "under two seconds" and "the link is left as a finished read leaves it". The first needs 262 kB/s; the second is false after any given-up window (B1). - m7
SimulatedNeuroLynQNodeSerialPortis insrc/main, ships in the production jar, and each instance starts a daemon thread and an executor. Use test fixtures. - n1
readBurstsets noerrorCodeon aDockExceptionfromtxPacket;ReadBurst.ended,ReadWindow.complete,duplicates,pastGapare never read. - n2
SensorPPGNeuroLynQ.java:56object-cluster name "PPG_Green" collides withSensorMAX86916.MAX86916_PPG_GREENwith a different meaning (raw counts). - n3 No driver test exercises the new
ShimmerObject.buildMsgPPG block (:2343) or the node timestamp gate (:941); only Advance-API's ADV_API_00078 does.
PPG parsing (u24 little-endian, 0 to 3 channels in mask bit order, GQ skipped) and the InfoMem 263 placement (radio settings end at 262, image is 384 bytes) check out. Unverified: "five faults put into the reader", codec vectors against the firmware, byte-identical hardware runs.
Cross-stack issues (shared by all six reviews)
- Stale heads in the DEV-1061 stack. Advance-API #482's head lacks the last two commits of its own base (the Span crash fix and the channel rating that adds
eventSpanManagerSpectrumAnalyzerScanFailed). ConsensysApps #493's head lacks its base's last commit. GitHub merges both cleanly, and the merged three-repo stack was compiled (main and test sources, all three repos), but the trees the author tested are not the trees that will land. - Master has moved. Driver master is 34 commits past #308's branch point, Advance-API master 13 past #483's. Both merge cleanly into their master-targeted PRs and the merged trees build and pass. But #482 textually conflicts with Advance-API master in
ShimmerSDLog.processSDLogHeader, where DEV-1123's pressure-ID warning and this PR'sparseNeuroLynQPpgChannelsland on the same lines. Resolve by keeping both. Also, Advance-API master now needs the driver'sSdHeaderPressureSensorId, so the final landing needs driver master plus #306 together. - DEV-1095 (SD timestamp anchor) now applies to node files via the
parseTimestampShimmer3gate #306 opens. It was verified on Shimmer3 only. Re-run ADV_API_00078 (golden node files) against a master-merged driver before merging. - CI never ran on #306, #482 or #493. The workflows trigger only on PRs into master. When #482 does reach master, ADV_API_00081 will go red on ubuntu (see #482 B1).
- CI clone URLs in ConsensysApps and Shimmer-Advance-API
gradle.ymlstill point atgithub.com/ShimmerEngineering/…and work only through GitHub's rename redirect. Advance-API master already fixed its own (DEV-1126); the #483 branch predates that fix. - Hardware ID 61 to 68. The PR text and javadocs say a node is "hardware ID 61 to 68", but
isShimmerGenVerisenselists only 61, 62, 63, 64 and 68. A node on 65 to 67 would be a plain GQ (SD access true, no PPG, no storage download). Either fix the text everywhere or the range. - Windows only. The nRF52840 Span and the NeuroLynQ node are detected only in
DeviceInfoWin.DeviceInfoLinuxandDeviceInfoMacare untouched, although the javadoc claims "every platform's detection calls this" and the release workflow builds macOS installers. - No JDK 11 here. Every build and test in this review ran on JDK 21 with source/target 11. One pre-existing Mockito test fails on 21 and was ignored.
Notes
All builds and tests in this review ran on JDK 21 with source/target 11; no JDK 11 was available. Nothing here was run on hardware. Full report with the verification table is held by the reviewer; findings are ordered most severe first. Verdict is given in the text; the approve/request-changes decision is left to the human reviewer.
Generated by Claude Code
Summary
A NeuroLynQ node (a Verisense running the GQ's firmware ID, DEV-1047) now logs sessions as GQ-format files under its own hardware ID, 61 to 68, and keeps them in NAND behind a new
$component, STORAGE (0x0C). The format and protocol are specified in verisense-firmwaredocs/VERISENSE_NEUROLYNQ_STORAGE.md. This PR is the driver's side: read such a file, and download a node's sessions over its dock link.Stacked on #304, whose branch it targets.
Reading a node's files
ShimmerVerObject:isSupportedSdDataImport(hwVer, fwId): a node's files are importable;isSupportedNodeStorageDownload();isSupportedSdCardAccess()stays false for a node, which has no card.ShimmerObject:NEUROLYNQ_PPG, marks 1 to 3 24-bit PPG counts per record, one per channel enabled in header byte 15;buildMsg's hardware gate admits a node;SensorPPGNeuroLynQ. A GQ's is unchanged.ConfigByteLayoutShimmerGq802154: the PPG channel mask at InfoMem 263, a byte no GQ reads.GSR needs nothing here: the node re-encodes its GSR into a GQ's ADC domain before writing it.
Downloading over the dock link
UartPacketDetails: the STORAGE component and its properties.ErrorCodesWiredProtocol: codes 7040 to 7045.AbstractCommsProtocolWired.readBurst(): a READ is answered by a burst of data frames, a NAND page at a time, then an END carrying the count and CRC-32. While a burst is read, those frames go on a queue of their own. Other requests are unchanged.comms/wiredProtocol/neurolynq/:NeuroLynQStorageCodec: a port of neurolynq-web-sdk's codec, held to the same vectors as it and the firmware's codec test;NeuroLynQStorageProtocol: the web SDK's read rule, with gap re-request and CRC-32;cancel()stopsreadFile()before its next window;SimulatedNeuroLynQNodeSerialPort: a node behind anAbstractSerialPortHal, able to drop, corrupt, cut short or NACK what it sends. It's inmainbecause Shimmer-Advance-API's dock tests drive it too.MsgDock:MSG_ID_DOCK_NODE_DOWNLOAD_PROGRESS(60) with a byte count and total.Two driver fixes the new tests found
Both affect every docked device.
processRxBuf()recursed once per packet. A read holding a whole burst, about 1,700 frames, ran the reader's thread out of stack.StackOverflowErrorescapedserialPortRxEvent()'s catch and killed the thread, and the port then went silent for good. It is now a loop; each packet is parsed exactly as before.shimmerUartCommandTxRx()clearedmThrownExceptionaftertxPacket(). A NACK parsed before the write returned was lost, and the command failed 500 ms later with TIMEOUT. It now clears first, as the three NoWait commands already do. It matters to a host: a node whose firmware predates storage answers INFO with BAD_CMD, and a TIMEOUT there would read as a failed scan.The other hosts of the
$protocol were checked for both. shimmer-web-sdk drains in a loop, neurolynq-web-sdk walks a cursor, and pyshimmer reads by length. The C# and Swift APIs share the CRC but not the dock protocol. None of them needs either fix.Verification
aNackFasterThanTheWriteIsNotLost) both fail before their fixes and pass after.Test numbers: API_00015 is taken by DEV-1067 (
SensorGSRVerisense). Whichever lands second should check its neighbour's.🤖 Generated with Claude Code