Skip to content

DEV-1095: anchor each SD file on its first packet's own counter time - #309

Merged
MAzalya merged 4 commits into
masterfrom
DEV-1095_sd_file_ts_anchor
Oct 5, 2026
Merged

MAzalya merged 4 commits into
masterfrom
DEV-1095_sd_file_ts_anchor

Conversation

@marknolan

@marknolan marknolan commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Fixes DEV-1095: SD-imported timestamps step at data-file splits. On a 66.8 h Shimmer3R recording, a permanent −158.8 ms step appears at the 000 → 001 split, and +15…+24 ms at every later split.

Cause

  • Parser: time = header initial TS + unwrapped ts − first packet's raw ts. This pins each file's first packet to the header time.
  • Firmware: the header holds the RTC when the file was created (sdFileSyncTs in log-and-stream-common), not the first packet's time. The firmware captures firstTsForSdFile but never writes it back to the header.
  • Why the step: packets buffered when a file opens were sampled before that moment, and the lead differs per file. File 000 is created after sampling has started (SD power-up, mkdir, header), so its lead is about 160 ms larger than at a mid-stream split.

Fix

  • New driverUtilities/SdTimestampAnchor, pure and static like TimestampUnwrap. It rebuilds the first packet's full counter value from the header's high bits plus the packet's own 24-bit timestamp, as a signed, wrap-corrected distance.
  • The result is exact whenever the header is within ±256 s of the first packet, and it doesn't depend on which moment the firmware records, so it stays right if the header's meaning ever changes.
  • Both timestamp paths use it: ShimmerObject.parseTimestampShimmer3 and SensorShimmerClock.
  • The javadoc on mFirstTsOffsetFromInitialTsTicks claimed the firmware writes the first-packet time into the header. It doesn't, and the javadoc is corrected.
  • Falls back to the old behaviour for: 2-byte timestamps, a zero initial timestamp, and a lead over 10 s.
  • StroKare workaround: it still gives the same result (there's a test for it).
  • RTC difference: added after the rebuild, as before, so the rebuild happens in the device-counter domain. That matters on Shimmer3, where the difference is an arbitrary offset rather than a multiple of the counter period.

Effect on output

Re-importing an existing recording shifts file 000 by roughly its SD start-up lead (about 160 ms on the recording above) and later files by a few ms. That's the correction, but exports made before and after this change will not match exactly.

Testing

  • New API_00012_SdTimestampAnchorTest (11 tests), added to API_00005_Suite_ShimmerDriver. It covers:
    • leads in both directions;
    • wraps forward and backward;
    • the StroKare case;
    • each fallback;
    • a two-file split, asserting the old rule steps there and the new one doesn't.
  • cd ShimmerDriver && ./gradlew test: 114 tests, 0 failures.
  • Not built locally: ShimmerDriverPC. Here it fails before compiling, resolving :ShimmerBluetoothManager ("No variants exist"), and an untouched origin/master fails the same way, so it's environmental. This PR's CI is the check for that project.
  • Verified on the recording's raw 000/001 files (Shimmer3R). Each file's first packet was sampled before its header timestamp: by 162.72 ms in 000 and 3.94 ms in 001. The old rule reproduces the Consensys export to the microsecond; with the fix, the gap across the 000 → 001 split is exactly one sample period. Those values are a test vector in API_00012 (realShimmer3rRecordingSplitsByOnePeriod).
  • Verified on Shimmer3 too (raw files of a 33-file recording, LogAndStream v1.1.5, 1024 Hz). File 000's first record was sampled 1.831 ms after its header, and every later file's 0.885 ms before. So the header's low 24 bits share the record counter on the MSP430 as well. With the fix, all 32 splits are exactly one sample period. The Consensys export matches the old rule on all 118,352,512 rows. On this recording the old rule's only artefact is +2.716 ms at 000 → 001, much smaller than the Shimmer3R's −158.8 ms, because a Shimmer3 takes file 000's header before its first record. The class doc now says so.

Related

🤖 Generated with Claude Code

The SD header's initial timestamp is the RTC when the file was created
(sdFileSyncTs), not the first packet's time: the firmware captures that in
firstTsForSdFile but never writes it back. Subtracting the first packet's
raw timestamp pinned each file's first packet to the header time, erasing
a creation-to-first-packet lead that differs per file. File 000 is created
after sampling starts, so a Shimmer3R recording stepped back -158.8 ms at
the 000 -> 001 split.

SdTimestampAnchor rebuilds the first packet's full counter value from the
header's high bits and the packet's 24-bit timestamp (signed, wrap-
corrected), exact within +-256 s of the header whatever moment the
firmware records. Both timestamp paths - ShimmerObject and
SensorShimmerClock - use it. 2-byte counters, a zero initial timestamp and
a lead over 10 s keep the previous behaviour.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
marknolan and others added 2 commits September 29, 2026 12:12
Header and first-packet values from the recording's raw files: file 000's
first packet predates its header by 5332 ticks (162.72 ms), file 001's by
129 (3.94 ms). The old rule reproduces the export's 158.78 ms step; the
anchored split is exactly one sample period.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Leads of 214 and 131 ticks gave the export's other backwards step,
-1.556 ms near hour 42. Both real splits now share one assertion helper.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Raw files from a 33-file Shimmer3 recording (LogAndStream v1.1.5,
1024 Hz) confirm the rule there. File 000's first record was sampled
1.831 ms after its header and every later file's 0.885 ms before, so
the header's low 24 bits share the record counter, and all 32 splits
close to exactly one sample period. The old rule's only artefact on
that recording is +2.7 ms at the 000 to 001 split.

The class doc said file 000's records predate its header by about
160 ms. That is the Shimmer3R; a Shimmer3 takes file 000's header
first.

Comment only.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

@MAzalya MAzalya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok looks good

@MAzalya
MAzalya merged commit 0918f19 into master Oct 5, 2026
2 checks passed
@MAzalya
MAzalya deleted the DEV-1095_sd_file_ts_anchor branch October 5, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants