Repository navigation
Conversation
9aa93d2 to
4acc697
Compare
c442587 to
49db111
Compare
4acc697 to
815408a
Compare
49db111 to
014ab1b
Compare
815408a to
70b6b0b
Compare
f4da635 to
13c02e0
Compare
70b6b0b to
eaab512
Compare
9928b36 to
6a929f6
Compare
eaab512 to
7c73b95
Compare
| clocks = _clocks(args.lance, {a["recording_id"] for a in todo.values()}) | ||
| rows = [] | ||
| for h, a in sorted(todo.items()): | ||
| rts, _ = clocks[a["recording_id"]] |
There was a problem hiding this comment.
low (correctness): Partial appends (the order is steps -> aliases -> tasks -> recordings) can leave alias rows whose recording has no recordings row. _clocks only covers recordings, so clocks[a['recording_id']] here (and clocks[rid] in from-zarr) raises KeyError and the whole conversion aborts. Rerunning such an append duplicates the recording's steps rows. _clocks keeps both copies, so every timestamp in rts appears twice and clip[s]/clip[e] land at the wrong times without any error. Skip aliases without a clock and list them in dropped. Build the clock from the recording's committed row range (row_start/num_frames, as the resolver does), or at least use np.unique.
| if not any(counts.values()): | ||
| return None | ||
| added = sorted(set(adds.column("recording_id").to_pylist())) | ||
| return {"rows": counts, **_record(root, "annotations", added, gone, operator)} |
There was a problem hiding this comment.
low (naming): For annotation commits added/replaced mean recordings gaining/losing annotation rows, which differs from delivery batches. Only this docstring says so. Document the per-source meaning in the CHANGES schema comment next to the added/replaced columns, so a reader of changes.lance can interpret them without reading commit.py.
|
|
||
|
|
||
| def _record( | ||
| root, source: str, added: list, replaced: list, operator, steps_tables=() |
There was a problem hiding this comment.
nit (api-design): Make the arguments after replaced keyword-only (*, operator, provenance=None, steps_tables=()) and pass them by name. #734 inserts provenance before steps_tables, and the positional call _record(root, source, added, replaced, operator, provenance, sorted(steps)) is easy to misorder.
| if r["recording_id"] in rids: | ||
| by_key.setdefault(tuple(r[c] for c in STEPS_KEY), []).append(r) | ||
| for key, mine in by_key.items(): | ||
| ids = ", ".join(commit._sql_str(r["recording_id"]) for r in mine) |
There was a problem hiding this comment.
nit (encapsulation): This script reaches into commit._open and commit._sql_str. Use dataset.open_table, and expose a public helper such as sql_in(column, values) for the IN (...) predicate, which commit_annotations and the migrate script also build by hand.
lance_pilot/annotations.py becomes egomimic/scripts/lance_annotations.py and works on s3:// roots. from-zarr (was migrate-zarr) maps the zarr backlog through each episode's frame timestamps; from-bucket maps scale_to_bucket span lists through aliases.lance. Commits go through commit.commit_annotations, which records a changes row (lab "annotations") and tags every table like a recording batch. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
Steps tables are now one per (lab, embodiment, end_effector), so the importer groups recordings by that key instead of by the old source-valued lab when reading frame times. The changes table names the committer stream in source, which annotation commits set to "annotations". Claude-Session: https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw
6a929f6 to
fab385b
Compare
7c73b95 to
d552315
Compare
|
Review findings: status after the fixup pass. The fixes are squashed into this PR's commits, and the whole stack was force-pushed. Fixed
Partial
|
lance_pilot/annotations.py becomes egomimic/scripts/lance_annotations.py and works on
s3:// roots. from-zarr (was migrate-zarr) maps the zarr backlog through each episode's
frame timestamps, reading them from each recording's (lab, embodiment, end_effector)
steps table; from-bucket maps scale_to_bucket span lists through aliases.lance.
Commits go through commit.commit_annotations, which records a changes row (source
"annotations") and tags every table like a recording batch.
https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw