Repository navigation
Require zarr episode metadata; add zarr/MCAP validators and vendor converter template - #736
Merged
Merged
Conversation
Only hdf5_to_zarr's opt-in --incremental flag used it, and that path bypassed create_and_write's metadata validation (no intrinsics).
create_and_write now takes task_name, operator, lab, scene and objects as
keyword-only arguments with no default, and rejects empty strings and
placeholders ("unknown", "n/a", "debug", ...). A field the source data truly
lacks must be waived explicitly with MissingMetadata(reason); it is stored as
null with the reason under zarr.attrs["missing_metadata"].
- egomimic/rldb/zarr/required_metadata.py: rules, MissingMetadata, converter CLI helpers
- egomimic/rldb/zarr/validate.py: `python -m egomimic.rldb.zarr.validate`,
replacing the copy-paste snippet in CONTRIBUTING_DATA.md §11.1; waived
fields fail unless --allow-missing
- run_conversion/ray_helper pass operator/lab/scene/objects from the DB row
to the aria/eva converters; empty legacy fields become explicit waivers
- aria/eva/hdf5 converters gain --operator/--lab/--scene/--objects/--missing-metadata
- CONTRIBUTING_DATA.md §6.3/§11.1/§12 document the fields and validator
Both called create_and_write with removed arguments (task=, enable_sharding=) and invalid embodiments, and embodiment_tutorial told human-data vendors to add an embodiment, which CONTRIBUTING_DATA.md §9.1 no longer asks for. The README now points vendors at CONTRIBUTING_DATA.md and the validator.
egomimic/scripts/convert_template/template_to_zarr.py: vendors copy it and replace find_raw_episodes() and load_episode(), which returns a RawEpisode (arrays in EgoVerse conventions plus that episode's required metadata). The template handles the episode-hash name, the ZarrWriter call, validation of every episode right after writing (failures are deleted and reported without stopping the batch), MP4 previews, and batch conversion via convert_all(paths, output_dir, workers=N) or a minimal --raw-root CLI. Ships with a synthetic load_episode so it runs end to end, and tests/unit/test_convert_template.py runs it in CI so it can't go stale like the deleted tutorials. CONTRIBUTING_DATA.md §6.0 and the README point here.
…ode edits - All validation (embodiment, intrinsics/extrinsics, required metadata) now runs at the start of ZarrWriter.write(), before anything touches disk. - ZarrWriter.__init__ takes embodiment, intrinsics and the five required metadata fields keyword-only with no default, so a missing one is a TypeError at the call site. create_and_write is a plain wrapper around ZarrWriter(...).write(...). - Editing an existing episode no longer needs a writer: module-level write_annotations / remove_key / key_exists replace the old methods and keep zarr.attrs["features"] in sync (append_annotations never did, so a new annotation key failed the validator). write_annotations(mode="a") now raises if the key already exists. - Annotation scripts, tests and CONTRIBUTING_DATA.md §8.3 updated.
A field the source data lacks is now marked with MissingMetadata() and written as null; the missing_metadata reasons dict is gone from zarr.attrs. The validator treats an absent key as an error and a null value as an explicit waiver (error unless --allow-missing). --missing-metadata takes just the field name, run_conversion passes a list of empty legacy fields, and the docstring, CLI help and template say to use it only if absolutely necessary.
What vendors deliver (one raw episode_<id>/episode.mcap per recording, plus optional annotation.mcap) and the check every delivery goes through before conversion to Lance: - SPEC.md: the vendor-facing spec; spec.py: the same rules as data (human / robot topic profiles, episode-info keys, frames, thresholds); schemas.py: the egoverse.* message types, rendered to proto/egoverse.proto for vendors. - validate.py: `python -m egomimic.rldb.mcap.validate episode_<id>/...` checks presence, video, clocks, calibration, IMU, poses and task/label spans, and prints the Lance recording_id each episode will get. - tests: synthetic episodes with one switch per mistake (each must report exactly its problem), proto/enum sync, and an integration run on real episodes. - pyproject: mcap, mcap-protobuf-support, foxglove-schemas-protobuf; ships the .proto. Checked on two real Aria recordings converted to the spec: both pass, and every column of aria_to_lance's steps rebuilds from the MCAP alone.
…er trips on Found in review; each passed validation and then failed in conversion: - a camera whose -info / -info-extra name another frame than its video frames (conversion multiplied through the wrong frame). The video, -info and -info-extra must now name the same frame. - a head-worn rig's steps camera not rigidly mounted on the head (e.g. its calibration in `world`): the converter places the steps through a /tf_static path from `head` to that camera, which is now required. (A frame placed nowhere is still reported once, as having no path to world.) - a NaN / Inf /tf_static translation (conversion: "SVD did not converge"); only the rotations were checked. Translations must now be finite. SPEC.md states the three rules. Both real Aria test recordings still validate clean.
The MCAP episode info now carries the same required metadata as a zarr episode (task_name, operator, lab, scene, objects), named and checked with the same rules via required_metadata.check_required_metadata, which the zarr validator now also uses: - lab means who the data came from (rl2, mecka); site and vendor are dropped, operator is a plain stable ID (no SHA-256 requirement). - task/instruction become task_name and an optional task_description. - objects stays comma-separated in the string and is checked as a list. - missing_metadata lists required keys the source truly lacks; those fail unless --allow-missing, which turns them into warnings.
…rence CONTRIBUTING_DATA §6.6: vendors deliver zarr (what we train on) and also upload the raw recordings, in any format, with video as stored or lightly compressed at original resolution. egomimic/rldb/mcap/SPEC.md is the checklist of what raw data should contain; MCAP deliveries follow it exactly and pass the MCAP validator. SPEC.md and the module docstrings now describe it as the raw-data reference rather than the deliverable, and allow one light compression pass instead of never re-encoding.
Required metadata (both validators and the writer share one rule set):
- lab must fully match the identifier pattern ("rl2\n" no longer passes);
values with leading/trailing whitespace are rejected.
- objects may be a list or a comma-separated string; each entry is checked
and the writer stores a list.
- waiver_errors is shared by the converters and MCAP; check_stored_metadata
(finished episodes) and check_writer_metadata (writer kwargs) name what
they check.
New egomimic/rldb/zarr/episode_rules.py holds the fps (30 or 60, plain int),
embodiment (torch-free list), camera-matrix and arm-side rules, so the writer
and validator can't drift and the validator no longer imports torch.
ZarrWriter.write checks everything before touching disk, builds the episode
in a sibling temp dir and swaps it in, so a failure never leaves a half
episode or destroys an existing one. Temp/old dirs left by a killed writer
are swept (an orphaned old copy is restored). metadata_override may add keys
but raises on keys the writer sets itself instead of silently dropping or
applying them. remove_key raises on an absent key.
The zarr validator reports malformed attrs, NaN/inf poses and 0-d arrays as
failures instead of crashing, and is split into focused checks.
MCAP validator: chunk CRCs are verified; lab cannot be waived (it feeds
recording_id); every video message gets structural checks, with full decode
on a sample run with err_detect=explode; /instruction is dropped (text comes
from task_name, task_description and annotations); episode-info keys are
snake_case and vendor-recording-id becomes source_recording_id;
annotation_vendor becomes annotation_source; unknown keys, placeholders in
any text key, duplicate metadata records, multi-channel topics and repeated
once-topics are problems. validate.py is restructured around a problem
builder and dispatch tables (1373 -> ~1120 lines).
run_conversion parses the DB objects column (Postgres array literal, plain comma string, or a list; None/NULL entries dropped, an empty result is waived) instead of writing the raw literal into zarr.json. A row whose metadata the writer would reject is skipped before extraction and recorded as zarr_processing_error = "Metadata: <reason>" rather than "Zero Frames". aria_to_zarr / eva_to_zarr check the metadata before extracting, take task_name inside one required_metadata dict, and ray_helper's zarr_job requires it (keyword-only, no default). mecka_to_zarr and external/scale's sfs_to_egoverse_zarr predate the required metadata and mandatory intrinsics (already broken on main); their entry points now raise NotImplementedError pointing to INTERNAL_DATA.md §4.3. The mecka pose helpers training imports are untouched. hdf5_to_zarr.py is deleted: it imported a module that does not exist, nothing referenced it, and it wrote identity intrinsics.
…g episodes - --allow-missing / allow_missing=True keeps episodes with MissingMetadata() fields; without it they fail validation and are removed, as before. - RawEpisode.source_recording_id is written to zarr.attrs so a trimmed episode links back to its raw recording (CONTRIBUTING_DATA.md §6.6). - The MP4 preview is part of the episode: if it fails, the zarr is removed. - A per-hash flock stops parallel workers that share a start time from overwriting each other; it is released if a worker is killed.
CONTRIBUTING_DATA.md is now only what an external vendor needs; the episode DB, ingest, internal converter CLIs, run_conversion, internal S3 prefixes, training data configs and the disabled converters move to INTERNAL_DATA.md. § numbers are kept so references from code stay valid. Vendor-facing changes: - vendors propose their own task_name; RL2 maps it to existing tasks. - the S3 prefix is the lab: zarr and previews go to processed_v3/<lab>/, raw recordings to raw/<lab>/<source_recording_id>/, linked from each zarr episode by source_recording_id. - no delivery note or manifest; waived fields are accepted with --allow-missing and stored as null. - operator is a stable pseudonymous ID (no hashing requirement). - fps 30 or 60; sentinel frames must be trimmed; NaN/inf fails validation. - fixed the §4 hash example, R2 vs AWS env vars, cartesian_6d shapes, annotation end index, ffmpeg install step, and checklist items vendors can't do.
Restore hdf5_to_zarr.py and the tutorial notebooks, and drop the norm-cache child-only test and lambda reformat. These move to a stacked cleanup branch so this PR only carries the raw format and converter validation. Claude-Session: https://claude.ai/code/session_01QY8Web4qc6ZYYrAAdw6TTa
- task_name, operator and lab can never be waived (writer, validators, template, run_conversion, converter CLIs); only scene and objects can be MissingMetadata(). - ZarrWriter stamps schema_version 1; the validator reports metadata problems on episodes without one as warnings and still checks arrays. - Reject non-finite camera matrices and non-positive focal lengths, more placeholder values and non-printable characters. Share quaternion tolerance 1e-3 between the zarr and MCAP validators. - Template: whole-episode annotation is (0, T); a failed MP4 preview keeps the episode; shipped lab="template" is rejected until replaced. - Mecka CLI raises before deleting the output dir; disable sfs_zarr_pipeline with sfs_to_egoverse_zarr (#738, #739). - MCAP: forbid and detect B-frames, fixed proto field numbers, CRCs checked in unchunked files, files without a metadata index, and no full message objects kept for time series or unknown topics. - Vendor guide no longer promises prefix-scoped credentials. Claude-Session: https://claude.ai/code/session_01QY8Web4qc6ZYYrAAdw6TTa
- ZarrWriter takes source_recording_id as a required argument (one path component, no placeholder) and stores it; aria/eva set it from the raw file name, the template from RawEpisode.source_recording_id. - The validator requires source_recording_id, obs_head_pose for human embodiments, and an <episode_hash>.zarr name between 2015 and now. - The template rejects a start_time_ms that is not epoch milliseconds before writing. Claude-Session: https://claude.ai/code/session_01QY8Web4qc6ZYYrAAdw6TTa
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR makes every zarr episode carry its required metadata. It adds validators for zarr episodes and for raw MCAP data, plus a converter template and guide for vendors.
task_name,operator,lab,sceneandobjects.task_name,operatorandlabare always required, in every converter.sceneandobjectscan be waived withMissingMetadata(). A waived field is stored asnulland accepted only with--allow-missing.unknown,debug,template,nan, …), whitespace and non-printable characters are rejected.source_recording_id(the raw recording an episode was cut from) is required. Aria and eva set it from the raw file name.schema_version: 1.metadata_overridecan't set keys the writer owns.frontcamera and finite values with positive focal lengths. Extrinsics must be finite.egomimic/rldb/zarr/validate.py):obs_head_poseis required for human embodiments.<episode_hash>.zarrname must decode to a start time between 2015 and now, which catches seconds or nanoseconds passed instead of milliseconds. The template checksstart_time_msbefore writing.schema_versionare legacy, so their metadata problems are warnings and their arrays are still checked.egomimic/rldb/mcap/,SPEC.md):episode_rules.pyandrequired_metadata.py.run_conversionparses DBobjects, including Postgres arrays.sceneorobjectsbecomes a waiver.task,operatororlab, or another bad value, is markedzarr_processing_error = "Metadata: <reason>"before extraction.sfs_to_egoverse_zarrandsfs_zarr_pipeline(Re-enable Scale SFS converters with required episode metadata #739), raiseNotImplementedErrorbefore touching disk (seeINTERNAL_DATA.md§4.3).write_annotations,key_existsandremove_keyreplace theZarrWritermethods, because the constructor now requires metadata. They keepfeaturesin sync.egomimic/scripts/convert_template/template_to_zarr.py. It shipslab="template", which is rejected on purpose until the vendor sets their own.(0, T).CONTRIBUTING_DATA.mdis now the vendor guide; internal notes moved toINTERNAL_DATA.md.Unrelated cleanup (
hdf5_to_zarr.pyand the tutorial notebook deletions, plus one norm-cache test) moved to #737.Review focus
zarr_writer.pyandrun_conversion.pychange existing production behaviour. The MCAP package is new and is not called anywhere else yet.Testing
On compute nodes:
test_zarr_required_metadata,test_run_conversion_metadataandtest_convert_templatepass.test_mcap_validateandtest_packagingpass (109 passed, 1 skipped).I didn't rerun the rest of
tests/unit; CI covers it.Rollout notes
Metadata:are skipped permanently, as with any otherzarr_processing_error. Fixing the row means clearing that column by hand.convert_one_bundlegains a requiredrequired_metadataargument, so the driver and the Ray workers must be deployed together.Follow-ups
create_and_write(the Lance stack lance migration #716–ingest: dashboard, accounts and upload CLI on the vendor buckets #735, and the history: learned null token for dropped and episode-start-padded history #715 line) need the five required fields andsource_recording_id.lab=aria/site=rl2.1e9sentinelee_poseframes, which the validator rejects.https://claude.ai/code/session_01QY8Web4qc6ZYYrAAdw6TTa