Skip to content

cache norm stats for data configs to minimize recomputation - #633

Merged
RyanPCo merged 1 commit into
mainfrom
ryanco/phase2-norm-cache
Sep 29, 2026
Merged

RyanPCo merged 1 commit into
mainfrom
ryanco/phase2-norm-cache

Conversation

@RyanPCo

@RyanPCo RyanPCo commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Co-Authored-By: Claude Opus 5 (1M context) [email protected]
Claude-Session: https://claude.ai/code/session_01UHDiN8yfERcsiFdz4mXuZi

@RyanPCo
RyanPCo force-pushed the ryanco/phase2-format-version branch from b998502 to d604c48 Compare September 13, 2026 17:47
@RyanPCo
RyanPCo force-pushed the ryanco/phase2-norm-cache branch from 089425f to c81cfc1 Compare September 13, 2026 17:47
@RyanPCo
RyanPCo force-pushed the ryanco/phase2-norm-cache branch from c81cfc1 to f717d16 Compare September 13, 2026 18:09
@RyanPCo
RyanPCo force-pushed the ryanco/phase2-format-version branch 2 times, most recently from 9046c7b to da2248d Compare September 13, 2026 20:11
@RyanPCo
RyanPCo force-pushed the ryanco/phase2-norm-cache branch from f717d16 to 6cadcff Compare September 13, 2026 20:11
@RyanPCo
RyanPCo force-pushed the ryanco/phase2-norm-cache branch from 6cadcff to 27eb832 Compare September 13, 2026 21:59
@RyanPCo
RyanPCo force-pushed the ryanco/phase2-format-version branch from da2248d to 424a204 Compare September 13, 2026 21:59
@RyanPCo RyanPCo changed the title norm stats: content-keyed cache under paths.cache_dir cache norm stats for data configs to minimize recomputation Sep 14, 2026
@RyanPCo
RyanPCo force-pushed the ryanco/phase2-format-version branch from 424a204 to 7d7c40f Compare September 15, 2026 13:31
@RyanPCo
RyanPCo force-pushed the ryanco/phase2-norm-cache branch from 27eb832 to 73e2073 Compare September 15, 2026 13:31
@RyanPCo

RyanPCo commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai

@greptile-apps

greptile-apps Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 4/5

[Medium risk] Adds caching layer for dataset normalization statistics.

The PR should not merge until chunk-only episode changes invalidate cached normalization statistics.

Findings

  1. P1 Chunk changes reuse stale stats ▶
  2. P2 Malformed cache entries pass validation ▶

Summary

The PR adds a shared, content-keyed normalization-stat cache, wires it into training, and documents and tests the new configuration.

  • Cache hits reuse a precomputed stats file; misses compute and write one.
  • The episode fingerprint does not detect chunk-only value changes.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Resolve training episodes] --> B[Fingerprint episode metadata]
  B --> C[Hash recipe and code]
  C --> D{Cache hit?}
  D -->|Yes| E[Load norm stats]
  D -->|No| F[Compute norm stats]
  F --> G[Write cache entry]
Loading

Reviews (1) · Last reviewed commit: "norm stats: content-keyed cache under pa..."

Comment thread egomimic/rldb/zarr/norm_cache.py Resolved
Comment thread egomimic/rldb/zarr/norm_cache.py Resolved
@RyanPCo
RyanPCo merged commit 42817f5 into main Sep 29, 2026
5 checks passed
@RyanPCo
RyanPCo deleted the ryanco/phase2-norm-cache branch September 29, 2026 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant