Repository navigation
perf(lakehouse): store hot-projection keys once, 6.9 GB to 1.1 GB - #48
Merged
Merged
Conversation
The SQLite hot projection repeated 64-character hashed keys in the table and in every index, plus an implicit primary-key index: about 813 bytes a race. Swimmer, meet and event keys now live once in dimension tables; the race table and its athlete-history and meet-results indexes hold integers, rows are inserted clustered by swimmer and date, and a `races` view keeps the original columns. Uniqueness of (source, race_id) is already enforced when the snapshot is read, so no index carries race_id. `lakehouse.py rebuild-hot` replaces the projection from the committed Iceberg snapshot without touching it. On the national checkpoint (8,506,767 races, snapshot 4441271835344284414) the store is 1,101,565,952 bytes, 129.5 bytes a race; per-course/category totals and 498 sampled swimmer histories match Iceberg, and warm history lookups take about 0.02 ms (0.012 before; the cost is the join to the swimmer table). Receipt, REPORT appendix and lakehouse notes updated. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The local SQLite hot projection repeated 64-character hashed keys in the table and in every index, plus an implicit primary-key index. That came to 6.9 GB, about 813 bytes a race.
athlete_historyandmeet_resultsindexes hold only integers. Rows are inserted clustered by swimmer and date, and aracesview keeps the original columns.lakehouse.py rebuild-hotreplaces the projection from the committed Iceberg snapshot. The snapshot ID is unchanged.Measured on the national checkpoint, 8,506,767 races:
Lookups are slightly slower because of the join to the swimmer table. The remaining bulk is the race ID, stored once per row.
Parity checks:
probepasses.Tests:
TURNMARK_LAKEHOUSE_TESTS=1 python -m unittest discover -s research/conversion-modelpasses (108), including a new test thatrebuild-hotrestores a corrupted projection from the pinned snapshot.render_report_tables.py --checkpasses.🤖 Generated with Claude Code