Repository navigation
Repair missing blob references during manual compaction (#15538) - #15538
mikechuangmeta wants to merge 1 commit into
Conversation
|
@mikechuangmeta has exported this pull request. If you are a Meta employee, you can view the originating Diff in D128836898. |
|
| Check | Count |
|---|---|
bugprone-argument-comment |
2 |
performance-no-automatic-move |
1 |
| Total | 3 |
Details
db/compaction/compaction_iterator.cc (1 warning(s))
db/compaction/compaction_iterator.cc:2169:14: warning: constness of 's' prevents automatic move [performance-no-automatic-move]
db/version_builder_test.cc (2 warning(s))
db/version_builder_test.cc:1395:32: warning: argument name 'file_number' in comment does not match parameter name 'file' [bugprone-argument-comment]
db/version_builder_test.cc:1421:35: warning: argument name 'file_number' in comment does not match parameter name 'file' [bugprone-argument-comment]
Claude Code Review - OBSOLETESuperseded by a newer AI review. Expand to see the original review.✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit 2b6e18b SummaryWell-structured feature with thorough test coverage. The decorator iterator approach is architecturally clean and follows existing patterns. The version builder change and the UDT seqno-zeroing fix are independent concerns bundled in. High-severity findings (1):
Full review (click to expand)Findings🔴 HIGHH1. CompactionPicker
|
| Context | Executes? | Assumptions hold? | Action |
|---|---|---|---|
| WritePreparedTxnDB | YES | YES - deletion respects snapshot visibility | Safe |
| ReadOnly DB | NO - CompactRange blocked | N/A | Safe |
| Universal compaction | YES | YES - PickCompactionForCompactRange handles it | Safe |
| User-defined timestamps | YES - extensively tested | YES - kTypeDeletionWithTimestamp, seqno zeroing handles blob refs | Safe |
| CompactionService | Bypassed - forced local | N/A | Safe |
| FIFO compaction | Blocked - NotSupported | N/A | Safe |
| MemPurge | NO - manual only | N/A | Safe |
Positive Observations
- Thorough test coverage: 500+ lines covering scalar repair, wide columns, SingleDelete, snapshots, subcompactions, UDT history, merges, compaction filters, error propagation, and re-openability.
- Clean architectural fit: Decorator iterator pattern matches ClippingIterator, BlobCountingIterator, HistoryTrimmingIterator.
- Defensive design: Manual-only, local-only, FIFO-blocked, defaults off, debug-asserted.
- Garbage flow handling: InstallCompactionResults correctly skips missing blob flows with logging.
- Stress test integration: Exposed via
--compact_range_skip_missing_blob_files. - Release notes: Both new feature and bug fix have entries.
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/ci_review_prompt.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
Summary: Pull Request resolved: meta#15538 What changed? - Adds `CompactRangeOptions::skip_missing_blob_files`, default `false`, for manual repair when referenced blob files are absent from the compaction's input Version. https://pxl.cl/j8V47 - Rewrites selected SSTs. Lost scalar values become deletions. Wide entities keep surviving columns. - Manual `CompactRange` only. Runs locally; compaction services are bypassed. FIFO is unsupported. Why? Best efforts recovery can leave dangling blob references in SSTs. These can cause later compactions to fail. Offline repair prepares the DB for normal reopening. Reviewed By: xingbowang Differential Revision: D128836898
2b6e18b to
8c714d8
Compare
Claude Code Review - OBSOLETESuperseded by a newer AI review. Expand to see the original review.✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit 8c714d8 SummaryWell-structured repair feature with careful handling of snapshot semantics, wide column entities, and the compaction pipeline. The design is sound: a wrapper iterator in the compaction pipeline is the right abstraction level, and forcing local-only compaction avoids compaction service complications. Test coverage is extensive. High-severity findings (2):
Full review (click to expand)Findings🔴 HIGHH1. Version builder
|
| Context | Does code execute? | Assumptions hold? | Action needed? |
|---|---|---|---|
| WritePreparedTxnDB | YES (manual compaction) | YES - repair iterator doesn't affect read_callback | No, but no test coverage |
| ReadOnly DB | NO - CompactRange not available | N/A | Safe |
| User-defined timestamps | YES | YES - uses kTypeDeletionWithTimestamp | Tested |
| Universal compaction | YES (manual) | YES - same pipeline | No specific test |
| FIFO compaction | NO - explicitly rejected | N/A | Tested |
| CompactionService | Bypassed | N/A | Tested |
| allow_ingest_behind | YES | YES - DeletionIsObsolete checks allow_ingest_behind | No specific test |
Positive Observations
- Excellent test coverage: 15+ unit tests covering scalars, wide columns, snapshots, single deletes, timestamps, merge operations, error handling, and seek correctness. 10+ integration tests covering end-to-end repair scenarios.
- Sound architectural choice: Using a wrapper iterator is clean and composable. The repair logic is isolated from the core compaction iterator.
- Careful snapshot semantics: Repair deletions preserve sequence numbers, correctly interact with snapshot boundaries, and handle SingleDelete+repair with proper conversion.
- DeletionIsObsolete refactor: Clean extraction that reduces duplication.
- Idempotent output: Tests verify repair output is well-formed by running it through a second compaction.
- Wide column handling: Partial entity repair with correct blob column index adjustment is well-implemented.
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/ci_review_prompt.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
8c714d8 to
f594ef9
Compare
Summary: Pull Request resolved: meta#15538 What changed? - Adds `CompactRangeOptions::skip_missing_blob_files`, default `false`, for manual repair when referenced blob files are absent from the compaction's input Version. https://pxl.cl/j8V47 - Rewrites selected SSTs. Lost scalar values become deletions. Wide entities keep surviving columns. - Manual `CompactRange` only. Runs locally; compaction services are bypassed. FIFO is unsupported. Why? Best efforts recovery can leave dangling blob references in SSTs. These can cause later compactions to fail. Offline repair prepares the DB for normal reopening. Reviewed By: xingbowang Differential Revision: D128836898
Claude Code Review - OBSOLETESuperseded by a newer AI review. Expand to see the original review.✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit f594ef9 The review has been completed. Here is the final output: SummarySolid implementation of a well-scoped repair feature for dangling blob references after best-efforts recovery. The design correctly limits repair to manual High-severity findings (1):
Full review (click to expand)Findings🔴 HIGHH1. PrepareOutput timestamp preservation is an unbounded behavioral change —
|
| Context | Affected? | Analysis |
|---|---|---|
| WritePreparedTxnDB | Safe | Repair deletions use the same seqno as the original value, so snapshot visibility through read_callback_ is preserved |
| ReadOnly DB | Not affected | skip_missing_blob_files is CompactRange-only; ReadOnly DB doesn't compact |
| CompactionService | Correctly handled | Forced to local compaction with explicit bypass |
| User-defined timestamps | Affected, tested | Creates kTypeDeletionWithTimestamp; tested by RepairPreservesTimestampHistoryAfterRecovery |
| FIFO compaction | Correctly rejected | Returns NotSupported before flush |
| Universal compaction | Handled | L0 code path in PickCompactionForCompactRange also sets the flag |
| Crash during repair | Safe | Standard compaction atomicity — if crash occurs, input SSTs are preserved |
allow_ingest_behind |
Not explicitly tested | DeletionIsObsolete() checks !allow_ingest_behind() which is correct |
Positive Observations
- The design correctly limits repair to manual CompactRange, preventing accidental data loss from automatic compactions
- The MissingBlobRepairIterator is cleanly layered in the iterator chain, after history trimming but before the CompactionIterator
- The SingleDelete + repair deletion handling is carefully implemented with proper snapshot stripe awareness
- The
RepairPropagatesKnownBlobFileIoFailureDuringGctest verifies that GC on existing blob files still reports real I/O errors (repair doesn't silently swallow legitimate failures) - The
RepairRetainsHealthyBlobsReferencedByUntouchedSststest verifies that partial-range repair doesn't orphan healthy blob files referenced by untouched SSTs - Good use of
SCOPED_TRACEfor parameterized test loops
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/ci_review_prompt.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
Summary: Pull Request resolved: meta#15538 What changed? - Adds `CompactRangeOptions::skip_missing_blob_files`, default `false`, for manual repair when referenced blob files are absent from the compaction's input Version. https://pxl.cl/j8V47 - Rewrites selected SSTs. Lost scalar values become deletions. Wide entities keep surviving columns. - Manual `CompactRange` only. Runs locally; compaction services are bypassed. FIFO is unsupported. Why? Best efforts recovery can leave dangling blob references in SSTs. These can cause later compactions to fail. Offline repair prepares the DB for normal reopening. Reviewed By: xingbowang Differential Revision: D128836898
f594ef9 to
5a2fe50
Compare
Summary: Pull Request resolved: meta#15538 What changed? - Adds `CompactRangeOptions::skip_missing_blob_files`, default `false`, for manual repair when referenced blob files are absent from the compaction's input Version. https://pxl.cl/j8V47 - Rewrites selected SSTs. Lost scalar values become deletions. Wide entities keep surviving columns. - Manual `CompactRange` only. Runs locally; compaction services are bypassed. FIFO is unsupported. Why? Best efforts recovery can leave dangling blob references in SSTs. These can cause later compactions to fail. Offline repair prepares the DB for normal reopening. Reviewed By: xingbowang Differential Revision: D128836898
5a2fe50 to
82d26a1
Compare
✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit 82d26a1 SummaryWell-designed feature that enables offline repair of dangling blob references left by best-effort recovery. The approach of using a wrapper iterator in the compaction input chain is architecturally clean, and the extensive test coverage is commendable. The VersionBuilder refactoring is a sound improvement. High-severity findings (1):
Full review (click to expand)Findings🔴 HIGHH1. DeletionIsObsolete() branch duplicates ProcessDeletion() logic —
|
| Context | Affected? | Analysis |
|---|---|---|
| WritePreparedTxnDB | Low risk | Repair runs in manual compaction; read_callback not involved |
| User-defined timestamps | Tested | Multiple tests cover UDT + repair |
| CompactionService | Handled | Forced local with logging |
| FIFO | Handled | Returns NotSupported before flush |
| Universal | Safe | Same compaction picker path |
| Snapshots | Tested | Multiple tests verify snapshot semantics |
| ReadOnly DB | N/A | Manual compaction unavailable |
Positive Observations
- Excellent test coverage (~35 tests) covering edge cases, snapshots, UDT, subcompactions, merge operators, wide columns, and error propagation
- Clean iterator chain architecture
BlobIndex::DecodeFromfix from assert to Status::Corruption is a genuine bug fixcovering_the_whole_rangereorder is a latent bug fix- Proper FIFO rejection before flush prevents unnecessary work
RepairPropagatesKnownBlobFileIoFailureDuringGctest verifies GC errors for present-but-deleted blob files aren't suppressed
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/ci_review_prompt.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
Summary: Pull Request resolved: meta#15538 What changed? - Adds `CompactRangeOptions::skip_missing_blob_files`, default `false`, for manual repair when referenced blob files are absent from the compaction's input Version. https://pxl.cl/j8V47 - Rewrites selected SSTs. Lost scalar values become deletions. Wide entities keep surviving columns. - Manual `CompactRange` only. Runs locally; compaction services are bypassed. FIFO is unsupported. Why? Best efforts recovery can leave dangling blob references in SSTs. These can cause later compactions to fail. Offline repair prepares the DB for normal reopening. Reviewed By: xingbowang Differential Revision: D128836898
82d26a1 to
ae30319
Compare
|
This pull request has been merged in a51c5e2. |
Summary:
What changed?
CompactRangeOptions::skip_missing_blob_files, defaultfalse, for manual repair when referenced blob files are absent from the compaction's input Version.https://pxl.cl/j8V47
CompactRangeonly. Runs locally; compaction services are bypassed. FIFO is unsupported.Why?
Best efforts recovery can leave dangling blob references in SSTs. These can cause later compactions to fail. Offline repair prepares the DB for normal reopening.
Reviewed By: xingbowang
Differential Revision: D128836898