Repository navigation
Conversation
A read writes `emitted` at depth 0, so a hydration snapshot rests uncompressed in the buffer pool until its feedback retractions arrive. On TPC-H sf100 that chain held 74 GiB of swap on its own, the run's peak once the chain merges were compressed. A read whose input reaches 256 MiB now writes `emitted` one generation deep, which lets the pool compress it, while a steady-state read stays at depth 0. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Large MV hydrations on swap-backed replicas vary by up to 1.7x in duration with identical configuration. The slow runs put far fewer correction chunks into the buffer pool and take 2.5x the major faults, so more of the buffer sits on the heap and pages through kernel swap. These gauges show where that heap comes from: chunks the pool declined at mint, or chunks a merge or split read took out of the pool. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The compute and storage instances kept each replica's dyncfg override in a map beside the replica state, so every path that removed and re-added a replica had to carry the override across. Compute's rehydration did not: `Instance::rehydrate_replica` removed the replica, which deleted its override, and the replay and every later configuration push reached the reconnected replica without its replica-scoped parameters. The coordinator re-pushes overrides only when the scoped configuration changes, so the loss lasted for the rest of the replica's life. The override is now a field of the replica state in both controllers. `add_replica` takes it as an argument, so creation and rehydration go through the same path and cannot drop it, and dropping a replica drops its override with it. Overrides for replicas a DDL transaction has not added yet stay in the controller-level maps, which `add_replica` reads from. Closes: CPU-288 Co-Authored-By: Claude Opus 5.5 <[email protected]>
Adds `column_paged_batcher_read_prefetch` to skip the buffer pool's `MADV_WILLNEED` before an extent read, `compute_correction_v2_single_read` to hand freshly merged `emitted` chunks to the persist batch builder while their bodies are in hand, and gauges for the MV sink write task's command queue. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Behind `compute_correction_v2_columnar_queue`, the MV sink's Timely operator consolidates each activation's desired and negated persist `ok` updates into a chain of chunks minted at the depth hint `compute_correction_v2_queue_depth`, and sends that chain to the write task instead of owned rows. The write task files the chain into the correction buffer's pending and bucket chains without sorting. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Small activations each minted a queued chain below the buffer pool's minimum spill size, which stayed on the heap. The operator now holds `ok` updates until they fill `compute_correction_v2_chunk_size` bytes, or until the batch carries frontier or consolidation work, or a batch write is about to start. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The pool's eviction queues are FIFO within a depth band, so under a backlog a fixed depth hint evicts the head of the write queue first. With `compute_correction_v2_queue_geometric_depth`, a queued chunk takes depth 0 while less than one unit of bytes is queued ahead of it, 1 below three units, 2 below seven, and 3 beyond. The unit, `compute_correction_v2_queue_depth_unit_bytes`, defaults to the pool budget. Filed chunks count as depth 0 for the correction buffer's merges. Co-Authored-By: Claude Opus 5.5 <[email protected]>
`compute_mv_sink_async_yield_interval` makes the async MV sink's write operator yield its Timely worker once its synchronous correction work since the last yield reaches the interval. Gauges count the yields and the longest single synchronous step. Co-Authored-By: Claude Opus 5.5 <[email protected]>
With `column_paged_batcher_overflow_handoff`, an insert the pool's budget denies even after enforcement takes its slot anyway and hands the chunk to the spill threads while their in-flight cap has room, instead of compressing it on the inserting thread. The overshoot above budget and slack is bounded by that cap, plus one chunk per inserter racing the cap check. `mz_column_pool_overflow_handoffs_total` counts the handoffs. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Records per-worker histograms of loop iterations and counters of time per phase, and logs iterations of 2 s or more with their phase breakdown. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…orrection-exp-interactive # Conflicts: # src/compute/src/compute_state.rs
Member
Author
|
Experiment build, done. Results are on CPU-293, CPU-298 and the CPU-279 thread. Closed by Claude Code. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment build for staging, not for merge. It merges the interactive runtime stack (#38678) into the correction write-path experiment branch (#39524), so the hydration stall measurements can be repeated with peeks and peek dataflows served by the interactive runtime.
Posted by Claude Code.
🤖 Generated with Claude Code