DO NOT MERGE: lxr improvements for julia - #1564
Draft
oscardssmith wants to merge 33 commits into
Draft
Conversation
Everything needed to run LXR as Julia's GC, as one commit to be split into reviewable pieces. Based on wenyuzhao/mmtk-core lxr-x/simplified. Grouped by where each piece should eventually go: Independent of LXR, targets mmtk-core master: - ConcurrentImmix schedules its root-nodes work with `add_deferred` and enables the Concurrent bucket for the pause, so packets added during the pause become available when the bucket flips. Bugs in LXR itself: - LargeObjectSpace released the object reference's address rather than the super page of `to_object_start()`. Correct only for VMs that set UNIFIED_OBJECT_REFERENCE_ADDRESS; for anyone else it hands the page resource an address it never allocated. Three sites. - Line-occupancy marks were wrong in three ways: the last line went unmarked when an object did not end on a boundary, the line holding the header was never marked when the reference address sits past the allocation start, and RC_STRADDLE_LINES was per line, so two objects sharing a line raced to clear each other's mark. It is now per granule, and marking and unmarking share one function so they cannot cover different sets of lines. - `rc_sweep_nursery` trusted the in-place-promotion flag and asserted `rc_dead()` only under debug_assert, so release builds freed blocks with live objects in them. It now checks the counts. - Root objects reported as nodes were incremented but not promoted, leaving a nursery root's referents at zero; and nodes the plan does not reference count were dropped instead of seeding the trace. - Slots outside the heap have no field unlog bit, and computing a metadata address from one reads unmapped memory. Immortal and VM space now arm the field unlog bits when the plan uses a field-granularity barrier. - `Slot::is_derived` lets the coalescing barrier decline to record a slot whose referent can only be recovered from state captured when the slot was made. Re-reading such a slot at the next pause yields an interior address that is not an object. Related to mmtk#1038. Not for upstream: - An object-granularity barrier variant behind `lxr_object_barrier`, for VMs that cannot name the written field. Incomplete: epoch re-arming is not implemented, so it is not correct to run. Julia does not need it. - Bring-up diagnostics: per-pause statistics, a reference-count event ledger, live-set tracking, and the retain-nursery/retain-mature switches, all behind MMTK_LXR_* environment variables. The retain switches reach into policy/immix/block.rs, which is the wrong place for them. Co-Authored-By: Claude Opus 5 <[email protected]>
`CollectNodeRoots` only ever traced its root set inside a stop-the-world pause. Roots reported as *slots* reach the concurrent workers through `ProcessIncs`, but a binding that reports roots as objects -- Julia does, via `create_process_pinning_roots_work` -- arrives here instead, so an `InitialMark` pause seeded no concurrent work at all. With nothing seeded, `concurrent_marking_packets_drained` is trivially true, so the trigger in `should_do_cycle_collection` fires immediately and the whole transitive closure lands in the following `FinalMark` pause. LXR then behaves as a stop-the-world marker, with a pause proportional to the live heap regardless of how much time the mutator had in between. Co-Authored-By: Claude Opus 5 <[email protected]>
`ImmortalSpace::prepare` and `VMSpace::prepare` bzero the mark bits of every region they own. For a plan that marks in a single pause that is right, but LXR marks between `InitialMark` and `FinalMark`, and `Plan::prepare` ran for both. Clearing at the closing pause discarded everything concurrent marking had established, so the pause re-traced the whole graph -- for Julia, whose sysimage lives in the VM space, about 1.77M objects re-marked in every `FinalMark`. `CommonPlan::prepare_ext` and `BasePlan::prepare_ext` take the decision as a parameter, and LXR resets only in the pauses that begin a mark cycle (`Full` and `InitialMark`). `prepare` keeps its old meaning, so no other plan changes. Co-Authored-By: Claude Opus 5 <[email protected]>
Two costs dominated the barrier for a VM whose fast path cannot always name the field written. `slot_has_unlog_bit` walks the mmapper's two-level chunk table, which is more expensive than everything else the barrier does, and callers ask it for field after field of one object -- all in the same chunk, since side metadata is mapped with the chunk its data belongs to. The last chunk that answered yes is remembered per mutator and reused. Only positive answers are cached: a chunk that is unmapped now may be mapped later, and skipping a slot that does have a bit loses an increment permanently, while metadata is never unmapped, so a cached yes cannot go stale. `object_probable_write_slow` snapshots every field of the object, and nothing stopped it running again on the next store to the same object. It now logs the object as well as its fields, which is what a fast path that can only test the per-object bit (Julia's `mmtk_gc_wb_fast`, for stores whose field it cannot name) keys on, so the walk happens once per object per epoch instead of once per store. Field bits re-arm lazily as each increment is processed; there is no per-object equivalent, because the increment buffer holds slots and a slot does not identify its owner, so the logged objects are remembered and re-armed by `flush` at the end of the epoch. Leaving one logged past its epoch would lose every later store to it. Co-Authored-By: Claude Opus 5 <[email protected]>
`Plan::release` ended every pause by re-arming the log bit of every object in the heap: `ImmixSpace::set_side_log_bits` walks every chunk single-threaded (its own `warn!` says as much), and `CommonPlan::set_side_log_bits` enumerates every LOS object one atomic store at a time. That makes a pause cost O(heap) rather than O(work done in the pause) -- 1.3ms of the 4.4ms median `RefCount` pause on `tree_mutable`, serially, inside a single `Release` packet, and it grows with the heap. Only bits a barrier actually cleared need re-arming. The field barrier already records the objects it logged and re-arms exactly those on flush, and mutators are flushed during every pause, so the bits are armed before mutators resume -- the only property the old location provided. The object barrier now does the same, recording pages rather than objects. The log bit is side metadata, so re-arming a whole page costs one `bset_metadata` of that page's metadata range, the same work as re-arming a single object on it, and it collapses the many objects a mutator typically logs on one page into a single entry. Over-arming is harmless: an armed bit only means the barrier fires once more for that object, which is the state every object starts an epoch in anyway. This is the epoch re-arming that `lxr_object_barrier` was missing, and without which it was not correct to run. Co-Authored-By: Claude Opus 5 <[email protected]>
`record_rc_event` sits on every processed increment and decrement, and `is_known_live` is consulted for every object a decrement takes to zero. The first inlined its whole body including the formatting and map lookups; the second took a global `RwLock` even when the verifier had never recorded a live set, so every GC worker contended one cache line for an answer that was always `false`. The recording is now out of line and `#[cold]`, and an atomic flag says whether the live set has ever been populated, so the disabled case is a predictable load and nothing else. Co-Authored-By: Claude Opus 5 <[email protected]>
`MMTK_LXR_STATS` now times each work packet and reports the types that accounted for the most time since the last report. `work_packet_stats` already measures this but only reports at harness end, which cannot answer "what is this one pause spending its time on" -- the question that matters while reducing a pause. The table is reset when the world stops, so it describes one pause; without that it spanned from the previous `resume_mutators` and absorbed all the concurrent work executed during the intervening mutator window, which is how entries came to report far more time than the pause they appeared under. Packet timing sums CPU across all workers, so it still cannot say where a pause's *wall* time goes: a 1ms pause can contain 30ms of packet time. `stage_timeline` records ordered marks against a single clock instead, so the gap between consecutive marks is elapsed time on the critical path -- including the park/wake handshake between work-bucket stages, which no packet timer sees. Marks are placed where every worker is provably parked (so the gap is a whole stage) and around the phases of LXR's `Release`. LXR's release also reports where the pause's own phases went (weak processing, `vm_release`, the common plan) and how many concurrent tracing packets are still outstanding. Zero packets at the end of an `InitialMark` means nothing was handed to the concurrent workers and the closure will fall into the next `FinalMark`. Co-Authored-By: Claude Opus 5 <[email protected]>
`Scanning::scan_chunk_count` reports how many pieces an object's reference fields can be visited in, and `Scanning::scan_object_chunks` visits one range of them. A VM that does not implement the pair keeps the existing behaviour: the default count is `None`, and MMTk scans whole objects. This exists so that one object need not pin one worker. An object whose field count grows without bound -- a 100M-element array of references -- takes hundreds of milliseconds to walk, and no amount of buffer-level splitting reaches it, because the cost is the walk itself rather than the work it discovers. The count is deliberately not a field count: a chunk is whatever unit the VM can start and stop at, so an array of multi-reference structs can report elements and still be split. Co-Authored-By: Claude Opus 5 <[email protected]>
`scan_nursery_object` walked every field of a promoted object inline, so one worker did the whole object while the rest of the pause's workers had nothing to do. For Julia's 100M-element array of `ZonedDateTime` that walk was a 311ms stop-the-world pause on its own, and buffer-level splitting could not touch it: the elements nearly all refer to the same already-counted object, so the walk takes the increment branch and spills almost nothing into the increment buffers that the existing packet splitting works on. An object the VM offers in chunks (`Scanning::scan_chunk_count`) is now handed to the other workers as one increment packet per 8192 chunks. The per-field work moved into `count_promoted_field` so the inline and packet paths cannot diverge, and the increments a chunk discovers are flushed by the same path as any other. Measured on GCBenchmarks' TimeZones, 16 workers: worst pause 311ms -> 98ms, total GC time 486ms -> 305ms. The pause that remains is the same object: 1.29s of worker CPU across 12324 packets, so it is now bounded by how many workers there are rather than by the array's length. Getting it below a millisecond needs the counting to move out of the pause altogether, which is a change to when increments may be applied, not to how they are split. Co-Authored-By: Claude Opus 5 <[email protected]>
The packet timings say which work packet types a pause spent its time in, but not what that time bought. These are the denominators: increments handed to `ProcessIncs`, objects it promoted, and the calls the slot-less write barrier had to answer by walking a whole object because the caller could not name the field written. Time divided by increments is what distinguishes a pause that is expensive because there are many increments from one where each is slow, and fields-per-call is the amplification factor of the barrier path that Julia's codegen falls back to. Each is maintained only under `MMTK_LXR_STATS`, and accumulated per packet or per call rather than per increment or per field: a global atomic on those paths is millions of contended updates inside the pause, enough to inflate the pause it is supposed to be measuring by 8x. The barrier's counter is one atomic per call, `ProcessIncs` publishes its tallies once when the packet finishes, and the plan reports and clears them at the end of the pause. `select_collection_kind` also reports the inputs to its own decision, since a pause kind that looks wrong is almost always a decision taken on numbers that look nothing like what the heap is doing. Co-Authored-By: Claude Opus 5 <[email protected]>
The promotion trace was a chain, not a tree. `add_new_slot` spills a packet every 1024 slots, and a packet consuming 1024 slots promotes about 512 objects whose fields are about 1024 new slots, so each packet produced almost exactly one successor. Measured on `tree_mutable`: 2.08M increments in 1984 packets of ~1048 slots each, formed as 13 chains -- one per root packet -- about 152 packets long. Parallelism was capped by the number of root packets rather than the number of workers, and measured about 3x (the root buffers are uneven); `ProcessIncs` CPU over pause wall time stayed at ~3x whether 4, 16 or 64 workers were available. `ProcessIncs` now hands half of each generation to another worker once the generation is larger than `MMTK_LXR_SPLIT_MIN` slots, which turns the chain into a binary tree whose depth is logarithmic in the generation size rather than linear in it. The dead `ACTIVE_PACKET_SPLIT` switch it replaces only split below a depth threshold, which is the wrong axis: depth is not what bounds the trace, width is. Co-Authored-By: Claude Opus 5 <[email protected]>
Every work-bucket boundary woke every parked worker, and several of the buckets an LXR pause opens hold exactly one packet (`ScheduleCollection`, `StopMutators`, `ScanMutatorRoots`, `FastRCPrepare`, `Release`). All but one worker immediately found nothing and re-parked, paying for the monitor lock twice more each. The bucket-opening path now reports how many packets it made available and the last parked worker wakes that many, which costs one `notify_one` per worker that actually has something to do. Under-waking cannot strand work: a worker that adds to an already-open bucket notifies through `WorkBucket::add`, so a packet appearing later gets its own wake-up. `MMTK_WAKE_ALL=1` restores the old behaviour for comparison. `notify_work_available` also took the monitor lock unconditionally, once per packet added, to notify condition variables that usually have no waiter at all -- every worker is busy while a GC is running. It now tracks how many workers are blocked and returns without locking when that is zero; the ordering argument for why a zero reading cannot lose a wake-up is on the function. Broadcasts to `active_worker_number_changed` are likewise gated on a worker actually waiting to be reactivated. The exhaustive victim scan in `poll_slow` looks like the obvious remaining O(workers^2) waste, and is left alone deliberately: probing a few random victims instead left the minimum pause unchanged -- so the scan is not on an empty pause's critical path -- while median pause and wall time got much worse (32 workers: `strings` median 6.9ms -> 21.7ms, `tree_mutable` wall 23.0s -> 34.0s), improving monotonically as the probe count rose back to exhaustive. `WorkBucket::poll` is a batch steal, so an opening bucket's packets sit in a few workers' local deques; finding them is what keeps the phase parallel. Co-Authored-By: Claude Opus 5 <[email protected]>
A concurrent phase whose work simply runs out has nobody to announce it. LXR's concurrent decrements and sweeping end that way -- unlike ConcurrentImmix, whose concurrent marking always finishes in a `FinalMark` pause and so reports through `resume_mutators`. Until now the collector stayed in `InConcurrentGC` and looked permanently mid-GC to everything that waits for it to be quiescent, `set_disabled` in particular. The last parked worker finding no goal to respond to is exactly that moment: a pause would have arrived here as a request. It now returns the status word to `NotInGC` and calls `Collection::concurrent_work_finished`, which a binding can use to release a thread that is waiting for the collector to go quiet. The transition reports whether it happened, so a mutator that requested a pause in the meantime is not overwritten. Co-Authored-By: Claude Opus 5 <[email protected]>
A plan that reclaims lazily has freed almost nothing by the time `on_gc_end` runs: LXR hands its nursery and mature block sweeping to the concurrent phase that follows the pause, and the page resource is not flushed until that phase drains. A trigger policy that sizes the next heap target from the reserved pages it sees at `on_gc_end` therefore concludes the collection freed nothing, and grows the heap without bound -- measured on Julia's `tree_mutable`, 1.6GB to 6.4GB peak RSS. `GCTriggerPolicy::on_lazy_reclaim_finished` is called once that phase has drained and flushed, which is the first moment the freed pages can be seen through `get_reserved_pages`. It takes the plan rather than the `MMTK` instance because the plans that reclaim lazily reach it from their own bookkeeping, which has no `&'static MMTK` to hand; a policy needing per-cycle facts from the instance should latch them in `on_gc_start`. A plan that reclaims everything before `on_gc_end` never calls it, so a policy must still leave a usable target behind at `on_gc_end`. Co-Authored-By: Claude Opus 5 <[email protected]>
The decision about the next collection is taken when the previous cycle's deferred sweeping drains, and it reads how much that cycle freed. Both halves of that were wrong. `num_clean_blocks_released_lazy` is a monotonic total, but `LXR::prepare` zeroed it at the start of every pause, so a decision taken just after a pause compared this cycle's reserved page count against a counter that had just been cleared. The heap looked full and the collection looked like it had freed nothing, which hinted an emergency and forced a whole-heap stop-the-world trace -- that accounted for every emergency pause on `tree_mutable`. The counter is now left alone and `gc_pause_end` snapshots it, and the LOS equivalent, alongside `HEAP_AFTER_GC`, so all three describe the same instant and the reclamation since then is a difference. `end_of_lazy` also fired for any wave of jobs that happened to reach zero. A wave is everything that became owed between two consecutive swaps, and `swap` runs once per pause, so a wave belongs to exactly one cycle -- but the wave still accumulating can hit zero transiently, and an old wave can drain late. Either would pair the newest `HEAP_AFTER_GC` with reclamation that has not happened yet. Waves now carry the cycle they belong to, only the current cycle's wave takes the decision, and it takes it once. A pause that deferred nothing decides immediately, since no wave will arrive to do it and `wait_for_decide_cycle_collection` would otherwise wait for nobody. Co-Authored-By: Claude Opus 5 <[email protected]>
`select_collection_kind` treated the low-headroom hint exactly like a genuine allocation failure and answered both with a full stop-the-world trace. Those full traces measured about 300ms each on `tree_mutable` and were 70% of all pause time, against about 2.4ms for an `InitialMark` and 17ms for the `FinalMark` that closes it. Low headroom is a reason to *trace*, not a reason to stop the world: tracing concurrently is what this plan exists for, and the hint says the heap needs its cycles found, not that memory has run out. It also fires readily, because the headroom test compares the budget against reserved pages less the whole blocks deferred sweeping returned, while LXR reclaims mostly by recycling lines inside blocks that stay reserved -- so reserved pages sit near the budget whether or not memory is actually short. A hint that fires that easily cannot carry the weight of a whole-heap stop, and does not need to: a real exhaustion still arrives as `emergency`, and a user-requested collection still gets its full pause. Co-Authored-By: Claude Opus 5 <[email protected]>
`increase_inc_buffer_size` was a load, an add and a store, and every mutator's barrier flush lands there. A read-modify-write that is not atomic loses almost every update under that much contention: measured on `tree_mutable`, the counter read back between 0 and 10918 for pauses whose `ProcessIncs` packets handled millions of increments. Any trigger built on `inc_buffer_size` was therefore inert. It is a `fetch_add` now. Co-Authored-By: Claude Opus 5 <[email protected]>
A pause's cost is dominated by `RCProcessIncs` promoting the young objects that survived -- 12.9ms of a 14.2ms `RefCount` pause and 5.5ms of a 7.9ms `FinalMark` on `tree_mutable`, with everything else under a millisecond. The two bounds meant to cap that were both ineffective. `MAX_SURVIVAL_MB` predicted survival from `SurvivalRatioPredictor::ratio`, which counts only promotions that *copied* the object. With evacuation disabled -- permanently, for the Julia binding -- that ratio is pinned at zero, so the prediction was zero against a 128MB limit and the bound never once fired, while about 887k objects and 27MB were promoted per pause. The predictor now also tracks the fraction of young allocation that survived at all, copied or promoted in place, and the bound uses that: a pause pays for every promotion, because `ProcessIncs` scans each promoted object's fields and generates further increments from them whether or not the object moved. `ratio` keeps its meaning for sizing a to-space, where being zero without evacuation is correct. `INC_BUFFER_LIMIT` has always been declared here and never read, so nothing bounded how much mutation could accumulate between pauses: heap occupancy is not a bound on that, and a mutation-heavy program reaches the heap target with an unbounded increment buffer queued. It is now consulted, off by default, via `MMTK_LXR_INC_BUFFER_LIMIT`. Co-Authored-By: Claude Opus 5 <[email protected]>
Sweeping a nursery block reads the block's reference count table to decide whether it is dead, reusable, or promoted in place. That is O(nursery bytes), it ran single-threaded in the `Release` packet, and it measured 7-8ms of a 27ms `RefCount` pause on `tree_mutable` -- second only to `RCProcessIncs`. None of it has to happen while mutators are stopped. A block being swept is not yet on any free list, so no mutator can allocate into it either way, and the concurrent phase that runs it (`RCLazySweepNurseryBlocks`) already reports what it frees through `num_clean_blocks_released_lazy`, which is what the next pause's sizing decision reads. The only cost of deferring is that the pages come back a mutator window later. `MMTK_LXR_STW_SWEEP_BLOCKS` caps how many blocks a non-`Full` pause sweeps itself, defaulting to none. A `Full` pause still sweeps everything: it is already a whole-heap stop, and it must not leave work owed to a concurrent phase. Co-Authored-By: Claude Opus 5 <[email protected]>
The forwarding bits are read and written by object forwarding, which only happens during evacuation. A build with both nursery and mature evacuation compiled out never forwards anything, so it has no reason to reject a binding that keeps its forwarding bits on the side. Julia's object model does exactly that, and its non-moving configuration could not start up. Co-Authored-By: Claude Opus 5 <[email protected]>
oscardssmith
marked this pull request as draft
August 24, 2026 21:56
# Conflicts: # src/plan/lxr/gc_work/mature_sweeping.rs # src/plan/lxr/gc_work/tracing.rs # src/policy/immix/immixspace.rs # src/util/rc.rs
# Conflicts: # src/plan/lxr/gc_work/tracing.rs # src/plan/lxr/global.rs # src/scheduler/scheduler.rs
…radual, upstream-audited rewrite Ports mmtk-core-lxr-followup3's fix from scratch as a series of small, individually-verified commits on top of current upstream master (see the lxr-julia-obj-ref-gradual branch on the gradual worktree for the full history), rather than as one large change. This caught two things the prior version missed: - object_is_in_straddle_line() had the same missing-position-check bug as object_is_in_straddle_line_no_rc_check(), just reached through count(o) != 0 instead of a raw metadata read. Fixed by delegating to the already-fixed function instead of re-deriving the check. - Current upstream master carries two straddle-line pre-filters (in ImmixSpace::trace_object_without_moving_rc and SweepDeadCycles::process_block) that assume a mark can only ever sit at a line's own start -- true for UNIFIED_OBJECT_REFERENCE_ADDRESS VMs, not for Julia, where a header mark can land in a line's last few bytes. Left unguarded, both would have silently misidentified a live header mark as a real, dead object during a sweep. Both are now gated on UNIFIED_OBJECT_REFERENCE_ADDRESS, keeping the exact same fast path for unified VMs and falling back to the exact check for Julia. The core design is unchanged from before: mark_straddle_object_with_size marks an object's header line (when the reference address trails the allocation start) and tail line (general LXR fix, not Julia-specific -- also needed by unified VMs, see the gradual branch's second commit) with synthetic RC entries, tagged with independently-addressable straddle bits so a real object sharing that line is never misread as a mark. Also restores the historical bring-up narrative this file carries (the inc_buffer_size fetch_add fix, the MAX_REF_COUNT-encoding alternative considered for the sweep crash) that the gradual rewrite deliberately leaves out, per JULIA_LXR_UPSTREAM.md's rule to keep that out of the destination repo. Verified: OpenJDK/fastdebug DaCapo chopin (sunflow x2, avrora x2) via the JULIA_LXR_UPSTREAM.md recipe, all passing, exercising the unified fast path with the two restored pre-filters active. Julia build (WITH_THIRD_PARTY_GC=mmtk MMTK_PLAN=LXR) passes; a GC-stress script mixing tiny/sub-line/exactly-one-line/multi-line/large allocations with a cohort kept alive across 40 rounds of GC.gc(true) (to force block reuse while still live) produces an identical checksum and intact objects across 3 runs, exercising the non-unified header-mark path this change adds.
This was referenced Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A collection of LXR fixes for optimizing with Julia