Repository navigation
jit: register _collections deque iterators as GC roots - #684
Conversation
WalkthroughThe change re-exports GC-managed deque iterator wrappers and registers both wrapper types in the JIT garbage-collector type registry. ChangesDeque iterator GC registration
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🤖 Codex parity reviewStatic analysis of this diff vs the local RPython/PyPy sources (commit 4bbc502). Files in the reviewed diff1. Regressions to PyPy parity introduced by this patchNone. 2. Other mismatches introduced by this patchNone. 3. Pre-existing mismatches (already present before this patch)
4. Structural adaptations
|
W_DequeIter and W_DequeRevIter are #[pyre_class] wrappers holding a traced `deque` edge but were never wired into build_gc, so the native GC-root completeness oracle aborts at init. They are `.allocate()`-immortal (malloc_typed), which the marker skips, so register both in the immortal `register_pyre_class_offsets` block alongside W_Enumerate/W_UnpackIter, and re-export the two types from the _collections module so build_gc can name them. Assisted-by: Claude
37824cd to
4bbc502
Compare
|
Force-pushed a corrected revision ( The first revision registered the two deque iterators via Corrected to the immortal-root path Note: PR #643 registers the same deque iterators via — commented by Claude |
#684 (register deque iterators as GC roots) and #685 (native _warnings, which bundled its own deque-iterator registration) both landed on main and each added a `_collections` re-export plus a build_gc registration for W_DequeIter / W_DequeRevIter. Merged together they collide: - `_collections/mod.rs` re-exports each type twice -> E0252, main fails to compile. - `eval.rs` registers each iterator twice: #684 via the immortal `register_pyre_class_offsets` list and #685 via the managed `register_pyre_class` marker path. W_DequeIter / W_DequeRevIter are `.allocate()`-immortal (malloc_typed), which the marker skips, so the immortal `register_pyre_class_offsets` registration (#684) is the effective one. Drop #685's managed registration and the duplicate re-export. Assisted-by: Claude
#685 and #684 each added `pub use deque_iter::W_DequeIter` / `pub use deque_rev_iter::W_DequeRevIter` to _collections/mod.rs at different offsets; both landed on main, so the names are now re-exported twice — E0252 `defined multiple times`, which breaks the Charon LLBC extraction and any fresh pyre-interpreter compile. Keep the documented #684 block and remove the bare duplicate. Assisted-by: Claude
…tration #684 (register deque iterators as GC roots) and #685 (native _warnings + _tokenize) both landed on main. #684 added the deque iterators to the immortal `register_pyre_class_offsets` list plus a `_collections` re-export; #685 added a second `_collections` re-export and managed `register_pyre_class` registrations for W_DequeIter, W_DequeRevIter, and W_TokenizerIter. Merged onto this branch: - `_collections/mod.rs` re-exports W_DequeIter / W_DequeRevIter twice. - `eval.rs` registers the deque iterators twice, once via the immortal offsets list and once via the managed marker path. W_DequeIter, W_DequeRevIter, and W_TokenizerIter are all `allocate`-immortal (malloc_typed), which the marker skips, so the managed `register_pyre_class` registrations neither root their inline children nor belong in `pytype_to_tid`; the extra tids 117-119 break the subclass-range alias and hierarchy asserts at startup. Drop the duplicate re-export and the three managed registrations, and add W_TokenizerIter to `register_pyre_class_offsets` so its `readline` field is rooted. The deque iterators were already in that list via #684. Assisted-by: Claude
#685 and #684 each added `pub use deque_iter::W_DequeIter` / `pub use deque_rev_iter::W_DequeRevIter` to _collections/mod.rs at different offsets; both landed on main, so the names are now re-exported twice — E0252 `defined multiple times`, which breaks the Charon LLBC extraction and any fresh pyre-interpreter compile. Keep the documented #684 block and remove the bare duplicate. Assisted-by: Claude
…ubclass ranges (#205 C2) (#646) * pyre-jit-trace: make the trace walker generic over a WalkSym symbolic state Route the FBW walker's symbolic-state access through a WalkSym trait and thread a `Sym: WalkSym` type parameter through the walk/step entry points, WalkContext, FbwWalkMode, and the dispatch helpers. The conversion is distributed across the jitcode_dispatch directory-module submodules (arith, branch, bridge_subwalk, diag, fbw_state, heapcache_ops, inline_call, mod, residual_call, resume_snapshot, specialize, tests, vable_ops, vstack_mirror). PyreSym field access becomes trait accessor calls; concrete PyreSym parameters become the generic Sym. jd0 (PyreSym) monomorphizes to the prior concrete walker. Assisted-by: Claude * majit: resolve build-time jitcode descrs via a shared global pool Add a process-global RuntimeBhDescr pool (set_global_build_descr_pool + GlobalDescrPool newtype) and JitCode::descr_at, which reads the per-jitcode exec.descrs first and falls back to the global pool. Route the JitCodeMachine descr reads (switch, untyped/typed inline_call, residual-call calldescrs, runtime_bh_descr fielddescrs, call_target/call_assembler_target) through descr_at, so build-time LLBC-extracted jitcodes with an empty per-jitcode pool resolve through the shared pool. Decode the typed inline_call_{r,ir,irf}_* family in JitCodeMachine dispatch. Assisted-by: Claude * jit: dormant jd1 unpackiterable driver groundwork and subclass-range GC numbering Add the dormant jd1 (unpackiterable_driver) scaffolding: UnpackSym / UnpackJitState (unpack_state.rs), register it as an inert second jitdriver, wire the no-op merge-point hook (baseobjspace.rs, call.rs, eval.rs), and resolve the portal jitcode by canonical key (jitcode_runtime.rs). Install the build-time descr pool from ALL_DESCRS (install_global_build_descr_pool) and add the resolution test. Match interpreter subclass ranges to GC numbering (pyobject.rs, typedef.rs, pyre-interpreter/lib.rs, eval.rs build_gc census). Add census entries for W_Accumulate / W_ZipLongest (tids 111-112), which shift the coroutine / coroutine-wrapper / dict-view-iterator tail to tids 113-115. check.py: dynasm 225/225, cranelift 225/225, wasm 224/224. Assisted-by: Claude * jit: thread jd1 reds (w_iterator, items) through the unpackiterable merge-point hook baseobjspace.py:31 unpackiterable_driver uses reds='auto', auto-collecting the loop locals w_iterator and items. The pyre meta-tracer cannot auto-collect Rust locals, so pass them explicitly from the marker (baseobjspace.rs) through unpack_merge_point (call.rs) to the hook (eval.rs unpack_merge_point_jit). The hook body remains a no-op; behavior unchanged. cargo check -p pyre-jit clean. Assisted-by: Claude * gc: restore W_Deque (tid 116) in the subclass-range census build_gc registers `collections.deque` W_Deque via register_pyre_class at the tail of the type-id chain, after the coroutine and dict-view-iterator slots, so it takes the current max GC type id 116. The subclass-range census enumerated the object hierarchy only through the dict-view iterators (tid 115), so the startup assertion in build_gc — actual GC vtable registrations against the shared alias census — failed on the extra tid-116 registration, panicking at every process start. Add (116, Some(0)) to SUBCLASS_RANGE_HIERARCHY and subclass_range_alias(116, W_Deque) to pyre_interpreter::all_subclass_range_aliases. Assisted-by: Claude * jit: WIP jd1 unpackiterable driver live-path drive (gated PYRE_JD1, walk SIGBUSes) Replace the no-op unpack_merge_point_jit hook with a real drive of jd1's extracted _unpackiterable_unknown_length loop, gated behind PYRE_JD1 (default off; PYRE_JD1_THRESHOLD overrides the warmup count, PYRE_JD1_DEBUG logs each stage). On the hot iterator-type green key the hook: - resolves jd1's portal jitcode via portal_jitcode_for_key ("baseobjspace::_unpackiterable_unknown_length") + installs the global build-time descr pool, - force_start_tracing with a descriptor carrying index=Some(2) so elect_active_jitdriver_sd elects the novable jd1 over jd0, seeding w_iterator/items as the two Ref reds (live_values), - drives trace_jitcode_with_args_and_runtime through the production with_trace_ctx_and_token_resolver runtime with the reds as InputArg argboxes, - abort_trace(false) to discard the recorded trace (residual side effects already landed on the shared reds). Verified so far (PYRE_JD1_THRESHOLD=1): the hook fires, the counter fires, and force_start_tracing returns StartedTracing — jd1 is elected and the trace session starts. The subsequent JitCodeMachine walk of the extracted body SIGBUSes inside trace_jitcode_with_args_and_runtime (before it returns), so residual execution is not yet working; the crash is independent of the runtime (bare ClosureRuntime and the resolver runtime both SIGBUS). Also found: _unpackiterable_unknown_length is rarely reached from Python — list()/tuple()/set()/sorted()/[*x]/{*x}/starred-assign all drain via direct next-loops that bypass this marker; only ~3 startup unpacks hit it. Assisted-by: Claude * jit: enter the jd1 trace at the merge point instead of the jitcode entry `trace_jitcode_with_args_and_runtime` resets the frame pc to 0 in `setup_call`, so driving the jd1 portal replayed `_unpackiterable_unknown_length`'s prologue (`length_hint` / `newlist` / `iterator_greenkey`) with `items` seeded where the function's `w_iterable` argument belongs, faulting inside `length_hint`. Add `trace_jitcode_from_merge_point`: it decodes the `jit_merge_point` op's green/red register lists at the header pc, seeds the green ref as a Const and each red ref as its InputArg into those registers, and starts the walk at the header pc (code_cursor) with the prologue skipped. Wire `drive_unpack_iterable_trace` to it, threading the raw greenkey pointer through the merge-point hook for the green seed. Add gated diagnostics: `MAJIT_OPTRACE` per-op cursor/pc/opcode trace in `run_to_end`, and `PYRE_JD1_DUMP` op-layout dump of the extracted body. The walk now executes the merge point and reaches the first residual call; the residual funcptr constant for `next` is unpatched (`jit_trace_fnaddrs` omits the unpack-loop callees), so the call still faults — fn-address resolution is a follow-up. Assisted-by: Claude * jit: bind jd1 unpack-loop residual callees (next, w_list_append) The `unpackiterable_driver` body residual-calls `next(w_iterator)` and `w_list_append(items, w_item)` directly in source, so the codewriter records the bare source paths. Without a runtime binding their funcptr constants fall back to a `symbolic_fnaddr_for_path` hash the residual handler cannot resolve, faulting when the merge-point walk executes the call. Add `bh_next`, a Ref-returning `#[jit_may_force]` bridge that publishes the raised exception (StopIteration included, unlike `jit_next`, whose null-for- StopIteration convention serves FOR_ITER's GuardNonnull). Register it under the `next` source paths and bind `w_list_append` (a `-> ()` residual) directly in `jit_trace_fnaddrs`. Assisted-by: Claude * interpreter: dedup and correct the #684/#685 deque/tokenizer GC registration #684 (register deque iterators as GC roots) and #685 (native _warnings + _tokenize) both landed on main. #684 added the deque iterators to the immortal `register_pyre_class_offsets` list plus a `_collections` re-export; #685 added a second `_collections` re-export and managed `register_pyre_class` registrations for W_DequeIter, W_DequeRevIter, and W_TokenizerIter. Merged onto this branch: - `_collections/mod.rs` re-exports W_DequeIter / W_DequeRevIter twice. - `eval.rs` registers the deque iterators twice, once via the immortal offsets list and once via the managed marker path. W_DequeIter, W_DequeRevIter, and W_TokenizerIter are all `allocate`-immortal (malloc_typed), which the marker skips, so the managed `register_pyre_class` registrations neither root their inline children nor belong in `pytype_to_tid`; the extra tids 117-119 break the subclass-range alias and hierarchy asserts at startup. Drop the duplicate re-export and the three managed registrations, and add W_TokenizerIter to `register_pyre_class_offsets` so its `readline` field is rooted. The deque iterators were already in that list via #684. Assisted-by: Claude
#684 re-exported W_DequeIter / W_DequeRevIter at the top of the module while #685 had already re-exported the same two names below the `mod deque_rev_iter` block, so the merged tree carries two `pub use` of each and fails to compile (E0252). Remove the lower pair; the top pair keeps its build_gc-root comment. Assisted-by: Claude
Problem
_collections._deque_iterator(W_DequeIter) and_collections._deque_reverse_iterator(
W_DequeRevIter) are#[pyre_class]wrappers that each hold a traceddeque: PyObjectRefmanaged child, but they were never wired into
build_gc. On the native backends the GC-rootcompleteness oracle (
eval.rs,#[cfg(not(target_arch = "wasm32"))], unconditional) thereforeaborts at GC init:
This fires before any user code runs, so every native JIT program panics — even a trivial
whileloop.main(014bc0fae6b) is affected.Fix
Both iterators are
.allocate()-immortal:allocatelowers tomalloc_typed, and the GCmarker skips malloc_typed objects. So they must be registered on the immortal-root path,
register_pyre_class_offsets, alongsideW_Enumerate/W_UnpackIter— the immortal-root walkerforce-forwards the
dequechild on every collection. Re-export the two types from_collections(they live in private submodules) so
build_gccan name them.Validation
gc.collect()(major collection). Withregister_pyre_class_offsetsthe sum matches CPython;the earlier
register_pyre_classdropped the deque (swept), so this is the test that separatesthe two paths — a light/minor-GC or churn workload passes either way.
check.py: dynasm 241/241, cranelift 241/241 green. One unrelated pre-existingwasm-only failure (
selfrec_tail_exception_unwind, tracked separately) passes on both nativesand is not touched by this change.
Summary by CodeRabbit
collections.dequeiterators.