Shapes: scale shared transition publication beyond two lanes

[chrisbbreuer]

Parent: #478
Split from: #530
Related: #461, #480, #493

Problem

#530 replaces deep linear lookup and predictable transition hashing with collision-free persistent AVL indexes. Its functional four-lane and suppression-free TSan witnesses pass, but a dedicated shared-realm publication diagnostic shows that total transition throughput stops scaling beyond two lanes even when lanes publish beneath independent parent shapes.

The pre-implementation fixture is intentionally split into:

• partitioned base shapes: each lane publishes unique fanout edges beneath its own prepared parent shape in one shared Context;
• anti-specialization variant: different base/key layout with identical logical work and checksum;
• contended control: every lane publishes beneath one parent, preserving the serialized worst case.

Each invocation uses a fresh base-shape generation and reuses one object per lane, deleting back to the base after every unique publication. This removes per-iteration object construction while preserving real transition publication, exact own-key order, deletion, and deterministic checksums.

Scope

• Attribute 1/2/4/8-lane time to Shape transition locks, Context arena allocation, object property locks, GC, and Thread lifecycle using shared attribution.
• Remove the causal independent-parent bottleneck with bounded Context-owned storage and an explicit synchronization design.
• Consider per-thread arena tranches, batched immutable-node allocation, or optimistic persistent-root publication only when exact profiles justify them.
• Preserve same-name convergence, collision-independent exact keys, stable slots, delete/re-add undo, OOM atomicity, moving GC, and Context teardown.
• Keep the same-parent contended row visible; do not describe intentional serialization as scalable.

No-workaround rules

• No GIL, hidden serialization outside the measured lock, prepublishing scored keys, reusing a result cache, reducing logical work at higher lane counts, unbounded per-thread reserves, benchmark-name/source recognition, or excluding the 4/8-lane rows.
• Do not trade exact same-name convergence or OOM rollback for duplicate visible shapes.

Acceptance

• Benchmark source, base/variant/contended identities, jobs, checksums, warmup, and timed boundary are committed before the runtime fix.
• Clean 1/2/4/8-lane order-balanced evidence reports transitions/second, instructions/transition, lock contention/wait, allocation bytes, GC, peak/retained RSS, and dispersion.
• Partitioned total publication throughput is monotonic across 1/2/4/8 lanes and the anti-specialization variant agrees; every row retains identical per-lane work.
• The contended row remains bounded and exact, with wait/serialization attributed rather than hidden.
• Same-name convergence, disjoint fanout, deep delete/re-add, allocation-failure rollback, 32 MiB bounded heap, moving/parallel GC, threadfuzz, and suppression-free TSan pass.
• Shallow classes, mixed application, dense Array, and Shapes: index deep named-property lookup without breaking transitions #530 deep lookup rows do not regress materially; retained memory remains bounded and published.

Security boundary

Property names, fanout, and thread schedules are attacker-controlled. The fix must improve independent publication without reintroducing predictable collision behavior, races, half-published roots, or unbounded thread-local retention.

[chrisbbreuer]

Benchmark-first commit landed on main as d9d54b5b: bench(shape): add shared transition publication rows (#531).

Frozen rows:

• representative_shape_transition_fanout: one fresh prepared base shape per lane;
• _variant: different base/key layout with identical logical work/checksum;
• _contended: all lanes deliberately share one parent shape.

Each lane reuses one object, publishes a unique edge, validates exact own-key order/value, deletes back to the prepared base, and checks absence. For 1,000 publications/lane the exact 1/2/4/8 checksums are 503,500 / 1,507,500 / 5,017,000 / 18,042,000. The 18/18 benchmark structural gates pass.

No runtime fix is included in this commit. Quick diagnostics confirm the partitioned row stops scaling beyond two lanes; accepted measurements and causal attribution will follow from this frozen parent.

[chrisbbreuer]

Benchmark boundary correction is now pushed in d30790f.

The committed row no longer charges Reflect.ownKeys or membership scans to each transition publication. Every operation still publishes a unique computed edge, validates the observed value, and validates deletion back to the prepared base shape; exhaustive key-order and absence semantics remain in the production unit suite.

For jobs=1000, the frozen exact checksums are now:

• 1 lane: 501500
• 2 lanes: 1503500
• 4 lanes: 5009000
• 8 lanes: 18026000

A quick parent diagnostic still loses aggregate throughput beyond two lanes (about 333k, 303k, 255k, and 137k publications/s at 1/2/4/8 lanes), so this commit is benchmark isolation only, not a runtime fix. I am tracing the remaining shared-realm environment/property ownership and synchronization costs before changing the engine.

[chrisbbreuer]

A second benchmark isolation correction is pushed in 6236f4b.

Unique property-name strings are now constructed in the existing untimed __benchmarkPrepare phase, one immutable key array per lane. The scored worker boundary remains one unique Shape-edge publication, exact value observation, and delete-to-base validation per operation. This removes integer/string formatting and its GC/backing allocations from the claimed transition-publication work.

The benchmark matrix structural tests pass 18/18, and all 20 quick samples across 1/2/4/8 lanes retain the frozen checksums (501500, 1503500, 5009000, 18026000). No runtime fix is included in this commit.

[chrisbbreuer]

Shape transition attribution is now first-class in 6ced428. Tier-attribution rows record requests, hits, misses, and writer yields alongside the existing Environment/Object/allocation/worker counters.

On the isolated 1,000-publication-per-lane row, the candidate records:

• 1 lane: 1,002 requests / 1,001 misses / 0 yields
• 2 lanes: 2,004 / 2,002 / 0
• 4 lanes: 4,008 / 4,004 / 0
• 8 lanes: 8,016 / 8,008 / 0

That evidence rules out contended per-Shape writer locks in the partitioned row. It instead exposed persistent-AVL path-copy amplification and the surrounding ordinary-property synchronization costs. The benchmark comparison self-tests and 18/18 representative matrix structural tests pass.

[chrisbbreuer]

The Shape publication implementation is pushed in 2708a80.

The persistent AVL remains the deep-shape property lookup index. Transition fanout now uses a separate append-only concurrent skip list:

• exact byte-key equality;
• one node per published edge instead of copying an AVL spine;
• one existing per-Shape writer lock for same-name convergence;
• lock-free readers over release-published immutable forward links;
• SipHash-2-4 secret level selection for attacker-controlled property names;
• secure-entropy failure is fail-closed, matching collection hash seeding.

Validation:

• Shape-focused units: 27 pass / 0 fail / 0 leak, including allocation-failure rollback, bounded + moving GC, concurrent same-name convergence, and shared-realm publication.
• focused no-GIL TSan publication: pass (23.56 s), no race/leak report.
• exact Shape-only test262: Object 3,411 + Array prototype 2,810 + Proxy 311 = 6,532 pass / 0 fail / 0 flips.

The partitioned JS row still does not meet the issue’s monotonic 1/2/4/8 throughput acceptance because surrounding ordinary-property synchronization remains dominant. I am keeping #531 open and continuing that work; the Shape writer-lock diagnosis itself is now closed by measured zero yields.

[chrisbbreuer]

The measured non-Shape bottleneck is split into #532: ordinary named-property synchronization and its private/publication proof. #531 remains the parent end-to-end scaling issue; #532 owns the lock-amortization work needed to satisfy its remaining monotonic 1/2/4/8 acceptance. This split does not relax or remove any #531 gate.