Priority exploration and a trace timeline

CHANGELOG.md

@@ -2,35 +2,77 @@
 
 ## 0.2.0 (unreleased)
 
+- Traces now say *where*, and that changes every hash. Each scheduling event
+  carries the host it belongs to — the machine that asked for a callback on
+  `schedule`, the machine that owns it on `run` and `cancel`, and nothing at
+  all for the simulation's own work, like a clock advance or the network's
+  delivery step — and every packet that reaches a machine records a `deliver`
+  event pairing with its `send` by uid. Both are format changes, so **every
+  recorded trace hash differs from the one 0.1.0 produced**, whatever the
+  workload does. That subsumes the narrower disclosure this list used to
+  carry, when recording crashes was the only thing that moved a hash: there
+  is now no workload whose trace is byte-identical to 0.1.0's. What a hash
+  means inside a version is untouched — same seed, same code, same
+  interpreter, same hash, checked across processes and hash-randomization
+  seeds as before.
+- `TraceEvent` is a `NamedTuple` rather than a frozen dataclass: one is built
+  for every callback the simulation schedules and runs, and a frozen
+  dataclass's per-field `object.__setattr__` cost more than twice as much on
+  the hottest path in the package. Fields, attribute access and immutability
+  are unchanged, and events now unpack and compare as plain tuples — but
+  `dataclasses.replace()` and `dataclasses.fields()` on an event raise
+  `TypeError` where they used to work.
+- Schedules can be searched by priority instead of by luck:
+  `--simloop-policy=pct` (also `explore(policy="pct")` and
+  `@sim_test(policy="pct")`) runs Burckhardt et al.'s PCT (ASPLOS 2010) —
+  random priorities per chain of work, highest ready one runs, a few randomly
+  placed demotions — which buys a stated per-run probability of hitting a bug
+  that needs `--simloop-pct-depth` ordering constraints (default 3). The
+  horizon those demotions spread over is measured rather than guessed: seed 0
+  runs the seeded schedule and its step count sizes it, which costs one extra
+  run of the workload and is what lets a found seed replay the schedule that
+  found it. PCT explores sequentially, so asking for it alongside
+  `--simloop-jobs` is refused rather than quietly ignored. It is a floor, not
+  a speed-up — on a shallow race uniform draws reach the bug some 46 times
+  sooner, and `docs/supported-api.md` publishes that measurement next to the
+  guarantee.
+- A failing seed can draw itself: `--simloop-timeline[=DIR]` writes
+  `simloop-timeline-seed<N>.html` for each failing seed and names the file in
+  the report, and `simloop.timeline_html(events)` renders any trace the same
+  way. One lane per machine and one for the simulation, virtual time left to
+  right, a dot per scheduling decision, an arrow for every packet that
+  crossed and a stub for every one that did not, with crashes and restarts
+  marking the lane they struck. The page is self-contained — inline CSS,
+  script and SVG, nothing fetched — and draws the last 5,000 events by
+  default, saying so when it dropped any.
 - Crashed hosts can come back: `loop.net.restart(name)` (or
   `host.restart()`) revives a machine as a fresh incarnation. It restores
   liveness and nothing else — the old tasks stay cancelled and the
   listeners are gone, so the caller boots the machine again the way it
   booted it the first time. Traffic due while the host was dead is lost,
   leaving peers to notice the outage from their own timeouts. Crashes are
-  now recorded too: `crash()` writes a trace event and consumes a uid, so
-  a workload that crashes a host hashes differently than it did in 0.1.0
-  (workloads that never crash stay byte-identical to 0.1.0).
+  recorded too: `crash()` writes a trace event and consumes a uid.
 - Every host now has `host.disk`, a mapping that survives its crashes:
   where state a real process would fsync belongs. Writes are atomic at
   assignment; there is no partial-write model.
 - Clocks can lie per host: `loop.net.set_clock(name, offset=...)` skews
   what that host's tasks read from `loop.time()`, and the deadlines they
   hand to `call_at` with it, while durations (`sleep`, `timeout`,
   `wait_for`, `call_later`) cost the same everywhere — which is what a
-  wrong wall clock does to a real machine. Traces stay on the true clock,
-  and runs that configure no offset are byte-identical to 0.1.0.
+  wrong wall clock does to a real machine. Traces stay on the true clock, so
+  skew never perturbs scheduling and a run that configures no offset makes
+  exactly the decisions it made without the feature.
 - A second flagship demo: `examples/raft/` is a teaching-sized Raft (leader
   election + log replication, plain asyncio on streams) tested only under
   simulation — four safety invariants checked over 50,000 chaos seeds, five
   safeguard ablations each caught and replayed from a seed, and failing
   schedules minimized toward FIFO — down to a single interesting step in the
   sharpest case.
 - Campaign evidence at scale, regenerable via `benchmarks/campaign.py`:
-  100,000 seeds of jobqueue chaos green in six minutes on a laptop, every
-  ablation caught with its failure density recorded, and 20 sampled failing
-  seeds replaying with identical trace hashes across 100 re-runs apiece.
-  A small nightly CI sweep keeps the numbers honest.
+  100,000 seeds of jobqueue chaos green in under six minutes on a laptop,
+  every ablation caught with its failure density recorded, and 20 sampled
+  failing seeds replaying with identical trace hashes across 100 re-runs
+  apiece. A small nightly CI sweep keeps the numbers honest.
 - Compatibility with third-party libraries is now measured instead of
   claimed: `probes/` drives aiohttp, anyio, websockets, httpx and the Redis
   wire protocol under a SimLoop, and `docs/compatibility.md` publishes what
@@ -50,9 +92,10 @@
   did at the first disagreement, and a window of context from both traces.
   On by default, costs one retained trace.
 - Every scheduling decision flows through a policy seam: seeded draws by
-  default (traces byte-identical to 0.1.0), with recorded choice lists that
-  can replay a schedule independently of its seed (internal, powers
-  shrinking).
+  default, making the same draws the loop made when it owned the PRNG itself,
+  with recorded choice lists that can replay a schedule independently of its
+  seed (internal, powers shrinking). Policies are shown who is ready, not
+  just how many, which is what a priority policy needs.
 - Seed exploration can use every core: `explore(fn, seeds, jobs=N)` and
   `--simloop-jobs=N` fan seed batches out over worker processes and report
   exactly what a sequential run would have — the earliest failing seed, with

README.md

@@ -83,31 +83,46 @@ replay: pytest 'tests/test_counter.py::test_two_increments_both_count' --simloop
 
 last 20 trace events:
   [t=1.3944] run      seq=58  SimNetwork._deliver
+  [t=1.3944] net      seq=21  deliver counter>b
+  [t=1.3944] schedule seq=63  b  Task.task_wakeup
+  [t=1.3944] run      seq=63  b  Task.task_wakeup
   [t=1.3944] net      seq=23  send b>counter
   ...
-  [t=1.3944] run      seq=70  SimLoop._stop_when_done
+  [t=1.3944] run      seq=70  driver  SimLoop._stop_when_done
 
-runs agree for 38 events; passing then ran StreamReaderProtocol.connection_made.<locals>.callback, failing ran list.remove
+runs agree for 41 events; passing then ran StreamReaderProtocol.connection_made.<locals>.callback, failing ran list.remove
 passing run:
-  [t=1.0944] schedule seq=13  SimNetwork._deliver
+  [t=1.0944] schedule seq=13  counter  SimNetwork._deliver
   ...
 failing run:
-  [t=1.1224] schedule seq=13  SimNetwork._deliver
+  [t=1.1224] schedule seq=13  counter  SimNetwork._deliver
   ...
 pending tasks by host:
-  counter  Task 'Task-1026'  awaiting serve  at tests/test_counter.py:26
+  counter  Task 'Task-1026'  awaiting serve  at tests/test_counter.py:24
 ```
 
 The report names the failing seed, prints the exact command that replays
 it — same scheduling decisions, same fault decisions, same trace — and
 diffs the failing run against the last passing seed, so the first thing
-the two runs did differently is one line of output. In CI, crank the
-search without touching code:
+the two runs did differently is one line of output. Trace lines name the
+machine whose work they were; the ones naming none are the simulation's own
+— the clock advancing, a packet crossing the wire. In CI, crank the search
+without touching code:
 
 ```
 pytest --simloop-seeds=1000
 ```
 
+Seeds are not the only way to search. `--simloop-policy=pct` schedules by
+priority instead of drawing uniformly — Burckhardt et al.'s PCT (ASPLOS
+2010): every chain of work gets a random priority, the highest ready one
+always runs, and a few randomly placed demotions shuffle the leader. That
+buys a stated probability, on every single run, of hitting a bug that needs
+`--simloop-pct-depth` scheduling constraints met in order. It is a floor,
+not a speed-up: on shallow races uniform draws find the bug sooner, and the
+[contract page](https://github.com/dhruvl/simloop/blob/main/docs/supported-api.md)
+publishes the measurement that says so.
+
 ## Shrink the schedule to the race
 
 Most steps of a failing schedule are noise. With `--simloop-shrink`, the
@@ -153,6 +168,27 @@ so look at the fault timings, not the task order. Shrinking is off by
 default (it costs extra runs, capped by `--simloop-shrink-budget`, default
 500) and experimental.
 
+## Watch the run that failed
+
+A schedule is easier to read as a picture than as a wall of trace lines:
+
+```
+pytest --simloop-timeline=artifacts
+```
+
+Every failing seed leaves `artifacts/simloop-timeline-seed<N>.html`, named in
+its failure report. The page is self-contained — inline CSS, inline script,
+inline SVG, nothing fetched — so it opens from a CI artifact store or an
+attachment as readily as from disk. It draws one lane per simulated machine
+and one for the simulation itself, virtual time running left to right, a dot
+for every scheduling decision on that machine, and an arrow for every packet
+that crossed. A packet that was sent and never arrived leaves a stub pointing
+nowhere, which is what a drop, a loss and a partition all look like from the
+sender's side; crashes and restarts mark the lane they struck. The last 5,000
+events are drawn, and the page says so when there were more.
+`simloop.timeline_html(events)` renders the same page from any trace you are
+holding.
+
 ## What the simulation gives you
 
 - **Seeded scheduling** — the ready queue's execution order comes from a
@@ -213,8 +249,8 @@ Simulation is cheap: SimLoop schedules a task step in ~4.4 µs (trace
 recording included — about 3.6× faster than the stock loop, which pays a
 selector syscall per iteration), compresses sleep-heavy workloads ~2,000×
 against wall clock, and with `--simloop-jobs` fanning seeds across
-processes, the full jobqueue chaos scenario sweeps 100,000 seeds in six
-minutes (~280 seeds/second) on an M4 MacBook Air. Methodology, numbers,
+processes, the full jobqueue chaos scenario sweeps 100,000 seeds in under
+six minutes (~300 seeds/second) on an M4 MacBook Air. Methodology, numbers,
 and the campaign results:
 [benchmarks/README.md](https://github.com/dhruvl/simloop/blob/main/benchmarks/README.md).
 

benchmarks/README.md

@@ -4,8 +4,9 @@ Three numbers matter for a simulation harness: what the simulated loop costs
 per scheduling step, how much simulated time it covers per wall-clock second,
 and how fast the explorer burns through seeds on a real test. Measured on a
 MacBook Air (Apple M4, 16 GB), macOS, CPython 3.12.13. Every number is the
-median of 5 runs after one warmup run, on an otherwise idle machine. Rerun
-them with the commands below; expect the ratios, not the absolute times, to
+median of at least 5 runs after one warmup run, with the machine as idle as
+a developer laptop gets. Rerun them with the commands below; expect the
+ratios, not the absolute times, to
 transfer to other machines.
 
 ## Scheduling overhead
@@ -25,16 +26,17 @@ hand-off — no I/O, no timers.
 
 SimLoop comes out about **3.6× faster per scheduling step**, trace recording
 included, and the ratio holds from 10×2000 to 500×200 task/round shapes
-(0.26–0.29× across the sweep). That is not because simloop is a faster event
-loop in any general sense — it is because a simulated loop never touches the
-OS. Profiling the stock run shows about half its time inside
-`select.kqueue.control`: the real loop pays a selector syscall on every
-iteration even when no I/O is pending, while SimLoop's iteration is pure
-Python — pop the PRNG-chosen callback, run it, append a trace event. The
-practical reading: replayable scheduling costs nothing at test time. (For
-contrast, trio's experimental deterministic-scheduling hook measured ~15%
-overhead on top of its normal loop — python-trio/trio#890; simloop sidesteps
-the comparison by replacing the loop instead of instrumenting it.)
+(0.26–0.30× across the sweep, widening with the task count). That is not
+because simloop is a faster event loop in any general sense — it is because a
+simulated loop never touches the OS. Profiling the stock run shows about half
+its time inside `select.kqueue.control`: the real loop pays a selector
+syscall on every iteration even when no I/O is pending, while SimLoop's
+iteration is pure Python — pop the PRNG-chosen callback, run it, append a
+trace event. The practical reading: replayable scheduling costs nothing at
+test time. (For contrast, trio's experimental deterministic-scheduling hook
+measured ~15% overhead on top of its normal loop — python-trio/trio#890;
+simloop sidesteps the comparison by replacing the loop instead of
+instrumenting it.)
 
 The stock-loop baseline is macOS/kqueue; an epoll or io_uring machine will
 price the syscall differently.
@@ -51,7 +53,7 @@ of simulated time:
 
 | simulated | wall | compression |
 |---|---|---|
-| 7164 s (1.99 h) | 3.55 s | **~2,000×** |
+| 7164 s (1.99 h) | 3.60 s | **~2,000×** |
 
 Virtual time never sleeps: between timers the clock jumps, so a suite full of
 `await asyncio.sleep(300)` costs only its callback processing. This is what
@@ -67,8 +69,17 @@ The jobqueue chaos campaign runs one full distributed scenario per seed —
 a broker, 3 workers, and 2 clients submitting 8 jobs (some poisoned) under
 randomized partitions and a worker crash, then settles for up to 600
 simulated seconds and checks every invariant. 300 seeds complete in
-**5.4–6.0 s**, about **55 seeds/second**. A thousand-seed overnight search is
-a 20-second coffee break.
+**5.3–6.3 s** across nine runs (median 5.8 s), about **52 seeds/second**,
+in one process. A thousand-seed overnight search is a 19-second coffee
+break.
+
+That is slower than the ~55 seeds/second (5.4–6.0 s) published for 0.1.0,
+and the cost is a feature: 0.2.0 records a trace event for every packet
+delivery, not
+just for every send, which is what lets a timeline draw both ends of a
+crossing — and a scenario this network-heavy pays for the extra events
+directly. The wire is where this benchmark spends its time, which is why
+it is the number that moved.
 
 ## Campaigns
 
@@ -109,9 +120,9 @@ Results, recorded 2026-08-01 on the M4 MacBook Air (10 jobs):
 
 | campaign | scale | result |
 |---|---|---|
-| green | 100,000 seeds, 6.0 min, 279.6 seeds/s | green — no invariant violated |
-| ablations | 6 mutations × 10,000 seeds, 2.6 min | every ablation caught, densities below |
-| replay stability | 20 failing seeds × 100 re-runs | identical trace hash on every run |
+| green | 100,000 seeds, 5.6 min, 300.1 seeds/s | green — no invariant violated |
+| ablations | 6 mutations × 10,000 seeds, 2.5 min | every ablation caught, densities below |
+| replay stability | 20 failing seeds × 100 re-runs, 13.5 s | identical trace hash on every run |
 
 Per-ablation failure density:
 

docs/design.md

@@ -99,10 +99,11 @@ properties are load-bearing:
 - **Completeness.** Even a *cancelled* handle's draw is recorded — the draw
   consumed PRNG state, so it is a scheduling decision, and a replay that made
   a different set of draws must produce a different hash.
-- **Injectivity.** Events serialize as `kind|when|seq|label` lines into a
+- **Injectivity.** Events serialize as `kind|when|seq|host|label` lines into a
   SHA-256. Labels are qualified callback names or network labels built from
-  validated host names, and host names may not contain `|`, `>` or newline —
-  so two distinct event streams cannot collide onto one byte sequence.
+  validated host names, the host field is a validated host name itself, and
+  host names may not contain `|`, `>` or newline — so two distinct event
+  streams cannot collide onto one byte sequence.
 
 Hash equality is therefore a cheap, sufficient check that a replay was
 *exact*, not merely same-outcome. It is asserted all over the test suite and
@@ -258,15 +259,25 @@ ten lines of task startup.
 
 **The policy seam.** The loop has exactly one ordering decision — which
 ready callback runs next — and it now flows through a policy object: seeded
-draws by default (byte-identical to owning the PRNG directly, held to that
-by the determinism suite), or a scripted replay of a recorded choice list.
-The recording makes a schedule *editable* where the seed only made it
-*repeatable*: a seed is a name for one schedule, but a choice list can be
-truncated, blanked to FIFO, or perturbed one decision at a time. Scripted
-runs tolerate drift on purpose — an out-of-range choice clamps, an
-exhausted recording falls back to FIFO — because an edited schedule that
-crashes the replayer answers nothing, while one that runs to a verdict
-answers the only question shrinking asks.
+draws by default (making the same draws the loop made when it owned the PRNG
+directly, held to that by the determinism suite), or a scripted replay of a
+recorded choice list. The recording makes a schedule *editable* where the
+seed only made it *repeatable*: a seed is a name for one schedule, but a
+choice list can be truncated, blanked to FIFO, or perturbed one decision at
+a time. Scripted runs tolerate drift on purpose — an out-of-range choice
+clamps, an exhausted recording falls back to FIFO — because an edited
+schedule that crashes the replayer answers nothing, while one that runs to a
+verdict answers the only question shrinking asks.
+
+The seam later grew a third policy and a wider view to feed it. Policies are
+handed the ready queue as `(owner, label)` views rather than a count, which
+is what a priority scheduler needs and what the two original policies still
+ignore; on top of that sits PCT (Burckhardt et al., ASPLOS 2010), reached by
+`--simloop-policy=pct`, which prices each chain of work with a random
+priority and demotes the leader at a few random steps to buy a stated per-run
+probability of hitting a bug of a given ordering depth. The contract page
+carries the guarantee, the calibration story and the measurement showing it
+is a floor rather than a faster search.
 
 **Shrinking is delta debugging over choices, judged by the exception.**
 Three passes, cheapest first: truncate the recording past the failure
@@ -302,7 +313,9 @@ so SimLoop schedules a task step in ~4.4 µs where the stock macOS loop
 spends ~16 µs — about half of that inside the per-iteration `kqueue` call —
 making the simulation roughly 3.6× faster per step *including* trace
 recording. Sleep-heavy workloads compress ~2,000× against wall clock, and
-the full jobqueue chaos scenario explores ~55 seeds/second on a laptop.
+the full jobqueue chaos scenario explores ~52 seeds/second in one process on
+a laptop — down from ~55 in 0.1.0, which is what the per-delivery trace
+events a timeline draws from cost a network-heavy workload.
 The trio thread priced deterministic scheduling at ~15% overhead; replacing
 the loop instead of instrumenting it turned the overhead negative.