From 97348e4cfebcc51a8684dfa3acefd15b6025caec Mon Sep 17 00:00:00 2001 From: Mike Grier Date: Tue, 3 Nov 2026 17:55:00 -0500 Subject: [PATCH 01/19] docs: plan a long-haul tier, a probation tier, and suites that do only what their claims need The engineer's direction, 2026-11-03, after sabotage controls timed out on runners that stalled under suites doing real file I/O: reduce each heavy test to the size its claim needs, keep its full scale in a long-haul tier that runs daily when main has moved and outside the sabotage sweep, and put the tests that have never run in CI on probation, where failures do not fail the run and results are gathered toward promotion. Recorded as a decision with its working positions (the promotion threshold, where results are kept), the measurements in the rationale, the work as M45, and the sweep-timeout question parked behind it as M45+.1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHECKLIST.md | 60 +++++++++++++++++++++++++++++++++++++++++++++ DESIGN-NOTES.md | 30 +++++++++++++++++++++++ DESIGN-RATIONALE.md | 20 +++++++++++++++ PARKED.md | 13 ++++++++++ PLANS.md | 4 +-- 5 files changed, 125 insertions(+), 2 deletions(-) diff --git a/CHECKLIST.md b/CHECKLIST.md index 8d05658b..69662ce4 100644 --- a/CHECKLIST.md +++ b/CHECKLIST.md @@ -601,3 +601,63 @@ yanks are needed, is [DE-D-8](crates/durable-ioring-events/DESIGN-NOTES.md#de-d- The engineer's own step, with their crates.io login: `cargo yank --version 0.1.1 durable-ioring-events`, then `cargo yank --version 0.5.0 win-durable-ioring` and `cargo yank --version 0.5.1 win-durable-ioring`. Both driver releases require the events crate at 0.1.1 or later within 0.1, so all three go, and the replacements must exist first. + +## M45 -- A long-haul tier, a probation tier, and suites that do only what their claims need + +Each pull request's CI, and each sabotage entry, runs a crate's whole suite, so every test pays its cost +on every one of them. Measured locally, 22 of the workspace's tests take a second or more and account +for nearly half of its serial test time; on GitHub runners, tests doing real file I/O have stalled for +many seconds while idle, which failed sabotage controls on the post-merge sweeps of `c714e83b` and +`13f06742`. So a test's scale is kept only where the claim needs it, and the scale for its own sake moves +to a long-haul tier that runs daily. Tests that have never run anywhere start on probation. The decision +is [DESIGN-NOTES.md](DESIGN-NOTES.md#long-haul-tier); the measurements behind it are in +[DESIGN-RATIONALE.md](DESIGN-RATIONALE.md#why-a-long-haul-tier). + +- [ ] **M45.1** -- The tiers' convention and its census. A long-haul test is + `#[ignore = "long-haul: "]` and sits in a module named `long_haul`; a probation test is + `#[ignore = "probation: "]` in a module named `probation`, since libtest filters by name + and not by ignore reason. A census script fails when a reason and its module path disagree in either + direction, so a test cannot be marked into a tier that never selects it; it runs in CI on every push, + with tests of its own for both directions. DEVELOPMENT.md states the convention. + +- [ ] **M45.2** -- `long-haul.yml`: daily, and only when `main` has moved since the last completed run, + plus manual dispatch. It runs `--ignored long_haul` across the workspace, and a failure there fails the + run. Then it runs `--ignored probation`, whose failures do not fail the run. Separate from the + sabotage sweep, which never passes `--ignored` and so never pays for either tier. + +- [ ] **M45.3** -- Keep each probation test's results across runs, and report them. The report gives each + probation test's runs, passes and current run of passes, which is what promoting it from `probation` + to `long_haul` -- a code change -- is decided on. Where the results live is this item's to decide + (artifacts expire; the sabotage record's branch is the pattern that does not), and so is the + promotion threshold, which is a working position until the first results exist. + +- [ ] **M45.4** -- Put the never-run tests on probation: `windows-ioring-sys`' five flush-barrier stress + tests ("saturates a disk with background writers"; "writes tens of MiB per trial") and + `win-durable-ioring`'s test that allocates a buffer of more than 4 GiB. + +- [ ] **M45.5** -- `win-durable-ioring`'s `blocking_push` example: write past a small declared capacity, not + three times 256 writes, and keep the full scale as a long-haul test. + +- [ ] **M45.6** -- `windows-file-enumeration-sys`: the directory of thousands of entries and the two large + running enumerations that are cancelled or dropped. Each to the size its claim needs, the full size + long-haul. + +- [ ] **M45.7** -- `win-multi-file-tail-watcher`: the large roster dropped during a poll, the job log under the + tiny and default options, the children whose output is reassembled, and the real-writer tests. Shrink + where the scale is not the claim; where it is a timing claim (a reader quiet between bursts), say so + and leave it. + +- [ ] **M45.8** -- `windows-platform-probes`: the tests that run every registered probe, or a cost probe at a + multiplier, at full workload. Run the probes at the smallest workload that still shows what each test + checks, with full workload long-haul. + +- [ ] **M45.9** -- The tests whose wait is their claim: `win-job-launcher`'s bound killing a descendant or a + grandchild, `win-debugging-sys`' process that outlives its bound, and the registration bounds in + `win-durable-ioring`. Shorten a bound only where the claim holds at the shorter one; otherwise leave it + and say why. + +- [ ] **M45.10** -- Review the remaining tests of a second or more against the same rule, starting with + `windows-file-watcher`'s scenario stress and `windows-ioring-sys`' properties under every resolution, by + re-measuring the workspace once M45.5 to M45.9 are in. + + > **-> UNBLOCKS PARKED WORK:** completing this unblocks `M45+` in [PARKED.md](PARKED.md#m45-plus). diff --git a/DESIGN-NOTES.md b/DESIGN-NOTES.md index 57f9807d..f3d6d9fc 100644 --- a/DESIGN-NOTES.md +++ b/DESIGN-NOTES.md @@ -3027,6 +3027,36 @@ last learned and not from nothing, which is what a clone's first scoped sweep ot The alternatives and what building it found are in [DESIGN-RATIONALE.md](DESIGN-RATIONALE.md#how-the-record-came-to-a-branch). + +**A test's scale is kept where its claim needs it; scale for its own sake runs daily in a long-haul tier, and a test that has never run starts on probation.** +The engineer, 2026-11-03, on the sabotage controls that timed out because a runner stalled under a +suite doing real file I/O: "do less in each of these tests", since a lighter suite helps all of CI, and +"perhaps we should have some heavy / long-haul runs for post-merge runs ... (but not as part of the +sabotage full runs)". Every pull request's CI and every sabotage entry runs a crate's whole suite, so a +test's cost is paid on each; a test whose claim holds at a small size is written at that size. + +- **The tiers are named, not inferred.** A long-haul test is `#[ignore = "long-haul: "]` + in a module named `long_haul`; a probation test is `#[ignore = "probation: "]` in a + module named `probation`. libtest selects by name, not by ignore reason, so the module path is what + runs a tier and the reason is what tells a reader why; a census fails when the two disagree, since a + test marked into a tier its name does not select would never run. The cost in the reason is + documentation (the engineer: "I don't know what the value of the 'cost' is other than pure + documentation but that's ok"). +- **The scale is moved, not dropped.** A test reduced to the size its claim needs keeps its full-scale + form as a long-haul test, so nothing that ran before stops running; it runs daily instead of on every + push. +- **Cadence: daily, and only when `main` has moved** (the engineer), plus manual dispatch, in a workflow + of its own. Not part of the sabotage sweep, which never passes `--ignored` and so never pays for either + tier. +- **Probation is for a test that has never run in CI** (the engineer: a set "which run after the main + tests which don't fail the run but we gather data on whether they succeed or fail and we can promote + them to non-probationary once they succeed for enough runs"). It runs after the long-haul tier, its + failures do not fail the run, and its results are kept across runs. Promotion is a code change, moving + the test to `long_haul`. **Working positions, not decisions:** how many passes are enough, and where the + results are kept; both are `M45.3`'s, decided once there are results to look at. + +The work is [CHECKLIST.md](CHECKLIST.md) `M45`. What prompted it, and the measurements, are in +[DESIGN-RATIONALE.md](DESIGN-RATIONALE.md#why-a-long-haul-tier). ## A `compile_fail` doctest names the error it exists to produce **Every `compile_fail` doctest pins its expected error code -- ```` ```compile_fail,E0502 ```` -- and CI diff --git a/DESIGN-RATIONALE.md b/DESIGN-RATIONALE.md index 13738e04..30ff178a 100644 --- a/DESIGN-RATIONALE.md +++ b/DESIGN-RATIONALE.md @@ -1057,3 +1057,23 @@ select fewer entries than a full one would. **Not decided.** Whether the cache should be removed once the branch is shown to work. It is kept beside the branch on purpose, so a defect in the new path cannot cost a run its record, and the merge-or-delete decision is parked with the post-merge observation that would settle it. + +## Why a long-haul tier, and what it was measured from + +The post-merge sabotage sweeps of `c714e83b` and `13f06742` each failed on a `win-durable-ioring` control +that timed out. Neither was a hang. The failed shards' transcripts showed the same code on the same runner +taking far longer than its own baseline: the library tests in one, the `blocking_push` example in the other, +with almost no CPU used while they waited. The stall moved between test binaries, so it was the runner and +not one test; but a suite doing less real file I/O gives a stall fewer places to land. + +Measured locally on 2026-11-03 by timing every test in the workspace one at a time: 22 tests take a second +or more and account for nearly half of the serial total. They fall into three kinds. Scale for its own sake +(a directory of thousands of entries, large running enumerations, a large roster), whose claims hold at a +size just past one batch. Probes run at full workload where the test checks the shape of their output. And +waits that are the claim, such as a bound that must kill a process, which shrink only if the bound can. + +**Alternatives not taken.** Raising the sweep's timeout for the crate, or re-running a control that times +out: both answer the symptom, and are parked behind the reduction (`M45+.1`) until it is known how often a +stall still reaches a control. Deleting the heavy tests: the scale is evidence the smaller form does not +give, so it moves to a daily tier instead. Running the heavy tier inside the sabotage sweep: the sweep runs +the whole suite once per entry, which is exactly the cost this removes. diff --git a/PARKED.md b/PARKED.md index 2b3732e5..4c621a58 100644 --- a/PARKED.md +++ b/PARKED.md @@ -19,3 +19,16 @@ milestone of [CHECKLIST.md](CHECKLIST.md), and the file is deleted when it is em `record` job's dependence on two stores; keeping it costs a second copy that can disagree with the first, which the merge by latest result already tolerates. Either way, the merge-or-delete is made when the new path is proven and is not pre-empted by a testing or efficiency shortcut. + +## M45+ -- What a sabotage sweep does with a control that times out (parked behind M45.10) + +- [ ] **M45+.1** -- Decide whether a sabotage entry declared `survives` that times out is run again once before + it is reported caught, or whether a crate whose suite does real I/O gets a manifest `timeoutSeconds` well + above its baseline, or neither. + + **Gate:** [M45.10](CHECKLIST.md), the re-measurement after the suites are reduced. The question arose from controls in + `win-durable-ioring` timing out on runners whose tests stalled while idle (post-merge sweeps of + `c714e83b` and `13f06742`); the engineer, 2026-11-03, chose first to make the suites do less, and this + is left until it is known how often a stall still reaches a control once they do. A control's patch + changes nothing, so a timeout on one cannot be a detection; a timeout on an entry declared `caught` + can be, and stays one either way. diff --git a/PLANS.md b/PLANS.md index 5b1de1ca..ad4a94a8 100644 --- a/PLANS.md +++ b/PLANS.md @@ -40,7 +40,7 @@ plans tracker: [crates/durable-ioring/PLANS.md](crates/durable-ioring/PLANS.md), | [CHECKLIST-mutation-survivors.md](CHECKLIST-mutation-survivors.md) | not started | Work queued from the workspace-wide cargo-mutants sweep of 2026-09-02, whose findings are kept in [mutation-sweeps/2026-09-02/](mutation-sweeps/2026-09-02/README.md) rather than re-derived -- the run took roughly fourteen hours. 2,792 caught, 1,112 survived, 198 timed out. **The headline numbers mislead in three ways and the README says how**: a timeout in a blocking-API crate is usually a detection that lost its name rather than a gap (measured: one of `windows-waitable-queues`' 120 timeouts fails four tests in 0.00s when re-injected alone), a low score on an executable probe crate is measuring the wrong thing, and three kinds of survivor -- equivalent mutants, unreachable code, and constants that want a `const` assertion -- are not missing tests at all. M1 covers the shipping crates; M2 holds the two crates that are not libraries and whose scope is an engineer's decision; M3 re-runs and prunes rather than hand-editing the tool's output into a second source of truth. | [mutation-sweeps/2026-09-02/README.md](mutation-sweeps/2026-09-02/README.md) | | [crates/topology-planner/CHECKLIST.md](crates/topology-planner/CHECKLIST.md) | in progress | **Planned, not built** -- the directory holds a plan and no code, and becomes a crate when M2 begins. Owns the mapping from a **dataflow description of an application** -- its input, output and processing code paths, the shape settled 2026-09-23 as `EP-D-6` and discharging the deferral `EP-D-4` named -- plus an abstracted idealized machine description, to **one or more suggested** sets of execution domains: which processors host a domain, where each thread pins, which memory node it allocates from, what channel connects each pair, and where each channel's buffer lives. `EP-D-6` also split planning into two stages -- connectivity inferred with no machine in hand, then realization against a machine model -- which adds `EP-1+.5` (the connectivity graph is a value and needs a type and a home) and `EP-1+.6` (the answer is a set, so what a caller receives and what `M2+.2` renders are sets). Filed because that mapping was **unowned**: [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) M32 lists the contracts "the runtime cannot be written without" and all of them concern the queue, while M33+.1 opens with "one pinned thread, its `IoRing`, its node-local registered pool, its shard" -- presupposing a plan nothing computed. Separate from `windows-topology-sys` because that crate states **facts** and this one applies **policy**; fusing them is what produced `outermost_partitioning_cache`, a policy answer sitting in the facts crate that three consumers then re-derived differently (SH-16.9). M1 was a *requirements* milestone -- it states what the topology must answer, and it fed the locality-model session, which has since concluded as `D-13`..`D-21`. **The component was deferred past PR #56 by direction**, contributing only planning documents there; #56 then closed unmerged on 2026-09-15 and its content is landing in peeled pieces instead, so the component is still unlanded and goes in its own pull request. Per `D-21` the topology reshape lands without it, since `windows-topology-sys` publishes a refined view of what the platform publishes and an adapter absorbs the rest. M2+ and M3+ are parked on that session concluding, and are additionally **awaiting a re-cut**: EP-D-4 and EP-D-5 re-scoped the component into four parts (`topology-model` holding the abstract machine description, the planner's traits and the plan type; `topology-planner`; an inward Windows adapter; an outward realizer), and only M1 has been reconciled with that. EP-1.1 is done and already earned its keep: checking the shard-set query against the model found `Processor::capacity` using `0` as both a valid efficiency class and a "not known" sentinel, which collide on every non-hybrid machine (filed as SH-16.12). | [crates/topology-planner/DESIGN-NOTES.md](crates/topology-planner/DESIGN-NOTES.md), [design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md](design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md) | | [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) | in progress | **M30 is complete and archived** in [COMPLETED-CHECKLIST.md](COMPLETED-CHECKLIST.md): the queue crate's name, skeleton and SPSC shape. M31 built the bounded-array MPSC (with a lazily created manual-reset doorbell whose reset cannot be separated from the observation that there is nothing to take -- achieved by ordering plus a re-check rather than by a lock, per D-9 and D-15) and is done but for `M31.6`, the `loom` verification, which is re-homed as `M30.4` in [CHECKLIST.md](CHECKLIST.md). M32 remains: the contract decisions -- ordering, correlation, backpressure among them -- the domain runtime cannot be written without. M33+ parks the runtime itself, the creation-time-affinity thread builder, the namespace `Outcome` extension, and the client-side `ThreadpoolWait` fan-in helper; the durability crate it also parked (`M33+.5`) was transferred on 2026-10-05 to [crates/durable-ioring](crates/durable-ioring/COMPONENT.md), reshaped to build on `IoRing` directly. M-inf holds items each gated on a specific measurement rather than on taste. The N=1 path is the whole first deliverable and depends on no NUMA hardware. | [DESIGN-NOTES.md](DESIGN-NOTES.md), [DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md](design-sessions/DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md) | -| [CHECKLIST.md](CHECKLIST.md) | in progress | M42: apply the documented-contract rule to the refusals made ahead of platform calls that already exist. M19: propagate the 2026-08-27 platform measurements (IoRing registration replaces the table; the completion-port/`IoRing` fork; `runs_long` as the growth mechanism; the measured 512 default maximum) into the crates whose code or documentation currently assumes otherwise. M20: decide the session-independent path form, now that path resolution is measured to follow the impersonated token's logon session. M21: reconcile with the impersonation and enumeration crates that landed during the session. M34 carries the review-driven repairs raised while shipping the placement tool: M34.1 (the reusable sabotage harness) is done, and M34.2 (route the placement tool's output through a sink rather than writing to stdout from many sites) and M34.3 (archive the completed item bodies still carried by the three root checklists) are open. M37: discharge the failable-call standard across the workspace. M30: find out how much of this workspace's algorithm correctness can be machine-checked -- a survey matching each argued-but-unchecked algorithm to a class of tool (TLA+/PlusCal, loom, bounded proof, `const` assertions), one pilot chosen because parameter shrinking makes an untestable property exhaustive, and a named list of what the pilot could not reach, which is the deliverable. Scoped as an instrument for narrowing hand-inspection rather than replacing it, and explicitly not a reversal of [D-31](crates/windows-waitable-queues/DESIGN-NOTES.md#d-31). Also re-homes `M31.6`, the `loom` verification the queue crate promises adopters before 1.0: it was previously untracked, referenced from that crate's design notes, a source file and its sabotage manifest with no live checklist item anywhere, and is now queued as M30.4. M30's rationale is in [DESIGN-RATIONALE.md](DESIGN-RATIONALE.md#machine-checking-what-is-argued) -- Tier 2, because no decision is taken yet; M30.5 is what produces one. M38 labels Model A and Model B at their first use in live files. M39 builds a locality helper that derives a file's flush domains from Windows' storage topology for durable-ioring. M43 corrects the `durable-ioring-events` 0.1.1 release, which should have been 0.2.0: release it, then yank 0.1.1 and `win-durable-ioring` 0.5.0 ([DE-D-8](crates/durable-ioring-events/DESIGN-NOTES.md#de-d-8)). | [DESIGN-NOTES.md](DESIGN-NOTES.md#remoting-synchronous-namespace-operations) for M19-M21 and M37; N/A for M30 | +| [CHECKLIST.md](CHECKLIST.md) | in progress | M42: apply the documented-contract rule to the refusals made ahead of platform calls that already exist. M19: propagate the 2026-08-27 platform measurements (IoRing registration replaces the table; the completion-port/`IoRing` fork; `runs_long` as the growth mechanism; the measured 512 default maximum) into the crates whose code or documentation currently assumes otherwise. M20: decide the session-independent path form, now that path resolution is measured to follow the impersonated token's logon session. M21: reconcile with the impersonation and enumeration crates that landed during the session. M34 carries the review-driven repairs raised while shipping the placement tool: M34.1 (the reusable sabotage harness) is done, and M34.2 (route the placement tool's output through a sink rather than writing to stdout from many sites) and M34.3 (archive the completed item bodies still carried by the three root checklists) are open. M37: discharge the failable-call standard across the workspace. M30: find out how much of this workspace's algorithm correctness can be machine-checked -- a survey matching each argued-but-unchecked algorithm to a class of tool (TLA+/PlusCal, loom, bounded proof, `const` assertions), one pilot chosen because parameter shrinking makes an untestable property exhaustive, and a named list of what the pilot could not reach, which is the deliverable. Scoped as an instrument for narrowing hand-inspection rather than replacing it, and explicitly not a reversal of [D-31](crates/windows-waitable-queues/DESIGN-NOTES.md#d-31). Also re-homes `M31.6`, the `loom` verification the queue crate promises adopters before 1.0: it was previously untracked, referenced from that crate's design notes, a source file and its sabotage manifest with no live checklist item anywhere, and is now queued as M30.4. M30's rationale is in [DESIGN-RATIONALE.md](DESIGN-RATIONALE.md#machine-checking-what-is-argued) -- Tier 2, because no decision is taken yet; M30.5 is what produces one. M38 labels Model A and Model B at their first use in live files. M39 builds a locality helper that derives a file's flush domains from Windows' storage topology for durable-ioring. M43 corrects the `durable-ioring-events` 0.1.1 release, which should have been 0.2.0: release it, then yank 0.1.1 and `win-durable-ioring` 0.5.0 ([DE-D-8](crates/durable-ioring-events/DESIGN-NOTES.md#de-d-8)). M45 adds a daily long-haul tier and a probation tier, and reduces each heavy test to the size its claim needs ([long-haul-tier](DESIGN-NOTES.md#long-haul-tier)). | [DESIGN-NOTES.md](DESIGN-NOTES.md#remoting-synchronous-namespace-operations) for M19-M21 and M37; N/A for M30 | | [CHECKLIST-ship-topology-and-queues.md](CHECKLIST-ship-topology-and-queues.md) | in progress | Release `windows-topology-sys` 0.2.0 and `windows-waitable-queues` 0.1.0, which everything shorter-term depends on. **Both reached crates.io on 2026-09-05**, by a route this file does not describe, since PR #56 closed unmerged; `SH-4.15` owns reconciling M4 with what shipped. Deliberately redundant with [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md): that file plans the design, this one plans the release, and a release has failure modes a design checklist does not surface. Two were found while writing it -- `windows-waitable-queues-v*` is missing from the publish workflow's tag list, so release-please would tag it and nothing would publish it, silently; and `windows-ioring-sys` is published against `windows-topology-sys = "0.1.0"`. (That second finding was later **corrected at SH-2.2**: the pin is a *dev*-dependency, which consumers never resolve, so it obliges a pin update but no release.) M1 settled the public surface before it was public (done, archived); M2 repairs the plumbing; M3 lands the branch; M4 releases; M5 verifies from outside the workspace; M6 is long-running validation and gates the queue crate's release specifically. M7-M13 were seven PR #56 review rounds (done, archived); M14, M15 and M16 are the three later rounds and carry the file's open work -- M15 owns the fix for an ABA hole that ships **disclosed rather than fixed**, so it does not block the release. M16 is the SH-3.1.1 diff review, the first to read the branch as a diff rather than react to a comment: seven findings, six fixed, including a publish-workflow regression this branch had introduced two commits earlier and a soundness hole in the crate about to freeze its API. Its remaining four are blocked on [design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md](design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md), which began by asking whether collapsing a seven-kind, any-depth topology onto a single cache boundary is the right projection and has since settled that presence and observation must be modeled rather than collapsed into an `Option`. **That work gated the merge, and has since discharged**: unlike M14 and M15, which concern a defect in an implementation that can ship disclosed, M16 concerned the shape of the public model `windows-topology-sys` 0.2.0 would publish, and a published model cannot be reshaped without another break. It became the `MMT-*` plan, which has landed; the session has concluded and 0.2.0 shipped the new model, so M3 no longer waits on M16. The file opens with a status table. | [crates/windows-topology-sys/DESIGN-NOTES.md](crates/windows-topology-sys/DESIGN-NOTES.md), [crates/windows-waitable-queues/DESIGN-NOTES.md](crates/windows-waitable-queues/DESIGN-NOTES.md) | | [CHECKLIST-placement-tool.md](CHECKLIST-placement-tool.md) | in progress | A small Windows tool a stranger can download, run once, and send back one structured result -- distributed as a GitHub release binary, never to a registry, per `PT-5.6` -- so this workspace can collect placement and NUMA-hop measurements from hardware it does not own. The gap is concrete: every host available here has one NUMA node, so the whole `cross NUMA node` row and the inter-node hop matrix are unmeasured and no local work will change that. Extracted rather than published in place, because [crates/windows-platform-probes](crates/windows-platform-probes) is deliberately "an experiment, not a component". **M1-M4 and M36 are complete and archived** in [COMPLETED-CHECKLIST.md](COMPLETED-CHECKLIST.md): the name and what the record says about a machine, `(group, number)` processor identity so a machine with more than 64 of them is not silently miscounted, each NUMA hop measured in both directions with the ring placed deliberately, the move of the measurement modules, a schema-versioned submission record, the runner's experience and trust, and redaction of the secondary metadata by default. What remains is M5 (distribute the binary), M6 (are "equivalent" processors actually equivalent?) and M7 (report what Windows contradicts about itself). **M5+ is WITHDRAWN**: the canonical artifact is a CI-built binary compiled from this repository through `path` dependencies, so it needs nothing published, and an earlier version of this row wrongly gated the whole tool on releasing `windows-topology-sys` and `windows-waitable-queues`. | [crates/windows-placement-probe/DESIGN-NOTES.md](crates/windows-placement-probe/DESIGN-NOTES.md), [crates/windows-platform-probes/DESIGN-NOTES.md](crates/windows-platform-probes/DESIGN-NOTES.md) | | [CHECKLIST-thread-ambient.md](CHECKLIST-thread-ambient.md) | in progress | **M22-M29 are complete and archived** in [COMPLETED-CHECKLIST.md](COMPLETED-CHECKLIST.md): `windows-thread-ambient-sys` (a standalone layer that captures a thread's ambient state and applies it on another thread), `windows-namespace-request-sys` (marshalable Win32 namespace call parameter sets, over an entry list audited from three real consumers rather than guessed), `windows-platform-probes` (a durable home for the measurements this workspace's designs rest on), and the defects the audit of those three found. What remains is **pending**: the three `M26+` items were each gated on the namespace-facility design branch reaching `main`, and it has, so the gate has lifted -- reconciling the imported design background, applying the M22.2 narrowing to M21.2, and making the merge-or-delete decision on the duplicated path preparation. The file is deleted outright once those land. | [crates/windows-thread-ambient-sys/DESIGN-NOTES.md](crates/windows-thread-ambient-sys/DESIGN-NOTES.md) | @@ -60,6 +60,6 @@ plans tracker: [crates/durable-ioring/PLANS.md](crates/durable-ioring/PLANS.md), | [crates/windows-file-watcher/PARKED.md](crates/windows-file-watcher/PARKED.md) | parked | The `M-inf` horizon: `ReadDirectoryChangesExW` extended records, digest-based change verification, and a per-volume capability cache, outside v1 by D-19. | [DESIGN-NOTES.md](crates/windows-file-watcher/DESIGN-NOTES.md) (D-19) | | [crates/wtf-string/PARKED.md](crates/wtf-string/PARKED.md) | parked | The `M-inf` horizon: a checked no-interior-NUL C-string companion type, optional by D-7. | [DESIGN-NOTES.md](crates/wtf-string/DESIGN-NOTES.md) (D-7) | | [crates/windows-file-enumeration-sys/PARKED.md](crates/windows-file-enumeration-sys/PARKED.md) | parked | REVIEW-1: re-read against that contract, which no longer caps ordinary paths or rewrites any to `\\?\` form, and review it against a traversal layer before one is built. Raised from the `windows-file-watcher` side; schedules no change of its own. | [DESIGN-NOTES.md](crates/windows-file-enumeration-sys/DESIGN-NOTES.md), [workspace DESIGN-NOTES.md](DESIGN-NOTES.md#path-contracts-follow-path-construction) | -| [PARKED.md](PARKED.md) | parked | M34.25: keep or remove the Actions cache path for the sabotage catcher record once the `sabotage-records` branch is shown to work, gated on M34.24. | [DESIGN-NOTES.md](DESIGN-NOTES.md#sabotage-records) | +| [PARKED.md](PARKED.md) | parked | M34.25: keep or remove the Actions cache path for the sabotage catcher record once the `sabotage-records` branch is shown to work, gated on M34.24. M45+.1: what a sweep does with a control that times out, gated on M45.10. | [DESIGN-NOTES.md](DESIGN-NOTES.md#sabotage-records) | Add a row here when new work is planned, against [CHECKLIST.md](CHECKLIST.md) or any crate's. From b3377ebeeecedcd60e1737c689804a1dbda482dd Mon Sep 17 00:00:00 2001 From: Mike Grier Date: Tue, 3 Nov 2026 18:43:08 -0500 Subject: [PATCH 02/19] chore(tools): check that a test's ignore reason and module agree about its tier Adds tools/check-test-tiers.ps1, which asks every test executable for its list and refuses a test whose ignore reason and module disagree: marked but not in the module, in the module but not marked, a path that contains the filter without a module of that name, a marker with no cost, and a misspelt marker. It runs in the all-features build + test job after cargo test. libtest writes an ignore reason into its JSON without escaping a backslash, so the administrative-share tests' reason is not valid JSON; the census reads such a line by its shape. tools/test-check-test-tiers.ps1 tests each rule in both directions, from recorded listings and against a fixture workspace, on PowerShell 7 and Windows PowerShell 5.1, and tools/sabotage-tiers.json re-injects each rule's defect (15 entries, all behaving as declared). DEVELOPMENT.md and the repository instructions state the convention. Completed item: M45.1: The tiers' convention and its census. A long-haul test is #[ignore = "long-haul: what it costs"] and sits in a module named long_haul; a probation test is #[ignore = "probation: what it costs"] in a module named probation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/copilot-instructions.md | 7 + .github/workflows/ci.yml | 19 ++ CHECKLIST.md | 7 +- COMPLETED-CHECKLIST.md | 27 +++ DEVELOPMENT.md | 33 ++++ tools/README-sabotage.md | 10 + tools/check-test-tiers.ps1 | 216 +++++++++++++++++++++ tools/sabotage-tiers-runner.cmd | 24 +++ tools/sabotage-tiers.json | 128 +++++++++++++ tools/test-check-test-tiers.ps1 | 320 ++++++++++++++++++++++++++++++++ 10 files changed, 785 insertions(+), 6 deletions(-) create mode 100644 tools/check-test-tiers.ps1 create mode 100644 tools/sabotage-tiers-runner.cmd create mode 100644 tools/sabotage-tiers.json create mode 100644 tools/test-check-test-tiers.ps1 diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 3daee057..3cee3300 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -2252,6 +2252,13 @@ The data does not have to be necessarily stable. A guideline might be that small a separate file or somehow encoded in source files. Larger data sets may be generated at run time, whether exhaustively or via random techniques. +### Test tiers + +A test too slow for every push is `#[ignore = "long-haul: "]` in a module named `long_haul`; one not yet +trusted is `#[ignore = "probation: "]` in a module named `probation`. A tier's run selects by the module +name and the reason names the tier, so the two must agree, and `tools/check-test-tiers.ps1` refuses a test where +they do not. The convention, and what an integration test does, is in [DEVELOPMENT.md](../DEVELOPMENT.md#test-tiers). + ## Architectural pre-steps **Never call `stdout`/`stderr`/`print`/`eprintln` (or the language equivalent) from diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fe56e1bd..c21e323a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -218,6 +218,16 @@ jobs: - name: Run test-sabotage-records.ps1 (Windows PowerShell 5.1) shell: powershell run: ./tools/test-sabotage-records.ps1 + # The check that a test's ignore reason and its module agree about its tier, driven + # both from recorded listings and against a real fixture workspace, because the + # JSON libtest writes is what it reads and is not always valid JSON. Both hosts: + # an ordinary script a developer may run under either. + - name: Run test-check-test-tiers.ps1 + shell: pwsh + run: ./tools/test-check-test-tiers.ps1 + - name: Run test-check-test-tiers.ps1 (Windows PowerShell 5.1) + shell: powershell + run: ./tools/test-check-test-tiers.ps1 # pwsh only, and deliberately not repeated under 5.1 like the step above: # check-encoding.ps1 carries curated mojibake with no BOM, so 5.1 decodes # it as ANSI and cannot parse it. The suite says so and refuses to run @@ -327,6 +337,15 @@ jobs: RUST_BACKTRACE: 1 RUST_LIB_BACKTRACE: 1 run: cargo test --workspace ${{ matrix.flags }} --locked --no-fail-fast + # A test's ignore reason and the module it sits in must agree about its tier, because + # the long-haul and probation runs select by module name: a test that disagrees is one + # nothing runs, and nothing else says so. It asks every test executable for its list, + # so it needs the all-features build above and nothing more. See DEVELOPMENT.md, + # "Test tiers". + - name: check test tiers + if: ${{ matrix.variant == 'all features' }} + shell: pwsh + run: ./tools/check-test-tiers.ps1 # The UNC current-directory test is ignored by default because it needs a # reachable administrative share (`\\localhost\C$`), which an ordinary # developer account does not have -- running it by default would report an diff --git a/CHECKLIST.md b/CHECKLIST.md index 69662ce4..defb99ed 100644 --- a/CHECKLIST.md +++ b/CHECKLIST.md @@ -613,12 +613,7 @@ to a long-haul tier that runs daily. Tests that have never run anywhere start on is [DESIGN-NOTES.md](DESIGN-NOTES.md#long-haul-tier); the measurements behind it are in [DESIGN-RATIONALE.md](DESIGN-RATIONALE.md#why-a-long-haul-tier). -- [ ] **M45.1** -- The tiers' convention and its census. A long-haul test is - `#[ignore = "long-haul: "]` and sits in a module named `long_haul`; a probation test is - `#[ignore = "probation: "]` in a module named `probation`, since libtest filters by name - and not by ignore reason. A census script fails when a reason and its module path disagree in either - direction, so a test cannot be marked into a tier that never selects it; it runs in CI on every push, - with tests of its own for both directions. DEVELOPMENT.md states the convention. +- [x] **M45.1** -- The tiers' convention and its census: `tools/check-test-tiers.ps1` refuses a test whose ignore reason and module disagree about its tier, in CI on every push. -> [completed 2026-11-03](COMPLETED-CHECKLIST.md#m451) - [ ] **M45.2** -- `long-haul.yml`: daily, and only when `main` has moved since the last completed run, plus manual dispatch. It runs `--ignored long_haul` across the workspace, and a failure there fails the diff --git a/COMPLETED-CHECKLIST.md b/COMPLETED-CHECKLIST.md index cd899a2d..03237dd0 100644 --- a/COMPLETED-CHECKLIST.md +++ b/COMPLETED-CHECKLIST.md @@ -4183,3 +4183,30 @@ crates' checklists; this is only the order and the links, and each item's own ch - [x] **M44.3** -- `windows-waitable-queues`: its doorbell's `duplicate` becomes `duplicate_handle`, and the crate depends on `win-handles-sys`. - [x] **M44.4** -- `win-job-launcher`: `duplicate_inheritable` becomes `duplicate_handle`, and the crate depends on `win-handles-sys`. + +## Moved 2026-11-03 18:41:21 -05:00 -- M45.1: the test tiers' convention and its census + +The decision is [DESIGN-NOTES.md](DESIGN-NOTES.md#long-haul-tier). + +### M45.1 -- The tiers' convention and its census: `tools/check-test-tiers.ps1` refuses a test whose ignore reason and module disagree about its tier, in CI on every push. *(completed 2026-11-03 18:41:21 -05:00)* + +- [x] **M45.1** -- The tiers' convention and its census. A long-haul test is + `#[ignore = "long-haul: "]` and sits in a module named `long_haul`; a probation test is + `#[ignore = "probation: "]` in a module named `probation`, since libtest filters by name + and not by ignore reason. A census script fails when a reason and its module path disagree in either + direction, so a test cannot be marked into a tier that never selects it; it runs in CI on every push, + with tests of its own for both directions. DEVELOPMENT.md states the convention. + +**Done.** [tools/check-test-tiers.ps1](tools/check-test-tiers.ps1) asks every test executable for its list +(`--list --format json`, which carries each test's ignore reason) and exits 0 consistent, 1 for a test that +breaks a rule, 2 when the census could not be taken: a `harness = false` target, a build that failed, or a +missing listing. It runs in the all-features `build + test` job after `cargo test`. +[tools/test-check-test-tiers.ps1](tools/test-check-test-tiers.ps1) tests each rule in both directions from +recorded listings, and against a fixture workspace whose real executable is asked, on PowerShell 7 and +Windows PowerShell 5.1; [tools/sabotage-tiers.json](tools/sabotage-tiers.json) re-injects each rule's +defect. The convention is in [DEVELOPMENT.md](DEVELOPMENT.md#test-tiers). + +libtest writes an ignore reason into its JSON without escaping a backslash, so the reason of the +administrative-share tests (`\\localhost\C$`) is not valid JSON and a strict parse refuses the line. The +census reads such a line by its shape. The branch that refuses a line that is neither is not reached by any +test, because no executable the suite can build writes one. diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 4b831f8e..90d62e5a 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -53,6 +53,39 @@ the directory override in `rustup`'s resolution order and forces those jobs onto the actual floating toolchain. The `msrv` job intentionally has no such override, since it is meant to run the pinned version. +## Test tiers + +A test too slow for every push, or not yet trusted, is `#[ignore]`d with a reason that names its tier and +sits in a module of the same name. The two spellings are what a run selects by, so they must agree: + +| Tier | Ignore reason starts with | Module | A run selects it with | +|---|---|---|---| +| long-haul | `long-haul:` | `long_haul` | `--ignored long_haul::` | +| probation | `probation:` | `probation` | `--ignored probation::` | + +```rust +mod long_haul { + #[test] + #[ignore = "long-haul: about 3s, a roster of 4096 files"] + fn the_whole_roster() { /* ... */ } +} +``` + +The text after the marker says what the test costs. Nothing reads it but a person deciding whether to run +the tier, so it is documentation, and it should say something. + +An integration test (a file under a crate's `tests/`) puts its tiered tests in a `mod long_haul` or +`mod probation` inside that file, the same as a unit test does. + +libtest filters by name and cannot filter by ignore reason, which is why the module is part of the +convention. [tools/check-test-tiers.ps1](tools/check-test-tiers.ps1) asks every test executable for its +list and refuses a test whose reason and module disagree: marked but not in the module (nothing would run +it), in the module but not marked (it would run on every push, or be swept into the tier for another +reason), a path that contains `long_haul::` without a module of that name, and a reason that reads like a +marker but is not spelled as one. CI runs it after the all-features `cargo test`, and +[tools/test-check-test-tiers.ps1](tools/test-check-test-tiers.ps1) tests it. The reasoning is in +[DESIGN-NOTES.md](DESIGN-NOTES.md#long-haul-tier). + ## Release process `release-please` owns version changes, tags, and changelog updates. Each crate diff --git a/tools/README-sabotage.md b/tools/README-sabotage.md index 3be5916e..2095e8c9 100644 --- a/tools/README-sabotage.md +++ b/tools/README-sabotage.md @@ -215,6 +215,16 @@ must stamp the file with the current time. The entry is caught. That is the argument for this file in one paragraph. A test suite cannot tell you what it fails to look at; only injecting the defect can. +[sabotage-tiers.json](sabotage-tiers.json) is the same arrangement for +[check-test-tiers.ps1](check-test-tiers.ps1), judged by +[test-check-test-tiers.ps1](test-check-test-tiers.ps1) through +[sabotage-tiers-runner.cmd](sabotage-tiers-runner.cmd): + +```powershell +.\tools\run-sabotage.ps1 -Manifest tools\sabotage-tiers.json ` + -CargoCommand (Resolve-Path tools\sabotage-tiers-runner.cmd).Path +``` + Runs on **Windows PowerShell 5.1 and PowerShell 7 alike**, like the other scripts in this directory, and the full sweep is verified on both. Worth stating because the two differ in ways that bite here specifically: 5.1 rejects diff --git a/tools/check-test-tiers.ps1 b/tools/check-test-tiers.ps1 new file mode 100644 index 00000000..0efe4ca2 --- /dev/null +++ b/tools/check-test-tiers.ps1 @@ -0,0 +1,216 @@ +# Copyright (c) Mike Grier. +# +# tools/check-test-tiers.ps1 -- a test's tier marker and the module it sits in agree. +# +# WHAT A TIER IS. DEVELOPMENT.md ("Test tiers") and DESIGN-NOTES.md ("long-haul-tier") say: a long-haul test is +# `#[ignore = "long-haul: "]` in a module named `long_haul`; a probation test is +# `#[ignore = "probation: "]` in a module named `probation`. The tier's own run selects by NAME, +# because libtest filters by name and cannot filter by ignore reason: `--ignored long_haul::` runs every +# ignored test whose path contains `long_haul::`. +# +# WHY A CENSUS. The marker and the path are two statements of one fact, and nothing makes them agree. A test +# marked into a tier its path does not select would be ignored everywhere and run by no one, which looks exactly +# like a test that passes; a test in the tier's module but not marked would run on every push, or, if ignored +# for another reason, be picked up by the tier's run for a reason that is not its own. So both directions are +# checked, with the rules in Get-TierViolation. +# +# THE CENSUS ASKS THE TEST BINARIES. It builds the workspace's test executables and has each one list its own +# tests, with their paths and ignore reasons, through libtest's JSON discovery. A scan of the source for +# `#[ignore]` and `mod long_haul` would be a proxy for what rustc and libtest decide -- macro-generated tests, +# `#[path]` modules and `cfg` all change the answer -- and a census of the tier's membership that disagreed +# with the tier's own run would be worse than none. Unlisted, and so unchecked: doctests, which the tiers do +# not apply to. +# +# The listing needs libtest's unstable `--format json`, which a test executable accepts under RUSTC_BOOTSTRAP=1 +# (the same arrangement as the compile_fail doctest check). That is set for the executables only. +# +# ./tools/check-test-tiers.ps1 # the workspace, all features (CI) +# ./tools/check-test-tiers.ps1 -ListingPath FILE # a recorded listing, for the tests +# ./tools/check-test-tiers.ps1 -RepositoryRoot DIR -CargoArgs @('--workspace') # a fixture workspace +# +# Exit: 0 consistent, 1 a test breaks a rule, 2 the census could not be taken. +# +# A listing file is one JSON object per line: {"binary":"","name":"","ignore":, +# "ignore_message":""}. tools/test-check-test-tiers.ps1 writes them. +[CmdletBinding()] +param( + [string] $RepositoryRoot, + [string] $ListingPath, + [string[]] $CargoArgs = @('--workspace', '--all-features', '--locked') +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +if (-not $RepositoryRoot) { $RepositoryRoot = Split-Path -Parent $PSScriptRoot } + +# The tiers, spelled once. `Marker` is what an ignore reason starts with, `Module` the module a tier's tests sit +# in, and `Pattern` what a reason that is MEANT as the marker looks like when it is not spelled right. The +# tier's run filters on `::`, and a test's path is `::::`, so the filter selects +# exactly the tests with that module segment and no other. +$tiers = @( + [pscustomobject]@{ Name = 'long-haul'; Marker = 'long-haul:'; Module = 'long_haul'; Pattern = '(?i)long[\s_-]*haul' } + [pscustomobject]@{ Name = 'probation'; Marker = 'probation:'; Module = 'probation'; Pattern = '(?i)probation' } +) + +# libtest writes an ignore reason into its JSON without escaping a backslash, so a reason that names a path +# (`\\localhost\C$`) is not valid JSON and a strict parser refuses the whole line. The fields read here are a +# test's path, whether it is ignored, and the reason's leading text, so a line the parser refuses is read by shape. +function ConvertFrom-LibtestLine { + param([string] $Line) + $parsed = $null + try { $parsed = $Line | ConvertFrom-Json -ErrorAction Stop } catch { $parsed = $null } + if ($null -ne $parsed) { + if ($parsed.type -ne 'test' -or $parsed.event -ne 'discovered') { return $null } + return [pscustomobject]@{ name = [string]$parsed.name; ignore = [bool]$parsed.ignore; ignore_message = [string]$parsed.ignore_message } + } + $shape = [regex]::Match($Line, '^\s*\{\s*"type":\s*"test",\s*"event":\s*"discovered",\s*"name":\s*"(?[^"]*)",\s*"ignore":\s*(?true|false)(?:,\s*"ignore_message":\s*"(?.*?)")?,\s*"source_path":') + if (-not $shape.Success) { + # No test reaches this: no executable the suite can build writes a line that is neither JSON nor this + # shape. It exists so a libtest that changes its format fails the census instead of being read as empty. + throw "A test listing line is neither JSON nor the shape libtest writes: $Line" + } + [pscustomobject]@{ name = $shape.Groups['name'].Value; ignore = ($shape.Groups['ignore'].Value -eq 'true'); ignore_message = $shape.Groups['message'].Value } +} + +function Get-TierViolation { + param([object[]] $Tests) + foreach ($test in $Tests) { + $reason = if ($test.ignore) { [string]$test.ignore_message } else { '' } + $segments = @($test.name -split '::') + $modules = if ($segments.Count -gt 1) { @($segments[0..($segments.Count - 2)]) } else { @() } + foreach ($tier in $tiers) { + $marked = $reason.StartsWith($tier.Marker, [System.StringComparison]::Ordinal) + $inModule = $modules -ccontains $tier.Module + $filter = $tier.Module + '::' + $selected = $test.name.Contains($filter) + $problem = $null + if ($marked -and -not $inModule) { + $problem = "is marked $($tier.Name) but is not in a module named $($tier.Module), so `"$filter`" never selects it and nothing runs it" + } + elseif ($marked -and $reason.Substring($tier.Marker.Length).Trim().Length -eq 0) { + $problem = "is marked $($tier.Name) but says nothing after `"$($tier.Marker)`": the reason is where what it costs is written" + } + elseif ($inModule -and -not $marked) { + $problem = if ($test.ignore) { + "is in a module named $($tier.Module) but ignored for another reason, so the $($tier.Name) run would pick it up for a reason that is not its own" + } + else { "is in a module named $($tier.Module) but is not ignored, so it runs on every push and not in the $($tier.Name) tier" } + } + elseif ($selected -and -not $inModule) { + $problem = "has `"$filter`" in its path without a module of that name, so the $($tier.Name) run would select it by accident" + } + elseif ($reason -match $tier.Pattern -and -not $marked) { + $problem = "has an ignore reason that reads like a $($tier.Name) marker but does not start with `"$($tier.Marker)`"" + } + if ($problem) { + [pscustomobject]@{ Binary = $test.binary; Name = $test.name; Reason = $reason; Problem = $problem } + } + } + } +} + +# The tests every executable lists, as records. Exits 2 when the census cannot be taken, because a census that +# silently saw fewer tests than exist would pass for the wrong reason. +function Get-TestListing { + # Every executable is asked for its list, and one that is not a libtest harness would instead run its own + # main with arguments it does not expect. None is today; one added later is a decision to make here, not a + # surprise to find in a CI run. + $custom = @(Get-ChildItem -LiteralPath $RepositoryRoot -Recurse -Filter Cargo.toml -File -ErrorAction SilentlyContinue | + Where-Object { $_.FullName -notmatch '[\\/](target|\.scratch|node_modules)[\\/]' } | + Where-Object { (Get-Content -LiteralPath $_.FullName -Raw) -match '(?m)^\s*harness\s*=\s*false' } | + ForEach-Object { $_.FullName }) + if ($custom.Count -gt 0) { + Write-Host "A target with harness = false is declared in: $($custom -join ', ')" + Write-Host 'This check asks each test executable for its list, which would run such a target instead; teach it about the target first.' + exit 2 + } + + # Function-local: on Windows PowerShell 5.1 cargo's progress on stderr is a terminating error under `Stop` + # even when it is discarded, and the exit code is what says whether the build worked. + $ErrorActionPreference = 'Continue' + Push-Location -LiteralPath $RepositoryRoot + try { + $built = @(& cargo test @CargoArgs --no-run --message-format=json 2>$null) + if ($LASTEXITCODE -ne 0) { + Write-Host "cargo could not build the tests (exit $LASTEXITCODE); run: cargo test $($CargoArgs -join ' ') --no-run" + exit 2 + } + } + finally { Pop-Location } + $executables = @($built | Where-Object { $_ -like '{*' } | ForEach-Object { + try { $_ | ConvertFrom-Json } catch { $null } + } | Where-Object { $null -ne $_ -and $_.reason -eq 'compiler-artifact' -and $_.executable -and $_.profile.test } | + ForEach-Object { [pscustomobject]@{ Name = $_.target.name; Path = $_.executable } } | + Sort-Object Path -Unique) + if ($executables.Count -eq 0) { + Write-Host 'cargo built no test executables, so there is nothing to ask.' + exit 2 + } + + foreach ($exe in $executables) { + $info = New-Object System.Diagnostics.ProcessStartInfo + $info.FileName = $exe.Path + $info.Arguments = '-Z unstable-options --format json --list' + $info.UseShellExecute = $false + $info.CreateNoWindow = $true + $info.RedirectStandardOutput = $true + $info.RedirectStandardError = $true + # The decoder is named, not the console's: a name or a reason need not be ASCII, and the console's code + # page is not always UTF-8 (see Get-GitBlobUtf8 in common.ps1). + $info.StandardOutputEncoding = New-Object System.Text.UTF8Encoding($false) + $info.EnvironmentVariables['RUSTC_BOOTSTRAP'] = '1' + $process = [System.Diagnostics.Process]::Start($info) + try { + $stderr = $process.StandardError.ReadToEndAsync() + $stdout = $process.StandardOutput.ReadToEndAsync() + if (-not $process.WaitForExit(120000)) { + try { $process.Kill() } catch { $null = $_ } + Write-Host "$($exe.Name) did not list its tests within 120 seconds." + exit 2 + } + $process.WaitForExit() + if ($process.ExitCode -ne 0) { + Write-Host "$($exe.Name) ($($exe.Path)) exited $($process.ExitCode) when asked to list its tests: $($stderr.Result.Trim())" + exit 2 + } + foreach ($line in ($stdout.Result -split "`r?`n")) { + if (-not $line.TrimStart().StartsWith('{')) { continue } + $event = ConvertFrom-LibtestLine -Line $line + if ($null -ne $event) { + [pscustomobject]@{ binary = $exe.Name; name = $event.name; ignore = $event.ignore; ignore_message = $event.ignore_message } + } + } + } + finally { $process.Dispose() } + } +} + +$tests = @(if ($ListingPath) { + if (-not (Test-Path -LiteralPath $ListingPath -PathType Leaf)) { Write-Host "No listing at $ListingPath."; exit 2 } + foreach ($line in [System.IO.File]::ReadAllLines($ListingPath)) { + if ($line.Trim().Length -eq 0) { continue } + $record = $line | ConvertFrom-Json + [pscustomobject]@{ binary = [string]$record.binary; name = [string]$record.name; ignore = [bool]$record.ignore; ignore_message = [string]$record.ignore_message } + } + } + else { Get-TestListing }) + +$violations = @(Get-TierViolation -Tests $tests | Sort-Object Binary, Name, Problem) +if ($violations.Count -gt 0) { + foreach ($v in $violations) { + Write-Host "$($v.Binary): $($v.Name)" + Write-Host " $($v.Problem)" + if ($v.Reason) { Write-Host " ignore reason: $($v.Reason)" } + } + Write-Host '' + Write-Host "$($violations.Count) test tier problem(s). The convention is in DEVELOPMENT.md, `"Test tiers`"." + exit 1 +} + +$summary = @($tiers | ForEach-Object { + $tier = $_ + "$($tier.Name) $(@($tests | Where-Object { $_.ignore -and $_.ignore_message.StartsWith($tier.Marker, [System.StringComparison]::Ordinal) }).Count)" + }) -join ', ' +Write-Host "Test tiers consistent: $($tests.Count) tests across $(@($tests | ForEach-Object { $_.binary } | Sort-Object -Unique).Count) executables; $summary." +exit 0 diff --git a/tools/sabotage-tiers-runner.cmd b/tools/sabotage-tiers-runner.cmd new file mode 100644 index 00000000..2e8e03c0 --- /dev/null +++ b/tools/sabotage-tiers-runner.cmd @@ -0,0 +1,24 @@ +@echo off +rem Copyright (c) Mike Grier. All rights reserved. +rem +rem The "cargo" for sabotage-tiers.json: makes test-check-test-tiers.ps1 look like +rem a cargo test run, so the sweep harness can judge the script that checks the +rem test tiers. The same arrangement as sabotage-records-runner.cmd, which is the +rem one to read for why it works. +rem +rem --no-run The build phase. There is nothing to compile, so it succeeds. +rem +rem --target-dir Meaningless here, and ignored. +rem +rem The harness runs this with the working directory set to its COPY of the tree, +rem so the suite invoked below is the copy's, and tests the copy's script -- the +rem one carrying the injected defect. + +echo %* | findstr /C:"--no-run" >nul && exit /b 0 + +rem PowerShell 7 where it exists, Windows PowerShell where it does not. +set "SABOTAGE_PS=pwsh" +where pwsh >nul 2>&1 || set "SABOTAGE_PS=powershell" + +%SABOTAGE_PS% -NoProfile -File "%CD%\tools\test-check-test-tiers.ps1" +exit /b %ERRORLEVEL% diff --git a/tools/sabotage-tiers.json b/tools/sabotage-tiers.json new file mode 100644 index 00000000..c867268e --- /dev/null +++ b/tools/sabotage-tiers.json @@ -0,0 +1,128 @@ +{ + "description": "Sabotages for tools/check-test-tiers.ps1, the check that a test's ignore reason and its module agree about which tier it is in, judged by test-check-test-tiers.ps1. Run with tools/run-sabotage.ps1 -Manifest tools/sabotage-tiers.json -CargoCommand tools/sabotage-tiers-runner.cmd; see tools/README-sabotage.md.", + "root": "..", + "package": "sabotage-tiers", + "testArgs": ["--suite"], + "sabotages": [ + { + "name": "a marked test outside its tier's module is accepted", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "The tier's run selects by the module name, so a test marked long-haul that sits anywhere else is one nothing runs: it is skipped on every push and never picked up by the run that exists for it, and the build stays green.", + "find": [" if ($marked -and -not $inModule) {"], + "replace": [" if ($false) {"] + }, + { + "name": "a marker with nothing after it is accepted", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "The text after the marker is where what a test costs is written. A bare marker passes the selection and leaves a reader of the ignore list nothing to judge it by.", + "find": ["$reason.Substring($tier.Marker.Length).Trim().Length -eq 0"], + "replace": ["$reason.Substring($tier.Marker.Length).Trim().Length -lt 0"] + }, + { + "name": "an unmarked test in a tier module is accepted", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "A test in the module that is not ignored runs on every push although it was meant for the tier; one ignored for another reason is picked up by the tier's run for a reason that is not its own. Both are the same unmarked test.", + "find": [" elseif ($inModule -and -not $marked) {"], + "replace": [" elseif ($false) {"] + }, + { + "name": "a path that only contains the filter is accepted", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "libtest's filter is a substring match, so `long_haul::` also selects `not_long_haul::t`. Without this check a test is swept into a tier by the spelling of an unrelated module.", + "find": [" elseif ($selected -and -not $inModule) {"], + "replace": [" elseif ($false) {"] + }, + { + "name": "a misspelt marker is accepted", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "`Long-haul:` or `long_haul:` is meant as the marker and is not one, so the test is ignored for a reason the tier does not recognise and nothing says so.", + "find": [" elseif ($reason -match $tier.Pattern -and -not $marked) {"], + "replace": [" elseif ($false) {"] + }, + { + "name": "the module name is matched without regard to case", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "libtest's filter is case-sensitive, so a module spelt `Long_Haul` is never selected by `long_haul::`. PowerShell's `-contains` ignores case and would call it in the module, which is the wrong answer exactly where the check exists to be right.", + "find": ["$inModule = $modules -ccontains $tier.Module"], + "replace": ["$inModule = $modules -contains $tier.Module"] + }, + { + "name": "the test's own name counts as one of its modules", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "The filter `long_haul::` selects a path with a segment of that name FOLLOWED by another, so a test that is itself called `long_haul` is not selected. Counting the last segment as a module accepts it.", + "find": ["@($segments[0..($segments.Count - 2)])"], + "replace": ["@($segments)"] + }, + { + "name": "the marker is matched without regard to case", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "The marker is spelt one way so that it can be searched for. Accepting `Long-haul:` makes the spelling a matter of who wrote the test, and it is the lookalike rule that would have said so.", + "find": ["$marked = $reason.StartsWith($tier.Marker, [System.StringComparison]::Ordinal)"], + "replace": ["$marked = $reason.StartsWith($tier.Marker, [System.StringComparison]::OrdinalIgnoreCase)"] + }, + { + "name": "a target with its own harness is ignored", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "The check runs every test executable with libtest's arguments; one with `harness = false` would run its own main instead. The refusal exists so that is a decision made here and not something found in a CI run.", + "find": [" if ($custom.Count -gt 0) {"], + "replace": [" if ($custom.Count -gt 99) {"] + }, + { + "name": "a build that failed is not noticed", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "A census taken over tests that did not build saw fewer tests than exist and would pass for the wrong reason. The exit code is what says the build worked.", + "find": [" if ($LASTEXITCODE -ne 0) {"], + "replace": [" if ($LASTEXITCODE -eq 99) {"] + }, + { + "name": "a test that breaks a rule exits zero", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "The exit code is the only thing CI reads.", + "find": [" exit 1"], + "replace": [" exit 0"] + }, + { + "name": "a missing listing exits zero", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "A census that could not be taken must not look like one that found nothing wrong.", + "find": ["Write-Host \"No listing at $ListingPath.\"; exit 2"], + "replace": ["Write-Host \"No listing at $ListingPath.\"; exit 0"] + }, + { + "name": "a line libtest wrote with an unescaped backslash reads every test as not ignored", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "libtest does not escape a backslash in an ignore reason, so a reason that names a path (the administrative-share tests have one) is not valid JSON and is read by shape. A reading that gets `ignore` wrong turns a marked test into an unmarked one in its module.", + "find": ["ignore = ($shape.Groups['ignore'].Value -eq 'true')"], + "replace": ["ignore = ($shape.Groups['ignore'].Value -eq 'false')"] + }, + { + "name": "a line libtest wrote with an unescaped backslash loses its reason", + "file": "tools/check-test-tiers.ps1", + "expect": "caught", + "why": "The same reading, losing the text: the marker is the start of the reason, so a reason read as empty is an unmarked test.", + "find": ["ignore_message = $shape.Groups['message'].Value }"], + "replace": ["ignore_message = '' }"] + }, + { + "name": "CONTROL: the problems are listed in another order", + "file": "tools/check-test-tiers.ps1", + "expect": "survives", + "why": "The manifest checking itself. This changes the order a reader sees problems in and nothing a test claims, so it MUST survive -- if it comes back caught, the suite is matching on incidental ordering rather than behaviour, and its other results are worth less than they look.", + "find": ["Sort-Object Binary, Name, Problem"], + "replace": ["Sort-Object Name, Binary, Problem"] + } + ] +} diff --git a/tools/test-check-test-tiers.ps1 b/tools/test-check-test-tiers.ps1 new file mode 100644 index 00000000..9ad958a2 --- /dev/null +++ b/tools/test-check-test-tiers.ps1 @@ -0,0 +1,320 @@ +# Copyright (c) Mike Grier. +<# +.SYNOPSIS + Tests for [check-test-tiers.ps1](check-test-tiers.ps1). + +.DESCRIPTION + The check guards the convention that lets a CI run select the tests that are too slow for every push + (`long_haul::`) or not yet trusted (`probation::`) by name, while the ignore reason says which tier a test + is in. A defect in it is quiet in the worst way: a test the run never selects is a test nothing runs, and + the build stays green. + + Cases are BIDIRECTIONAL. A check that flagged every ignored test would pass "an unmarked test in the module + is refused" and a check that flagged nothing would pass "an ordinary ignored test is accepted"; both are + pinned, as is each lookalike spelling against the near-miss names that must NOT be flagged. + + The rules are driven through a recorded listing (`-ListingPath`), which needs no build. The end-to-end cases + build a one-crate fixture workspace and ask its real test executable, because the JSON libtest writes is + what the check reads, and it is not always valid JSON (a reason that names a path leaves its backslashes + unescaped). + +.PARAMETER Name + Optional wildcard filter over case names, to re-run just one. +#> +[CmdletBinding()] +param( + [string] $Name = '*' +) + +$ErrorActionPreference = 'Stop' + +$checker = Join-Path $PSScriptRoot 'check-test-tiers.ps1' +if (-not (Test-Path -LiteralPath $checker)) { throw "$checker not found beside this script" } + +function Assert-True { + param([bool] $Condition, [string] $Message) + if (-not $Condition) { throw $Message } +} + +function Assert-Equal { + param($Expected, $Actual, [string] $Message) + if ($Expected -ne $Actual) { throw "$Message (expected '$Expected', got '$Actual')" } +} + +function Assert-Match { + param([string] $Pattern, [string] $Text, [string] $Message) + if ($Text -notmatch $Pattern) { throw "$Message (no match for /$Pattern/ in: $Text)" } +} + +function Assert-NoMatch { + param([string] $Pattern, [string] $Text, [string] $Message) + if ($Text -match $Pattern) { throw "$Message (unexpected /$Pattern/ in: $Text)" } +} + +$script:roots = New-Object System.Collections.Generic.List[string] + +function New-Root { + $path = Join-Path ([System.IO.Path]::GetTempPath()) ('test-tiers-' + [guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Force -Path $path | Out-Null + $script:roots.Add($path) | Out-Null + return $path +} + +function Remove-Roots { + foreach ($path in $script:roots) { Remove-Item -LiteralPath $path -Recurse -Force -ErrorAction SilentlyContinue } + $script:roots.Clear() +} + +# The messages are captured as strings and matched, so they must not wrap: on Windows PowerShell 5.1 +# `Out-String` wraps captured host output at the host's width, which on a CI runner is narrow. +$captureWidth = 4096 + +function Invoke-Checker { + param([hashtable] $Arguments) + $text = & $checker @Arguments 6>&1 | Out-String -Width $captureWidth + return [pscustomobject]@{ Exit = $LASTEXITCODE; Output = $text } +} + +# A recorded listing: each test is @(name, ignored, reason). +function Invoke-OnListing { + param([object[]] $Tests) + $path = Join-Path (New-Root) 'listing.ndjson' + $lines = foreach ($t in $Tests) { + ConvertTo-Json -Compress -InputObject ([ordered]@{ binary = 'fixture'; name = $t[0]; ignore = [bool]$t[1]; ignore_message = [string]$t[2] }) + } + [System.IO.File]::WriteAllText($path, (($lines -join "`n") + "`n")) + return Invoke-Checker @{ ListingPath = $path } +} + +function Test-Flagged { + param([object[]] $Tests, [string] $Pattern, [string] $Message) + $r = Invoke-OnListing $Tests + Assert-Equal 1 $r.Exit "$Message -- $($r.Output)" + Assert-Match $Pattern $r.Output $Message +} + +function Test-Accepted { + param([object[]] $Tests, [string] $Message) + $r = Invoke-OnListing $Tests + Assert-Equal 0 $r.Exit "$Message -- $($r.Output)" + Assert-Match 'Test tiers consistent' $r.Output $Message +} + +# A one-crate workspace whose test executable is real. `Source` is the crate's lib.rs. +function New-Fixture { + param([string] $Source, [string] $ExtraManifest = '') + $root = Join-Path (New-Root) 'fixture' + New-Item -ItemType Directory -Force -Path (Join-Path $root 'src') | Out-Null + [System.IO.File]::WriteAllText((Join-Path $root 'Cargo.toml'), "[package]`nname = `"tiers-fixture`"`nversion = `"0.0.0`"`nedition = `"2021`"`n$ExtraManifest") + [System.IO.File]::WriteAllText((Join-Path $root 'src\lib.rs'), $Source) + Push-Location -LiteralPath $root + try { & cargo generate-lockfile 2>&1 | Out-Null } finally { Pop-Location } + return $root +} + +$cases = @( + @{ + Name = 'ordinary-tests-and-an-ordinarily-ignored-test-are-accepted' + Run = { + Test-Accepted @( + @('a', $false, ''), + @('m::b', $false, ''), + @('m::slow_for_another_reason', $true, 'needs the administrative share, which a CI runner has'), + @('bench::timing', $true, 'run deliberately: a timing run') + ) 'nothing is tiered' + } + }, + @{ + Name = 'a-long-haul-test-in-its-module-with-a-cost-is-accepted-and-counted' + Run = { + $r = Invoke-OnListing @( + @('long_haul::big_roster', $true, 'long-haul: about 3s'), + @('outer::long_haul::nested::deep', $true, 'long-haul: about 9s')) + Assert-Equal 0 $r.Exit $r.Output + Assert-Match 'long-haul 2, probation 0' $r.Output 'the tier counts what is marked' + } + }, + @{ + Name = 'a-probation-test-in-its-module-with-a-cost-is-accepted-and-counted' + Run = { + $r = Invoke-OnListing @(, @('probation::stress', $true, 'probation: never run in CI yet')) + Assert-Equal 0 $r.Exit $r.Output + Assert-Match 'long-haul 0, probation 1' $r.Output 'the tier counts what is marked' + } + }, + @{ + Name = 'a-marked-test-outside-its-module-is-refused' + Run = { + Test-Flagged @(, @('tests::heavy', $true, 'long-haul: about 3s')) 'not in a module named long_haul' 'nothing would select it' + Test-Flagged @(, @('plain_test', $true, 'probation: new')) 'not in a module named probation' 'a bare test has no module at all' + } + }, + @{ + Name = 'the-module-must-be-an-exact-segment-of-the-parents-and-not-the-test-itself' + Run = { + Test-Flagged @(, @('outer::long_haul', $true, 'long-haul: x')) 'not in a module named long_haul' 'the test`s own name is not a module, and the filter `long_haul::` does not select it' + Test-Flagged @(, @('not_long_haul::t', $true, 'long-haul: x')) 'not in a module named long_haul' 'a longer module name is not the module' + } + }, + @{ + Name = 'the-module-name-is-matched-exactly-as-the-filter-matches-it' + Run = { + Test-Flagged @(, @('Long_Haul::t', $true, 'long-haul: x')) 'not in a module named long_haul' 'libtest`s filter is case-sensitive, so a differently cased module is never selected' + Test-Flagged @(, @('PROBATION::t', $true, 'probation: x')) 'not in a module named probation' 'and so is the other tier' + } + }, + @{ + Name = 'a-marker-with-no-cost-after-it-is-refused' + Run = { + Test-Flagged @(, @('long_haul::t', $true, 'long-haul:')) 'says nothing after' 'the bare marker' + Test-Flagged @(, @('probation::t', $true, 'probation: ')) 'says nothing after' 'the marker and whitespace' + } + }, + @{ + Name = 'an-unmarked-test-in-a-tier-module-is-refused-whether-or-not-it-is-ignored' + Run = { + Test-Flagged @(, @('long_haul::t', $true, 'needs an admin share')) 'ignored for another reason' 'ignored for a reason that is not its own' + Test-Flagged @(, @('long_haul::t', $false, '')) 'is not ignored' 'it would run on every push' + Test-Flagged @(, @('probation::t', $true, '')) 'ignored for another reason' 'ignored with no reason at all' + } + }, + @{ + Name = 'a-path-that-only-contains-the-filter-is-refused-as-an-accidental-selection' + Run = { + Test-Flagged @(, @('not_long_haul::t', $false, '')) 'would select it by accident' 'the filter matches a longer module name' + Test-Flagged @(, @('a_probation::t', $true, 'needs admin')) 'would select it by accident' 'and so does the other tier' + } + }, + @{ + Name = 'a-reason-that-reads-like-a-marker-but-is-not-spelled-as-one-is-refused' + Run = { + foreach ($reason in @('Long-haul: x', 'long haul: x', 'long_haul: x', 'longhaul: x', 'LONG-HAUL x', 'Probation: x', 'on probation')) { + Test-Flagged @(, @('some::test', $true, $reason)) 'reads like a' "the reason `"$reason`" is a misspelt marker" + } + } + }, + @{ + Name = 'names-that-merely-mention-a-tier-are-not-flagged' + Run = { + Test-Accepted @( + @('a_user_on_probation', $false, ''), + @('long_haul_is_documented', $false, ''), + @('m::probation_is_not_a_module_here', $false, ''), + @('m::t', $true, 'needs a long-lived server; run deliberately') + ) 'only an ignore reason or a module segment is a marker' + } + }, + @{ + Name = 'every-problem-is-reported-and-counted' + Run = { + $r = Invoke-OnListing @( + @('long_haul::a', $false, ''), + @('tests::b', $true, 'probation: x'), + @('fine', $false, '')) + Assert-Equal 1 $r.Exit $r.Output + Assert-Match '2 test tier problem' $r.Output 'both are counted' + Assert-Match 'long_haul::a' $r.Output 'the first is named' + Assert-Match 'tests::b' $r.Output 'the second is named' + Assert-NoMatch '\bfine\b' $r.Output 'a conforming test is not mentioned' + } + }, + @{ + Name = 'a-listing-that-is-missing-is-a-census-that-could-not-be-taken' + Run = { + $r = Invoke-Checker @{ ListingPath = (Join-Path (New-Root) 'absent.ndjson') } + Assert-Equal 2 $r.Exit $r.Output + Assert-Match 'No listing' $r.Output 'it says what is missing' + } + }, + @{ + Name = 'a-real-workspace-with-tiered-tests-is-accepted-including-a-reason-that-names-a-path' + Run = { + $root = New-Fixture @' +#[test] fn ordinary() {} +#[test] +#[ignore = r"needs the administrative share \\localhost\C$, which a CI runner has"] +fn names_a_path() {} +mod long_haul { + #[test] + #[ignore = "long-haul: about 3s"] + fn heavy() {} + #[test] + #[ignore = r"long-haul: reads \\localhost\C$, about 3s"] + fn heavy_and_names_a_path() {} +} +mod probation { + #[test] + #[ignore = "probation: never run in CI yet"] + fn untried() {} +} +'@ + $r = Invoke-Checker @{ RepositoryRoot = $root } + Assert-Equal 0 $r.Exit $r.Output + Assert-Match '5 tests' $r.Output 'it saw all five, including the two whose lines are not valid JSON' + Assert-Match 'long-haul 2, probation 1' $r.Output 'and read the marked one that names a path as marked, ignored and costed' + } + }, + @{ + Name = 'a-real-workspace-with-an-unmarked-test-in-a-tier-module-is-refused' + Run = { + $root = New-Fixture @' +mod long_haul { + #[test] + fn heavy_but_not_ignored() {} +} +'@ + $r = Invoke-Checker @{ RepositoryRoot = $root } + Assert-Equal 1 $r.Exit $r.Output + Assert-Match 'long_haul::heavy_but_not_ignored' $r.Output 'it names the test' + Assert-Match 'is not ignored' $r.Output 'and why' + } + }, + @{ + Name = 'a-target-with-a-custom-harness-is-a-census-that-could-not-be-taken' + Run = { + $root = New-Fixture '#[test] fn t() {}' "`n[[test]]`nname = `"custom`"`npath = `"src/lib.rs`"`nharness = false`n" + $r = Invoke-Checker @{ RepositoryRoot = $root } + Assert-Equal 2 $r.Exit $r.Output + Assert-Match 'harness = false' $r.Output 'it names the declaration' + } + }, + @{ + Name = 'tests-that-do-not-build-are-a-census-that-could-not-be-taken' + Run = { + $root = New-Fixture 'this is not rust' + $r = Invoke-Checker @{ RepositoryRoot = $root } + Assert-Equal 2 $r.Exit $r.Output + Assert-Match 'could not build' $r.Output 'it says the build failed' + } + } +) + +$selected = $cases | Where-Object { $_.Name -like $Name } +if (-not $selected) { + Write-Host "No cases matched '$Name'." -ForegroundColor Red + exit 1 +} + +$failures = 0 +foreach ($case in $selected) { + try { + & $case.Run + Write-Host " PASS $($case.Name)" -ForegroundColor Green + } + catch { + $failures++ + Write-Host " FAIL $($case.Name)" -ForegroundColor Red + Write-Host " $($_.Exception.Message)" -ForegroundColor Red + } + finally { + Remove-Roots + } +} + +Write-Host '' +if ($failures -gt 0) { + Write-Host "test-check-test-tiers: $failures of $($selected.Count) case(s) failed." -ForegroundColor Red + exit 1 +} +Write-Host "test-check-test-tiers: all $($selected.Count) case(s) passed." -ForegroundColor Green +exit 0 From e40d6d1e15ec34854d8f139f3ee282266a93fe34 Mon Sep 17 00:00:00 2001 From: Mike Grier Date: Tue, 3 Nov 2026 19:19:46 -0500 Subject: [PATCH 03/19] chore(tools): add the long-haul workflow and the planner that decides whether main has moved Adds .github/workflows/long-haul.yml: a plan job and a tiers job. The tiers job runs the long-haul tier and then the probation tier with --ignored and the tier's module filter, over the library, binary, integration and example test targets. A failure in the long-haul tier fails the run; the probation step is continue-on-error and its outcome goes in the run summary. tools/plan-long-haul.ps1 decides whether the schedule has anything to do: a dispatch always runs, and a schedule runs unless the most recent earlier run that completed was on the commit being run now. A cancelled run did not complete. When the earlier runs cannot be listed it runs, with a warning. tools/test-plan-long-haul.ps1 tests each rule in both directions on PowerShell 7 and Windows PowerShell 5.1, including the gh path through a fake gh and the arguments it receives, and runs in CI with the other harness suites. tools/sabotage-long-haul.json re-injects each rule's defect (16 entries, one a declared equivalent mutant). Windows PowerShell 5.1 returns a parsed JSON array as one object, which the planner now enumerates. The workflow has not run on a runner; the commands were run on the workspace and on a fixture crate. Completed item: M45.2: long-haul.yml: daily, and only when main has moved since the last completed run, plus manual dispatch. It runs the long-haul tier across the workspace and a failure there fails the run, then the probation tier, whose failures do not. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/ci.yml | 9 + .github/workflows/long-haul.yml | 102 ++++++++++ CHECKLIST.md | 6 +- COMPLETED-CHECKLIST.md | 35 ++++ DEVELOPMENT.md | 9 + tools/README-sabotage.md | 10 + tools/plan-long-haul.ps1 | 119 ++++++++++++ tools/sabotage-long-haul-runner.cmd | 24 +++ tools/sabotage-long-haul.json | 136 +++++++++++++ tools/test-plan-long-haul.ps1 | 284 ++++++++++++++++++++++++++++ 10 files changed, 730 insertions(+), 4 deletions(-) create mode 100644 .github/workflows/long-haul.yml create mode 100644 tools/plan-long-haul.ps1 create mode 100644 tools/sabotage-long-haul-runner.cmd create mode 100644 tools/sabotage-long-haul.json create mode 100644 tools/test-plan-long-haul.ps1 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c21e323a..07328206 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -228,6 +228,15 @@ jobs: - name: Run test-check-test-tiers.ps1 (Windows PowerShell 5.1) shell: powershell run: ./tools/test-check-test-tiers.ps1 + # The planner that decides whether the long-haul workflow runs: a defect in it is quiet + # in both directions, the tier never running while the schedule stays green or running + # daily on an unchanged main. Both hosts, as above. + - name: Run test-plan-long-haul.ps1 + shell: pwsh + run: ./tools/test-plan-long-haul.ps1 + - name: Run test-plan-long-haul.ps1 (Windows PowerShell 5.1) + shell: powershell + run: ./tools/test-plan-long-haul.ps1 # pwsh only, and deliberately not repeated under 5.1 like the step above: # check-encoding.ps1 carries curated mojibake with no BOM, so 5.1 decodes # it as ANSI and cannot parse it. The suite says so and refuses to run diff --git a/.github/workflows/long-haul.yml b/.github/workflows/long-haul.yml new file mode 100644 index 00000000..ff5fd12d --- /dev/null +++ b/.github/workflows/long-haul.yml @@ -0,0 +1,102 @@ +# Copyright (c) Mike Grier. +# +# The long-haul and probation test tiers. See DEVELOPMENT.md, "Test tiers", and +# DESIGN-NOTES.md, "long-haul-tier". +# +# A test whose scale is the point rather than the claim is `#[ignore = "long-haul: ..."]` +# in a module named `long_haul`, so ci.yml and every sabotage entry skip it and this +# workflow runs it. A test that has never run anywhere is the same with `probation`. +# libtest selects by name and not by ignore reason, which is why the module is part of +# the convention and why tools/check-test-tiers.ps1 holds the two in step. +# +# WHEN IT RUNS. Daily, and only when `main` has moved since the last run that +# completed: on an unchanged `main` a run would repeat the last one's result at the cost +# of a workspace build and every long test. tools/plan-long-haul.ps1 decides, and has +# tests of its own. A manual dispatch always runs, on the branch it is dispatched from. +# +# WHAT FAILS IT. The long-haul tier: a failure there fails the run, since those tests +# are the full-scale form of claims the push-time suites hold at a small size. The +# probation tier does not: its tests have no record yet, and a failure is data rather +# than a verdict. It runs after the long-haul tier, and whatever the long-haul tier did. +# +# WHY IT IS NOT PART OF THE SABOTAGE SWEEP. A sweep runs a crate's whole suite once per +# entry, so a tier that costs minutes would be paid for hundreds of times. A sweep never +# passes `--ignored` and so never pays for either tier. +name: long-haul + +on: + schedule: + # Off the hour, when scheduled runs are least delayed. + - cron: '47 8 * * *' + workflow_dispatch: + +permissions: + contents: read + +# One at a time, never cancelling: a run cancelled by the next one would count as not +# completed, and the commit it was on would be run again. +concurrency: + group: long-haul + cancel-in-progress: false + +jobs: + # Asks whether `main` has moved. Needs `actions: read` to list the earlier runs. + plan: + name: plan + runs-on: windows-latest + permissions: + contents: read + actions: read + outputs: + run: ${{ steps.plan.outputs.run }} + steps: + - uses: actions/checkout@v7 + - name: Decide whether main has moved + id: plan + shell: pwsh + env: + GH_TOKEN: ${{ github.token }} + GH_REPO: ${{ github.repository }} + EVENT_NAME: ${{ github.event_name }} + RUN_SHA: ${{ github.sha }} + RUN_ID: ${{ github.run_id }} + run: ./tools/plan-long-haul.ps1 -EventName "${env:EVENT_NAME}" -Sha "${env:RUN_SHA}" -CurrentRunId "${env:RUN_ID}" -GitHubOutput "${env:GITHUB_OUTPUT}" + + tiers: + name: long-haul and probation + needs: plan + if: ${{ needs.plan.outputs.run == 'true' }} + runs-on: windows-latest + # The long tier's tests cost seconds each, not hours; a run past this has stalled. + timeout-minutes: 120 + # See ci.yml: without this, rustup's directory-scoped toolchain file would win over + # the toolchain just installed. + env: + RUSTUP_TOOLCHAIN: stable + RUST_BACKTRACE: 1 + RUST_LIB_BACKTRACE: 1 + steps: + - uses: actions/checkout@v7 + - uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 + with: + key: long-haul + # Library, binary, integration and example test targets, as `cargo test` builds + # them, and not the doctests: a doctest cannot sit in a `long_haul` module. + # `--no-fail-fast` so every crate's tier reports rather than the first to fail. + - name: cargo test (long-haul tier) + run: 'cargo test --workspace --all-features --locked --no-fail-fast --lib --bins --tests --examples -- --ignored long_haul::' + # After the long-haul tier, and whether or not it passed, but never cancelled. + - name: cargo test (probation tier) + id: probation + if: ${{ !cancelled() }} + continue-on-error: true + run: 'cargo test --workspace --all-features --locked --no-fail-fast --lib --bins --tests --examples -- --ignored probation::' + # A failed probation step leaves the job green, so say what it did where it is read. + - name: Report the probation tier + if: ${{ !cancelled() }} + shell: pwsh + env: + PROBATION_OUTCOME: ${{ steps.probation.outcome }} + run: | + "probation tier: ${env:PROBATION_OUTCOME}" | Tee-Object -FilePath $env:GITHUB_STEP_SUMMARY -Append diff --git a/CHECKLIST.md b/CHECKLIST.md index defb99ed..531008bb 100644 --- a/CHECKLIST.md +++ b/CHECKLIST.md @@ -615,10 +615,8 @@ is [DESIGN-NOTES.md](DESIGN-NOTES.md#long-haul-tier); the measurements behind it - [x] **M45.1** -- The tiers' convention and its census: `tools/check-test-tiers.ps1` refuses a test whose ignore reason and module disagree about its tier, in CI on every push. -> [completed 2026-11-03](COMPLETED-CHECKLIST.md#m451) -- [ ] **M45.2** -- `long-haul.yml`: daily, and only when `main` has moved since the last completed run, - plus manual dispatch. It runs `--ignored long_haul` across the workspace, and a failure there fails the - run. Then it runs `--ignored probation`, whose failures do not fail the run. Separate from the - sabotage sweep, which never passes `--ignored` and so never pays for either tier. +- [x] **M45.2** -- `long-haul.yml`: daily when `main` has moved, plus dispatch; the long-haul tier fails the run, the probation tier does not. -> [completed 2026-11-03](COMPLETED-CHECKLIST.md#m452) + - [ ] **M45.3** -- Keep each probation test's results across runs, and report them. The report gives each probation test's runs, passes and current run of passes, which is what promoting it from `probation` diff --git a/COMPLETED-CHECKLIST.md b/COMPLETED-CHECKLIST.md index 03237dd0..bf4ada62 100644 --- a/COMPLETED-CHECKLIST.md +++ b/COMPLETED-CHECKLIST.md @@ -4210,3 +4210,38 @@ libtest writes an ignore reason into its JSON without escaping a backslash, so t administrative-share tests (`\\localhost\C$`) is not valid JSON and a strict parse refuses the line. The census reads such a line by its shape. The branch that refuses a line that is neither is not reached by any test, because no executable the suite can build writes one. + +## Moved 2026-11-03 19:18:51 -05:00 -- M45.2: the long-haul workflow + +The decision is [DESIGN-NOTES.md](DESIGN-NOTES.md#long-haul-tier). + +### M45.2 -- `long-haul.yml`: daily when `main` has moved, plus dispatch; the long-haul tier fails the run, the probation tier does not. *(completed 2026-11-03 19:18:51 -05:00)* + +- [x] **M45.2** -- `long-haul.yml`: daily, and only when `main` has moved since the last completed run, + plus manual dispatch. It runs `--ignored long_haul` across the workspace, and a failure there fails the + run. Then it runs `--ignored probation`, whose failures do not fail the run. Separate from the + sabotage sweep, which never passes `--ignored` and so never pays for either tier. + +**Done.** [.github/workflows/long-haul.yml](.github/workflows/long-haul.yml) has a `plan` job and a `tiers` job. +`plan` runs [tools/plan-long-haul.ps1](tools/plan-long-haul.ps1): a dispatch always runs, and a schedule runs unless +the most recent earlier run that completed (succeeded, failed or timed out; a cancelled run did not complete) was on +the commit being run now. When the earlier runs cannot be listed it runs, with a warning, since not being able to ask +is not evidence that nothing changed. `tiers` runs the long-haul tier and then the probation tier, which is +`continue-on-error` and reports its outcome in the run summary, after the long-haul tier whether or not that passed. +The commands select the library, binary, integration and example test targets and not the doctests, since a doctest +cannot sit in a `long_haul` module. + +[tools/test-plan-long-haul.ps1](tools/test-plan-long-haul.ps1) tests each rule in both directions on PowerShell 7 and +Windows PowerShell 5.1, in CI with the other harness suites, including the path that asks GitHub through a fake `gh` and +the arguments it was given. [tools/sabotage-long-haul.json](tools/sabotage-long-haul.json) re-injects each rule's defect. +Windows PowerShell 5.1 returns a parsed JSON array as one object, which the planner enumerates; the suite found that on +its first 5.1 run, and the real `gh` output would have hit it. + +One entry is a declared equivalent mutant: a dispatch never has the earlier runs loaded, so the decision's own check for +a non-schedule event gives the same answer as the rule for a schedule with no earlier run. The entry is expected to +survive and says it will come back caught if the loader's check is ever removed. + +The workflow itself has never run: nothing in the repository can run a GitHub schedule, so the first dispatch after it +reaches `main` is the first time its two jobs and the `gh` call run on a runner. The commands were run locally, on the +workspace, where nothing is tiered yet and every executable reports no tests selected, and on a fixture crate whose +long-haul tests were selected and whose other ignored tests were not. diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 90d62e5a..9db10211 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -86,6 +86,15 @@ marker but is not spelled as one. CI runs it after the all-features `cargo test` [tools/test-check-test-tiers.ps1](tools/test-check-test-tiers.ps1) tests it. The reasoning is in [DESIGN-NOTES.md](DESIGN-NOTES.md#long-haul-tier). +[.github/workflows/long-haul.yml](.github/workflows/long-haul.yml) runs the tiers: `cargo test --workspace +--all-features --locked --no-fail-fast --lib --bins --tests --examples -- --ignored long_haul::`, and then the +same with `probation::`. It runs daily when `main` has moved since the last run that completed, and on a manual +dispatch (from any branch, always). A failure in the long-haul tier fails the run. A failure in the probation +tier does not: that step is `continue-on-error`, and the run's summary says what it did. The decision whether +`main` has moved is [tools/plan-long-haul.ps1](tools/plan-long-haul.ps1), tested by +[tools/test-plan-long-haul.ps1](tools/test-plan-long-haul.ps1). To run a tier locally, pass the same arguments +for the crate in hand: `cargo test -p --all-features -- --ignored long_haul::`. + ## Release process `release-please` owns version changes, tags, and changelog updates. Each crate diff --git a/tools/README-sabotage.md b/tools/README-sabotage.md index 2095e8c9..c0a0f98c 100644 --- a/tools/README-sabotage.md +++ b/tools/README-sabotage.md @@ -225,6 +225,16 @@ you what it fails to look at; only injecting the defect can. -CargoCommand (Resolve-Path tools\sabotage-tiers-runner.cmd).Path ``` +[sabotage-long-haul.json](sabotage-long-haul.json) is the same for [plan-long-haul.ps1](plan-long-haul.ps1), the planner +behind the long-haul workflow, judged by [test-plan-long-haul.ps1](test-plan-long-haul.ps1) through +[sabotage-long-haul-runner.cmd](sabotage-long-haul-runner.cmd). It carries one declared equivalent mutant, an entry +that is expected to survive and says why. + +```powershell +.\tools\run-sabotage.ps1 -Manifest tools\sabotage-long-haul.json ` + -CargoCommand (Resolve-Path tools\sabotage-long-haul-runner.cmd).Path +``` + Runs on **Windows PowerShell 5.1 and PowerShell 7 alike**, like the other scripts in this directory, and the full sweep is verified on both. Worth stating because the two differ in ways that bite here specifically: 5.1 rejects diff --git a/tools/plan-long-haul.ps1 b/tools/plan-long-haul.ps1 new file mode 100644 index 00000000..0b002613 --- /dev/null +++ b/tools/plan-long-haul.ps1 @@ -0,0 +1,119 @@ +# Copyright (c) Mike Grier. +<# +.SYNOPSIS + Decides whether the long-haul workflow has anything to do. + +.DESCRIPTION + `.github/workflows/long-haul.yml` runs the long-haul and probation test tiers daily, and only when `main` has + moved since the last run that completed. A run on an unchanged `main` would repeat the last one's result at + the cost of a workspace build and every long test, so the schedule asks this first. + + The rule: + + - A run the engineer asked for (`workflow_dispatch`), or any trigger but the schedule, always runs. + - A scheduled run does not run when the most recent earlier run of the workflow that completed -- succeeded, + failed or timed out -- was on the commit being run now. It runs when there is no such run, or when that + run was on another commit. + - A cancelled run did not complete, and a run that was only the gate skipping its work has the commit it + skipped on, so it counts: the schedule would otherwise run again the next day on the same commit. + - When the earlier runs cannot be listed the answer is to run, with a warning. The tier exists to run, and + a failure to ask is not evidence that nothing changed. + + The runs come from `gh run list`, or from `-RunsPath` for a test: a JSON array of objects with + `databaseId`, `headSha`, `conclusion` and `createdAt`. + +.PARAMETER EventName + The triggering event, `github.event_name`. + +.PARAMETER Sha + The commit this run is on, `github.sha`. + +.PARAMETER CurrentRunId + This run's id, `github.run_id`, so it is never counted as an earlier one. + +.PARAMETER Workflow + The workflow file to list the earlier runs of. + +.PARAMETER Branch + The branch the earlier runs are listed for. + +.PARAMETER RunsPath + A JSON file of earlier runs to use instead of asking GitHub. + +.PARAMETER GitHubOutput + The Actions step-output file. Receives `run=true|false` and `reason=`. + +.EXAMPLE + ./tools/plan-long-haul.ps1 -EventName schedule -Sha $sha -CurrentRunId 1 -RunsPath runs.json +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)][string] $EventName, + [Parameter(Mandatory = $true)][string] $Sha, + [string] $CurrentRunId = '', + [string] $Workflow = 'long-haul.yml', + [string] $Branch = 'main', + [string] $RunsPath, + [string] $GitHubOutput +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +. (Join-Path $PSScriptRoot 'common.ps1') + +# What counts as having completed: it ran to a result. A cancelled run, a skipped one and a run that never +# started say nothing about the commit. +$completedConclusions = @('success', 'failure', 'timed_out') + +function Get-LongHaulPlan { + param([string] $EventName, [string] $Sha, [string] $CurrentRunId, [object[]] $Runs) + if ($EventName -ne 'schedule') { + return [pscustomobject]@{ Run = $true; Reason = "a $EventName run is always taken" } + } + $earlier = @($Runs | + Where-Object { "$($_.databaseId)" -ne $CurrentRunId -and $completedConclusions -contains [string]$_.conclusion } | + Sort-Object { [datetime]$_.createdAt } -Descending) + if ($earlier.Count -eq 0) { + return [pscustomobject]@{ Run = $true; Reason = 'no earlier run completed' } + } + $last = $earlier[0] + if ([string]$last.headSha -eq $Sha) { + return [pscustomobject]@{ Run = $false; Reason = "main has not moved since run $($last.databaseId), which completed ($($last.conclusion)) on $Sha" } + } + return [pscustomobject]@{ Run = $true; Reason = "main moved from $($last.headSha) (run $($last.databaseId)) to $Sha" } +} + +$runs = @() +$plan = $null +if ($EventName -eq 'schedule') { + $json = $null + if ($RunsPath) { + if (-not (Test-Path -LiteralPath $RunsPath -PathType Leaf)) { Write-Host "No runs file at $RunsPath."; exit 2 } + $json = [System.IO.File]::ReadAllText($RunsPath) + } + else { + $asked = Invoke-NativeSplit { gh run list --workflow $Workflow --branch $Branch --status completed --limit 30 --json databaseId,headSha,conclusion,createdAt } + if ($asked.ExitCode -ne 0) { + Write-Host "::warning::could not list earlier runs of $Workflow (gh exited $($asked.ExitCode): $($asked.Stderr)); running" + $plan = [pscustomobject]@{ Run = $true; Reason = 'the earlier runs could not be listed' } + } + else { $json = ($asked.Stdout -join "`n") } + } + # Windows PowerShell 5.1 hands back a parsed array as ONE object, so it is enumerated to get the runs. + if ($null -ne $json -and $json.Trim().Length -gt 0) { + $parsed = $json | ConvertFrom-Json + $runs = @($parsed | ForEach-Object { $_ } | Where-Object { $null -ne $_ }) + } +} + +if ($null -eq $plan) { + $plan = Get-LongHaulPlan -EventName $EventName -Sha $Sha -CurrentRunId $CurrentRunId -Runs $runs +} + +$verdict = if ($plan.Run) { 'run' } else { 'skip' } +Write-Host "long-haul: $verdict -- $($plan.Reason)" +if ($GitHubOutput) { + [System.IO.File]::AppendAllText($GitHubOutput, "run=$($plan.Run.ToString().ToLowerInvariant())`nreason=$($plan.Reason)`n") +} +exit 0 diff --git a/tools/sabotage-long-haul-runner.cmd b/tools/sabotage-long-haul-runner.cmd new file mode 100644 index 00000000..7d5aaca0 --- /dev/null +++ b/tools/sabotage-long-haul-runner.cmd @@ -0,0 +1,24 @@ +@echo off +rem Copyright (c) Mike Grier. All rights reserved. +rem +rem The "cargo" for sabotage-long-haul.json: makes test-plan-long-haul.ps1 look like +rem a cargo test run, so the sweep harness can judge the planner behind the long-haul +rem workflow. The same arrangement as sabotage-records-runner.cmd, which is the one +rem to read for why it works. +rem +rem --no-run The build phase. There is nothing to compile, so it succeeds. +rem +rem --target-dir Meaningless here, and ignored. +rem +rem The harness runs this with the working directory set to its COPY of the tree, +rem so the suite invoked below is the copy's, and tests the copy's script -- the +rem one carrying the injected defect. + +echo %* | findstr /C:"--no-run" >nul && exit /b 0 + +rem PowerShell 7 where it exists, Windows PowerShell where it does not. +set "SABOTAGE_PS=pwsh" +where pwsh >nul 2>&1 || set "SABOTAGE_PS=powershell" + +%SABOTAGE_PS% -NoProfile -File "%CD%\tools\test-plan-long-haul.ps1" +exit /b %ERRORLEVEL% diff --git a/tools/sabotage-long-haul.json b/tools/sabotage-long-haul.json new file mode 100644 index 00000000..442bf9fc --- /dev/null +++ b/tools/sabotage-long-haul.json @@ -0,0 +1,136 @@ +{ + "description": "Sabotages for tools/plan-long-haul.ps1, the planner that makes the long-haul workflow run daily and only when main has moved, judged by test-plan-long-haul.ps1. Run with tools/run-sabotage.ps1 -Manifest tools/sabotage-long-haul.json -CargoCommand tools/sabotage-long-haul-runner.cmd; see tools/README-sabotage.md.", + "root": "..", + "package": "sabotage-long-haul", + "testArgs": ["--suite"], + "sabotages": [ + { + "name": "a manual dispatch asks GitHub for the earlier runs too", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "A dispatch always runs, so the listing is of no use to it, and a failure to list would put a warning on a run it could not have changed.", + "find": ["if ($EventName -eq 'schedule') {"], + "replace": ["if ($true) {"] + }, + { + "name": "EQUIVALENT: a manual dispatch is held to the schedule's rule in the decision itself", + "file": "tools/plan-long-haul.ps1", + "expect": "survives", + "why": "An equivalent mutant, declared so that it is not looked for again. A dispatch never has the earlier runs loaded (the entry above is what guards that), so the decision's own check for a non-schedule event and the rule for a schedule with no earlier run give the same answer, run. If the loader's check is ever removed this becomes the only guard and the entry will come back caught, which is the signal to move it to `caught`.", + "find": [" if ($EventName -ne 'schedule') {"], + "replace": [" if ($false) {"] + }, + { + "name": "a cancelled run counts as completed", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "A cancelled run ran no tests. Counting it would skip the commit it was on, and the tier would not run on it until main moved again.", + "find": ["$completedConclusions = @('success', 'failure', 'timed_out')"], + "replace": ["$completedConclusions = @('success', 'failure', 'timed_out', 'cancelled')"] + }, + { + "name": "a timed-out run does not count as completed", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "A run that timed out reported a result. Without it the tier runs again each day on an unchanged main, to time out again.", + "find": ["$completedConclusions = @('success', 'failure', 'timed_out')"], + "replace": ["$completedConclusions = @('success', 'failure')"] + }, + { + "name": "a failed run does not count as completed", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "A failed run is a result and the commit has been run. Repeating it on an unchanged main is a daily rerun that says the same thing and spends a workspace build on it.", + "find": ["$completedConclusions = @('success', 'failure', 'timed_out')"], + "replace": ["$completedConclusions = @('success', 'timed_out')"] + }, + { + "name": "this run is counted among the earlier ones", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "The run being planned is on the commit being run, so a planner that counted it would find a run on this commit and skip, every time it could see itself.", + "find": ["Where-Object { \"$($_.databaseId)\" -ne $CurrentRunId -and $completedConclusions -contains [string]$_.conclusion }"], + "replace": ["Where-Object { $completedConclusions -contains [string]$_.conclusion }"] + }, + { + "name": "the oldest completed run decides, not the latest", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "The question is whether main has moved since the LAST run. Judged against the first one ever, the answer is yes from the second day on and the tier runs daily regardless.", + "find": ["Sort-Object { [datetime]$_.createdAt } -Descending)"], + "replace": ["Sort-Object { [datetime]$_.createdAt })"] + }, + { + "name": "any earlier run on this commit skips, not only the latest", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "A commit that ran, then was left behind, then came back as main's tip (a revert) has not been run since it came back. Matching any earlier run skips it though the last run was on something else.", + "find": [" if ([string]$last.headSha -eq $Sha) {"], + "replace": [" if (@($earlier | Where-Object { [string]$_.headSha -eq $Sha }).Count -gt 0) {"] + }, + { + "name": "there being no earlier run skips", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "The first run, and the first after the listing is empty, must run: a planner that skipped would never run the tier at all, and the schedule would stay green.", + "find": ["Run = $true; Reason = 'no earlier run completed'"], + "replace": ["Run = $false; Reason = 'no earlier run completed'"] + }, + { + "name": "main having moved skips", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "The case the schedule exists for.", + "find": [" return [pscustomobject]@{ Run = $true; Reason = \"main moved from"], + "replace": [" return [pscustomobject]@{ Run = $false; Reason = \"main moved from"] + }, + { + "name": "a listing that could not be had skips", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "Not being able to ask is not evidence that nothing changed. A planner that skipped on an API failure would stop the tier for as long as the failure lasted and say nothing but a warning.", + "find": ["$plan = [pscustomobject]@{ Run = $true; Reason = 'the earlier runs could not be listed' }"], + "replace": ["$plan = [pscustomobject]@{ Run = $false; Reason = 'the earlier runs could not be listed' }"] + }, + { + "name": "the earlier runs of every workflow are listed, not this one's", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "ci.yml runs on the same commits, so listing its runs finds one on this commit and skips. The tier would then run only when a commit reached main without a CI run, which is never.", + "find": ["gh run list --workflow $Workflow --branch $Branch"], + "replace": ["gh run list --branch $Branch"] + }, + { + "name": "the earlier runs of every branch are listed, not main's", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "A dispatch from a branch is a run of this workflow too, on that branch's commit. Taken as the last run, it would make main look as though it had moved when it had not.", + "find": ["gh run list --workflow $Workflow --branch $Branch"], + "replace": ["gh run list --workflow $Workflow"] + }, + { + "name": "the step output says run=true whatever was decided", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "The workflow reads only this file, so a decision printed to the log and not written is a decision that changes nothing.", + "find": ["\"run=$($plan.Run.ToString().ToLowerInvariant())`nreason=$($plan.Reason)`n\""], + "replace": ["\"run=true`nreason=$($plan.Reason)`n\""] + }, + { + "name": "a missing runs file exits zero", + "file": "tools/plan-long-haul.ps1", + "expect": "caught", + "why": "A planner that could not be given its input must not look like one that decided.", + "find": ["Write-Host \"No runs file at $RunsPath.\"; exit 2"], + "replace": ["Write-Host \"No runs file at $RunsPath.\"; exit 0"] + }, + { + "name": "CONTROL: the reason for the first run is reworded", + "file": "tools/plan-long-haul.ps1", + "expect": "survives", + "why": "The manifest checking itself. This changes what a reader sees and nothing a test claims, so it MUST survive -- if it comes back caught, the suite is matching on incidental wording rather than behaviour, and its other results are worth less than they look.", + "find": ["Reason = 'no earlier run completed'"], + "replace": ["Reason = 'there is no earlier run to compare with'"] + } + ] +} diff --git a/tools/test-plan-long-haul.ps1 b/tools/test-plan-long-haul.ps1 new file mode 100644 index 00000000..fba3b26a --- /dev/null +++ b/tools/test-plan-long-haul.ps1 @@ -0,0 +1,284 @@ +# Copyright (c) Mike Grier. +<# +.SYNOPSIS + Tests for [plan-long-haul.ps1](plan-long-haul.ps1). + +.DESCRIPTION + The planner is what makes the long-haul workflow run "daily, and only when main has moved". A defect in it + is quiet in both directions: one that always skips leaves the tier never running while the schedule stays + green, and one that always runs spends a workspace build and every long test each day to repeat a result. + + Cases are BIDIRECTIONAL: each rule that skips is paired with the neighbouring case that must run, so a + planner that did either unconditionally fails one of them. The earlier runs come from a recorded file, and + the path that asks GitHub is driven through a fake `gh` on PATH. + +.PARAMETER Name + Optional wildcard filter over case names, to re-run just one. +#> +[CmdletBinding()] +param( + [string] $Name = '*' +) + +$ErrorActionPreference = 'Stop' + +$planner = Join-Path $PSScriptRoot 'plan-long-haul.ps1' +if (-not (Test-Path -LiteralPath $planner)) { throw "$planner not found beside this script" } + +function Assert-True { + param([bool] $Condition, [string] $Message) + if (-not $Condition) { throw $Message } +} + +function Assert-Equal { + param($Expected, $Actual, [string] $Message) + if ($Expected -ne $Actual) { throw "$Message (expected '$Expected', got '$Actual')" } +} + +function Assert-Match { + param([string] $Pattern, [string] $Text, [string] $Message) + if ($Text -notmatch $Pattern) { throw "$Message (no match for /$Pattern/ in: $Text)" } +} + +$script:roots = New-Object System.Collections.Generic.List[string] + +function New-Root { + $path = Join-Path ([System.IO.Path]::GetTempPath()) ('long-haul-' + [guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Force -Path $path | Out-Null + $script:roots.Add($path) | Out-Null + return $path +} + +function Remove-Roots { + foreach ($path in $script:roots) { Remove-Item -LiteralPath $path -Recurse -Force -ErrorAction SilentlyContinue } + $script:roots.Clear() +} + +# Captured messages must not wrap: on Windows PowerShell 5.1 `Out-String` wraps captured host output at the +# host's width, which on a CI runner is narrow. +$captureWidth = 4096 + +$a = 'a' * 40 +$b = 'b' * 40 +$c = 'c' * 40 + +# A recorded run: @(id, sha, conclusion, hour). The hour orders them, since the planner must not rely on the +# order they are listed in. +function New-RunsFile { + param([object[]] $Runs) + $path = Join-Path (New-Root) 'runs.json' + $items = foreach ($r in $Runs) { + [ordered]@{ databaseId = $r[0]; headSha = $r[1]; conclusion = $r[2]; createdAt = ('2026-01-01T{0:00}:00:00Z' -f $r[3]) } + } + $json = if (@($items).Count -eq 0) { '[]' } else { ConvertTo-Json -InputObject @($items) } + [System.IO.File]::WriteAllText($path, $json) + return $path +} + +function Invoke-Planner { + param([string] $EventName, [string] $Sha, [string] $RunId = '100', [string] $RunsPath, [string] $Output) + $arguments = @{ EventName = $EventName; Sha = $Sha; CurrentRunId = $RunId } + if ($RunsPath) { $arguments.RunsPath = $RunsPath } + if ($Output) { $arguments.GitHubOutput = $Output } + $text = & $planner @arguments 6>&1 | Out-String -Width $captureWidth + return [pscustomobject]@{ Exit = $LASTEXITCODE; Output = $text } +} + +function Assert-Plan { + param([string] $Expected, $Result, [string] $Message) + Assert-Equal 0 $Result.Exit "$Message -- $($Result.Output)" + Assert-Match "long-haul: $Expected --" $Result.Output $Message +} + +# A `gh` that records its arguments and prints a file, or fails, standing in for the real one while `Run` +# executes. Run is given the directory the arguments are recorded in, as args.txt. +function Use-FakeGh { + param([string] $Prints, [int] $ExitCode, [scriptblock] $Run) + $dir = New-Root + $record = "@echo %* > `"%~dp0args.txt`"`r`n" + $body = if ($Prints) { "$record@type `"$Prints`"`r`n@exit /b $ExitCode`r`n" } else { "$record@echo gh: no luck 1>&2`r`n@exit /b $ExitCode`r`n" } + [System.IO.File]::WriteAllText((Join-Path $dir 'gh.cmd'), "@echo off`r`n$body") + $saved = $env:PATH + $env:PATH = "$dir;$saved" + try { & $Run $dir } finally { $env:PATH = $saved } +} + +$cases = @( + @{ + Name = 'a-dispatch-always-runs-even-on-a-commit-that-already-ran' + Run = { + $runs = New-RunsFile @(, @(1, $a, 'success', 1)) + Assert-Plan 'run' (Invoke-Planner -EventName workflow_dispatch -Sha $a -RunsPath $runs) 'the engineer asked' + Assert-Plan 'run' (Invoke-Planner -EventName push -Sha $a -RunsPath $runs) 'and so is any trigger but the schedule' + } + }, + @{ + Name = 'a-dispatch-does-not-ask-github-so-a-failing-gh-cannot-warn-about-it' + Run = { + Use-FakeGh -Prints '' -ExitCode 1 -Run { + param($dir) + $r = Invoke-Planner -EventName workflow_dispatch -Sha $a + Assert-Plan 'run' $r 'a dispatch runs' + Assert-True (-not ($r.Output -match '::warning::')) "a dispatch has no use for the listing, so its failure is not reported -- $($r.Output)" + Assert-True (-not (Test-Path -LiteralPath (Join-Path $dir 'args.txt'))) 'gh was not run at all' + } + } + }, + @{ + Name = 'a-schedule-with-no-earlier-run-runs' + Run = { + Assert-Plan 'run' (Invoke-Planner -EventName schedule -Sha $a -RunsPath (New-RunsFile @())) 'the first run ever' + } + }, + @{ + Name = 'a-schedule-on-the-commit-the-last-run-completed-on-skips' + Run = { + $runs = New-RunsFile @(, @(1, $a, 'success', 1)) + $r = Invoke-Planner -EventName schedule -Sha $a -RunsPath $runs + Assert-Plan 'skip' $r 'main has not moved' + Assert-Match 'run 1' $r.Output 'it names the run it matched' + } + }, + @{ + Name = 'a-schedule-after-main-moved-runs' + Run = { + $runs = New-RunsFile @(, @(1, $a, 'success', 1)) + $r = Invoke-Planner -EventName schedule -Sha $b -RunsPath $runs + Assert-Plan 'run' $r 'main moved' + Assert-Match 'main moved from' $r.Output 'it says so' + } + }, + @{ + Name = 'a-failed-or-timed-out-run-counts-as-completed' + Run = { + foreach ($conclusion in @('failure', 'timed_out', 'success')) { + $runs = New-RunsFile @(, @(1, $a, $conclusion, 1)) + Assert-Plan 'skip' (Invoke-Planner -EventName schedule -Sha $a -RunsPath $runs) "a $conclusion run on an unchanged main is not repeated" + } + } + }, + @{ + Name = 'a-cancelled-or-skipped-run-does-not-count-as-completed' + Run = { + foreach ($conclusion in @('cancelled', 'skipped', 'startup_failure', 'action_required', '')) { + $runs = New-RunsFile @(, @(1, $a, $conclusion, 1)) + Assert-Plan 'run' (Invoke-Planner -EventName schedule -Sha $a -RunsPath $runs) "a '$conclusion' run did not complete, so the commit has not been run" + } + } + }, + @{ + Name = 'a-cancelled-run-is-passed-over-for-the-completed-one-before-it' + Run = { + $runs = New-RunsFile @(@(1, $a, 'success', 1), @(2, $b, 'cancelled', 2)) + Assert-Plan 'skip' (Invoke-Planner -EventName schedule -Sha $a -RunsPath $runs) 'the last completed run is the one on a' + Assert-Plan 'run' (Invoke-Planner -EventName schedule -Sha $b -RunsPath $runs) 'the cancelled run on b ran nothing' + } + }, + @{ + Name = 'only-the-most-recent-completed-run-decides' + Run = { + # main went a, then b. A run on a is old news now, so a is not skipped just because it once ran. + $runs = New-RunsFile @(@(1, $a, 'success', 1), @(2, $b, 'success', 2)) + Assert-Plan 'skip' (Invoke-Planner -EventName schedule -Sha $b -RunsPath $runs) 'b is the last one run' + Assert-Plan 'run' (Invoke-Planner -EventName schedule -Sha $a -RunsPath $runs) 'a ran before b, and the last run was on b' + Assert-Plan 'run' (Invoke-Planner -EventName schedule -Sha $c -RunsPath $runs) 'c is new' + } + }, + @{ + Name = 'the-order-the-runs-are-listed-in-does-not-matter' + Run = { + $oldestFirst = New-RunsFile @(@(1, $a, 'success', 1), @(2, $b, 'success', 2)) + $newestFirst = New-RunsFile @(@(2, $b, 'success', 2), @(1, $a, 'success', 1)) + foreach ($file in @($oldestFirst, $newestFirst)) { + Assert-Plan 'skip' (Invoke-Planner -EventName schedule -Sha $b -RunsPath $file) 'b was last' + Assert-Plan 'run' (Invoke-Planner -EventName schedule -Sha $a -RunsPath $file) 'a was not' + } + } + }, + @{ + Name = 'this-run-is-never-counted-as-an-earlier-one' + Run = { + $runs = New-RunsFile @(, @(100, $a, 'success', 1)) + Assert-Plan 'run' (Invoke-Planner -EventName schedule -Sha $a -RunId 100 -RunsPath $runs) 'the only run listed is this one' + Assert-Plan 'skip' (Invoke-Planner -EventName schedule -Sha $a -RunId 101 -RunsPath $runs) 'the same listing, for a different run' + } + }, + @{ + Name = 'the-answer-is-written-to-the-step-output-file' + Run = { + $out = Join-Path (New-Root) 'output.txt' + $runs = New-RunsFile @(, @(1, $a, 'success', 1)) + Assert-Plan 'skip' (Invoke-Planner -EventName schedule -Sha $a -RunsPath $runs -Output $out) 'skipping' + $text = [System.IO.File]::ReadAllText($out) + Assert-Match '(?m)^run=false$' $text 'a skip is run=false' + Assert-Match '(?m)^reason=main has not moved' $text 'with its reason' + $out2 = Join-Path (New-Root) 'output.txt' + Assert-Plan 'run' (Invoke-Planner -EventName schedule -Sha $b -RunsPath $runs -Output $out2) 'running' + Assert-Match '(?m)^run=true$' ([System.IO.File]::ReadAllText($out2)) 'a run is run=true' + } + }, + @{ + Name = 'a-missing-runs-file-is-a-planner-that-could-not-be-asked' + Run = { + $r = Invoke-Planner -EventName schedule -Sha $a -RunsPath (Join-Path (New-Root) 'absent.json') + Assert-Equal 2 $r.Exit $r.Output + Assert-Match 'No runs file' $r.Output 'it says what is missing' + } + }, + @{ + Name = 'the-runs-come-from-gh-when-no-file-is-given' + Run = { + $runs = New-RunsFile @(, @(1, $a, 'success', 1)) + Use-FakeGh -Prints $runs -ExitCode 0 -Run { + param($dir) + Assert-Plan 'skip' (Invoke-Planner -EventName schedule -Sha $a) 'gh listed a completed run on this commit' + Assert-Plan 'run' (Invoke-Planner -EventName schedule -Sha $b) 'and one on another' + $asked = [System.IO.File]::ReadAllText((Join-Path $dir 'args.txt')) + Assert-Match '\brun list\b' $asked 'it lists runs' + Assert-Match '--workflow long-haul\.yml\b' $asked 'of this workflow and no other, whose commits would match for the wrong reason' + Assert-Match '--branch main\b' $asked 'on main, which is the branch whose movement is asked about' + Assert-Match '--status completed\b' $asked 'that have completed' + } + } + }, + @{ + Name = 'a-gh-that-fails-runs-with-a-warning-rather-than-skipping' + Run = { + Use-FakeGh -Prints '' -ExitCode 1 -Run { + $r = Invoke-Planner -EventName schedule -Sha $a + Assert-Plan 'run' $r 'not being able to ask is not evidence that nothing changed' + Assert-Match '::warning::could not list earlier runs' $r.Output 'and it says it could not' + } + } + } +) + +$selected = $cases | Where-Object { $_.Name -like $Name } +if (-not $selected) { + Write-Host "No cases matched '$Name'." -ForegroundColor Red + exit 1 +} + +$failures = 0 +foreach ($case in $selected) { + try { + & $case.Run + Write-Host " PASS $($case.Name)" -ForegroundColor Green + } + catch { + $failures++ + Write-Host " FAIL $($case.Name)" -ForegroundColor Red + Write-Host " $($_.Exception.Message)" -ForegroundColor Red + } + finally { + Remove-Roots + } +} + +Write-Host '' +if ($failures -gt 0) { + Write-Host "test-plan-long-haul: $failures of $($selected.Count) case(s) failed." -ForegroundColor Red + exit 1 +} +Write-Host "test-plan-long-haul: all $($selected.Count) case(s) passed." -ForegroundColor Green +exit 0 From 42636ac7538c919e40553996ac39f5fdad6d6f9f Mon Sep 17 00:00:00 2001 From: Mike Grier Date: Tue, 3 Nov 2026 21:37:53 -0500 Subject: [PATCH 04/19] chore(tools): keep the probation tier's results across runs on a branch and report them Adds tools/run-tier.ps1, which owns the command that selects a tier and reads each test's outcome from libtest's JSON events, keyed by the source path of the target its executable was built from. A test that started and never reported (its executable aborted) is a failure. long-haul.yml now runs both tiers through it. Adds tools/publish-probation-results.ps1, which applies a run to results.json on a probation-results branch that shares no history with main: once per run id, as a fast-forward, never forced, retried from a fresh fetch. A record job holding the only write access publishes it from the artifact the read-only tests job uploads, and tools/report-probation.ps1 prints each test's runs, passes, current run of passes and latest result, with a candidate threshold that is a working position. tools/test-probation.ps1 tests the parse, the record, the report, the publisher against a bare remote and the runner against real workspaces on PowerShell 7 and Windows PowerShell 5.1, and runs in CI. tools/sabotage-probation.json re-injects 31 defects and a control, all behaving as declared. The record keeps tests in an ordinal dictionary and a list, since PowerShell's ordered dictionaries and object properties compare keys without regard to case. The workflow has not run on a runner. The publisher is a copy of the sabotage record's on purpose; merging them is parked as M45+.2. Completed item: M45.3: Keep each probation test's results across runs, and report them. Where the results live is decided and the promotion threshold stays a working position until the first results exist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/copilot-instructions.md | 2 +- .github/workflows/ci.yml | 11 + .github/workflows/long-haul.yml | 74 ++- CHECKLIST.md | 8 +- COMPLETED-CHECKLIST.md | 41 ++ DESIGN-NOTES.md | 49 +- DESIGN-RATIONALE.md | 23 + DEVELOPMENT.md | 23 +- PARKED.md | 13 + tools/README-sabotage.md | 11 + tools/probation-common.ps1 | 304 ++++++++++++ tools/publish-probation-results.ps1 | 203 ++++++++ tools/report-probation.ps1 | 70 +++ tools/run-tier.ps1 | 114 +++++ tools/sabotage-probation-runner.cmd | 24 + tools/sabotage-probation.json | 394 ++++++++++++++++ tools/test-probation.ps1 | 689 ++++++++++++++++++++++++++++ 17 files changed, 2024 insertions(+), 29 deletions(-) create mode 100644 tools/probation-common.ps1 create mode 100644 tools/publish-probation-results.ps1 create mode 100644 tools/report-probation.ps1 create mode 100644 tools/run-tier.ps1 create mode 100644 tools/sabotage-probation-runner.cmd create mode 100644 tools/sabotage-probation.json create mode 100644 tools/test-probation.ps1 diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 3cee3300..e864099b 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -2257,7 +2257,7 @@ via random techniques. A test too slow for every push is `#[ignore = "long-haul: "]` in a module named `long_haul`; one not yet trusted is `#[ignore = "probation: "]` in a module named `probation`. A tier's run selects by the module name and the reason names the tier, so the two must agree, and `tools/check-test-tiers.ps1` refuses a test where -they do not. The convention, and what an integration test does, is in [DEVELOPMENT.md](../DEVELOPMENT.md#test-tiers). +they do not. The convention, what an integration test does, how the tiers run and where probation results are kept are in [DEVELOPMENT.md](../DEVELOPMENT.md#test-tiers). ## Architectural pre-steps diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 07328206..79c0269c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -237,6 +237,17 @@ jobs: - name: Run test-plan-long-haul.ps1 (Windows PowerShell 5.1) shell: powershell run: ./tools/test-plan-long-haul.ps1 + # The scripts that run a tier, keep what each probation test did on a branch, and report + # it. A defect in them is quiet: a test counted as passing that did not, a run counted + # twice, or a test that killed its process counted as never having run. The publisher + # runs against a bare repository, and the runner against real one-crate workspaces. Both + # hosts, as above. + - name: Run test-probation.ps1 + shell: pwsh + run: ./tools/test-probation.ps1 + - name: Run test-probation.ps1 (Windows PowerShell 5.1) + shell: powershell + run: ./tools/test-probation.ps1 # pwsh only, and deliberately not repeated under 5.1 like the step above: # check-encoding.ps1 carries curated mojibake with no BOM, so 5.1 decodes # it as ANSI and cannot parse it. The suite says so and refuses to run diff --git a/.github/workflows/long-haul.yml b/.github/workflows/long-haul.yml index ff5fd12d..8d729795 100644 --- a/.github/workflows/long-haul.yml +++ b/.github/workflows/long-haul.yml @@ -18,6 +18,8 @@ # are the full-scale form of claims the push-time suites hold at a small size. The # probation tier does not: its tests have no record yet, and a failure is data rather # than a verdict. It runs after the long-haul tier, and whatever the long-haul tier did. +# Its results are kept across runs on the probation-results branch, by the last job, and +# reported in the run's summary (DESIGN-NOTES.md, "probation-results"). # # WHY IT IS NOT PART OF THE SABOTAGE SWEEP. A sweep runs a crate's whole suite once per # entry, so a tier that costs minutes would be paid for hundreds of times. A sweep never @@ -81,22 +83,70 @@ jobs: - uses: Swatinem/rust-cache@v2 with: key: long-haul - # Library, binary, integration and example test targets, as `cargo test` builds - # them, and not the doctests: a doctest cannot sit in a `long_haul` module. - # `--no-fail-fast` so every crate's tier reports rather than the first to fail. - - name: cargo test (long-haul tier) - run: 'cargo test --workspace --all-features --locked --no-fail-fast --lib --bins --tests --examples -- --ignored long_haul::' + # tools/run-tier.ps1 owns the command that selects a tier: the library, binary, + # integration and example test targets with `--ignored` and the tier's module + # filter, and not the doctests, since a doctest cannot sit in a `long_haul` + # module. `--no-fail-fast` so every crate's tier reports rather than the first + # to fail. It writes what each test did, and exits with cargo's own code. + - name: Run the long-haul tier + shell: pwsh + run: ./tools/run-tier.ps1 -Module long_haul -OutputDirectory .scratch/tiers/long_haul # After the long-haul tier, and whether or not it passed, but never cancelled. - - name: cargo test (probation tier) + - name: Run the probation tier id: probation if: ${{ !cancelled() }} continue-on-error: true - run: 'cargo test --workspace --all-features --locked --no-fail-fast --lib --bins --tests --examples -- --ignored probation::' - # A failed probation step leaves the job green, so say what it did where it is read. - - name: Report the probation tier + shell: pwsh + run: ./tools/run-tier.ps1 -Module probation -OutputDirectory .scratch/tiers/probation + # What each probation test did, for the job below that keeps it. Passed on as an + # artifact so that the job holding write access never runs the tests, which run + # here with read-only access. + - name: Upload what each probation test did if: ${{ !cancelled() }} + uses: actions/upload-artifact@v7 + with: + name: probation-outcomes + path: .scratch/tiers/probation/outcomes.json + if-no-files-found: warn + retention-days: 1 + overwrite: true + + # Keeps what each probation test did, across runs, on the `probation-results` branch, + # and reports it (DESIGN-NOTES.md, "probation-results"). The only job here that writes + # to the repository, and it writes one branch, which holds nothing but this record. It + # runs the scripts of the checkout this run is for and none of the tests, which ran in + # the job above with read-only access. + # + # Not fatal, as the sweep's equivalent is not: the record only ever gains or misses a + # run, and a red run for a publish that failed would be read as a test that did. + record: + name: keep the probation results + needs: [plan, tiers] + if: ${{ always() && needs.plan.outputs.run == 'true' }} + runs-on: windows-latest + permissions: + contents: write + steps: + - uses: actions/checkout@v7 + - name: Download what each probation test did + continue-on-error: true + uses: actions/download-artifact@v8 + with: + name: probation-outcomes + path: .scratch/probation + # The id of the run and its attempt: a re-run of the jobs is another run of the + # tests, and applying the same id twice is applying it once. + - name: Publish the results to the probation-results branch + if: ${{ hashFiles('.scratch/probation/outcomes.json') != '' }} + continue-on-error: true shell: pwsh env: - PROBATION_OUTCOME: ${{ steps.probation.outcome }} - run: | - "probation tier: ${env:PROBATION_OUTCOME}" | Tee-Object -FilePath $env:GITHUB_STEP_SUMMARY -Append + RUN_ID: ${{ github.run_id }}.${{ github.run_attempt }} + RUN_SHA: ${{ github.sha }} + run: ./tools/publish-probation-results.ps1 -Outcomes .scratch/probation/outcomes.json -RunId "${env:RUN_ID}" -Commit "${env:RUN_SHA}" + # Read back from the branch, so what is reported is what is kept, and so that a + # run that could not publish still reports what the branch holds. + - name: Report the probation tier + continue-on-error: true + shell: pwsh + run: ./tools/report-probation.ps1 -SummaryPath "${env:GITHUB_STEP_SUMMARY}" diff --git a/CHECKLIST.md b/CHECKLIST.md index 531008bb..b7ed40d6 100644 --- a/CHECKLIST.md +++ b/CHECKLIST.md @@ -618,11 +618,7 @@ is [DESIGN-NOTES.md](DESIGN-NOTES.md#long-haul-tier); the measurements behind it - [x] **M45.2** -- `long-haul.yml`: daily when `main` has moved, plus dispatch; the long-haul tier fails the run, the probation tier does not. -> [completed 2026-11-03](COMPLETED-CHECKLIST.md#m452) -- [ ] **M45.3** -- Keep each probation test's results across runs, and report them. The report gives each - probation test's runs, passes and current run of passes, which is what promoting it from `probation` - to `long_haul` -- a code change -- is decided on. Where the results live is this item's to decide - (artifacts expire; the sabotage record's branch is the pattern that does not), and so is the - promotion threshold, which is a working position until the first results exist. +- [x] **M45.3** -- Probation results are kept on the `probation-results` branch and reported; the promotion threshold stays a working position. -> [completed 2026-11-03](COMPLETED-CHECKLIST.md#m453) - [ ] **M45.4** -- Put the never-run tests on probation: `windows-ioring-sys`' five flush-barrier stress tests ("saturates a disk with background writers"; "writes tens of MiB per trial") and @@ -653,4 +649,4 @@ is [DESIGN-NOTES.md](DESIGN-NOTES.md#long-haul-tier); the measurements behind it `windows-file-watcher`'s scenario stress and `windows-ioring-sys`' properties under every resolution, by re-measuring the workspace once M45.5 to M45.9 are in. - > **-> UNBLOCKS PARKED WORK:** completing this unblocks `M45+` in [PARKED.md](PARKED.md#m45-plus). + > **-> UNBLOCKS PARKED WORK:** completing this unblocks `M45+` in [PARKED.md](PARKED.md#m45-plus), and the publishers' merge-or-delete in [PARKED.md](PARKED.md#m45-plus-publishers). diff --git a/COMPLETED-CHECKLIST.md b/COMPLETED-CHECKLIST.md index bf4ada62..31f03327 100644 --- a/COMPLETED-CHECKLIST.md +++ b/COMPLETED-CHECKLIST.md @@ -4245,3 +4245,44 @@ The workflow itself has never run: nothing in the repository can run a GitHub sc reaches `main` is the first time its two jobs and the `gh` call run on a runner. The commands were run locally, on the workspace, where nothing is tiered yet and every executable reports no tests selected, and on a fixture crate whose long-haul tests were selected and whose other ignored tests were not. + +## Moved 2026-11-03 21:35:43 -05:00 -- M45.3: the probation tier's results + +The decision is [DESIGN-NOTES.md](DESIGN-NOTES.md#probation-results). + +### M45.3 -- Probation results are kept on the `probation-results` branch and reported; the promotion threshold stays a working position. *(completed 2026-11-03 21:35:43 -05:00)* + +- [x] **M45.3** -- Keep each probation test's results across runs, and report them. The report gives each + probation test's runs, passes and current run of passes, which is what promoting it from `probation` + to `long_haul` -- a code change -- is decided on. Where the results live is this item's to decide + (artifacts expire; the sabotage record's branch is the pattern that does not), and so is the + promotion threshold, which is a working position until the first results exist. + +**Done.** Where the results live is decided: a branch of their own, `probation-results`, holding one generated +`results.json`, written by a `record` job in [long-haul.yml](.github/workflows/long-haul.yml) and read by +[report-probation.ps1](tools/report-probation.ps1). The tests run in a job with read-only access and hand what each +did to the `record` job as an artifact, so the job that can write never runs a test. +[run-tier.ps1](tools/run-tier.ps1) now owns the command that selects a tier, and both of the workflow's tiers go through +it; it reads each test's outcome from libtest's JSON events, keys it by the source path of the target its executable was +built from, and counts a test that started and never reported (its executable aborted) as a failure. +[publish-probation-results.ps1](tools/publish-probation-results.ps1) applies a run once, whatever the number of retries, +and never forces. The reasoning is in [DESIGN-NOTES.md](DESIGN-NOTES.md#probation-results) and +[DESIGN-RATIONALE.md](DESIGN-RATIONALE.md#how-probation-results-are-kept). + +The promotion threshold is **not decided**: the report takes it as `-Threshold` and labels its default a working +position, because there is no result yet to suggest a number. It is the one part of this item left open on purpose, and +nothing is scheduled for it: the first results are what settle it. + +[tools/test-probation.ps1](tools/test-probation.ps1) tests the parse, the record, the report, the publisher against a bare +repository (including a race and a branch whose name merely ends the same) and the runner against real one-crate +workspaces, one of which has an executable that aborts, on PowerShell 7 and Windows PowerShell 5.1, in CI with the other +harness suites. [tools/sabotage-probation.json](tools/sabotage-probation.json) re-injects 31 defects and a control, and all +behave as declared. Its sweep found two things: a surviving entry showed that `[ordered]@{}` compares keys without regard to +case, so the record would have merged two tests that differ only by case (it is now a list on disk and an ordinal dictionary +in memory); and `RUSTC_BOOTSTRAP=0` still turns on libtest's unstable options, because libtest only checks that the +variable is set, so the entry that was meant to switch them off removes the variable instead. + +The workflow has not run on a runner, so the first dispatch after it reaches `main` is the first time the `record` job, its +artifact hand-off and the push to the new branch run for real. Nothing is on probation yet (`M45.4` puts the first tests +there), so that run will publish nothing and report an empty record. The `record` job's publisher is a copy of the sabotage +record's, on purpose; the merge-or-delete decision is parked as `M45+.2`. diff --git a/DESIGN-NOTES.md b/DESIGN-NOTES.md index f3d6d9fc..e0d4b92f 100644 --- a/DESIGN-NOTES.md +++ b/DESIGN-NOTES.md @@ -3052,11 +3052,56 @@ test's cost is paid on each; a test whose claim holds at a small size is written tests which don't fail the run but we gather data on whether they succeed or fail and we can promote them to non-probationary once they succeed for enough runs"). It runs after the long-haul tier, its failures do not fail the run, and its results are kept across runs. Promotion is a code change, moving - the test to `long_haul`. **Working positions, not decisions:** how many passes are enough, and where the - results are kept; both are `M45.3`'s, decided once there are results to look at. + the test to `long_haul`. Where the results are kept is [probation-results](#probation-results). + **A working position, not a decision:** how many passes are enough, which is `M45.3`'s to settle once + there are results to look at. The work is [CHECKLIST.md](CHECKLIST.md) `M45`. What prompted it, and the measurements, are in [DESIGN-RATIONALE.md](DESIGN-RATIONALE.md#why-a-long-haul-tier). +**Each probation test's results are kept across runs on a branch of their own, `probation-results`, and reported; how many passes are enough is a working position.** +`M45.3` left this to its own work, having said only that artifacts expire and that the sabotage record's +branch is the pattern that does not. The record, `results.json`, is derived data in the same sense as the +[sabotage record](#sabotage-records), so it has the same home and the same reasons: a branch that shares no +history with `main`, written by the `record` job of +[long-haul.yml](.github/workflows/long-haul.yml) with +[publish-probation-results.ps1](tools/publish-probation-results.ps1), and read by +[report-probation.ps1](tools/report-probation.ps1), which prints each test's runs, passes, current run of +passes and latest result. The tests run in a job with read-only access and hand what each did to the `record` +job as an artifact, so the job that can write never runs a test. + +- **What a result is.** [run-tier.ps1](tools/run-tier.ps1) runs a tier through libtest's JSON events, which are + unstable and are asked for as [check-test-tiers.ps1](tools/check-test-tiers.ps1) asks for them, and reads + what each test did from them and not from text meant for a person. A test is keyed by the source path of the + target its executable was built from, relative to the workspace, and its path inside it: cargo's own + `Running` header names a path relative to the package, which two crates share, so the executable is mapped + to its target from the build's JSON. An executable that cargo ran and the build did not produce is an error. + cargo's output and the events are one file by one redirect, because the header says which executable the + events after it belong to and two pipes read on two threads do not keep that order. +- **A test's key is case-sensitive.** Two Rust tests can differ only by case, and PowerShell's ordered + dictionaries and object properties both treat those as one, which would merge two tests' counts. The record + is a list on disk and an ordinal-keyed dictionary in memory for that reason. +- **A test that started and was never reported is a failure.** The executable stopped under it, by a crash or + an abort; counting that as a pass, or as nothing, would promote exactly the test probation is for. A test + of an executable that stopped before it started is not known to exist and is not counted. +- **A run is applied once.** Each run has an id (the workflow's run and its attempt, since a re-run of the + jobs is another run of the tests), and a record whose latest run is that id is left as it is, so a retry + from a fresh fetch cannot count a run twice. The record's latest run moving on is also how a test that was + not in it is known: it was promoted, renamed or deleted, and the report says so and does not call it a + candidate for what it no longer is. Its counts are kept; nothing prunes the record. +- **The threshold is the report''s, and is a working position.** The report calls a test a candidate when its + latest result is a pass and its current run of passes has reached `-Threshold`. The + default is a guess made before a result exists, it decides nothing -- promotion is a change to the test -- + and it is a parameter so that what the first results suggest can be tried without a code change. It has no + decision ID and nothing binds to it. +- **Not fatal.** The publish and the report are `continue-on-error`, as the sweep''s record is: a result the + record misses is a run it lacks, and a red run for a publish that failed would be read as a test that did. +- **The plumbing is a copy, deliberately.** [publish-probation-results.ps1](tools/publish-probation-results.ps1) + and `Get-ProbationBranch` follow [publish-sabotage-catchers.ps1](tools/publish-sabotage-catchers.ps1) and + `Get-SabotageRecordBranch` rather than generalising them, so that building this did not touch the sabotage + harness, which a change to `common.ps1` would have re-swept in full. Whether to merge them is parked, in + [PARKED.md](PARKED.md#m45-plus-publishers), for when the new one has been shown to work. + +What was weighed is in [DESIGN-RATIONALE.md](DESIGN-RATIONALE.md#how-probation-results-are-kept). ## A `compile_fail` doctest names the error it exists to produce **Every `compile_fail` doctest pins its expected error code -- ```` ```compile_fail,E0502 ```` -- and CI diff --git a/DESIGN-RATIONALE.md b/DESIGN-RATIONALE.md index 30ff178a..4d55b722 100644 --- a/DESIGN-RATIONALE.md +++ b/DESIGN-RATIONALE.md @@ -1077,3 +1077,26 @@ out: both answer the symptom, and are parked behind the reduction (`M45+.1`) unt stall still reaches a control. Deleting the heavy tests: the scale is evidence the smaller form does not give, so it moves to a daily tier instead. Running the heavy tier inside the sabotage sweep: the sweep runs the whole suite once per entry, which is exactly the cost this removes. + +## How probation results are kept, and what was weighed + +The decision is [DESIGN-NOTES.md](DESIGN-NOTES.md#probation-results). The engineer asked for tests that have never +run to run after the main tests, not fail the run, and be promoted "once they succeed for enough runs", which needs a +count that outlives a run. + +- **An artifact** expires, which is the whole objection to it, and is why it carries a run's outcomes to the job that + keeps them and no further. +- **The cache** evicts and no clone can read it, the reasons the sabotage record left it for a branch. +- **The `sabotage-records` branch itself** holds one generated file about a different kind of fact, and a second file + there would couple two unrelated publishers to one history and one retry loop. A branch of its own costs a readme. +- **Reading the human-readable test output** was the first thing tried and gives up what matters: a test's printed + line says it passed or failed and nothing of which executable, and a test that kills its process prints nothing at + all, which is the case that must not read as a pass. libtest's JSON names the test when it starts and again when it + finishes, so a start with no finish is visible. Its format is unstable; the census already depends on it, and a + change to it fails the run of this script loudly where a parse of text would quietly find nothing. +- **Running the executables directly** would give the JSON and the executable together, and lose what `cargo test` + sets for a test: its working directory and the variables it reads. A test would then pass or fail by how it was + started. +- **What building it found.** The tests caught two things a reading would not have: `@()` over an empty generic list + throws on PowerShell 7 here, so the empty case of every parse failed; and PowerShell 7 reads an ISO 8601 string out + of JSON as a DateTime, and written back it differs from the branch's text, so every publish would have committed. diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 9db10211..b8f72114 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -86,14 +86,21 @@ marker but is not spelled as one. CI runs it after the all-features `cargo test` [tools/test-check-test-tiers.ps1](tools/test-check-test-tiers.ps1) tests it. The reasoning is in [DESIGN-NOTES.md](DESIGN-NOTES.md#long-haul-tier). -[.github/workflows/long-haul.yml](.github/workflows/long-haul.yml) runs the tiers: `cargo test --workspace ---all-features --locked --no-fail-fast --lib --bins --tests --examples -- --ignored long_haul::`, and then the -same with `probation::`. It runs daily when `main` has moved since the last run that completed, and on a manual -dispatch (from any branch, always). A failure in the long-haul tier fails the run. A failure in the probation -tier does not: that step is `continue-on-error`, and the run's summary says what it did. The decision whether -`main` has moved is [tools/plan-long-haul.ps1](tools/plan-long-haul.ps1), tested by -[tools/test-plan-long-haul.ps1](tools/test-plan-long-haul.ps1). To run a tier locally, pass the same arguments -for the crate in hand: `cargo test -p --all-features -- --ignored long_haul::`. +[.github/workflows/long-haul.yml](.github/workflows/long-haul.yml) runs the tiers, the long-haul tier and then the +probation tier, each through [tools/run-tier.ps1](tools/run-tier.ps1), which owns the command that selects a tier. +It runs daily when `main` has moved since the last run that completed, and on a manual dispatch (from any branch, +always). A failure in the long-haul tier fails the run. A failure in the probation tier does not: that step is +`continue-on-error`. The decision whether `main` has moved is [tools/plan-long-haul.ps1](tools/plan-long-haul.ps1), +tested by [tools/test-plan-long-haul.ps1](tools/test-plan-long-haul.ps1). To run a tier locally: +`./tools/run-tier.ps1 -Module long_haul -OutputDirectory .scratch/tiers/long_haul`, or for one crate, +`cargo test -p --all-features -- --ignored long_haul::`. + +What each probation test did is kept across runs on the `probation-results` branch, which holds one generated file and +shares no history with `main`. To read it, run `./tools/report-probation.ps1`: each test's runs, passes, current run of +passes and latest result, and which tests have passed enough runs in a row (`-Threshold`) to be candidates for promotion. +Promoting one is a change to the test, moving it from the `probation` module to `long_haul`; the report decides nothing, +and the threshold is a working position (see [DESIGN-NOTES.md](DESIGN-NOTES.md#probation-results)). The scripts are +tested by [tools/test-probation.ps1](tools/test-probation.ps1). ## Release process diff --git a/PARKED.md b/PARKED.md index 4c621a58..5b9197c6 100644 --- a/PARKED.md +++ b/PARKED.md @@ -32,3 +32,16 @@ milestone of [CHECKLIST.md](CHECKLIST.md), and the file is deleted when it is em is left until it is known how often a stall still reaches a control once they do. A control's patch changes nothing, so a timeout on one cannot be a detection; a timeout on an entry declared `caught` can be, and stays one either way. + +## M45+ -- The two publishers of a record to a branch (parked behind M45.10) + +- [ ] **M45+.2** -- Merge or delete the copy. `tools/publish-probation-results.ps1` and `Get-ProbationBranch` in + `tools/probation-common.ps1` repeat the approach of `tools/publish-sabotage-catchers.ps1` and + `Get-SabotageRecordBranch` in `tools/common.ps1`: ask the remote for the branch, build a blob, a tree and a + commit with plumbing in an index of its own, push a fast-forward, retry from a fresh fetch. Decide whether + one parameterised publisher serves both, or whether the two stay apart. + + **Gate:** [M45.10](CHECKLIST.md). The probation publisher is a copy so that building it left the sabotage harness + alone (`DESIGN-NOTES.md`, "probation-results"), and it is to be judged after the scheduled workflow has run it + against the real remote, which nothing before M45.10 does. A merge edits `tools/common.ps1`, which every + sabotage sweep takes as an input. diff --git a/tools/README-sabotage.md b/tools/README-sabotage.md index c0a0f98c..4d5b3402 100644 --- a/tools/README-sabotage.md +++ b/tools/README-sabotage.md @@ -235,6 +235,17 @@ that is expected to survive and says why. -CargoCommand (Resolve-Path tools\sabotage-long-haul-runner.cmd).Path ``` +[sabotage-probation.json](sabotage-probation.json) is the same for the scripts that keep the probation tier's results -- +[probation-common.ps1](probation-common.ps1), [run-tier.ps1](run-tier.ps1), +[publish-probation-results.ps1](publish-probation-results.ps1) and [report-probation.ps1](report-probation.ps1) -- judged by +[test-probation.ps1](test-probation.ps1) through [sabotage-probation-runner.cmd](sabotage-probation-runner.cmd). That suite +builds real one-crate workspaces, so each entry costs close to a minute. + +```powershell +.\tools\run-sabotage.ps1 -Manifest tools\sabotage-probation.json ` + -CargoCommand (Resolve-Path tools\sabotage-probation-runner.cmd).Path +``` + Runs on **Windows PowerShell 5.1 and PowerShell 7 alike**, like the other scripts in this directory, and the full sweep is verified on both. Worth stating because the two differ in ways that bite here specifically: 5.1 rejects diff --git a/tools/probation-common.ps1 b/tools/probation-common.ps1 new file mode 100644 index 00000000..324d13c3 --- /dev/null +++ b/tools/probation-common.ps1 @@ -0,0 +1,304 @@ +# Copyright (c) Mike Grier. +<# + Shared support for the probation tier's results, dot-sourced by + run-tier.ps1, publish-probation-results.ps1 and report-probation.ps1. Not a script. + + A probation test is one that has never run anywhere, so it starts on probation and is + promoted to the long-haul tier once it has passed enough runs (DESIGN-NOTES.md, + "long-haul-tier"). These functions turn a tier run's libtest output into outcomes, keep + a record of each test's runs across runs, and report that record. The record is derived + data and is published to a branch of its own, `probation-results`; the decision is in + DESIGN-NOTES.md, "probation-results". + + The record, `results.json`: + + { "version": 1, "latestRun": "", "latestAt": "