Add a long-haul tier, a probation tier, and suites that do only what their claims need

.github/copilot-instructions.md

@@ -2252,6 +2252,13 @@ The data does not have to be necessarily stable. A guideline might be that small
 a separate file or somehow encoded in source files. Larger data sets may be generated at run time, whether exhaustively or
 via random techniques.
 
+### Test tiers
+
+A test too slow for every push is `#[ignore = "long-haul: <cost>"]` in a module named `long_haul`; one not yet
+trusted is `#[ignore = "probation: <cost>"]` in a module named `probation`. A tier's run selects by the module
+name and the reason names the tier, so the two must agree, and `tools/check-test-tiers.ps1` refuses a test where
+they do not. The convention, what an integration test does, how the tiers run and where probation results are kept are in [DEVELOPMENT.md](../DEVELOPMENT.md#test-tiers).
+
 ## Architectural pre-steps
 
 **Never call `stdout`/`stderr`/`print`/`eprintln` (or the language equivalent) from

.github/release/workflows.test.cjs

@@ -0,0 +1,36 @@
+// Copyright (c) Mike Grier.
+
+// Every workflow file is valid YAML. GitHub does not run a workflow it cannot parse, and says so only as a failed
+// run with no jobs: the pull request's checks simply never appear, so a required one is missing and nothing says why.
+// An unquoted `run:` value holding `: ` is enough (`--skip long_haul:: --skip probation::`), and no other check here
+// reads a workflow as YAML.
+
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const path = require('node:path');
+const YAML = require('yaml');
+
+const workflows = path.join(__dirname, '..', 'workflows');
+
+function problems(text) {
+  const document = YAML.parseDocument(text);
+  return [...document.errors, ...document.warnings].map((e) => e.message);
+}
+
+test('the check reports a plain scalar that holds ": "', () => {
+  const broken = 'jobs:\n  t:\n    steps:\n      - run: cargo test -- --skip long_haul:: --skip probation::\n';
+  assert.notEqual(problems(broken).length, 0);
+});
+
+test('the check accepts the same command quoted', () => {
+  const quoted = "jobs:\n  t:\n    steps:\n      - run: 'cargo test -- --skip long_haul:: --skip probation::'\n";
+  assert.deepEqual(problems(quoted), []);
+});
+
+for (const file of fs.readdirSync(workflows).filter((f) => /\.ya?ml$/.test(f))) {
+  test(`${file} is valid YAML`, () => {
+    const found = problems(fs.readFileSync(path.join(workflows, file), 'utf8'));
+    assert.deepEqual(found, [], `${file}: ${found.join('; ')}`);
+  });
+}

.github/workflows/ci.yml

@@ -218,6 +218,36 @@ jobs:
       - name: Run test-sabotage-records.ps1 (Windows PowerShell 5.1)
         shell: powershell
         run: ./tools/test-sabotage-records.ps1
+      # The check that a test's ignore reason and its module agree about its tier, driven
+      # both from recorded listings and against a real fixture workspace, because the
+      # JSON libtest writes is what it reads and is not always valid JSON. Both hosts:
+      # an ordinary script a developer may run under either.
+      - name: Run test-check-test-tiers.ps1
+        shell: pwsh
+        run: ./tools/test-check-test-tiers.ps1
+      - name: Run test-check-test-tiers.ps1 (Windows PowerShell 5.1)
+        shell: powershell
+        run: ./tools/test-check-test-tiers.ps1
+      # The planner that decides whether the long-haul workflow runs: a defect in it is quiet
+      # in both directions, the tier never running while the schedule stays green or running
+      # daily on an unchanged main. Both hosts, as above.
+      - name: Run test-plan-long-haul.ps1
+        shell: pwsh
+        run: ./tools/test-plan-long-haul.ps1
+      - name: Run test-plan-long-haul.ps1 (Windows PowerShell 5.1)
+        shell: powershell
+        run: ./tools/test-plan-long-haul.ps1
+      # The scripts that run a tier, keep what each probation test did on a branch, and report
+      # it. A defect in them is quiet: a test counted as passing that did not, a run counted
+      # twice, or a test that killed its process counted as never having run. The publisher
+      # runs against a bare repository, and the runner against real one-crate workspaces. Both
+      # hosts, as above.
+      - name: Run test-probation.ps1
+        shell: pwsh
+        run: ./tools/test-probation.ps1
+      - name: Run test-probation.ps1 (Windows PowerShell 5.1)
+        shell: powershell
+        run: ./tools/test-probation.ps1
       # pwsh only, and deliberately not repeated under 5.1 like the step above:
       # check-encoding.ps1 carries curated mojibake with no BOM, so 5.1 decodes
       # it as ANSI and cannot parse it. The suite says so and refuses to run
@@ -327,6 +357,15 @@ jobs:
           RUST_BACKTRACE: 1
           RUST_LIB_BACKTRACE: 1
         run: cargo test --workspace ${{ matrix.flags }} --locked --no-fail-fast
+      # A test's ignore reason and the module it sits in must agree about its tier, because
+      # the long-haul and probation runs select by module name: a test that disagrees is one
+      # nothing runs, and nothing else says so. It asks every test executable for its list,
+      # so it needs the all-features build above and nothing more. See DEVELOPMENT.md,
+      # "Test tiers".
+      - name: check test tiers
+        if: ${{ matrix.variant == 'all features' }}
+        shell: pwsh
+        run: ./tools/check-test-tiers.ps1
       # The UNC current-directory test is ignored by default because it needs a
       # reachable administrative share (`\\localhost\C$`), which an ordinary
       # developer account does not have -- running it by default would report an
@@ -560,7 +599,7 @@ jobs:
         env:
           RUST_BACKTRACE: 1
           RUST_LIB_BACKTRACE: 1
-        run: cargo test -p windows-platform-probes --locked --no-fail-fast -- --include-ignored
+        run: 'cargo test -p windows-platform-probes --locked --no-fail-fast -- --include-ignored --skip long_haul:: --skip probation::'
       - name: cargo test (windows-platform-probes oracle-in-renderer integration target)
         env:
           RUST_BACKTRACE: 1

.github/workflows/long-haul.yml

@@ -0,0 +1,163 @@
+# Copyright (c) Mike Grier.
+#
+# The long-haul and probation test tiers. See DEVELOPMENT.md, "Test tiers", and
+# DESIGN-NOTES.md, "long-haul-tier".
+#
+# A test whose scale is the point rather than the claim is `#[ignore = "long-haul: ..."]`
+# in a module named `long_haul`, so ci.yml and every sabotage entry skip it and this
+# workflow runs it. A test that has never run anywhere is the same with `probation`.
+# libtest selects by name and not by ignore reason, which is why the module is part of
+# the convention and why tools/check-test-tiers.ps1 holds the two in step.
+#
+# WHEN IT RUNS. Daily, and only when `main` has moved since the last run that
+# completed: on an unchanged `main` a run would repeat the last one's result at the cost
+# of a workspace build and every long test. tools/plan-long-haul.ps1 decides, and has
+# tests of its own. A manual dispatch always runs, on the branch it is dispatched from.
+#
+# WHAT FAILS IT. The long-haul tier: a failure there fails the run, since those tests
+# are the full-scale form of claims the push-time suites hold at a small size. The
+# probation tier does not: its tests have no record yet, and a failure is data rather
+# than a verdict. It runs after the long-haul tier, and whatever the long-haul tier did.
+# Its results are kept across runs on the probation-results branch, by the last job, and
+# reported in the run's summary (DESIGN-NOTES.md, "probation-results").
+#
+# WHY IT IS NOT PART OF THE SABOTAGE SWEEP. A sweep runs a crate's whole suite once per
+# entry, so a tier that costs minutes would be paid for hundreds of times. A sweep never
+# passes `--ignored` and so never pays for either tier.
+name: long-haul
+
+on:
+  schedule:
+    # Off the hour, when scheduled runs are least delayed.
+    - cron: '47 8 * * *'
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+# One at a time, never cancelling: a run cancelled by the next one would count as not
+# completed, and the commit it was on would be run again.
+concurrency:
+  group: long-haul
+  cancel-in-progress: false
+
+jobs:
+  # Asks whether `main` has moved. Needs `actions: read` to list the earlier runs.
+  plan:
+    name: plan
+    runs-on: windows-latest
+    permissions:
+      contents: read
+      actions: read
+    outputs:
+      run: ${{ steps.plan.outputs.run }}
+    steps:
+      - uses: actions/checkout@v7
+      - name: Decide whether main has moved
+        id: plan
+        shell: pwsh
+        env:
+          GH_TOKEN: ${{ github.token }}
+          GH_REPO: ${{ github.repository }}
+          EVENT_NAME: ${{ github.event_name }}
+          RUN_SHA: ${{ github.sha }}
+          RUN_ID: ${{ github.run_id }}
+        run: ./tools/plan-long-haul.ps1 -EventName "${env:EVENT_NAME}" -Sha "${env:RUN_SHA}" -CurrentRunId "${env:RUN_ID}" -GitHubOutput "${env:GITHUB_OUTPUT}"
+
+  tiers:
+    name: long-haul and probation
+    needs: plan
+    if: ${{ needs.plan.outputs.run == 'true' }}
+    runs-on: windows-latest
+    # The long tier's tests cost seconds each, not hours; a run past this has stalled.
+    timeout-minutes: 120
+    # See ci.yml: without this, rustup's directory-scoped toolchain file would win over
+    # the toolchain just installed.
+    env:
+      RUSTUP_TOOLCHAIN: stable
+      RUST_BACKTRACE: 1
+      RUST_LIB_BACKTRACE: 1
+    steps:
+      - uses: actions/checkout@v7
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+        with:
+          key: long-haul
+      # tools/run-tier.ps1 owns the command that selects a tier: the library, binary,
+      # integration and example test targets with `--ignored` and the tier's module
+      # filter, and not the doctests, since a doctest cannot sit in a `long_haul`
+      # module. `--no-fail-fast` so every crate's tier reports rather than the first
+      # to fail. It writes what each test did, and exits with cargo's own code.
+      - name: Run the long-haul tier
+        shell: pwsh
+        run: ./tools/run-tier.ps1 -Module long_haul -OutputDirectory .scratch/tiers/long_haul
+      # After the long-haul tier, and whether or not it passed, but never cancelled.
+      - name: Run the probation tier
+        id: probation
+        if: ${{ !cancelled() }}
+        continue-on-error: true
+        shell: pwsh
+        run: ./tools/run-tier.ps1 -Module probation -OutputDirectory .scratch/tiers/probation
+      # The id of the run and attempt that ran the tests, kept beside what they did. The job
+      # below that keeps the results can be re-run alone, which is a new attempt of that job
+      # and not another run of the tests, so it must not name the run by its own attempt.
+      - name: Name the run that produced these results
+        if: ${{ !cancelled() && hashFiles('.scratch/tiers/probation/outcomes.json') != '' }}
+        shell: pwsh
+        env:
+          RUN_ID: ${{ github.run_id }}.${{ github.run_attempt }}
+        run: ' [System.IO.File]::WriteAllText(".scratch/tiers/probation/run-id.txt", $env:RUN_ID) '
+      # What each probation test did, for the job below that keeps it. Passed on as an
+      # artifact so that the job holding write access never runs the tests, which run
+      # here with read-only access.
+      - name: Upload what each probation test did
+        if: ${{ !cancelled() }}
+        uses: actions/upload-artifact@v7
+        with:
+          name: probation-outcomes
+          path: |
+            .scratch/tiers/probation/outcomes.json
+            .scratch/tiers/probation/run-id.txt
+          if-no-files-found: warn
+          retention-days: 1
+          overwrite: true
+
+  # Keeps what each probation test did, across runs, on the `probation-results` branch,
+  # and reports it (DESIGN-NOTES.md, "probation-results"). The only job here that writes
+  # to the repository, and it writes one branch, which holds nothing but this record. It
+  # runs the scripts of the checkout this run is for and none of the tests, which ran in
+  # the job above with read-only access.
+  #
+  # Not fatal, as the sweep's equivalent is not: the record only ever gains or misses a
+  # run, and a red run for a publish that failed would be read as a test that did.
+  record:
+    name: keep the probation results
+    needs: [plan, tiers]
+    if: ${{ always() && needs.plan.outputs.run == 'true' }}
+    runs-on: windows-latest
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v7
+      - name: Download what each probation test did
+        continue-on-error: true
+        uses: actions/download-artifact@v8
+        with:
+          name: probation-outcomes
+          path: .scratch/probation
+      # The run is named by the id the job that ran the tests wrote beside the outcomes, not
+      # by this job's own attempt: a re-run of this job alone has the same outcomes, and
+      # applying the same id twice is applying it once.
+      - name: Publish the results to the probation-results branch
+        if: ${{ hashFiles('.scratch/probation/outcomes.json') != '' && hashFiles('.scratch/probation/run-id.txt') != '' }}
+        continue-on-error: true
+        shell: pwsh
+        env:
+          RUN_SHA: ${{ github.sha }}
+        run: ./tools/publish-probation-results.ps1 -Outcomes .scratch/probation/outcomes.json -RunId (Get-Content -Raw .scratch/probation/run-id.txt).Trim() -Commit "${env:RUN_SHA}"
+      # Read back from the branch, so what is reported is what is kept, and so that a
+      # run that could not publish still reports what the branch holds.
+      - name: Report the probation tier
+        continue-on-error: true
+        shell: pwsh
+        run: ./tools/report-probation.ps1 -SummaryPath "${env:GITHUB_STEP_SUMMARY}"

CHECKLIST.md

@@ -603,3 +603,35 @@ yanks are needed, is [DE-D-8](crates/durable-ioring-events/DESIGN-NOTES.md#de-d-
   The engineer's own step, with their crates.io login: `cargo yank --version 0.1.1 durable-ioring-events`, then
   `cargo yank --version 0.5.0 win-durable-ioring` and `cargo yank --version 0.5.1 win-durable-ioring`.
   Both driver releases require the events crate at 0.1.1 or later within 0.1, so all three go, and the replacements must exist first.
+
+## M45 -- A long-haul tier, a probation tier, and suites that do only what their claims need
+
+Each pull request's CI, and each sabotage entry, runs a crate's whole suite, so every test pays its cost
+on every one of them. Measured locally, 22 of the workspace's tests take a second or more and account
+for nearly half of its serial test time; on GitHub runners, tests doing real file I/O have stalled for
+many seconds while idle, which failed sabotage controls on the post-merge sweeps of `c714e83b` and
+`13f06742`. So a test's scale is kept only where the claim needs it, and the scale for its own sake moves
+to a long-haul tier that runs daily. Tests that have never run anywhere start on probation. The decision
+is [DESIGN-NOTES.md](DESIGN-NOTES.md#long-haul-tier); the measurements behind it are in
+[DESIGN-RATIONALE.md](DESIGN-RATIONALE.md#why-a-long-haul-tier).
+
+- [x] **M45.1** -- The tiers' convention and its census: `tools/check-test-tiers.ps1` refuses a test whose ignore reason and module disagree about its tier, in CI on every push. -> [completed 2026-11-03](COMPLETED-CHECKLIST.md#m451)
+
+- [x] **M45.2** -- `long-haul.yml`: daily when `main` has moved, plus dispatch; the long-haul tier fails the run, the probation tier does not. -> [completed 2026-11-03](COMPLETED-CHECKLIST.md#m452)
+
+
+- [x] **M45.3** -- Probation results are kept on the `probation-results` branch and reported; the promotion threshold stays a working position. -> [completed 2026-11-03](COMPLETED-CHECKLIST.md#m453)
+
+- [x] **M45.4** -- The six never-run tests (five flush-barrier stress instruments and the 4 GiB buffer test) are on probation, and a tier runs one test at a time. -> [completed 2026-11-03](COMPLETED-CHECKLIST.md#m454)
+
+- [x] **M45.5** -- `blocking_push`'s capacity test writes past a declared capacity of 32 and its full-scale form is long-haul. -> [completed 2026-11-03](COMPLETED-CHECKLIST.md#m455)
+
+- [x] **M45.6** -- `windows-file-enumeration-sys`'s three large-directory tests run over 600 files with the minimum buffer, and their full-scale forms are long-haul. -> [completed 2026-11-03](COMPLETED-CHECKLIST.md#m456)
+
+- [x] **M45.7** -- `win-multi-file-tail-watcher`'s seven scale-bearing tests run smaller with their full-scale forms long-haul; the quiet-between-bursts waits are left and say why. -> [completed 2026-11-03](COMPLETED-CHECKLIST.md#m457)
+
+- [x] **M45.8** -- The census and the cost-probe tests run the cost probes at a twentieth of their workload through a new `--iteration-divisor`, with full-length forms long-haul. -> [completed 2026-11-04](COMPLETED-CHECKLIST.md#m458)
+
+- [x] **M45.9** -- The four waits that are their claim were measured, none was shortened, and each says why beside its constant. -> [completed 2026-11-04](COMPLETED-CHECKLIST.md#m459)
+
+- [x] **M45.10** -- The workspace was re-measured: 17 file-watcher stress tests that never ran are on probation, and the 13 tests still over a second are each left with a reason. -> [completed 2026-11-04](COMPLETED-CHECKLIST.md#m4510)