Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHECKLIST.md
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,8 @@ last learned. The decision and its alternatives are in

> **-> UNBLOCKS PARKED WORK:** completing this unblocks `M34.25` in [PARKED.md](PARKED.md#m3425-plus).

- [x] **M34.26** -- Add `expect: hardware-dependent` to the sabotage harness, and declare the RS-C-4 covering-flag entry with it.

## M37 -- Discharge the failable-call standard across the workspace

Numbered M37, not M22. This section arrived from PR #84, which numbered it M22 without knowing
Expand Down
18 changes: 18 additions & 0 deletions DESIGN-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -3027,6 +3027,24 @@ last learned and not from nothing, which is what a clone's first scoped sweep ot

The alternatives and what building it found are in
[DESIGN-RATIONALE.md](DESIGN-RATIONALE.md#how-the-record-came-to-a-branch).

<a id="sabotage-hardware-dependent"></a>**An entry whose result depends on the machine declares `hardware-dependent`.**
The engineer, 2026-11-03, choosing among taking such an entry out of its manifest into documentation, a new
expectation, and leaving it to fail at random. The case was `windows-ioring-sys`' entry that drops a flush's
covering flag: it was declared to survive from the machine it was written on, and GitHub runners have since
both caught it and not, so every full sweep failed or passed on which runner it drew. `expect:
hardware-dependent` scores caught and survived alike and puts `-- on this machine; either is declared` on
the verdict line, so which one a machine gave is kept; a patch that does not build, or a run that never
reached the tests, still fails, since it measured nothing.

- **It is for the machine, not for flakiness.** An entry whose result varies because the tests are
timing-sensitive, or because the runner was slow, is a defect in the entry or the tests, and
`hardware-dependent` would hide it. The `why` of an entry declaring it says what about the machine
decides the result.
- **Why not documentation alone.** The entry still runs everywhere and still fails a sweep on a patch that
stops compiling; moved into prose, nothing would notice the day the patch stopped applying.

The work is [CHECKLIST.md](CHECKLIST.md) `M34.26`.
## <a id="compile-fail-pins-its-code"></a>A `compile_fail` doctest names the error it exists to produce

**Every `compile_fail` doctest pins its expected error code -- ```` ```compile_fail,E0502 ```` -- and CI
Expand Down
2 changes: 1 addition & 1 deletion crates/windows-ioring-sys/DESIGN-NOTES.md

Large diffs are not rendered by default.

24 changes: 6 additions & 18 deletions crates/windows-ioring-sys/sabotage.json
Original file line number Diff line number Diff line change
Expand Up @@ -726,25 +726,13 @@
]
},
{
"name": "DECLARED BLIND SPOT: dropping the covering flag does not fire RS-C-4 on this machine",
"name": "HARDWARE-DEPENDENT: dropping the covering flag fires RS-C-4 only where the device does not order the flush",
"file": "tests/flush_barrier.rs",
"expect": "survives",
"why": "A hardware-dependent limit on the conformance assertion's SENSITIVITY, measured rather than assumed. Running the treatment case with FlushCoverage::Unordered removes the barrier entirely, and on a machine where the device stack orders a flush behind that file's outstanding writes by itself -- which this file's own header documents, and which is the case here -- no preceding write completes after the flush even unflagged, so the assertion stays green. It is therefore NOT a portable demonstration that the assertion can go red: on the spike's machine it would be caught, on this one it survives. Recorded because a reader would otherwise reasonably expect this to be the calibration for RS-C-4, and it cannot be. What the assertion still does everywhere is report a violation if one occurs, which is the conformance job; what it cannot do everywhere is prove it would notice.",
"testArgs": [
"test",
"-p",
"windows-ioring-sys",
"--locked",
"--all-features",
"--test",
"flush_barrier"
],
"find": [
" let covering = run_case(&mut ring, handle, FlushCoverage::CoversPrecedingOperations);"
],
"replace": [
" let covering = run_case(&mut ring, handle, FlushCoverage::Unordered);"
]
"expect": "hardware-dependent",
"why": "A hardware-dependent limit on the conformance assertion's SENSITIVITY, measured rather than assumed. Running the treatment case with FlushCoverage::Unordered removes the barrier entirely. On a machine whose device stack orders a flush behind that file's outstanding writes by itself -- which this file's own header documents -- no preceding write completes after the flush even unflagged, so the assertion stays green and the entry survives; on one whose stack does not, the assertion fires and it is caught. It was declared survives from the machine it was written on, and GitHub runners have since given both answers: survived in the nightly sweep of 2026-11-03 (run 41595881867), caught in the post-merge sweeps of c714e83b (run 41569818547) and 13f06742 (run 41714398521). So it is declared hardware-dependent (D-65), and each sweep's line records which this machine gave. It is NOT a portable demonstration that the assertion can go red. What the assertion still does everywhere is report a violation if one occurs, which is the conformance job; what it cannot do everywhere is prove it would notice.",
"testArgs": ["test", "-p", "windows-ioring-sys", "--locked", "--all-features", "--test", "flush_barrier"],
"find": [" let covering = run_case(&mut ring, handle, FlushCoverage::CoversPrecedingOperations);"],
"replace": [" let covering = run_case(&mut ring, handle, FlushCoverage::Unordered);"]
},
{
"name": "M26.7: a kernel test goes back to asserting a completion is already poppable",
Expand Down
15 changes: 13 additions & 2 deletions tools/README-sabotage.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ the field, never a raw PowerShell error: the tool is a diagnostic instrument, so
its own failures must not need diagnosing, and must not take a calling script
down with them. Missing file, invalid JSON, absent `package` or `sabotages`, an
empty `sabotages`, an unresolvable `root`, a missing per-sabotage field, an `expect` that is not
one of `caught`, `survives` and `refused-by-build`, and a `buildError` that is
one of `caught`, `survives`, `refused-by-build` and `hardware-dependent`, and a `buildError` that is
missing where it is required or present where it is ignored are each reported this way, as
are running from outside a git repository, an `-OutputDirectory` git does not
ignore, and a `testArgs` carrying `--target-dir`.
Expand Down Expand Up @@ -352,6 +352,17 @@ If such an entry's build fails with some *other* error, it is reported as
`MANIFEST DOES NOT COMPILE (the build failed, but not with '...')`; if it builds
cleanly, the guard did not fire, and the result says `BUILT CLEANLY` first.

**`... -- on this machine; either is declared`** -- the entry declared
`hardware-dependent`, and this is the verdict the tests gave on the machine that
ran them. Both `caught` and `survived` count as declared: the sabotage measures
the device as much as the tests, so one machine catches it and another does not,
and a single `expect` would fail every sweep on whichever machine disagreed. The
line keeps which it was, so two runs can still be compared. Anything that is not a
verdict on the tests -- `MANIFEST DOES NOT COMPILE`, `INFRASTRUCTURE` -- still
fails, since it measured nothing. Use it only where the `why` can say what about
the machine decides the result; an entry whose result is merely unreliable is a
defect in the entry or the tests, not a property of the hardware.

**`MANIFEST DOES NOT COMPILE`** -- the patch is not valid Rust, so the tests
never got a chance to notice it. Reported for a build-phase failure and,
separately, for `a doctest would not build`: `--no-run` cannot pre-build
Expand Down Expand Up @@ -535,7 +546,7 @@ Inside each entry of `sabotages`:
|---|---|---|
| `name` | yes | Unique; also the `-Name` filter key and the transcript filename. Two names that differ only in punctuation are rejected, as is `baseline`. |
| `file` | yes | Source to patch, relative to `root`. |
| `expect` | yes | `caught` for a defect the tests must notice, `survives` for a control, `refused-by-build` for a defect a compile-time guard must reject. |
| `expect` | yes | `caught` for a defect the tests must notice, `survives` for a control, `refused-by-build` for a defect a compile-time guard must reject, `hardware-dependent` for one whose result depends on the machine it runs on (either verdict counts, and the line says which). |
| `buildError` | with `refused-by-build` only | Literal text the build's error output must contain -- the guard's own message, e.g. `a policy is missing from Policy::ALL`. Required for `refused-by-build` and rejected for anything else. |
| `why` | yes | What breaks, and why the suite should or should not notice. This is the part a future reader needs; the patch only says what changed. |
| `find` | yes | Lines to replace. Must match **exactly once**. |
Expand Down
26 changes: 21 additions & 5 deletions tools/run-sabotage.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,14 @@
it -- so a harness with no timeout hangs too, and a lost-wakeup defect that
hangs the suite has been detected exactly as intended.

AN ENTRY CAN DECLARE ITS RESULT HARDWARE-DEPENDENT. `expect` is normally
one verdict, but some sabotages measure the machine as much as the tests: a
device that orders a flush behind outstanding writes hides a missing barrier,
and one that does not exposes it. Such an entry declares
`hardware-dependent`, which counts caught and survived alike and reports
which one this machine gave; a patch that does not build, or a run that
never reached the tests, still fails, since neither is a measurement.

THE BASELINE MUST BE GREEN FIRST. Against an already-red suite every
sabotage "fails" and the whole sweep means nothing while looking like a
clean bill of health. This script refuses to start until the unmodified
Expand Down Expand Up @@ -1148,14 +1156,14 @@ foreach ($entry in @($spec.sabotages)) {
) -join "`n") 2
}
}
if (@('caught', 'survives', 'refused-by-build') -notcontains $entry.expect) {
if (@('caught', 'survives', 'refused-by-build', 'hardware-dependent') -notcontains $entry.expect) {
# Checked because an unrecognised value is not inert: `expect` is
# compared for equality when scoring, so anything else can never match
# and the entry would be reported as misbehaving on every run, whatever
# the suite actually did.
Exit-WithMessage (@(
"The sabotage '$($entry.name)' declares expect = '$($entry.expect)'."
"It must be 'caught', 'survives' or 'refused-by-build'."
"It must be 'caught', 'survives', 'refused-by-build' or 'hardware-dependent'."
) -join "`n") 2
}
# `buildError` is what makes a build failure count, so it is required
Expand Down Expand Up @@ -1866,10 +1874,18 @@ foreach ($sabotage in $selected) {
if ($sabotage.expect -eq 'refused-by-build' -and @('passed', 'failed', 'hung') -contains $run.Outcome) {
$actual = "BUILT CLEANLY: the guard did not fire; then $actual"
}
# A hardware-dependent entry's result is a measurement of the machine it ran
# on, so either verdict on the tests is what it declared; the line says which
# this machine gave. Anything that is not a verdict on the tests -- a patch
# that does not build, a run that never started -- is still a failure.
$hardware = $sabotage.expect -eq 'hardware-dependent' -and @('passed', 'failed', 'hung') -contains $run.Outcome
if ($hardware) {
$actual = "$actual -- on this machine; either is declared"
}
$ok = switch ($run.Outcome) {
'passed' { $sabotage.expect -eq 'survives' }
'failed' { $sabotage.expect -eq 'caught' }
'hung' { $sabotage.expect -eq 'caught' }
'passed' { $sabotage.expect -eq 'survives' -or $hardware }
'failed' { $sabotage.expect -eq 'caught' -or $hardware }
'hung' { $sabotage.expect -eq 'caught' -or $hardware }
'build-failed' { $refusedAsNamed }
default { $false }
}
Expand Down
32 changes: 32 additions & 0 deletions tools/sabotage2.json
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,38 @@
"why": "The two flags choose the entries by different rules, so a run given both would use one and silently ignore the other.",
"find": ["if (-not [string]::IsNullOrWhiteSpace($Since) -and -not [string]::IsNullOrWhiteSpace($Diff)) {"],
"replace": ["if ($false) {"]
},
{
"name": "a hardware-dependent entry that is caught is scored as misbehaving",
"file": "tools/run-sabotage.ps1",
"expect": "caught",
"why": "An entry declared hardware-dependent is a measurement: on one machine the tests catch it and on another they do not. Scoring only the survived side would make every machine that catches it a red sweep, which is the failure that prompted the declaration. Added with hardware-dependent, the engineer's ruling of 2026-11-03, for an entry whose result depends on the device it runs on.",
"find": [" 'failed' { $sabotage.expect -eq 'caught' -or $hardware }"],
"replace": [" 'failed' { $sabotage.expect -eq 'caught' }"]
},
{
"name": "a hardware-dependent entry whose patch does not build is scored as behaving",
"file": "tools/run-sabotage.ps1",
"expect": "caught",
"why": "A patch that does not compile measured nothing, on any machine. Letting hardware-dependent absorb it would turn a broken manifest entry into a permanent green that no sweep ever reports. Added with hardware-dependent, the engineer's ruling of 2026-11-03, for an entry whose result depends on the device it runs on.",
"find": [" 'build-failed' { $refusedAsNamed }"],
"replace": [" 'build-failed' { $refusedAsNamed -or $sabotage.expect -eq 'hardware-dependent' }"]
},
{
"name": "a hardware-dependent result does not say it was this machine's",
"file": "tools/run-sabotage.ps1",
"expect": "caught",
"why": "The verdict line is where the measurement is kept: without the note it reads as an ordinary caught or survived, and a reader comparing two runs cannot tell a machine difference from a change in the tests. Added with hardware-dependent, the engineer's ruling of 2026-11-03, for an entry whose result depends on the device it runs on.",
"find": [" $actual = \"$actual -- on this machine; either is declared\""],
"replace": [" $actual = \"$actual\""]
},
{
"name": "hardware-dependent is refused as an unknown expect",
"file": "tools/run-sabotage.ps1",
"expect": "caught",
"why": "The validator holds the list of values scoring understands; one missing from it refuses every manifest that uses it before a single entry runs. Added with hardware-dependent, the engineer's ruling of 2026-11-03, for an entry whose result depends on the device it runs on.",
"find": [" if (@('caught', 'survives', 'refused-by-build', 'hardware-dependent') -notcontains $entry.expect) {"],
"replace": [" if (@('caught', 'survives', 'refused-by-build') -notcontains $entry.expect) {"]
}
]
}
27 changes: 27 additions & 0 deletions tools/test-run-sabotage.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -1591,6 +1591,8 @@ Test-Case 'a sweep of planned overruns kills what overruns and spares what is gi
New-FauxRun 'sleeps past the bound, but its own bound allows it' 'sleep 5 fail' 'caught' 'caught \(suite failed, exit 101\)' `
-Extra @{ timeoutSeconds = 9 }
New-FauxRun 'finishes under the bound' 'sleep 1 pass' 'survives' 'survived \(NOT caught\)'
New-FauxRun 'hardware-dependent, and the tests hang' 'hang' 'hardware-dependent' `
("caught \(tests HUNG past ${b}s\)" + [regex]::Escape(' -- on this machine; either is declared'))
)
$root = New-FauxFixture -Runs $runs
try {
Expand Down Expand Up @@ -1638,6 +1640,31 @@ Test-Case 'a sweep whose runs do not do what their entries declared fails, and n
}
}

Test-Case 'a hardware-dependent entry behaves as declared caught or survived, says which, and still fails when the tests never ran' {
# Both sides of the declaration, and its edge: a run that survives and one that
# fails are each what it declared, and the line says which this machine gave; a
# patch that does not build is not a measurement of the machine, so it fails as
# it would under any other declaration. The hang is left to the overrun cases,
# where its cost is paid: here it would cost a whole bound.
$either = ' -- on this machine; either is declared'
$runs = @(
New-FauxRun 'the device orders the flush itself' 'pass' 'hardware-dependent' ('survived \(NOT caught\)' + [regex]::Escape($either))
New-FauxRun 'the device does not' 'fail' 'hardware-dependent' ('caught \(suite failed, exit 101\)' + [regex]::Escape($either))
New-FauxRun 'the patch does not build' 'build-fail' 'hardware-dependent' 'MANIFEST DOES NOT COMPILE \(tests never ran\)' -Ok $false
)
$root = New-FauxFixture -Runs $runs
try {
$result = Invoke-FauxSweep -Root $root
Assert-Equal 1 $result.ExitCode $result.Output
Assert-FauxPlanReported -Runs $runs -Output $result.Output
Assert-Match '1 of 3 sabotages did not behave as declared\.' $result.Output
$listed = $result.Output.Substring($result.Output.IndexOf('UNEXPECTED RESULTS'))
Assert-Match '(?m)^\s+the patch does not build\s*$' $listed 'the build failure is listed'
Assert-False ($listed -match 'orders the flush itself|the device does not') 'neither measurement is listed'
}
finally { Remove-Fixture $root }
}

Test-Case 'the same plan gives the same verdicts on every sweep' {
# Nothing in a faux run is random, so two sweeps of one plan must agree
# line for line, apart from the seconds each took. Overruns are left out
Expand Down
Loading