Skip to content

feat(gate): fail the rule, ledger, and owned blind spots - #40

Merged
AlexNegrescu26 merged 3 commits into
mainfrom
fix/gate-blind-spots
Oct 31, 2026
Merged

AlexNegrescu26 merged 3 commits into
mainfrom
fix/gate-blind-spots

Conversation

@AlexNegrescu26

Copy link
Copy Markdown
Collaborator

Stacked on #33 (sync/upstream-c742dd6, head bb0e701).

Every defect in #33's two fix rounds shipped while check-port printed PASS. This PR makes each of those defect classes fail the gate, with a mutation case in test-gate-mutations.sh that proves the check fires. A maintainer syncing upstream now gets a red gate where they used to get a spliced sentence or a dropped one. The upstream-sync workflow runs the same checks against the candidate sha before it auto-merges.

What fails now

Check Fails on Shipped example
rule replacement & a bare & in a replacement 73f30d9 rule 512 spliced its sentence into opening-a-pr.md three times
rule paren groups an ERE group around literal parentheses with no backreference, when the replacement adds ( 73f30d9 rules 538, 551, 560 shipped ((
rule wildcards a mid-line .* with no $ anchor and no swallow.tsv row, or a row naming a plain rule 4275556 rule 34, (\b.*)?\.
swallow ledger a wildcard rule with no row, or raw upstream spans that no longer match the row's sha256. It prints the spans rules 35 and 89 silently dropped new c742dd6 text
rule staleness an exemption whose rule is not stale. It now probes d, a and addressed rules too 12 live-rule exemptions at 4275556; rule 109 never probed
rule shadowing a rule whose own raw match an earlier rule rewrites, unless shadow-exempt.tsv records it 4275556 rule 228 behind 227
shadow exemptions a shadow-exempt.tsv row naming no rule or two new ledger
owned drift an owned path whose upstream object differs from the one owned.txt records as read ab1f702's setup-pstack edit, unread through the c742dd6 sync

rule liveness now prints a cause count instead of 116 rule lines: 88 stale, 2 shadowed, 25 consumed nets, 1 catch-all.

Other changes

  • apply_patches runs patch --force. Plain patch waits at File to patch: or Assume -R? when sync-upstream.sh prints the build to a terminal. --batch was not an option, because it answers Assume -R? yes and reverts the hunk.
  • apply_rules runs one sed -E -i per batch of 64 files, in parallel. -i implies -s, so range addresses still end at each file. The rebuild is byte-identical, and reproducible passes.
  • Cheap fixes from the gate review: the readonly negation has a word boundary, guide claims grep feature map and the label drops "pin", four retired-label terms that matched nothing are gone, CHECK_PORT_DOCS_ROOT is gone, the frontmatter check has a label, and lib.sh's dead pipe in the staleness probe is gone with the function it lived in. CHECK_PORT_SKIP_MUTATIONS became CHECK_PORT_NESTED, which also skips the worktree audit in nested runs.
  • The staleness, blank-exemption, patch-failure and target-builds mutations are back. The harness now has 37 cases and runs them in parallel.
  • rules.sed translates the two false transcript claims. eval.md:22 now says "(history://<id> is the documented surface)", like session-pickup. automate-me/SKILL.md:29 resolves the transcript tree through skill://omp-mechanics/references/session-store-on-omp.md.
  • stale-exempt.tsv gains a row for rule 109 and a final newline. Without the newline, an appended mutation row edited the last row instead.

ab1f702 and the owned setup-pstack

Commit fae861e records owned.txt with the 519d36a tree, d533da1, and its gate fails on exactly that, skills/setup-pstack: upstream moved at c742dd6, reviewed d533da1…, now 800811d…. git diff d533da1 800811d shows one sentence changed in step 3(c). Arena's cross-judge pool used to pick one value whose family differs from the parent's when possible. Since ab1f702 Arena runs one judge, tries only other-family values in list order, and never judges with inherit-parent or auto. The owned skill configures no cross-judge pool list. On omp the judge is an exact discovered agent from another family, and the arena rule already says an agent that inherits the parent's model never judges. The owned copy needs nothing, so f381939 records 800811d.

Recorded, not fixed

rules.sed was open to this PR only for the transcript claims, so two shadowed rules are recorded in shadow-exempt.tsv instead of fixed.

  • Rule 156 (agent mode (readonly: false)) sits behind rule 155's readonly: false token rule. It has no effect at c742dd6, because the reflect sentence rules consume both upstream lines. Moving it above rule 155 makes it fire.
  • Rule 170 (subagent_type: "poteto-agent") duplicates rule 149's output. Deleting it changes nothing.

Runtime

Measured on this 16-core machine, two runs each, CANON=/tmp/cursor-plugins bash omp-port/check-port.sh. bb0e701 took 87.9 s and 87.7 s. f381939 took 37.4 s and 37.6 s. The worktree audit is about 20 s in both. The rest of the drop is the batched sed: one build went from about 10 s to about 1.4 s. The harness went from 13.7 s with 17 cases to 9.9 s with 37. CI runners have fewer cores, so the parallel part will save less there (inferred, not measured).

Verification

  • CANON=/tmp/cursor-plugins bash omp-port/check-port.sh at f381939 ends check-port: PASS, including gate mutations PASS 37 mutations rejected, each by its own check.
  • bash omp-port/test-gate-mutations.sh at f381939 ends mutation tests: PASS, 37 cases rejected. New cases: bare & in a replacement, parentheses as a group, greedy mid-line wildcard, swallow row on a plain rule, swallowed text changed, wildcard rule with no row, exemption on a live rule, delete rule matching no line, rule behind its own duplicate, shadow row naming no rule, owned path moved upstream, patch prompt on a terminal, readonly after another, guide feature map, bare skill command, required item field. Restored: rule matching no upstream text, whitespace exemption row, patch that no longer applies, target that no longer builds.
  • bash -n passes on every omp-port/*.sh.
  • CANON=/tmp/cursor-plugins bash omp-port/sync-upstream.sh rebuild changed only automate-me/SKILL.md and poteto-mode/playbooks/eval.md.
  • The new awk programs give byte-identical rule_table, mark_rules and span output under Ubuntu's mawk 1.3.4 20240123. The readonly check fails under that mawk, and bb0e701's regex misses the negation on why/SKILL.md:13 there too, so that predates this PR. CI passes it, so CI's awk is probably gawk (inferred, not checked).

Each new check fails on the historical input, run in throwaway copies of this branch with CHECK_PORT_NESTED=1 bash omp-port/check-port.sh.

73f30d9's rules.sed:

rule replacement &           FAIL
  rules.sed:512: s#or `git fetch && git reset --hard origin/<branch>` between them\.#or `git fetch && git reset ...
rule paren groups            FAIL
  rules.sed:538: s#(open files, recent edits, cursor location, what was just discussed)#(open files, recent edits, what was just discussed)#
  rules.sed:551: s#(the system prompt names the path)#(`history://<id>` is the documented surface)#
  rules.sed:560: s#(force-push, delete production data, send external messages)#(force-push to a shared or trunk branch, ...)#

4275556's rules.sed and stale-exempt.tsv:

rule wildcards               FAIL
  rules.sed:34 ends a `.*` mid-line with no `$` anchor and has no swallow.tsv row: s#Pick each subagent's ... section(\b.*)?\.#...
rule staleness               FAIL
  rules.sed:109: /^- `subagent_type`: `generalPurpose`$/d
  stale-exempt.tsv:15 exempts rules.sed:80, which matches upstream text at c742dd6; it fires nothing, cause consumed
  stale-exempt.tsv:33 exempts rules.sed:132, which matches upstream text at c742dd6; it fires in the build
  ... 12 such rows, matching the gate review's list
rule shadowing               FAIL
  rules.sed:228 never fires: rules.sed:227 rewrite its own match first, e.g. Run workers on a new cloud VM (`environment: "cloud"`)
swallow ledger               FAIL
  rules.sed:34 has no swallow.tsv row; its spans hash to cc641dfd...: s#Pick each subagent's ... section(\b.*)?\.#...

swallow.tsv hashed at 519d36a and gated at c742dd6 fails on rules 25, 35, 89, 93, 140 and 141, which are the rules whose swallowed text upstream changed:

swallow ledger               FAIL
  swallow.tsv:15 rules.sed:35 swallows different text at c742dd6: reviewed a67eaf1b9c74, now 801f64a1...
    now: **Match the model to the job.** Before each spawn, start from the role's line or default ...
  swallow.tsv:29 rules.sed:89 swallows different text at c742dd6: reviewed abf2a7527816, now 72d0bb18...
    now: **Defaults for every `Task` call.** `run_in_background: true` (a steer or resume of a running agent too, ...

apply_patches with the old patch ... <"$p" and the terminal deadline cut to 20 s:

mutation tests: patch prompt on a terminal hung until the deadline, a prompt is waiting for an answer
mutation tests: FAIL

With --batch instead of --force:

mutation tests: patch that no longer applies failed for the wrong reason
mutation tests: patch prompt on a terminal unexpectedly passed
mutation tests: FAIL

Replaces #38, which GitHub closed when its base branch was deleted on merging #33. Same commits rebased onto main; the tree is byte-identical to the reviewed head f381939.

No omp system prompt names a transcript path, so eval.md:22 and
automate-me/SKILL.md:29 sent agents to a source that does not exist.

automate-me now resolves the session transcript tree through
skill://omp-mechanics/references/session-store-on-omp.md, with a rule that
reads the raw upstream sentence before the directory token rewrites it.
The history://<id> rule now covers eval's "this path" as well as "the
path", so eval reads the same as session-pickup and show-me-your-work.
Each defect below shipped while check-port printed PASS. Each check now
fails on it, and test-gate-mutations.sh has a case that proves it.

- rule replacement &: a bare & in a replacement splices the match back in
  (73f30d9 rule 512).
- rule paren groups: an ERE group around literal parentheses that the
  replacement doubles (73f30d9 rules 538, 551, 560).
- rule wildcards, swallow ledger: omp-port/swallow.tsv holds the sha256
  of every raw upstream span each .* or .+ rule matches. A mid-line
  wildcard with no row, a missing row, a changed hash, or a row on a plain
  rule fails (4275556 rule 34; rules 35 and 89 at c742dd6).
- rule staleness: fails an exemption on a rule that is not stale, and
  marks d, a, and addressed rules too, so rule 109 is now probed.
- rule shadowing: fails a rule whose own raw match an earlier rule
  rewrites, unless omp-port/shadow-exempt.tsv records it (4275556 rule
  228). Rules 156 and 170 are recorded.
- owned drift: owned.txt records the upstream object last read for each
  owned path. setup-pstack still records the 519d36a tree, so this commit
  fails on ab1f702's unread edit.

apply_patches runs patch --force, so a patch that no longer applies
exits instead of waiting at a prompt on a terminal. apply_rules runs one
sed per batch of files in parallel. Also: the readonly negation word
boundary, the feature map grep, four retired-label terms and
CHECK_PORT_DOCS_ROOT removed, a frontmatter name label, and the
staleness, blank-exemption, patch-failure, and target-builds mutations
restored. The harness runs its cases in parallel.
`git diff d533da1 800811d` shows one changed sentence in step 3(c).
Arena's cross-judge pool used to pick one value whose family differs
from the parent's when possible. Since ab1f702 Arena runs one judge,
tries only values from another family in list order, and never judges
with an `inherit-parent` or `auto` value.

The owned setup-pstack has no cross-judge pool list to configure. On omp
the judge is an exact discovered agent from another family, and the
arena rule in rules.sed already says an agent that inherits the parent's
model never judges. The owned copy needs nothing, so owned.txt now
records the c742dd6 tree.
@AlexNegrescu26
AlexNegrescu26 merged commit 21ebd7e into main Oct 31, 2026
2 checks passed
@AlexNegrescu26
AlexNegrescu26 deleted the fix/gate-blind-spots branch October 31, 2026 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant