Repository navigation
Conversation
Bind early permission refusals without claiming fixture qualification, retain raw native evidence, and independently verify invocation termination before parsing assertions. Keep direct Accessibility observations separate from recording readiness. Apply the measured four-pixel Notch animation label allowance and adversarial coverage. Preserve all fifteen cases and native acceptance blockers for #50 and #106.
|
Remote architecture checkpoint, 2026-10-04: proposed, not implemented. Published at the owner's request before pausing. Source paths below refer to the pinned c8 source tree. Machine-local root aliases identify evidence locations, not uploaded files. title: "Notch immutable regression engine grounding"
|
| ID | Observation | Evidence | Confidence | Status |
|---|---|---|---|---|
| O1 | User requires new tests/suites without repeated intervention, and now explicitly requires avoiding runner changes. | Current user message; local session turn 168. | High | Confirmed |
| O2 | Existing guidance defines normal operation as building and promoting new Products under compatible identity. | S1, lines 243-258. | High | Confirmed |
| O3 | Feature scenarios, expected labels, and OCR tolerance policy are compiled into the signed test target. | S2, S3, S4. | High | Confirmed |
| O4 | Prepared source identity spans all regression sources, requiring suite.json and build_runner.py; provenance is not separated by engine versus test. | S5, lines 216-237. | High | Confirmed |
| O5 | PR106 samples direct Accessibility and Screen Recording in the same test process. | S3, lines 201-208. | High | Confirmed |
| O6 | Earlier stable-signing updates and cold boots proved bounded Screen Recording reuse. Full001 later refused direct Accessibility. A rebuild revoking permission is not established as the cause of that refusal. | S6, current/prior-attempt and historical P2 sections. | High | Confirmed historical evidence |
| O7 | The race requires native event ordering, 1 ms polling and observation inside 0.14 s. Remote per-step orchestration is not equivalent. | S3, lines 687-705 and 731-751. | High | Confirmed |
| O8 | Both dedicated VMs are stopped. No new VM/UI/permission operation occurred in this architecture investigation. | S7. | High | Confirmed current observation |
| O9 | Existing termination/ownership checks are stronger than accepting a raw PASS or successful dispatch. They must survive protocol migration. | S8. | High | Confirmed |
Architectural model
| Element | Kind | Definition / boundary | Owner | Relationships / lifecycle | Status |
|---|---|---|---|---|---|
| Fixed native engine | Permission-bearing package | Generic XCTest plan executor plus frozen runner, test bundle and bundled dependencies at a fixed guest path. No feature names, expected labels or per-test Swift. | Infrastructure maintainer | Explicit engine releases only; ordinary suites never build, sign, replace or mutate it. | Proposed |
| Capability protocol | Data-only interface | Versioned allowlist: exact-app identity, bounded app-owned UI queries, native pointer/click/scroll, window capture, local waits and timed action batches. | Infrastructure maintainer | Reject unknown version/capability before side effects; cannot add powers through a plan. | Proposed |
| Test package | Evolving code/data | Journeys, fixtures, selectors, expectations, negative controls and output-oracle logic outside installed engine Products. | Test author | Arbitrary test composition and analysis run in the non-permission-bearing controller; engine receives only validated plans. | Proposed |
| Controller | Orchestration | Reuse strict SSH, owned guest Aqua jobs, exclusive suite ownership and offline recovery. No public listener needed. | Regression worker | Upload bounded plans into a private per-run directory; execute existing engine without building; collect attributable output and prove termination. | Proposed reuse |
| Evidence evaluator | External assertions | Consumes exact captures, raw observations, event details and native timestamps; computes product verdict separately from executor status. | Independent verification workflow | OCR/geometry policy changes require no engine rebuild. Must not learn expected answers from observed output. | Proposed |
| Candidate/fixtures | Independent inputs | Immutable candidate and synthetic producer identities; no product hooks or test-only bypasses. | Parent preparation | Separately pinned, qualified before use, restored afterward. | Existing boundary retained |
| Baseline | Recoverable VM state | New offline baseline containing qualified fixed engine, genuine setup and known fixture state. | VM owner/controller | Restore without downgrading engine or losing setup. Preserve previous images and evidence. | Proposed migration |
Execution and provenance contract
- Independently bind engine package digest, engine protocol/capability version,
test package/oracle digest, exact plan digest, candidate digest, fixture
identity, worker identity and fresh run ID. A test revision must not
invalidate the engine's own source/package identity. - Reject physical-host execution, wrong user/session, unsupported capability,
stale identities, unknown plan fields and unowned paths before actions.
Discover supported capabilities without requesting consent. - Execute only scoped operations against freshly qualified app-owned targets.
Revalidate original process identity across actions; never silently relaunch
a substituted candidate. Scope optional producer operations separately. - Execute tightly timed sequences wholly inside the native engine. For the
existing race, send one bounded plan containing scroll, mouse down/up and
local state sampling. Preserve actual NSEvent sign/magnitude evidence,
monotonic timestamps and the measured <0.14 s boundary, not a controller
round-trip approximation. - Return observations and scoped captures, not invented product success.
XCTest success means executor completion only; external assertions decide
product PASS/FAIL. Environmental refusal is BLOCKED. A qualified product
failure remains visible through later collection/restoration trouble. - Preserve independent job/process/fixture identity, termination proof,
evidence completeness, negative-control semantics and cleanup. Introduce
a versioned adapter: do not reinterpret existing c8 receipts or silently
relax its exact XCTest-count contract. - No arbitrary shell/eval/dynamic libraries/test plug-ins inside the engine.
Native privacy operations stay in the fixed process, not newly spawned
per-test executables. Plans are bounded data, not a permission bypass. - Engine-wide capture returns only explicitly selected app-owned shareable
windows. No desktop fallback, unrelated window capture or privacy reset.
Assumptions and tensions
| ID | Assumption or contradiction | Consequence if wrong | Resolution path | Status |
|---|---|---|---|---|
| A1 | "Stable signer" was treated as the requested "unchanged runner." | Routine tests still cause package rebuild/promotion and permission uncertainty. | Separate immutable engine from changing test/oracle manifests. | Confirmed design mismatch |
| A2 | Generic existing native primitives can express current 15 scenarios. | Feature-specific native logic might leak back into engine releases. | Coverage matrix and full migration proof before freezing engine v1. Include media producer observation, restoration and timed batches. | To prove |
| A3 | A fixed package can run test-without-building with external plans across cold boots. | Per-run Xcode outputs or dependency churn might mutate the frozen package. | Keep all jobs/manifests/results outside Products; inventory complete package before/after and pin toolchain compatibility. | To prove |
| A4 | Freezing binaries removes per-test permission-bearing code churn, not all OS setup. | A new OS capability, OS upgrade, or actual engine defect may still require maintenance/setup. | Distinguish engine releases from test releases; explicit upgrade policy and measured readiness, no automatic regrant. | Explicit limitation, not accepted proof |
| A5 | Same-UID immutable modes are an operational guard, not isolation from a malicious owner. | Overstated security guarantee. | Preserve package verification and bounded commands; do not claim stronger sandboxing from 0555 alone. | Explicit limitation |
Options and tradeoffs
| Option | Benefits | Costs / risks | Evidence | Disposition |
|---|---|---|---|---|
| Continue rebuilding signed scenario runner | Small immediate code delta; some capture-consent persistence already measured. | Does not meet unchanged-runner requirement; keeps tests and permissions coupled. | S1-S6. | Reject for routine architecture |
| Fixed native engine plus external test plans/oracles | Matches requirement; preserves native evidence and existing XCTest plumbing; changes assertions without replacing engine. | One-time migration; protocol design and full native equivalence proof required. | S2-S5 show separable responsibilities; S3 timing constrains boundary. | Recommend |
| New custom automation daemon/framework | Could provide a generic API. | New transport, lifecycle and permission-responsibility surface; no verified compatibility here; unnecessary to start. | No validated local evidence. | Defer, not default |
Decision register
| Decision | Rationale | Evidence | Owner | Unlocks | Status |
|---|---|---|---|---|---|
| Routine new tests must leave installed engine bytes unchanged. | Direct user requirement. | Current user message. | Dylan | Correct architectural invariant. | Confirmed requirement |
| Prefer generic executor in existing XCTest environment, with external plans and assertions. | Minimize unproved new plumbing; preserve actual native interactions. | S2-S4, S8. | Dylan / infrastructure author | Bounded infrastructure proposal. | Proposed |
| Do not resume the old consent handoff as the next step. | It does not address requested architecture. | Current user correction. | Parent agent | Architecture-first work sequence. | Current task direction |
| Preserve c8, failed-attempt evidence and blocked PR disposition. | No new native acceptance or merge readiness exists. | S6-S8, current source identity. | Parent agent | Honest subsequent comparison. | Retained constraint |
| No permanent "never update engine" claim. | Engine defects and new OS capabilities are real maintenance, unlike routine tests. | Boundary reasoning. | Dylan / infrastructure maintainer | Explicit, non-routine upgrade lane. | Proposed limitation |
Evidence register
Source worktree: <source-worktree>;
HEAD c8af8918fa78da3b77eacc977b97d80451d5229d, observed clean.
All code paths below are relative to that worktree.
| Source | What it establishes | Freshness | Limitations |
|---|---|---|---|
| S1: experiments/tart-regression/RUNNER.md:243-334 | Existing stable-identity rebuild/promotion and consent contract. | Read 2026-10-04 at c8. | Guidance, not current permission observation. |
| S2: experiments/tart-regression/GuestRegressionProbe/GuestRegressionProbe.xcodeproj/project.pbxproj:8-44 | Feature probes and oracles compiled into one UI-test target. | Read at c8. | No native run this turn. |
| S3: experiments/tart-regression/GuestRegressionProbe/PR106Probe.swift:11-208,675-755 | Hardcoded journeys, permission checks and native timing/input. | Read at c8. | Existing interactions remain unaccepted natively. |
| S4: experiments/tart-regression/GuestRegressionProbe/SettingsRemovalOutputOracle.swift:4-126 | Compiled labels, feature cases and OCR tolerance policy. | Read at c8. | Does not itself prove rendered results. |
| S5: experiments/tart-regression/build_runner.py:216-237 | Entire regression-source package provenance. | Read at c8. | Needs a new split manifest contract, not weakened verification. |
| S6: /AGENTS.md:1-155 | Installed c8 provenance, preserved old package, historical capture reuse and Accessibility refusal. | Read 2026-10-04. | Current permissions unobserved. |
| S7: /vm.sh list | notch-proof and notch-proof-run02 stopped. | Current architecture investigation. | No guest boot or UI observation. |
| S8: experiments/tart-regression/run-suite.py:165-196 | Attribution and termination precede result interpretation. | Read at c8. | New executor status cannot impersonate old product verdict. |
| S9: local session 707f46a0-b37e-4d9e-b6b0-d78f10b44423, turn 168 | Earlier explicit priority: adding tests/running suites without intervention. | Retrieved from local history. | Current message supplies the stricter unchanged-runner requirement. |
Frontier
| Priority | Material question | Why it matters | Ready when |
|---|---|---|---|
| 1 | Does the fixed capability set express every existing scenario without feature-specific native branches? | Avoid moving the same coupling into renamed operations. | Infrastructure author maps all 15 cases, their controls and restoration to generic bounded operations. |
| 2 | Can external plans execute with exactly unchanged installed code and runtime inputs? | This is the requested property, not signer compatibility. | Add a genuinely new test after engine installation, run it plus existing suite, and prove the complete engine inventory unchanged. |
| 3 | Are evidence/assertion semantics equivalent, including the race? | Moving oracle logic must not weaken acceptance. | Existing native actions, measured timing, failure controls and independently attributable cleanup pass through the new versioned adapter. |
| 4 | What genuine initial setup is necessary for the finished engine? | Avoid repeated setup for unfinished prototypes. | Final engine/capabilities fixed; actual responsible process diagnosed; any required guest-only human consent prepared visibly once, then unattended proof. |
Accepted risks and follow-ups
No new risks are approved by this artifact. This design remains proposed.
- Infrastructure author: implement the split as separate infrastructure
work, not another implicit PR106 repair cycle. No implementation occurred in
this investigation. - Infrastructure author: migrate existing tests and OCR/label policy out of
permission-bearing code; create separate engine, test and candidate manifests;
keep all per-run mutable output outside the installed package. - Independent verifier: install the completed engine once, establish genuine
setup if needed, then add a new test and an oracle change without building,
signing or replacing engine Products. Run positives and negative controls,
cold boot and repeat; compare all executable/bundle contents before/after.
Missing consent remains BLOCKED, not permission to alter privacy databases. - Independent verifier: prove native 140 ms race, direction/magnitude,
app/producer identity, pixels and cleanup; archive evidence. No retries until
green, substituted historical evidence or success-shaped fallback. - VM controller: create the new offline baseline only after engine/setup
qualification; old Snapshot A restores old code and earlier consent state. - Parent: both PRs remain blocked pending actual acceptance. The corrected
next step is infrastructure architecture/migration, not repeating the old
guest-Accessibility handoff. If human setup later becomes necessary, prepare
and verify a visible guest before asking the user to interact.
HandoffGoal
Current Progress
What Worked
What Didn't Work
Next Steps
|
Paused - unchanged-runner architecture takes priority
Owner paused work on 2026-10-04. Read the remote handoff and full architecture proposal first.
Routine new tests must not rebuild or replace the permission-bearing engine. That separation is proposed, not implemented. This supersedes the older 'grant Accessibility next' handoff below; historical evidence remains preserved. Both VMs and watchers are stopped. No native acceptance, merge readiness or permission change is claimed. Resume only after the owner continues.
Current continuation - draft, native acceptance BLOCKED
Current source:
c8af8918fa78da3b77eacc977b97d80451d5229d, containingpocketat84f067862fdef48c225891e1977f9cb9684a99a9(Calendar #97, Shared #109 and stable runner #108). Related to #50. The panel-swipe removal and retained media/hover/click/keyboard behavior remain unchanged. All fifteen registered scenarios remain present.Latest native attempt, preserved unchanged: About admission plus eleven of the fifteen suite cases actually executed: twelve native tests, raw 4 PASS / 4 FAIL / 4 BLOCKED, zero skips/retries. All twelve before/after Screen Recording observations were true. Both General scenarios failed because the Vision box for the visible
Notch animationlabel began four pixels before its native text frame. Both receipts had every other retained label true and the removed controls absent. The scenarios share one byte-identical top/bottom capture pair; this is not two independent renderings.The first interaction stopped before candidate/fixture/UI qualification because the runner lacked separate direct Accessibility trust. The previous combined guard did not record a separate Accessibility Boolean; the recording observation was true. The launcher then obscured that refusal, omitted its final result file and retained the suite lock. The remaining four interaction cases never dispatched. Original report SHA-256:
41d743de8d6d61cbf78ac6d3661dbb92fe570557206dbfc1b942b8d3d5eed3fb. No failed report or missing native outcome has been converted to a pass.Correction at this head: only the measured label receives the existing four-pixel allowance, with captured-coordinate and rejection controls for both General descriptors. The runner binds fixture inputs before early refusals without claiming native qualification, samples Screen Recording and Accessibility separately before/after, preserves raw diagnostics, and persists a bound BLOCKED result for launcher collection errors. The suite separates attributable process/job termination from assertion parsing: malformed assertions may continue only after proven termination; missing ownership/cleanup evidence retains the lock. The 140-ms real-event challenge, exact process/candidate/fixture identity, sticky primary failures, capture/restoration gates and direct Accessibility guard remain intact.
Current source evidence: all 17 safe local gates passed / 18 planned on the exact committed head, including 159 regression Python tests, 182 workflow contracts, 69 helper tests, app/helper builds and lint, executable pixel/failure-policy contracts, and the standalone native runner build. Full app-hosted tests were deliberately deferred to isolated hosted runners; the original local envelope remains
incomplete, not 18/18. The final patch initially exposed one leaked mocked cleanup flag; the unchanged retry reproduced it, then an explicit test-fixture reset fixed it. That failure history is retained.Two fresh independent static reviews covered the complete correction. Roast found no substantive defect. The second reviewer initially questioned other-label evidence, then inspected the preserved receipts and withdrew that blocker; it accepted the documented conservative termination-pin binding and unverified raw-receipt reporting. Recovery cost, possible misreading of
rawVerdicton a BLOCKED diagnostic, and future OCR variability remain explicit limitations, not acceptance-gate defects.Hosted evidence now complete for this head: build run 37233760446 passed all three matrix legs (macOS 15, macOS 26 and Xcode 27), including each actual app-hosted Test step. The three downloaded XCTest result bundles each report 238 passed, 0 failed, 0 skipped, 0 expected failures. Checkout logs identify the synthetic merge
56d02571cdf4010087ed91bb71a41cd719de79d0; its parents are this head and the abovepocketbase, and its tree equals the reviewed head's treee7cee17241cdff6e54dc4d84a74345ccc1ba939a. CodeQL run 37233760498 completed successfully for Swift, Python and Actions, including Swift's actual analysis step. The final exact-head check-run response has fourteen completed successful checks, including the aggregate CodeQL check that was earlier pending/skipped. Helper, lint, contract and target-policy/guidance checks also succeeded. This closes the deferred hosted-app-test evidence gap; it does not satisfy native interaction acceptance or turn an empty configured required-check set into a required-green gate.Recoverable VM model: the offline snapshot/atomic restore/reboot model was exercised successfully. After the failed media attempt, the exact pre-media snapshot was restored; original scoped preferences/data, candidate and protected runner matched, and the three temporary preference keys plus producer/run preparation roots were absent. Failed-run state and prior packages remain preserved. Separate MACL/ctime changes remain recorded without claims about cause, harmlessness or permission equivalence.
Reviewed runner promoted: the complete stable-signed
c8af8918package is installed at the fixed guest root/Users/notch/.local/vm-regression/work/stable-probe, manifestff1f9cc4858681b19922743c842c8d0613d9a7e9a7b8c039823dc83b4251706c. Exact package/source/signatures/protected containers were verified before and after exclusive whole-root moves; the prior8e649d5package remains preserved. Both executable/code hashes changed while the approved signer, role identities, requirements and entitlements remained compatible. Thirteen scoped profile paths, effective preferences and conditions matched after promotion. Normal guest shutdown completed. No app/test/UI launch, permission sample, privacy grant or reset occurred during promotion; current permission readiness remains unverified.The installed candidate executable remains
ee221a5666b0f989b31c2b19ab255d76a1b58fc1eaab13081f69dd28bf041e3c, actually built at30a1bb4; all product/build inputs are unchanged through this head. It was not rebuilt atc8af8918.Still required: guest-only direct Accessibility authorization for
/Users/notch/.local/vm-regression/work/stable-probe/Products/Debug/GuestRegressionProbe-Runner.app, verified permission behavior with the changed runner, fresh full fifteen-case native outcomes including producer/consumer/real-input/pixel controls, and verified fixture restoration. Screen Recording is separate; no reset is requested. The existing pre-media snapshot contains the old runner and pre-Accessibility state; establish a new offline baseline after setup rather than restoring away the new runner/consent. The earlier eleven-test baseline subset matched all ten selected expectations but never covered the five new interactions. Neither hosted tests, source review nor signing compatibility satisfies these native gates. PR107 remains dependent on this acceptance.No merge, release, notarization, host installation, automatic privacy grant/reset or permission bypass. The current disposition is an owner-prerequisite handback, not readiness or completed PR clearance; observation-only monitoring is being stopped rather than represented as useful unattended progress. The parent retains the separately authorized conditional disposition after the missing evidence exists.
Historical evidence below - not acceptance of the current head
The following records describe earlier candidates and preserved failures. They are retained for provenance, not relabeled as current-source validation.
Draft: supplemental panel and media preservation proof remains
Related to #50. All thirty registered native outcomes matched. A later independent contract reassessment confirmed that the administrative comparison of every filesystem attribute is stricter than the declared signed-candidate/output contract. The original worker's BLOCKED report and exact 72-zero-byte
com.apple.maclobservation remain immutable; they are not silently converted to PASS.Code/file bytes, modes, symlinks, thirteen architecture signatures and recorded preferences stayed unchanged. This supports the narrower measured integrity claim, not universal metadata or permission-state equality. No attribute was stripped, no blanket exception adopted, and the exact MACL semantics remain unknown.
The runner prerequisite is complete: #108 is merged into
pocketat
b8b70228d82bd0e8b208c2e34b4c94f6670fd9b2. Protected stable-signedrunners, exact-source/package verification and the unattended update model
are delivered. The updated runner passed admission plus the full nine-case
registry on two fresh headless boots: 20 native tests, 18/18 expected suite
outcomes, zero skips, and true recording preflight before and after every test.
No additional human interaction, grant or reset was needed during that proof.
Artifact inventory and app/UI/process/job/lock cleanup were verified.
This is the configured runner's bounded acceptance, not this feature's
candidate-specific acceptance or whole-app coverage. The old timeout,
restoration and metadata failures remain preserved; they are no longer the
current runner blocker.
Historical hosted checks passed at
d9b02661b8cf2a8c8815e0e6902be878077e4ba6(observed before the #108 merge); refresh current head/base readiness before landing.#50: remove panel swipes, preserve media gestures
Validation and identity
d9b02661b8cf2a8c8815e0e6902be878077e4ba6.e966f1f01316c8cd505e0a54118a435676e241943dc9dfc0b270f5980740b2cd.e652e9e8ee5f6fea950230571e8d7daae52545602f313229d7d111537b4d85dc.Settings coverage does not certify physical trackpad behavior, live media playback or the pulse transition through real gestures. Required follow-up acceptance is not replaced by source contracts. The generic runner blocker is resolved, but this draft still needs its own supplemental panel/media evidence and current-base readiness; it is not merge-ready.
Integration freshness after the runner merge
Before landing, reconcile this change against current
pocket, includingmerged #108, and refresh actual head/base review and CI evidence. Prepare new
exact-source candidates/fixtures whenever integration changes their inputs.
Historical green checks are not automatically current-base readiness.
This reconciliation changes the description only; it does not update this
branch, rerun tests, mark the draft ready, or authorize a merge.
Factual reconciliation after #108 merged into
pocketatb8b70228d82bd0e8b208c2e34b4c94f6670fd9b2on 2026-10-04 UTC(2026-10-03 evening EDT). Remaining issue/PR states, priorities and ownership
are unchanged. This update does not run tests, integrate branches, merge
other work, change permissions, install an app or publish a release.