Skip to content

ci: attribute partial index failure in combined hosted-adapter proof #2403

Open

Description

@schlessera

Supports #736; found during #1364 / #2377 qualification. Native parent attachment was refused because the epic already has GitHub’s maximum of 100 direct children; this remains a standalone investigation pending hierarchy reconciliation.

Context

Required preflight on head d54b10667a91d799289f16560731d669fdf943b0, base 775eb17aab740fa759c2182f306c38fc17acbcbe, passed lint/metadata and all 548 fast invariants. Complete import-graph feedback then ran 2,134 tests across 158 files: 2,133 passed, one failed, 19,012 assertions, 3,691.40 seconds; preflight exited 1.

The sole failure was pi hosted curated tools capture, append, archive and index in one step: add.ok was true, but add.indexed was false (packages/ui-server/tests/brain-application-adapters.test.ts:77). The preceding Claude curated-tool case and subsequent Pi raw-write case passed. This is failed current-candidate proof, not an unavailable browser or module-load failure.

The production application commits Markdown before indexing and intentionally returns partial success plus an indexError when indexing fails (packages/ui-server/src/brain/application.ts:391-396). The original test asserted only booleans, so that failed log did not retain the actual returned indexError, CLI output or runtime refusal. The subsequently delivered failure-only diagnostics are described in Current diagnostic delivery below. No quota, lease, native-storage, concurrency, provider or source cause is established.

The adapter fixture already supplies its own outside-brain UI state path (adapter, packages/ui-server/tests/brain-application-adapters.test.ts:33-38), preserving the published #1885 correction. Do not reopen that historical attribution or infer the old default-domain cause without new evidence.

One bounded diagnostic run added temporary result/receipt logging immediately before the original assertions and selected only the named Pi curated-tool case. Source/guards/deadlines were otherwise unchanged. It passed (1 test / 15 assertions / 3.99 seconds), returned indexed=true, and logged a successful receipt. Instrumentation was restored byte-for-byte. This isolated recovery neither reproduces nor diagnoses the combined failure and does not replace the failed full receipt.

An earlier full preflight on e51ca9be5 / base 82a64a13a passed all 2,134 affected cases. The subsequent base refresh changed proof/test harness inputs, with no production-source delta in the relevant server, SDK, backend or React packages. Both results retain their actual identities; equivalence is not a cause attribution.

Scope

Capture and attribute the actual post-write indexing failure under source-valid combined ordering/load. Record the complete returned application result and retained receipt, indexError, actual CLI/runtime exit and relevant owner/refusal evidence before selecting a correction. Recover the original ordering conditions rather than retrying an isolated pass into a repair claim.

Out of scope

No #1364 per-app-state redesign, weaker indexed=true assertion, change to truthful partial-success semantics, larger budget/deadline, skipped case, retries as correction, foreign reservation deletion, speculative lifecycle fix or real provider/network access.

Acceptance criteria

  • A controlled failing receipt identifies why indexing failed after the authoritative Markdown write, or names the exact missing reproduction evidence.
  • Preserve original full-run failure and isolated recovery separately, including source/base/runtime and command identities.
  • Preserve curated capture/update/archive/index, real maintenance boundary, existing fixture-owned state and keyless/offline guards.
  • Any correction is supported by an intended failing-first/mutation receipt, restored pass, same-class audit and complete affected current-candidate proof.

Verification

Use the repository test wrapper and original named adapter file plus the recorded combined import-graph selection. Keep frozen dependencies, test-owned temporary storage and real maintenance execution. A successful isolated named test is recovery evidence only. Under the October 21 maintainer-approved triage, #1364 has no dependency on this attribution item and proceeds on its own complete current-candidate proof; the original failed result remains failed.

Contract impact

None selected; attribution is missing. Preserve priority p2. The October 22 maintainer ruling removes the 0.44.0 commitment: this investigation is unscheduled until recurrence supplies the missing diagnostics. This investigation is not agent-ready and no longer blocks #1364.

Bounded combined-order attempt — 2026-10-18

At frozen head d54b10667a91d799289f16560731d669fdf943b0 and base 775eb17aab740fa759c2182f306c38fc17acbcbe, a separate checkout used a fresh frozen-lockfile install, Bun 1.4.3 and Chrome for Testing 153.0.8010.12. Temporary adapter-only instrumentation forwarded the original maintenance request unchanged and would retain complete results/receipts, stdout/stderr, exit/timedOut, boundary status and thrown error properties. Assertions, fixture-owned state, production sources, inner deadlines and offline guards were unchanged. The outer diagnostic bound was 90 minutes.

A first discovery check selected the same 158 files but started them in a different order in the fresh checkout. It was stopped during the first fixture build, exit 143, before assertions; this is a setup mismatch, not reproduction or passing proof. A two-file runner probe then confirmed explicit argv ordering. The single combined replay used all 158 original file paths in their recorded order; the repeated external-speech heading was its existing nested subprocess, not another selected file. Every observed distinct file heading matched the original ordered prefix exactly.

That replay ran from 01:22:14Z to 02:09:14Z (2,820 seconds), then the test wrapper reported exit 137 during packages/ui-server/tests/sync-mirror.test.ts, file 78 of 158. The adapter file is position 135 and was never reached: zero adapter diagnostic lines. The log contains 1,150 pass lines and zero failing assertion lines, including nested subprocess output; there is no final suite report, completed case/assertion total or passing combined receipt. Exit 137 does not identify its initiating signal/process/resource cause. No retry was performed. Existing #2396 investigates a separate exit-137 observation; this attempt establishes no common cause.

The log explicitly reports a shared server cgroup and unavailable per-server memory/pids/cpu enforcement. No host-wide process/environment scrape or causal resource observer was used. Historical host scheduling, maintenance ownership state and the initiating termination event remain unobserved. No process outside this task was killed and no foreign reservation was deleted.

Temporary instrumentation was restored byte-for-byte, both checkouts remain clean, and the diagnostic checkout's workspace output lease is released. The original failed full receipt and isolated recovery retain their original hashes. The clean frozen branch repro/2403-combined-index-a1, original inventory, instrumentation patch, setup-mismatch log and incomplete ordered log remain preserved locally. No implementation, mutation/red-restored repair evidence, PR or current-candidate qualification is claimed.

Still missing: a reached combined-order Pi curated-add failure retaining its complete indexError and application receipt, the corresponding maintenance stdout/stderr and exit/refusal/ownership evidence, and the actual boundary that caused indexing to fail after Markdown committed. This attempt did not reach that observation point. Keep needs: repro, priority p1 and milestone 0.44.0; #1364 / #2377 remain unqualified by this investigation. Resume from specific reached failure evidence or an attributed condition that justifies a bounded causal replay, rather than another unchanged full retry.

Current campaign diagnostic validation — 2026-10-19

Claim release044-2403-r2-20261019 prepared a failure-only diagnostic in local commit d425d6651cea558b51c73e4afe8da6400d855206, base 685d83e22cc452e97a2c92ef7ae683f34ceb2371. It preserves the original indexed assertions and real maintenance boundary, retains application/indexError/maintenance result and fixture-memory receipts on failure, omits credential values and restricts error metadata. Fixture records are in memory, not durable production storage. Relevant indexing source remains unchanged from the historical failing candidate.

Exactly three bounded arms ran on Bun 1.4.3. The original fixture with a controlled copied-view indexing error exits 1 at the original indexed assertion without the diagnostic; the candidate with the same control exits 1 at that assertion and retains the expected error, indexError, maintenance result and fixture receipt. Restoring the production source byte-identically and running the full adapter file once passes 14 tests / 101 assertions with quiet success. Source is restored after every arm, the worktree is clean, owned fixture storage is removed and the workspace lock is released.

This proves diagnostic visibility only. It neither reproduces the natural historical failure nor selects a cause or product repair. Preserve needs: repro, P1 and 0.44.0. Actual default preflight now passes on head 5e78322958e1234ad7324da7ac15ac42ded2b839, base 90a95123cfceb169bd2ecf7c213544db742eaed7: 53 fast tests / 1,371 assertions in three files, followed by 14 adapter tests / 101 assertions in one file; command exit 0 in 43.800 seconds. #2426 supplied this partial diagnostic only, uses Refs #2403 and retains milestone 0.44.0; its later merged state is reconciled below. The main refresh changed only the independent native-diagram test; diagnostic source bytes are unchanged from the earlier focused receipt. The owned namespace preserved normal HOME, UID/GID and original temporary storage, used a private empty-on-completion maintenance ledger and unchanged IPv4/IPv6 route guards; no browser, Docker, privileged storage/prover or provider work ran. Source is clean, the workspace descriptor is released and owned temporary remnants are archived. At that campaign stage, hosted confirmation, readiness transition, merge and diagnostic delivery were still pending; that pending-delivery statement is superseded by the verified merge below. No historical cause or complete qualification is claimed. #1364 remains blocked on causal evidence and its own complete qualification.

Current diagnostic delivery — triage 2026-10-21

PR #2426 is merged (2026-10-20T07:53:57Z), squash c488979d8a48160a957ef004fc7a57c22aa7c61b, verified as an ancestor of inspected main b393bf45994b5416867cbb372420b2b5be364892. Current adapter (packages/ui-server/tests/brain-application-adapters.test.ts:33-78) keeps its existing outside-brain state path (createFixtureHostBoundary, line47), forwards the original maintenance request unchanged and records its result or bounded error fields (lines49-63). indexFailure (lines65-76) adds the application result, fixture-memory receipts and maintenance observations only on failed success/indexed assertions; the curated add assertion is now at line117, with update/archive at121/126. Success stays quiet. Do not implement this diagnostic again.

This is delivery of diagnostic visibility, not the original indexing repair. The authoritative Markdown/partial-index result remains (packages/ui-server/src/brain/application.ts:389-396), and the original combined failure’s indexError/cause is still unknown. Preserve needs: repro, p2, all original failure identities and the copied-view/ownership evidence requirement. The October 22 maintainer ruling makes this attribution item unscheduled; the delivered #2426 diagnostic retains its own historical release attribution. The maintainer-approved October 21 triage drops #1364’s dependency and sets p2; #1364 proceeds on its own complete candidate proof. Earlier dated p1/blocking statements describe their historical handoffs. The triage inspected merged identity, ancestry and current code; it did not independently audit every historical hosted checkout, run a new reproduction or claim full current-main green.

2026-10-21 brief reconciliation: folded the later maintainer-approved priority/dependency disposition into the current specification. Historical failures and diagnostic receipts remain attributed to their original sources; needs: repro remains and no repair or proof waiver is claimed.

2026-10-22 triage at main ca8842828ed8634a84c4e8aa4a88e487238721b4: Aligned Verification and Contract impact with the already-recorded October 21 p2/nonblocking ruling. Historical failed qualification remains failed, delivered diagnostics are not a repair, and needs: repro remains.

2026-10-22 triage: reconciled the active release wording with the maintainer’s later removal from 0.44.0. Earlier dated campaign receipts describe their historical scheduling; this open p2 needs: repro investigation has no current release commitment and does not block #1364.

Activity

  1. added this to the 0.44.0 milestone on Oct 18, 2026
  2. added
    type: fixSomething is broken and must behave the way it is documented.
    area: ui-serverHono app factory: turn coordinator, auth, session catalog, routes, cron.
    area: ciCI workflows, the gates they run, release scripts, packaging.
    priority: p1Scheduled into the current or next milestone.
    needs: reproBlocked on a reproduction.
    on Oct 18, 2026
  3. self-assigned this
    on Oct 18, 2026
  4. schlessera commented on Oct 18, 2026

    @schlessera
    OwnerAuthor

    Claim token: p1-2403-diagnostic-a1-20261018. I own only this bounded reproduction investigation, using branch repro/2403-combined-index-a1 at frozen failing source. I will inspect the preserved combined failure, retain full indexing and native refusal diagnostics, and preserve the original assertions, offline guards and inner deadlines. No cause or correction is selected yet; needs: repro remains until evidence supports attribution.

  5. schlessera commented on Oct 18, 2026

    @schlessera
    OwnerAuthor

    Claim p1-2403-diagnostic-a1-20261018 released: the single 158-file ordered diagnostic ended with wrapper exit 137 at sync-mirror (file 78), before the adapter at position 135. It retained 1,150 pass lines and no assertion-failure lines, without a final suite report. The body now records the exact source/runtime/order and missing target evidence. Instrumentation is restored byte-for-byte, the frozen checkout is clean, and its workspace lease is released. Original receipts remain intact. No correction, PR, retry-to-green or common cause with #2396 is claimed; needs: repro, p1 and 0.44.0 remain.

  6. removed their assignment
    on Oct 18, 2026
  7. schlessera commented on Oct 18, 2026

    @schlessera
    OwnerAuthor

    2026-10-18 — second reproduction pass at 775eb17aab740fa759c2182f306c38fc17acbcbe.

    Different approach: Trace real process/file-lock/publication syscalls through the complete hosted-adapter file, retaining full add results and recorded application receipts before each original indexed assertion. Earlier evidence isolated the Pi case or attempted an incomplete 158-file replay; this executes every adapter case with kernel-level SQLite/file/process observation and valid owned fixture storage. No index behavior or assertion changes.

    Observed: Complete hosted-adapter file passes 14 tests / 101 assertions. Both Claude and Pi add results and durable application receipts report ok=true and indexed=true before their unchanged assertions; no partial index error occurs. Native process/lock/fsync/rename trace is retained. Outer invocation exit 0, elapsed 22.662 s; outer watchdog fired: false.

    Command (working directory: repository root; any diagnostic source changes are described above):

    strace -f -qq -e trace=process,signal,fcntl,flock,fsync,fdatasync,rename,renameat,renameat2 -o /tmp/brain-repro-audit-2/2403-native.strace bun run test packages/ui-server/tests/brain-application-adapters.test.ts

    Explicit environment: TMPDIR=/dev/shm/odysseus-repro2.

    The complete current file plus syscall trace differs from the earlier isolated Pi pass and incomplete 158-file history. The original broad failure and historical candidate remain separate; this does not establish a database-lock cause or a repair.

    This is focused local evidence, not historical-run equivalence, a repair or full candidate proof. Existing assertions, deadlines and offline guards were retained. Temporary source instrumentation was restored byte for byte. Keep needs: repro; no readiness or closure change. Full log, command/exit receipt and diagnostic patch are retained in task-owned scratch.

  8. schlessera commented on Oct 18, 2026

    @schlessera
    OwnerAuthor

    2026-10-18 — third reproduction pass at 775eb17aab740fa759c2182f306c38fc17acbcbe.

    Different approach: Hold a real SQLite BEGIN IMMEDIATE writer on the authoritative fixture brain.db only during the actual boundary planner index call, release it in finally, and retain the complete add result and durable receipt before the unchanged indexed=true assertion. This supplies a concrete lock condition instead of another isolated successful syscall trace; native copied-view behavior may distinguish the actual indexed target.

    Observed: The actual Pi curated add/update/archive case passes all 15 assertions despite an owned authoritative brain.db BEGIN IMMEDIATE writer held around each real planner index call. Add result and recorded receipt both report ok=true/indexed=true. The native fixture uses a copied execution view; this does not establish contention on the database actually indexed by the worker. Outer invocation exit 0, elapsed 1.518 s; outer watchdog fired: false.

    Command (working directory: repository root; any diagnostic source changes are described above):

    unshare --user --map-current-user --keep-caps --mount --net sh /tmp/brain-repro-audit-3/offline-mounted.sh bun run test packages/ui-server/tests/brain-application-adapters.test.ts -t 'pi hosted curated tools capture'

    Explicit environment: BRAIN_REQUIRE_CHROME=1, PUPPETEER_EXECUTABLE_PATH=$PLAYWRIGHT_CHROME, DEBUG=pw:browser, TMPDIR=/tmp/brain-repro-audit-3/native-state.

    Original historical failure and broad qualification remain unresolved. The supplied lock did not produce partial indexing, so no database-lock cause or remedy is inferred.

    This is focused local evidence, not historical-run equivalence, a repair or full candidate proof. Existing assertions, deadlines and offline guards were retained. Temporary source instrumentation was restored byte for byte. Keep needs: repro; no readiness or closure change. Full log, command/exit receipt and diagnostic patch are retained in task-owned scratch.

  9. self-assigned this
    on Oct 18, 2026
  10. schlessera commented on Oct 18, 2026

    @schlessera
    OwnerAuthor

    2026-10-19 campaign claim release044-2403-r2-20261019, intended branch repro/2403-campaign-r2.

    Fresh readback confirms the issue is open and unassigned, and the previous claim released custody. Bounded attribution for post-write partial indexing in the hosted-adapter proof. Recover original artifacts, inspect the actual copied-view indexing target and retain full application/indexError/maintenance receipt evidence. No unchanged 158-file retry, speculative database-lock repair, weaker indexed assertion or inferred common cause with #2396 is authorized. Keep needs: repro until causal evidence establishes readiness.

    A dedicated GPT-6.1-Sol worker owns this scoped continuation. The campaign coordinator serializes claims, local heavy-runtime access, final hosted confirmation and merge windows. No full current-candidate proof or delivery is claimed. Other issue branches and stopped investigations remain untouched.

  11. added
    priority: p2Wanted, not scheduled. Pulled in when a milestone has room.
    and removed
    priority: p1Scheduled into the current or next milestone.
    on Oct 21, 2026
  12. schlessera commented on Oct 21, 2026

    @schlessera
    OwnerAuthor

    Priority lowered to p2 and #1364's block dropped (maintainer-approved triage, 2026-10-21). This is one local partial-index (indexed=false) observation from 2026-10-18, in the leaked-mount period, with no hosted occurrence; five attempts did not reproduce it. Since #2426 (merged 2026-10-20), a recurrence records its own indexError diagnostics. It was p1 only because it blocked #1364. #1364 now proceeds on its own proof, and this stays open as an ordinary p2 attribution item.

  13. removed this from the 0.44.0 milestone on Oct 22, 2026
  14. schlessera commented on Oct 22, 2026

    @schlessera
    OwnerAuthor

    Maintainer ruling 2026-10-22: moved out of 0.44.0, so this is now unscheduled. There is no reproduction. The partial index failure was seen once, locally, during #1364 qualification. Bounded attempts, including a combined-order replay at the frozen head, did not reproduce it. The diagnostics from #2426 (c488979) now make a recurrence report its indexError. Since 2026-10-21 it no longer blocks #1364. needs: repro and priority: p2 stay. Schedule it again if the new diagnostics fire.

  15. schlessera commented on Oct 22, 2026

    @schlessera
    OwnerAuthor

    2026-10-22 triage: folded the later unscheduling ruling into the active Contract impact and diagnostic-delivery brief. This p2 needs: repro investigation has no milestone and does not block #1364; the delivered #2426 diagnostic keeps its own historical attribution. Its current failure-only indexFailure/maintenance observations are present at inspected main aa8c896. No natural recurrence, cause, repair or readiness is inferred.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area: ciCI workflows, the gates they run, release scripts, packaging.area: ui-serverHono app factory: turn coordinator, auth, session catalog, routes, cron.needs: reproBlocked on a reproduction.priority: p2Wanted, not scheduled. Pulled in when a milestone has room.type: fixSomething is broken and must behave the way it is documented.

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions