knowledge: python data-file packaging (new backend/python/packaging), orchestrator-injected env in test suites, reflow vs substring anchors (3 verified, mechanism corrected, 1 half folded into #47)

.dev-loop/INGEST_REPORT.md

@@ -1,53 +1,45 @@
-# Knowledge consolidation — 15 open PRs (#17–#40) → one reconciled state
-
-The 15 open `knowledge/*` PRs (created 2026-08-04 → 2026-08-05, before the
-harvest processed-store dedupe fix in #41) contained 123 file-versions of ~75
-unique pages, with the same insight landing at up to 3 different paths across
-up to 8 PRs. Per-PR review would re-import those duplicates, so — as with the
-#6–#13 consolidation — this branch carries the reconciled end-state and the 15
-PRs are closed in its favor.
+# Knowledge flush — 3 insight(s)
 
 ## Verified best-practice
 
-Every adopted page's sources were carried from its originating PR's flush, where
-they were live-verified at flush time; no new URLs were introduced during
-consolidation (checked mechanically: every `http(s)` URL in every merged page
-appears in a source PR's diff; every added body line in amended pages traces to
-a source PR hunk — orphan-line verification). Confidence fields were kept as the
-originating flushes set them, except client-side-rate-limiting where the union
-of provider-doc citations (Okta, Auth0, GitHub, OpenAI, RFC 6585) supports
-`verified` for the load-bearing claims. One subagent's fabricated content (12
-files matching neither main nor any PR, with invented source URLs) was detected
-by the same verification and replaced with true PR content.
+**1. `__file__`-relative data files break after non-editable install → package data + `importlib.resources`** (hash `a9e3db01b9b1ceb1`, linkly session)
+- Claim: a Python package locating runtime data files via `__file__`-relative paths works in editable/`PYTHONPATH` runs but fails after `pip install .`; data files must be declared as package data and resolved via `importlib.resources`.
+- Sources checked (fetched this flush): https://setuptools.pypa.io/en/latest/userguide/datafiles.html — "It is strongly recommended that, if you are using data files, you should use `importlib.resources` to access them"; `__file__` manipulation "isn't compatible with PEP 302-based import hooks, including importing from zip files"; `include_package_data`/`package_data` control wheel contents. https://docs.python.org/3/library/importlib.resources.html — resources "do not have to exist as physical files and directories"; `as_file()` yields a real path and cleans up extractions.
+- Field evidence: linkly v0.4.0 `lnpl build` rc=4, grammar path resolved to `.venv/lib/python3.13/mlir/lnpl.irdl.mlir` (absent from the wheel) while `PYTHONPATH=impl` built fine.
+- **Confidence: verified** (official docs + field reproduction).
+
+**2. Orchestrator-injected coordination env leaks into harness test suites** (hash `03b0983bb4a5d9d8`, dev-loop session)
+- Claim: when an orchestration worker runs the harness's own test suite, injected `LO_*`/`GROUNDWORK_*` vars leak into the scripts under test — tests fail against the unset-env baseline AND test writes land in the live run's shared state (watcher then monitors the wrong session). Unset injected vars in setup; pass tempdir state paths explicitly.
+- Sources checked: dev-loop issue #100 (OPEN, root cause section marked "verified 2026-08-14") — 6 deterministic `launch-session.bats` failures reproduced with `env LO_RUN_ID=… bats` on a clean checkout; live `t90.json` found rewritten with bats tempdir paths and a foreign session name. The env-merge mechanism (`env` merges into the inherited environment unless `-i`) is already sourced on the target page (pubs.opengroup.org env spec).
+- **Confidence: field-tested** (reproducible in-repo evidence; no additional external doc needed beyond what the page already cites).
+
+**3. Reflowing prose that tests assert as a single-line substring** (hash `4b8d97490dc0ed5e`, dev-loop session)
+- Claim as harvested: "macOS bash 3.2 matches a newline-split phrase in `[[ ]]`, so only ubuntu CI fails."
+- **Mechanism correction — the harvested claim is false.** Measured this flush on bash 3.2.57: `[[ "$s" == *"return to step 1"* ]]` does NOT match a newline-split phrase (NOMATCH on both old and new bash). What actually differs by platform: `set -e; [[ 1 -eq 2 ]]; echo REACHED` prints REACHED on bash 3.2 — under bash ≤4.0 a failing mid-test `[[ ]]` doesn't abort, so the broken assertion passes silently on macOS while bash ≥4.2 CI fails it. `tests/orchestrate-review-pass.bats:81` uses exactly this mid-test `[[ ]]` form.
+- Field evidence: two reproductions (dev-loop PR #94 §O3; PR #102 'return to step 1 of the dispatch', fixed by reflow commit 9cbc065).
+- **Confidence: verified** (directive verified with the corrected mechanism; the wrong mechanism was NOT ingested).
 
 ## Existing-layer check
 
-- Merged-main near-dup scan before consolidation: pairwise Jaccard over
-  title + "When this applies" across all 141 merged pages → **0 flagged pairs**;
-  previously merged content carries no duplication.
-- Cross-PR dedup during consolidation: 10 duplicate clusters collapsed to one
-  canonical page each (rate limiting 8→1, call-site enumeration 7→folded into
-  the canonical merged in #20, stderr/exit-0 diagnostics 4→1, sysroot 2→1,
-  env-off-switch 2→1, completion predicates 2→1, robots.txt 2→1,
-  harness-mediated results 2→1, leaked artifacts 2→1, orchestration category
-  naming unified). Three near-pairs kept distinct after trigger comparison,
-  with mutual `related:` links (differential setup vs interpretation; expansion
-  semantics vs off-switch design; import-time tactics vs level choice).
-- 24 existing pages received union-merged amendments; additions already present
-  in main (from #16/#20) were skipped, and all non-canonical `related:` ids
-  were remapped to canonical page ids (post-merge broken-link scan: 0).
+Pages read: qa-document-verification-editing-a-gated-document, testing-data-test-data-and-isolation, infrastructure-agent-orchestration-shared-run-state, infrastructure-agent-orchestration-worktree-isolated-workers
+
+- Grepped main's `wiki/` for `__file__`/`importlib`: zero hits → insight 1 is not covered anywhere; **created new page** (a stale local-only branch `knowledge/dch0202-20260805-144711` once drafted a related page but has no PR and is not in main — not an open-PR obligation).
+- `testing-data-test-data-and-isolation` already carries the generic absent-variable `unset` row and the env-derived-write-path row; insight 2's *injected-orchestration-env → live-state corruption* angle was missing → **merged** (+1 Do row, +1 edge case, +1 field-incident source), related-links added both ways with `infrastructure-agent-orchestration-shared-run-state`. No conflict with existing directives.
+- `qa-document-verification-editing-a-gated-document` covers anchor inventory before editing; its Substring anchor row lacked the reflow/line-wrap breakage mode and the platform-invisible-failure twist → **merged** (extended Substring row, +1 edge case, +1 Instead-of row, +2 sources), cross-linking existing `testing-quality-tests-that-cannot-fail`. No conflict.
+- `node scripts/wiki-lint-prohibitions.js`: 61 directives, 0 violations after edits.
+
+## Open-PR check
+
+Listed 27 open `knowledge/*` heads (gh pr list, head:knowledge/) and grepped `git diff origin/main...<head> -- wiki/` across all fetched knowledge branches for `importlib|__file__|package data|LO_|GROUNDWORK_|ambient env|unset|bats|bash 3.2|normalize_ws|line-wrap|whitespace`.
+
+- Insight 1: no open head touches Python packaging/`__file__` → **new**.
+- Insight 2: no open head touches injected-env test isolation (the generic absent-var row in old branch `dch0202-20260805-103148` / closed PR #28 is already in main) → **new**.
+- Insight 3: **partial fold into PR #47** (`knowledge/dch0202-20260806-130040`) — #47 already carries the runtime mechanism (bats mid-test `[[ ]]` trap under bash ≤4.0, tests-that-cannot-fail row, COMPAT + bats-gotchas sources). That part is NOT re-ingested here; a comment noting the two additional field reproductions (PR #94/#102) is left on #47. The document-editing side (reflow breaks single-line substring anchors; whitespace-normalize new phrase gates) is absent from #47 and from main → merged into `editing-a-gated-document` here as **new**.
 
 ## Routing decision
 
-- New categories: `infrastructure/agent-orchestration` (5 pages; unified the
-  competing `orchestration`/`agent-orchestration` names), `databases/data-survey`
-  (1), `qa/deliverables` (1). All other pages route into existing categories.
-- Canonical-path decisions: rate limiting → `backend/common/reliability/`
-  (sits beside timeouts-and-retries; 6 of 8 variants chose it); stderr
-  diagnostics → `platforms/processes/` (concern spans beyond shells); leaked
-  artifacts → `testing/data/artifact-leakage-from-a-suite`; call-site
-  enumeration → the existing `backend/common/change-impact/` page.
-- All 38 new pages listed in their domain indexes (nearest-index rule; backend
-  routes via its python sub-index for bytecode-cache-staleness); INDEX.md domain
-  summaries updated for infrastructure/qa/databases. Full-wiki lint: frontmatter,
-  ids, related-links, index coverage, size, qualifiers, staleness → 0 findings.
+| Insight | Target | Decision |
+|---------|--------|----------|
+| 1 — data files & install paths | `backend/python/packaging/data-files-and-install-paths.md` | **New page, NEW category `packaging`** — existing backend/python categories (concurrency, boundaries, serving, language) cover runtime behavior, not distribution/wheel contents; harvested "platforms" hint rejected (platforms = OS-level differences, this is install-mode-level). backend/python index + backend index + root INDEX updated |
+| 2 — injected coordination env | `testing/data/test-data-and-isolation.md` | **Merge** — trigger is "how a suite isolates state", same page that owns the absent-var and env-write-path rows; orchestration cross-link to shared-run-state rather than a new agent-orchestration page (the actor is the test author, not the brief author) |
+| 3 — reflow vs substring anchors | `qa/document-verification/editing-a-gated-document.md` | **Merge** — exact trigger match ("editing a document automated text gates check"); gate-authoring half folded as guidance pointing at whitespace-normalized comparison; runtime-mechanism half deferred to open PR #47 (fold) |

INDEX.md

@@ -12,7 +12,7 @@ follow the cross-pointers in their index or take the next matching seeded domain
 | Domain | Status | Route here when |
 |--------|--------|-----------------|
 | [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior, surveying live data to derive a rule, verifying additive migrations |
-| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, call-site enumeration before a contract change, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, consuming LLM APIs (completion validation, context budgeting), consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) |
+| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, call-site enumeration before a contract change, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, consuming LLM APIs (completion validation, context budgeting), consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps, packaging data files with `importlib.resources`) |
 | [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility |
 | [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting), per-environment/path-valued config, multi-agent orchestration (worker liveness signals, shared run state, tmux pane delivery, completion gates, worktree-isolated workers) |
 | [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) |

log.md

@@ -44,3 +44,4 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
 ## [2026-08-06] dedup | Cross-PR duplicate clusters collapsed: client-side rate limiting (8 PR versions at 3 paths → 1), call-site enumeration (7 versions at 7 paths → additions folded into the already-merged canonical page), stderr/exit-0 diagnostics (4 versions at 4 paths → 1), macOS sysroot (2→1), env-var off switch (2→1), completion predicates (2→1), robots.txt (2→1), harness-mediated results (2→1), leaked test artifacts (2→1), orchestration category naming (orchestration vs agent-orchestration → agent-orchestration). Kept distinct after trigger comparison: differential-testing vs differential-run-agreement (setup vs interpretation), unset-versus-empty-parameters vs env-var-off-switches (semantics vs design, cross-linked), import-time-side-effects vs test-level-choice edge (tactics vs level choice, cross-linked). Merged-main near-dup scan (Jaccard over title+trigger, 141 pages): 0 pairs — no duplication among previously merged content.
 ## [2026-08-06] ingest | Reconciled post-consolidation flush PRs #42–#43 (created against pre-#44 main). Folded unique content: platforms/filesystems/permissions-and-exec-bits (+`sh "$SCRIPT"` interpreter-invocation rows — stubs need no exec bit, avoids EDR chmod+x heuristics), platforms/shells/command-text-inspected-before-execution (+gate-AUTHOR step 8: parse all three POSIX quoting forms, expand only ~/$HOME/${HOME}; red-then-green bats evidence), testing/quality/checks-that-cannot-pass (+vacuously-green pre-implementation test row), platforms/processes/tool-diagnostics-without-a-failing-exit-code (+repeat-suppression edge, +clang reproduction source), infrastructure/ci-cd/secrets-handling (+independent credential channels row: gh token vs git SSH vs API token, git push --dry-run / author:@me probes), testing/mocking/what-to-mock (+ESM read-only-bindings DI rows, +PATH-emptied tripwire negative-proof). Dropped as duplicates of the #44 state: #42's rate-limit row (already in timeouts-and-retries + client-side-rate-limiting), #42's warning-only-diagnostics page (same trigger as tool-diagnostics-without-a-failing-exit-code), #42's leak-attribution row (already in test-data-and-isolation).
 ## [2026-08-12] revise | routing: disjoint scopes for the doc-gate cluster (testing/quality ↔ qa/document-verification) and the flaky pair (testing/flaky ↔ debugging/concurrency); INDEX backend LLM phrasing; databases→backup cross-pointer (#37)
+## [2026-08-14] ingest | knowledge-flush of 3 queued insights. New: backend/python/packaging/data-files-and-install-paths (NEW category packaging — `__file__`-relative data files break after non-editable install; declare package data + resolve via importlib.resources files()/as_file(); verify against a wheel installed into a scratch venv; setuptools + CPython docs verified, linkly v0.4.0 rc=4 reproduction). Merged: testing/data/test-data-and-isolation +1 Do row / +1 edge case (orchestrator-injected coordination env — unset in setup, pass tempdir state paths explicitly; leaked values also corrupt the live run's shared state; dev-loop #100 reproduction) with related↔shared-run-state; qa/document-verification/editing-a-gated-document +reflow breakage mode on the Substring anchor row, +platform-invisible-failure edge (bash ≤4.0 mid-test `[[ ]]` gap), +Instead-of row, measured 2026-08-14 + dev-loop PR #94/#102 field reproductions. Mechanism correction during verification: the harvested claim "macOS bash 3.2 matches a newline-split phrase in [[ ]]" was falsified by measurement — the wrap breaks the match on every platform; macOS only hides the failure (errexit compound-command gap, cross-linked to tests-that-cannot-fail / open PR #47).

wiki/backend/index.md

@@ -8,7 +8,7 @@ three stack subtrees — route by concern first, stack second:
 | [common](#common-language-agnostic) (below) | The concern is language-agnostic: API contracts, enumerating call sites before a contract change, idempotency, JWT issuance, outbound calls, caching, jobs, transactions in app code, shared state/pools, exception structure, consuming LLM APIs (completion validation, context budgeting), consuming external-API responses, externally-owned defaults, object-storage references |
 | [java](java/index.md) | You are writing/reviewing JVM backend code (Java/Kotlin, Spring, JPA/Hibernate) and the concern is stack-specific: entity mapping, persistence context, proxy pitfalls, JVM threads/memory |
 | [node](node/index.md) | You are writing/reviewing Node.js/TypeScript backend code: event-loop blocking, promise error handling, runtime validation at boundaries, graceful shutdown |
-| [python](python/index.md) | You are writing/reviewing Python backend code: GIL/concurrency model, pydantic validation, WSGI/ASGI workers, language traps |
+| [python](python/index.md) | You are writing/reviewing Python backend code: GIL/concurrency model, pydantic validation, WSGI/ASGI workers, language traps, packaging data files and resolving them after install |
 
 Load a stack page IN ADDITION to the matching common page when both apply — common
 owns the principle, the stack page owns the mechanics. SQL, index, and DB-side

wiki/backend/python/index.md

@@ -24,6 +24,12 @@ Match your situation to a "load when" line; load only matching pages.
 |------|-----------|
 | [app-servers-and-workers](serving/app-servers-and-workers.md) | Deploying a Python web app behind gunicorn/uvicorn; choosing worker count, worker class (sync/gthread/ASGI), or worker timeout; requests queue or time out while CPU sits idle; workers killed mid-request; worker memory growth and preload/max_requests recycling decisions |
 
+## packaging
+
+| Page | Load when |
+|------|-----------|
+| [data-files-and-install-paths](packaging/data-files-and-install-paths.md) | A Python package reads non-code files (grammars, templates, KBs) located via `__file__`-relative paths; an installed console script cannot find a data file that exists in the repo; deciding how data files get into the wheel and how code resolves them after `pip install .` |
+
 ## language
 
 | Page | Load when |