Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 119 additions & 43 deletions .dev-loop/INGEST_REPORT.md
Original file line number Diff line number Diff line change
@@ -1,53 +1,129 @@
# Knowledge consolidation15 open PRs (#17–#40) → one reconciled state
# Knowledge flush4 insight(s)

The 15 open `knowledge/*` PRs (created 2026-08-04 → 2026-08-05, before the
harvest processed-store dedupe fix in #41) contained 123 file-versions of ~75
unique pages, with the same insight landing at up to 3 different paths across
up to 8 PRs. Per-PR review would re-import those duplicates, so — as with the
#6–#13 consolidation — this branch carries the reconciled end-state and the 15
PRs are closed in its favor.
Queue contained 5 pending rows across 4 session files; two rows (hashes
`e44b870e42ca9d9a` / `f5bba19794bcdae8`) were English/Korean duplicates of the
same insight harvested in one session, folded into one candidate. Net: 4 unique
insights, all processed.

## Verified best-practice

Every adopted page's sources were carried from its originating PR's flush, where
they were live-verified at flush time; no new URLs were introduced during
consolidation (checked mechanically: every `http(s)` URL in every merged page
appears in a source PR's diff; every added body line in amended pages traces to
a source PR hunk — orphan-line verification). Confidence fields were kept as the
originating flushes set them, except client-side-rate-limiting where the union
of provider-doc citations (Okta, Auth0, GitHub, OpenAI, RFC 6585) supports
`verified` for the load-bearing claims. One subagent's fabricated content (12
files matching neither main nor any PR, with invented source URLs) was detected
by the same verification and replaced with true PR content.
**1. Gitignored coordinator-state dirs are absent from git worktrees — hand
workers absolute paths.**
Claim: a repo-relative path under a gitignored directory (`.orchestration/`)
resolves in the coordinator's main checkout but silently misses in every
worker's worktree, because `git worktree add` checks out tracked files only.
Verified by local reproduction this session (git, macOS): repo with
`.gitignore: .orchestration/` + populated `.orchestration/status/`;
`git worktree add ../wt1 -b wt1` → `ls ../wt1/.orchestration` = "No such file
or directory", `cat .orchestration/status/run.json` from the worktree cwd
failed, `git ls-files .orchestration` = 0. Consistent with
https://git-scm.com/docs/git-worktree (linked worktrees are separate checkouts
of the branch). **Confidence: verified.**

**2. Verify a documented EDR claim on the host before reasoning from it.**
Claim: docs asserting "this host runs SentinelOne/EDR" must be checked against
the host (vendor dir + process grep + `systemextensionsctl list` together);
all-empty means "not installed", not "failed to detect". Mechanism verified
against https://www.elastic.co/blog/mac-system-extensions-for-threat-detection-part-3
and https://github.com/redcanaryco/mac-monitor/wiki/5.-Endpoint-Security-Overview
(post-kext-deprecation, macOS EDRs ship Endpoint Security clients as system
extensions, enumerable via `systemextensionsctl list`); all three commands
re-run this session on this macOS host (Darwin 25.1.0) reproducing the
incident's empty results. Field incident: root XMRig ran 4d8h on a host whose
global CLAUDE.md claimed SentinelOne. **Confidence: verified.**

**3. Judge a miner-suspect process by executable path + signature, never by
name.**
Claim: malware adopts legitimate daemon names; macOS ships a genuine
`/usr/libexec/sysmond`, so name-based kills hit the Apple daemon or miss the
miner. Verified this session: `ls -l /usr/libexec/sysmond` (root:wheel),
`man -w sysmond` → man8 page, `codesign -vv` → valid / satisfies Designated
Requirement, `codesign -dv` → `Identifier=com.apple.sysmond`,
`Authority=Software Signing`. Technique documented at
https://attack.mitre.org/techniques/T1036/005/ ("giving it the name of a
legitimate, trusted program", fetched and confirmed this session). Field
incident: quarantined `~/.config/sysmond` was XMRig 6.26.0 by its own log
header. **Confidence: verified.**

**4. For prod-only failures the client swallows, grep the service logs for the
endpoint path before reading more code.**
Claim: when the frontend `.catch()`-swallows errors, a 500 and a no-op are
indistinguishable from the UI; one server-side exception line kills hypothesis
families. Field evidence: chungyak-alimi bookmark bug — backend/proxy/browser
all verified normal, then one `journalctl | grep bookmark` surfaced
PostgreSQL's "no unique or exclusion constraint matching the ON CONFLICT
specification" → deployed DB on an old schema. The PostgreSQL mechanism
(ON CONFLICT arbiter inference raises an error without a matching unique
index) confirmed against https://www.postgresql.org/docs/current/sql-insert.html.
The directive itself is production experience aligned with
https://sre.google/sre-book/effective-troubleshooting/ (already a page source).
**Confidence: field-tested** (marked as such in the added source line).

## Existing-layer check

- Merged-main near-dup scan before consolidation: pairwise Jaccard over
title + "When this applies" across all 141 merged pages → **0 flagged pairs**;
previously merged content carries no duplication.
- Cross-PR dedup during consolidation: 10 duplicate clusters collapsed to one
canonical page each (rate limiting 8→1, call-site enumeration 7→folded into
the canonical merged in #20, stderr/exit-0 diagnostics 4→1, sysroot 2→1,
env-off-switch 2→1, completion predicates 2→1, robots.txt 2→1,
harness-mediated results 2→1, leaked artifacts 2→1, orchestration category
naming unified). Three near-pairs kept distinct after trigger comparison,
with mutual `related:` links (differential setup vs interpretation; expansion
semantics vs off-switch design; import-time tactics vs level choice).
- 24 existing pages received union-merged amendments; additions already present
in main (from #16/#20) were skipped, and all non-canonical `related:` ids
were remapped to canonical page ids (post-merge broken-link scan: 0).
Pages read: infrastructure-agent-orchestration-worktree-isolated-workers, infrastructure-agent-orchestration-shared-run-state, debugging-methodology-reproduce-first, debugging-signals-logs-and-correlation

Also read: root `INDEX.md`, `wiki/infrastructure/index.md`,
`wiki/security/index.md`, `wiki/debugging/index.md` (routing), and
`skills/wiki-ingest/SKILL.md` + `templates/page.md` + `AGENTS.md` format rules.

- Insight 1 overlaps `worktree-isolated-workers` (same trigger family, new
failure mode) → **merged** there: +1 edge case, +1 instead-of row, +1
reproduction source. No conflict: the page's "write produced paths
worktree-relative" directive concerns worker *output*; this case concerns
coordinator-state paths workers must *read*, and reads of absolute main-root
paths pass the guardrail per the page's own table. Related link added
worktree-isolated-workers → shared-run-state (reverse link already existed).
- Insight 4 overlaps `reproduce-first`'s prod-only evidence row → **merged**
there: +1 edge case (silent-swallow), +1 field source. Related links now
bidirectional with `logs-and-correlation` (that page already pointed at
reproduce-first in an edge-case row).
- Insights 2–3: read every security category in `wiki/security/index.md`
(input, api-exposure, authn, authz, secrets, dependencies, data) — all cover
designing trust boundaries in code; none covers live host-compromise triage.
No duplicate, no conflicting directive. → **new pages** (see Routing).

## Open-PR check

Listed 25 open `knowledge/*` heads (#47–#91). Same-branch heads fetched
directly; fork PRs (#49, #52, #72, #73, #74, #76, #78, #86, #91) fetched via
`pull/<n>/head`. All 25 diffs against `origin/main -- wiki/` were concatenated
and keyword-swept (gitignore, EDR, sentinel, sysmond, xmrig, miner, incident,
systemextensionsctl, swallow, `.catch(`, server/production log, prod-only).

- Insight 1: #47 and #51 also edit `worktree-isolated-workers.md`, but both add
guardrail read-escalation version behavior — no overlap with the
gitignored-dir-absent-in-worktree case (their hunks inspected in full).
Verdict: **new** (file-level merge conflict possible with #47/#51; content
disjoint — flagged here for the owner's merge ordering).
- Insight 2: EDR/SentinelOne keyword hits were pre-existing removed lines in
#47/#51's `permissions-and-exec-bits` context, unrelated. Verdict: **new**.
- Insight 3: no hit for sysmond/miner/masquerade in any open head. Verdict:
**new**.
- Insight 4: no open head touches `reproduce-first.md` or
`logs-and-correlation.md`; silent-failure hits were unrelated env-var pages.
Verdict: **new**.

No candidate was folded into an in-flight branch; none dropped as a pending
duplicate.

## Routing decision

- New categories: `infrastructure/agent-orchestration` (5 pages; unified the
competing `orchestration`/`agent-orchestration` names), `databases/data-survey`
(1), `qa/deliverables` (1). All other pages route into existing categories.
- Canonical-path decisions: rate limiting → `backend/common/reliability/`
(sits beside timeouts-and-retries; 6 of 8 variants chose it); stderr
diagnostics → `platforms/processes/` (concern spans beyond shells); leaked
artifacts → `testing/data/artifact-leakage-from-a-suite`; call-site
enumeration → the existing `backend/common/change-impact/` page.
- All 38 new pages listed in their domain indexes (nearest-index rule; backend
routes via its python sub-index for bytecode-cache-staleness); INDEX.md domain
summaries updated for infrastructure/qa/databases. Full-wiki lint: frontmatter,
ids, related-links, index coverage, size, qualifiers, staleness → 0 findings.
- Insight 1 → `infrastructure/agent-orchestration/worktree-isolated-workers`
(merge; harvested domain hint "infrastructure" confirmed).
- Insight 2 → `security/incident-response/verifying-assumed-security-agents`
(**new category** `incident-response`): existing security categories all
govern designing/reviewing trust boundaries in code (input, api-exposure,
authn, authz, secrets, dependencies, data); responding to a live host
compromise fits none of them, and debugging/platforms were rejected
(debugging owns diagnosing code failures; platforms owns cross-OS code
breakage, not threat triage). Domain index + root INDEX.md security row
updated.
- Insight 3 → `security/incident-response/process-identity-by-path-and-hash`
(same new category; the two pages cross-reference via `related:`).
- Insight 4 → `debugging/methodology/reproduce-first` (merge; harvested domain
hint "debugging" confirmed).

Queue retirement: all 5 rows (4 insights + 1 language duplicate) appended to
`~/.dev-loop/queue/.processed.jsonl` and their session files rewritten/deleted
after PR creation.
2 changes: 1 addition & 1 deletion INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ follow the cross-pointers in their index or take the next matching seeded domain
| [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) |
| [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, exploratory testing (guarded-path coverage, override matrices), scope-purity gates, sourcing deliverable documents from generated artifacts, automated verification of document deliverables (spec/RFC gates) (writing automated test code → testing) |
| [debugging](wiki/debugging/index.md) | **seeded** | Diagnosing a failure — finding what is wrong and why: reproducing, bisection, hypothesis testing, traces/logs, intermittent failures (fixing the diagnosed fault → its owning domain) |
| [security](wiki/security/index.md) | **seeded** | Trust-boundary decisions: input validation, session-vs-token auth choice, per-resource authorization (IDOR), secrets hygiene, dependency trust, PII handling (XSS rendering → frontend; CI secrets → infrastructure; JWT implementation → backend/frontend auth) |
| [security](wiki/security/index.md) | **seeded** | Trust-boundary decisions: input validation, session-vs-token auth choice, per-resource authorization (IDOR), secrets hygiene, dependency trust, PII handling; host-compromise triage / incident response (verifying assumed security agents, identifying masquerading processes) (XSS rendering → frontend; CI secrets → infrastructure; JWT implementation → backend/frontend auth) |
| [platforms](wiki/platforms/index.md) | **seeded** | OS-level differences breaking code across macOS/Linux/Windows: shell portability, BSD-vs-GNU CLI, filesystem case/line endings, Unicode normalization in text/file-name matching, commands inspected before execution, background services/cron, invoking prompt-capable CLIs non-interactively, toolchain version pinning |
| [mobile](wiki/mobile/index.md) | **seeded** | App-side iOS/Android/cross-platform: process death/state survival, offline-first sync, mobile-network calls, store rollout/hotfix strategy, startup time |

Expand Down
1 change: 1 addition & 0 deletions log.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,4 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
## [2026-08-06] revise | Union-merged amendments from the same 15 PRs into 24 existing pages (largest: tests-that-cannot-fail — 7 PRs: per-assertion mutation granularity, restore-mechanism-by-commit-state, suite-total-drop detection; portable-shell-scripts — 8 PRs: POSIX inline set-- reordering, ${VAR:-} vs ${VAR-} colon semantics, quote-by-what-the-text-is; test-data-and-isolation — 6 PRs: env-derived write paths, absent-variable cases, exec-bit fixtures, leak attribution; change-impact/call-site-enumeration — 8 unique additions from 7 path-variants: rename-in-place, same-type reorder, *args forwarding, two-searches-agree fallacy, codemod, coverage completion check). All non-canonical cross-page ids remapped to canonical pages; every added line traced to its source PR hunk (fabricated worker output was detected by orphan-line verification and replaced with true PR content).
## [2026-08-06] dedup | Cross-PR duplicate clusters collapsed: client-side rate limiting (8 PR versions at 3 paths → 1), call-site enumeration (7 versions at 7 paths → additions folded into the already-merged canonical page), stderr/exit-0 diagnostics (4 versions at 4 paths → 1), macOS sysroot (2→1), env-var off switch (2→1), completion predicates (2→1), robots.txt (2→1), harness-mediated results (2→1), leaked test artifacts (2→1), orchestration category naming (orchestration vs agent-orchestration → agent-orchestration). Kept distinct after trigger comparison: differential-testing vs differential-run-agreement (setup vs interpretation), unset-versus-empty-parameters vs env-var-off-switches (semantics vs design, cross-linked), import-time-side-effects vs test-level-choice edge (tactics vs level choice, cross-linked). Merged-main near-dup scan (Jaccard over title+trigger, 141 pages): 0 pairs — no duplication among previously merged content.
## [2026-08-06] ingest | Reconciled post-consolidation flush PRs #42–#43 (created against pre-#44 main). Folded unique content: platforms/filesystems/permissions-and-exec-bits (+`sh "$SCRIPT"` interpreter-invocation rows — stubs need no exec bit, avoids EDR chmod+x heuristics), platforms/shells/command-text-inspected-before-execution (+gate-AUTHOR step 8: parse all three POSIX quoting forms, expand only ~/$HOME/${HOME}; red-then-green bats evidence), testing/quality/checks-that-cannot-pass (+vacuously-green pre-implementation test row), platforms/processes/tool-diagnostics-without-a-failing-exit-code (+repeat-suppression edge, +clang reproduction source), infrastructure/ci-cd/secrets-handling (+independent credential channels row: gh token vs git SSH vs API token, git push --dry-run / author:@me probes), testing/mocking/what-to-mock (+ESM read-only-bindings DI rows, +PATH-emptied tripwire negative-proof). Dropped as duplicates of the #44 state: #42's rate-limit row (already in timeouts-and-retries + client-side-rate-limiting), #42's warning-only-diagnostics page (same trigger as tool-diagnostics-without-a-failing-exit-code), #42's leak-attribution row (already in test-data-and-isolation).
## [2026-08-13] ingest | knowledge-flush of 5 queued rows (4 unique insights; one row was a same-session Korean duplicate). New category security/incident-response +2 pages: verifying-assumed-security-agents (a documented EDR claim is verified on the host — vendor dir + process grep + `systemextensionsctl list` together, all-empty ⇒ "not installed" not "failed to detect"; derived from the SentinelOne-claimed / XMRig-4d8h incident, commands re-run this session) and process-identity-by-path-and-hash (a miner-suspect name matching a system daemon is judged by executable path + codesign, never name — genuine `/usr/libexec/sysmond` vs `~/.config/sysmond` XMRig; MITRE T1036.005). Merged: infrastructure/agent-orchestration/worktree-isolated-workers +1 edge case / +1 instead-of / +1 source (a repo-relative path to a gitignored coordinator-state dir resolves only in the main checkout — `git worktree add` never materializes ignored dirs; substitute absolute paths into worker briefs; local worktree reproduction) and debugging/methodology/reproduce-first +1 edge case / +1 field source (prod-only failure the client swallows: grep the service logs for the endpoint path before reading more code; ON CONFLICT schema-drift field case). Related links added both ways (worktree↔shared-run-state, reproduce-first↔logs-and-correlation, the two new pages to each other).
6 changes: 4 additions & 2 deletions wiki/debugging/methodology/reproduce-first.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ sources:
- https://sscce.org/
- https://www.debuggingbook.org/html/DeltaDebugger.html
- https://sre.google/sre-book/effective-troubleshooting/
last_verified: 2026-07-10
related: [debugging-methodology-hypothesis-testing, debugging-concurrency-intermittent-failures]
last_verified: 2026-08-13
related: [debugging-methodology-hypothesis-testing, debugging-concurrency-intermittent-failures, debugging-signals-logs-and-correlation]
---

# Building a Reproduction Before Investigating a Bug
Expand Down Expand Up @@ -54,6 +54,7 @@ When a full local reproduction is impossible, capture evidence instead:
| Reproduction needs data you are not allowed to copy | Reproduce the shape, not the content: synthesize data matching the schema, volume, and the specific values named in the failure (nulls, empty lists, boundary sizes) |
| The report names the exact line to fix | Reproduce anyway before editing; a reproduction that survives the claimed fix disproves the report's diagnosis cheaply |
| Bug reproduces only on the reporter's machine | Diff the two environments one variable at a time — versions, locale, config — moving your environment toward theirs until it fails ([debugging-methodology-isolate-by-bisection]) |
| Prod-only failure with no visible error — the client swallows it (a `.catch()` that ignores, an empty error handler) and the action just "does nothing" | Grep the production service logs for the endpoint path before reading more code: from the UI a 500 and a no-op are indistinguishable, and one server-side exception line kills whole families of hypotheses that local code reading cannot ([debugging-signals-logs-and-correlation]) |

## Instead of

Expand All @@ -68,3 +69,4 @@ When a full local reproduction is impossible, capture evidence instead:
- https://sscce.org/ — minimal, self-contained example discipline
- https://www.debuggingbook.org/html/DeltaDebugger.html — systematically reducing failure-inducing inputs
- https://sre.google/sre-book/effective-troubleshooting/ — "simplify and reduce"; reproduction as the basis of diagnosis
- Field context 2026-08 (silent-swallow row, field-tested): a prod-only bookmark bug where backend code, proxy, and browser click were all verified normal from the outside; one `journalctl | grep bookmark` surfaced PostgreSQL's "no unique or exclusion constraint matching the ON CONFLICT specification", pinning the cause to a deployed DB left on an old schema — a cause invisible in the repo's code
Loading
Loading