knowledge: headless UA bot-block probe, estimate-derived threshold re-anchoring, permanent-suite scope evidence, shallow-clone history guard (4 verified)

.dev-loop/INGEST_REPORT.md

@@ -1,53 +1,53 @@
-# Knowledge consolidation — 15 open PRs (#17–#40) → one reconciled state
-
-The 15 open `knowledge/*` PRs (created 2026-08-04 → 2026-08-05, before the
-harvest processed-store dedupe fix in #41) contained 123 file-versions of ~75
-unique pages, with the same insight landing at up to 3 different paths across
-up to 8 PRs. Per-PR review would re-import those duplicates, so — as with the
-#6–#13 consolidation — this branch carries the reconciled end-state and the 15
-PRs are closed in its favor.
+# Knowledge flush — 4 insight(s)
 
 ## Verified best-practice
 
-Every adopted page's sources were carried from its originating PR's flush, where
-they were live-verified at flush time; no new URLs were introduced during
-consolidation (checked mechanically: every `http(s)` URL in every merged page
-appears in a source PR's diff; every added body line in amended pages traces to
-a source PR hunk — orphan-line verification). Confidence fields were kept as the
-originating flushes set them, except client-side-rate-limiting where the union
-of provider-doc citations (Okta, Auth0, GitHub, OpenAI, RFC 6585) supports
-`verified` for the load-bearing claims. One subagent's fabricated content (12
-files matching neither main nor any PR, with invented source URLs) was detected
-by the same verification and replaced with true PR content.
+**1. Headless-browser QA shows empty data → probe with a desktop Chrome UA before diagnosing an outage** (from chungyak-alimi session, 2026-08-14)
+- Claim: commercial sites' API gateways classify the headless default UA as a bot and reject only the data APIs (4xx) while the page shell/static assets load, so the failure masquerades as a server outage; swapping to a regular Chrome UA is the cheapest discriminating probe.
+- Verified: fetched Chromium source `headless/lib/browser/headless_browser_impl.cc` (raw.githubusercontent.com, 2026-08-14) — `const char kHeadlessProductName[] = "HeadlessChrome";` with the comment "Product name for building the default user agent string", i.e. the headless UA is distinguishable by construction. Checked developer.chrome.com/docs/chromium/headless (no UA statement there — not cited). Field evidence from the session: dabangapp.com/map/apt markers/room-list APIs all 400 under default headless UA, 200 after a Chrome/131 UA override.
+- Confidence: **verified** (source code + field reproduction).
+
+**2. Re-anchor SL/TP-style absolute triggers to the actual fill at the confirmation-recording point** (from auto-trading-bot session, 2026-08-14)
+- Claim: absolute triggers derived from a pre-execution price estimate survive slippage unchanged, collapsing the designed band; recompute them ratio-preserving from the actual fill, once, in the function that records the fill.
+- Verified: freqtrade.io/en/stable/stoploss/ (fetched) — stoploss defined as a ratio of entry ("a stoploss of -10% is placed exactly 10% below the entry point"), and with stoploss-on-exchange the order is placed after the buy order fills; en.wikipedia.org/wiki/Slippage_(finance) — decision-time vs execution price divergence is inherent to market orders. Live incident: 6 real positions exited TAKE_PROFIT at +0.06%–+0.42% against a +2% design; `mark_filled()` re-anchor restored the band, regression tests red on pre-fix code (this repo's PR #3, 464 tests green).
+- Confidence: **verified** (official docs pattern + reproduced incident + discriminating regression tests).
+
+**3. A permanent test suite must not assert ambient working-tree state; prove scope from the introducing commit's diff** (from dev-loop session, 2026-08-14)
+- Claim: the tree a permanent suite inspects is whoever-runs-it's in-progress state; any uncommitted sibling file fails it spuriously.
+- Verified: bazel.build/reference/test-encyclopedia (fetched) — "Tests should be hermetic: that is, they ought to access only those resources on which they have a declared dependency"; "If tests are not properly hermetic then they do not give historically reproducible results". Field evidence: dev-loop reviews/i83-insight-emission-r1.md — reproduced spurious failure, rewrite to commit-diff evidence → 521/521.
+- Confidence: **verified**.
+
+**4. Guard history-dependent checks with `git rev-parse --is-shallow-repository`; a depth-1 boundary commit reports every tracked file as added** (from dev-loop session, 2026-08-14)
+- Claim: actions/checkout's default `fetch-depth: 1` grafts a parentless boundary commit, so `git log --diff-filter` / `merge-base` answer falsely rather than erroring; skip honestly on shallow clones or deepen the fetch per job.
+- Verified: actions/checkout README ("Only a single commit is fetched by default"); git-scm.com/docs/git-rev-parse (`--is-shallow-repository`); **fresh local reproduction this session** (git 2.50.1, 2026-08-14): `--depth 1` clone of the 306-file dev-loop repo → `--is-shallow-repository` = true, `git log -1 --diff-filter=A --name-only` listed 306/306 files as added.
+- Confidence: **verified**.
 
 ## Existing-layer check
 
-- Merged-main near-dup scan before consolidation: pairwise Jaccard over
-  title + "When this applies" across all 141 merged pages → **0 flagged pairs**;
-  previously merged content carries no duplication.
-- Cross-PR dedup during consolidation: 10 duplicate clusters collapsed to one
-  canonical page each (rate limiting 8→1, call-site enumeration 7→folded into
-  the canonical merged in #20, stderr/exit-0 diagnostics 4→1, sysroot 2→1,
-  env-off-switch 2→1, completion predicates 2→1, robots.txt 2→1,
-  harness-mediated results 2→1, leaked artifacts 2→1, orchestration category
-  naming unified). Three near-pairs kept distinct after trigger comparison,
-  with mutual `related:` links (differential setup vs interpretation; expansion
-  semantics vs off-switch design; import-time tactics vs level choice).
-- 24 existing pages received union-merged amendments; additions already present
-  in main (from #16/#20) were skipped, and all non-canonical `related:` ids
-  were remapped to canonical page ids (post-merge broken-link scan: 0).
+Pages read: qa-process-scope-purity-checks, qa-environments-test-environment-parity, testing-e2e-e2e-stability, testing-quality-checks-that-cannot-pass, backend-common-integrations-robots-txt-and-source-selection
+
+- Insight 1: `qa/environments/test-environment-parity` is staging-vs-prod parity — different trigger; `backend/common/integrations/robots-txt-and-source-selection` covers UA group matching for crawler policy (adjacent, not duplicate) → **new page**, related-linked both ways to robots-txt and to `testing-e2e-e2e-stability`.
+- Insight 2: no backend/common page covers estimate-vs-actual anchoring (integrations pages cover externally-owned names and robots policy) → **new page** under the existing `integrations` category (the gap is between your request/estimate and the external system's confirmed outcome).
+- Insight 3: `qa/process/scope-purity-checks` owns proving scope purity from `git status` — same territory, complementary directive (which *evidence source* per gate lifetime) → **merged** into that page (Do #4 table, edge-case row, instead-of row, Bazel source, related link). No conflict: the existing page's working-tree guidance remains correct for one-shot gates that own their tree.
+- Insight 4: no existing page covers shallow-clone history semantics (`checks-that-cannot-pass` is unwritten-target gates; `harness-reverse-controls` is harness scoring) → **new page** `testing/quality/history-dependent-checks-on-shallow-clones`, related-linked both ways to scope-purity-checks and checks-that-cannot-pass.
+- Domain indexes updated: qa (new environments row; scope-purity "load when" extended), testing (new quality row), backend (new integrations row). `log.md` ingest entry appended. Prohibition lint: 62/62 directives compliant, 0 violations.
+
+## Open-PR check
+
+Listed 26 open `knowledge/*` heads (#47–#92) via `gh pr list --search "head:knowledge/"`, fetched all heads, and diffed each against `origin/main -- wiki/` grepping for overlap terms (user-agent/headless/bot-detect, shallow/fetch-depth, slippage/stop-loss/take-profit/re-anchor/fill price, working-tree/git status/diff-filter/scope-purity). Hits were incidental only:
+- #64 (choiyounggi-20260808-004155): "headless" in an AskUserQuestion-in-Docker citation; "working tree"/"git status" in orchestration liveness context — different insights.
+- #61 (choiyounggi-20260807-213244): "re-anchor" refers to test-fixture literal anchors — different concept.
+- #50/#51 heads: `git status` appears in worker-resume prompts and the harness-reverse-controls index line — different insights.
+
+Verdict per candidate: **all 4 new** (no fold, no drop). No sibling PR carries any of these insights.
 
 ## Routing decision
 
-- New categories: `infrastructure/agent-orchestration` (5 pages; unified the
-  competing `orchestration`/`agent-orchestration` names), `databases/data-survey`
-  (1), `qa/deliverables` (1). All other pages route into existing categories.
-- Canonical-path decisions: rate limiting → `backend/common/reliability/`
-  (sits beside timeouts-and-retries; 6 of 8 variants chose it); stderr
-  diagnostics → `platforms/processes/` (concern spans beyond shells); leaked
-  artifacts → `testing/data/artifact-leakage-from-a-suite`; call-site
-  enumeration → the existing `backend/common/change-impact/` page.
-- All 38 new pages listed in their domain indexes (nearest-index rule; backend
-  routes via its python sub-index for bytecode-cache-staleness); INDEX.md domain
-  summaries updated for infrastructure/qa/databases. Full-wiki lint: frontmatter,
-  ids, related-links, index coverage, size, qualifiers, staleness → 0 findings.
+| Insight | Target | Rationale |
+|---------|--------|-----------|
+| 1 headless UA bot-block | `qa/environments/headless-browser-bot-blocking` (new page, existing category) | Harvest hint qa; the failing dimension is the *test client environment* vs a real user's browser — environments category; exploratory/bug-reports rejected (it is a diagnosis-of-environment case, not a session-design or report-format case) |
+| 2 SL/TP re-anchor | `backend/common/integrations/estimate-derived-thresholds` (new page, existing category) | Language-agnostic server-side concern; `integrations` already owns consuming external systems' responses — the insight is precisely the estimate/actual gap in an external system's confirmed outcome. No new category needed |
+| 3 working-tree assertions | merge into `qa/process/scope-purity-checks` | Merge-before-create: same territory (proving scope purity), complementary directive; harvested "testing" hint re-routed to the page that owns the case (testing index already cross-points release-process → qa) |
+| 4 shallow-clone history | `testing/quality/history-dependent-checks-on-shallow-clones` (new page, existing category) | The artifact being written is automated-check code — testing/quality alongside checks-that-cannot-pass/harness-reverse-controls; infrastructure rejected (the fix lives in the check and its workflow stanza, not in pipeline design) |
+
+One PR, no auto-merge. Queue rows for all 4 candidates will be retired to `.processed.jsonl` after the PR opens.

log.md

@@ -44,3 +44,4 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
 ## [2026-08-06] dedup | Cross-PR duplicate clusters collapsed: client-side rate limiting (8 PR versions at 3 paths → 1), call-site enumeration (7 versions at 7 paths → additions folded into the already-merged canonical page), stderr/exit-0 diagnostics (4 versions at 4 paths → 1), macOS sysroot (2→1), env-var off switch (2→1), completion predicates (2→1), robots.txt (2→1), harness-mediated results (2→1), leaked test artifacts (2→1), orchestration category naming (orchestration vs agent-orchestration → agent-orchestration). Kept distinct after trigger comparison: differential-testing vs differential-run-agreement (setup vs interpretation), unset-versus-empty-parameters vs env-var-off-switches (semantics vs design, cross-linked), import-time-side-effects vs test-level-choice edge (tactics vs level choice, cross-linked). Merged-main near-dup scan (Jaccard over title+trigger, 141 pages): 0 pairs — no duplication among previously merged content.
 ## [2026-08-06] ingest | Reconciled post-consolidation flush PRs #42–#43 (created against pre-#44 main). Folded unique content: platforms/filesystems/permissions-and-exec-bits (+`sh "$SCRIPT"` interpreter-invocation rows — stubs need no exec bit, avoids EDR chmod+x heuristics), platforms/shells/command-text-inspected-before-execution (+gate-AUTHOR step 8: parse all three POSIX quoting forms, expand only ~/$HOME/${HOME}; red-then-green bats evidence), testing/quality/checks-that-cannot-pass (+vacuously-green pre-implementation test row), platforms/processes/tool-diagnostics-without-a-failing-exit-code (+repeat-suppression edge, +clang reproduction source), infrastructure/ci-cd/secrets-handling (+independent credential channels row: gh token vs git SSH vs API token, git push --dry-run / author:@me probes), testing/mocking/what-to-mock (+ESM read-only-bindings DI rows, +PATH-emptied tripwire negative-proof). Dropped as duplicates of the #44 state: #42's rate-limit row (already in timeouts-and-retries + client-side-rate-limiting), #42's warning-only-diagnostics page (same trigger as tool-diagnostics-without-a-failing-exit-code), #42's leak-attribution row (already in test-data-and-isolation).
 ## [2026-08-12] revise | routing: disjoint scopes for the doc-gate cluster (testing/quality ↔ qa/document-verification) and the flaky pair (testing/flaky ↔ debugging/concurrency); INDEX backend LLM phrasing; databases→backup cross-pointer (#37)
+## [2026-08-14] ingest | knowledge-flush of 4 queued insights. New: qa/environments/headless-browser-bot-blocking (page shell renders but data APIs alone 4xx under a headless browser — probe with a desktop Chrome UA before diagnosing an outage; Chromium builds the headless default UA from kHeadlessProductName="HeadlessChrome"; field repro dabangapp 400→200), backend/common/integrations/estimate-derived-thresholds (absolute SL/TP-style triggers derived from a pre-execution estimate must be re-anchored ratio-preserving at the single confirmation-recording point — live incident: +2% design band collapsed to +0.06–0.42% via slippage; freqtrade anchors stoploss to entry after fill), testing/quality/history-dependent-checks-on-shallow-clones (guard git-history checks with --is-shallow-repository and skip honestly, or deepen fetch-depth per job; local repro: depth-1 boundary commit reported 306/306 files as added). Merged: qa/process/scope-purity-checks +Do#4 evidence-source-by-gate-lifetime table, +permanent-suite edge/instead-of rows, +Bazel test-encyclopedia hermeticity source (a permanent suite asserting `git status` reads someone else's in-progress tree — prove scope from the introducing commit's diff or skip). All URLs live-checked this session; two local reproductions (Chromium source constant via raw fetch; git 2.50.1 shallow-clone diff-filter).

wiki/backend/common/integrations/estimate-derived-thresholds.md

@@ -0,0 +1,65 @@
+---
+id: backend-common-integrations-estimate-derived-thresholds
+domain: backend
+category: integrations
+applies_to: [general]
+confidence: verified
+sources:
+  - https://www.freqtrade.io/en/stable/stoploss/
+  - https://en.wikipedia.org/wiki/Slippage_(finance)
+  - "Live incident 2026-08-13/14 (KIS auto-trading bot): 6 positions exited TAKE_PROFIT at +0.06%–+0.42% against a +2% design; re-anchor in mark_filled() restored the band, regression tests red on pre-fix code"
+last_verified: 2026-08-14
+related: []
+---
+
+# Absolute Thresholds Derived from a Pre-Execution Estimate
+
+## When this applies
+
+You submit an action to an external system whose actual outcome can differ
+from the estimate you decided on (market order → fill price vs the quote at
+decision time), and you persist **absolute** trigger values derived from that
+estimate — stop-loss/take-profit prices, alert thresholds, budget cutoffs.
+The actual outcome arrives later as a separate confirmation event (fill
+report, webhook, reconciliation).
+
+## Do this
+
+1. **Persist the intent as ratios/offsets relative to the anchor** (e.g.
+   SL −2% / TP +2% of entry), alongside any precomputed absolutes. The ratio
+   is the durable design value; the absolute is a cache of it.
+2. **Re-anchor at the confirmation-recording point.** In the single function
+   that records the action as confirmed (`mark_filled()`, the fill-webhook
+   handler), recompute the absolute triggers from the actual outcome,
+   preserving the stored ratios. Fix it there — not per entry path: entry
+   paths multiply (daily run, intraday redeploy, manual), while every one of
+   them funnels through the confirmation recorder.
+3. **Treat the estimate/actual gap as expected behavior, not an error.**
+   Slippage — execution at a price different from the one at decision time —
+   is a normal property of market orders and widens at opens and in volatile
+   periods (Wikipedia: Slippage). Reference practice: freqtrade defines
+   stoploss as a ratio of the entry ("a stoploss of -10% is placed exactly
+   10% below the entry point") and places the exchange stoploss order only
+   after the buy order fills — the anchor follows the fill, not the quote.
+
+## Edge cases
+
+| Case | Then |
+|------|------|
+| Partial fills | Re-anchor on the volume-weighted average fill price — either at each fill event or once on completion; pick one and record which in the recorder |
+| The confirmation event can arrive twice (webhook redelivery, reconciliation re-run) | Make the re-anchor idempotent: derive from stored ratios + fill price, never by mutating the previous absolutes incrementally |
+| The estimate-derived absolute was already shown or notified to users | Recompute the display from the same anchor (or emit a correction); a UI still quoting the stale absolute contradicts the triggers actually armed |
+| The confirmation payload lacks the actual value (no fill price reported) | Keep the estimate-derived values and log loudly that triggers are estimate-anchored; silently treating the estimate as the actual hides the gap this page exists to close |
+
+## Instead of
+
+| If you are about to | Do this instead | Why |
+|---------------------|-----------------|-----|
+| Fix a stale-anchor bug in each entry path that computes SL/TP | Re-anchor once where the fill is recorded | Entry paths keep multiplying; the confirmation recorder is the funnel they all pass through |
+| Keep absolute triggers computed from the pre-order quote after the fill confirms | Recompute from the fill price, preserving the design ratios | Slippage/gap eats the margin: a +2% designed take-profit band collapsed to +0.06%–+0.42% observed triggers, exiting positions for less than fees |
+
+## Sources
+
+- https://www.freqtrade.io/en/stable/stoploss/ — stoploss defined as a ratio of the entry price ("a stoploss of -10% is placed exactly 10% below the entry point"); with stoploss-on-exchange, the stoploss order "is placed on the exchange immediately after buy order fills"
+- https://en.wikipedia.org/wiki/Slippage_(finance) — execution price differing from the decision-time price is inherent to market orders, larger under volatility
+- Live incident 2026-08-13/14, Korea Investment & Securities auto-trading bot: SL/TP absolutes computed from the pre-order signal price survived fills at higher prices; 6 real positions triggered TAKE_PROFIT at +0.06%–+0.42% against a +2% design. Re-anchoring in `mark_filled()` (ratio-preserving recompute from the actual fill price) restored the band; regression tests fail on the pre-fix code and pass after (464-test suite green)

wiki/backend/common/integrations/robots-txt-and-source-selection.md

@@ -11,7 +11,7 @@ sources:
   - https://housing.seoul.go.kr/robots.txt
   - https://apply.gh.or.kr/robots.txt
 last_verified: 2026-08-05
-related: [backend-common-integrations-externally-owned-defaults, backend-common-reliability-timeouts-and-retries]
+related: [backend-common-integrations-externally-owned-defaults, backend-common-reliability-timeouts-and-retries, qa-environments-headless-browser-bot-blocking]
 ---
 
 # Choosing a Source to Crawl by Reading robots.txt