From 19528ca44a4c97fcca579a3c5e8694c7ced6b982 Mon Sep 17 00:00:00 2001 From: choiyounggi <74581798+choiyounggi@users.noreply.github.com> Date: Fri, 14 Aug 2026 13:54:13 +0900 Subject: [PATCH] knowledge: ingest 4 verified insights (headless UA bot-block probe, estimate-derived threshold re-anchoring, permanent-suite scope evidence, shallow-clone history guard) --- .dev-loop/INGEST_REPORT.md | 88 +++++++++---------- log.md | 1 + .../estimate-derived-thresholds.md | 65 ++++++++++++++ .../robots-txt-and-source-selection.md | 2 +- wiki/backend/index.md | 1 + .../headless-browser-bot-blocking.md | 63 +++++++++++++ wiki/qa/index.md | 3 +- wiki/qa/process/scope-purity-checks.md | 26 +++++- wiki/testing/e2e/e2e-stability.md | 2 +- wiki/testing/index.md | 1 + .../quality/checks-that-cannot-pass.md | 2 +- ...tory-dependent-checks-on-shallow-clones.md | 63 +++++++++++++ 12 files changed, 266 insertions(+), 51 deletions(-) create mode 100644 wiki/backend/common/integrations/estimate-derived-thresholds.md create mode 100644 wiki/qa/environments/headless-browser-bot-blocking.md create mode 100644 wiki/testing/quality/history-dependent-checks-on-shallow-clones.md diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 55ccfd1..8e54191 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,53 +1,53 @@ -# Knowledge consolidation — 15 open PRs (#17–#40) → one reconciled state - -The 15 open `knowledge/*` PRs (created 2026-08-04 → 2026-08-05, before the -harvest processed-store dedupe fix in #41) contained 123 file-versions of ~75 -unique pages, with the same insight landing at up to 3 different paths across -up to 8 PRs. Per-PR review would re-import those duplicates, so — as with the -#6–#13 consolidation — this branch carries the reconciled end-state and the 15 -PRs are closed in its favor. +# Knowledge flush — 4 insight(s) ## Verified best-practice -Every adopted page's sources were carried from its originating PR's flush, where -they were live-verified at flush time; no new URLs were introduced during -consolidation (checked mechanically: every `http(s)` URL in every merged page -appears in a source PR's diff; every added body line in amended pages traces to -a source PR hunk — orphan-line verification). Confidence fields were kept as the -originating flushes set them, except client-side-rate-limiting where the union -of provider-doc citations (Okta, Auth0, GitHub, OpenAI, RFC 6585) supports -`verified` for the load-bearing claims. One subagent's fabricated content (12 -files matching neither main nor any PR, with invented source URLs) was detected -by the same verification and replaced with true PR content. +**1. Headless-browser QA shows empty data → probe with a desktop Chrome UA before diagnosing an outage** (from chungyak-alimi session, 2026-08-14) +- Claim: commercial sites' API gateways classify the headless default UA as a bot and reject only the data APIs (4xx) while the page shell/static assets load, so the failure masquerades as a server outage; swapping to a regular Chrome UA is the cheapest discriminating probe. +- Verified: fetched Chromium source `headless/lib/browser/headless_browser_impl.cc` (raw.githubusercontent.com, 2026-08-14) — `const char kHeadlessProductName[] = "HeadlessChrome";` with the comment "Product name for building the default user agent string", i.e. the headless UA is distinguishable by construction. Checked developer.chrome.com/docs/chromium/headless (no UA statement there — not cited). Field evidence from the session: dabangapp.com/map/apt markers/room-list APIs all 400 under default headless UA, 200 after a Chrome/131 UA override. +- Confidence: **verified** (source code + field reproduction). + +**2. Re-anchor SL/TP-style absolute triggers to the actual fill at the confirmation-recording point** (from auto-trading-bot session, 2026-08-14) +- Claim: absolute triggers derived from a pre-execution price estimate survive slippage unchanged, collapsing the designed band; recompute them ratio-preserving from the actual fill, once, in the function that records the fill. +- Verified: freqtrade.io/en/stable/stoploss/ (fetched) — stoploss defined as a ratio of entry ("a stoploss of -10% is placed exactly 10% below the entry point"), and with stoploss-on-exchange the order is placed after the buy order fills; en.wikipedia.org/wiki/Slippage_(finance) — decision-time vs execution price divergence is inherent to market orders. Live incident: 6 real positions exited TAKE_PROFIT at +0.06%–+0.42% against a +2% design; `mark_filled()` re-anchor restored the band, regression tests red on pre-fix code (this repo's PR #3, 464 tests green). +- Confidence: **verified** (official docs pattern + reproduced incident + discriminating regression tests). + +**3. A permanent test suite must not assert ambient working-tree state; prove scope from the introducing commit's diff** (from dev-loop session, 2026-08-14) +- Claim: the tree a permanent suite inspects is whoever-runs-it's in-progress state; any uncommitted sibling file fails it spuriously. +- Verified: bazel.build/reference/test-encyclopedia (fetched) — "Tests should be hermetic: that is, they ought to access only those resources on which they have a declared dependency"; "If tests are not properly hermetic then they do not give historically reproducible results". Field evidence: dev-loop reviews/i83-insight-emission-r1.md — reproduced spurious failure, rewrite to commit-diff evidence → 521/521. +- Confidence: **verified**. + +**4. Guard history-dependent checks with `git rev-parse --is-shallow-repository`; a depth-1 boundary commit reports every tracked file as added** (from dev-loop session, 2026-08-14) +- Claim: actions/checkout's default `fetch-depth: 1` grafts a parentless boundary commit, so `git log --diff-filter` / `merge-base` answer falsely rather than erroring; skip honestly on shallow clones or deepen the fetch per job. +- Verified: actions/checkout README ("Only a single commit is fetched by default"); git-scm.com/docs/git-rev-parse (`--is-shallow-repository`); **fresh local reproduction this session** (git 2.50.1, 2026-08-14): `--depth 1` clone of the 306-file dev-loop repo → `--is-shallow-repository` = true, `git log -1 --diff-filter=A --name-only` listed 306/306 files as added. +- Confidence: **verified**. ## Existing-layer check -- Merged-main near-dup scan before consolidation: pairwise Jaccard over - title + "When this applies" across all 141 merged pages → **0 flagged pairs**; - previously merged content carries no duplication. -- Cross-PR dedup during consolidation: 10 duplicate clusters collapsed to one - canonical page each (rate limiting 8→1, call-site enumeration 7→folded into - the canonical merged in #20, stderr/exit-0 diagnostics 4→1, sysroot 2→1, - env-off-switch 2→1, completion predicates 2→1, robots.txt 2→1, - harness-mediated results 2→1, leaked artifacts 2→1, orchestration category - naming unified). Three near-pairs kept distinct after trigger comparison, - with mutual `related:` links (differential setup vs interpretation; expansion - semantics vs off-switch design; import-time tactics vs level choice). -- 24 existing pages received union-merged amendments; additions already present - in main (from #16/#20) were skipped, and all non-canonical `related:` ids - were remapped to canonical page ids (post-merge broken-link scan: 0). +Pages read: qa-process-scope-purity-checks, qa-environments-test-environment-parity, testing-e2e-e2e-stability, testing-quality-checks-that-cannot-pass, backend-common-integrations-robots-txt-and-source-selection + +- Insight 1: `qa/environments/test-environment-parity` is staging-vs-prod parity — different trigger; `backend/common/integrations/robots-txt-and-source-selection` covers UA group matching for crawler policy (adjacent, not duplicate) → **new page**, related-linked both ways to robots-txt and to `testing-e2e-e2e-stability`. +- Insight 2: no backend/common page covers estimate-vs-actual anchoring (integrations pages cover externally-owned names and robots policy) → **new page** under the existing `integrations` category (the gap is between your request/estimate and the external system's confirmed outcome). +- Insight 3: `qa/process/scope-purity-checks` owns proving scope purity from `git status` — same territory, complementary directive (which *evidence source* per gate lifetime) → **merged** into that page (Do #4 table, edge-case row, instead-of row, Bazel source, related link). No conflict: the existing page's working-tree guidance remains correct for one-shot gates that own their tree. +- Insight 4: no existing page covers shallow-clone history semantics (`checks-that-cannot-pass` is unwritten-target gates; `harness-reverse-controls` is harness scoring) → **new page** `testing/quality/history-dependent-checks-on-shallow-clones`, related-linked both ways to scope-purity-checks and checks-that-cannot-pass. +- Domain indexes updated: qa (new environments row; scope-purity "load when" extended), testing (new quality row), backend (new integrations row). `log.md` ingest entry appended. Prohibition lint: 62/62 directives compliant, 0 violations. + +## Open-PR check + +Listed 26 open `knowledge/*` heads (#47–#92) via `gh pr list --search "head:knowledge/"`, fetched all heads, and diffed each against `origin/main -- wiki/` grepping for overlap terms (user-agent/headless/bot-detect, shallow/fetch-depth, slippage/stop-loss/take-profit/re-anchor/fill price, working-tree/git status/diff-filter/scope-purity). Hits were incidental only: +- #64 (choiyounggi-20260808-004155): "headless" in an AskUserQuestion-in-Docker citation; "working tree"/"git status" in orchestration liveness context — different insights. +- #61 (choiyounggi-20260807-213244): "re-anchor" refers to test-fixture literal anchors — different concept. +- #50/#51 heads: `git status` appears in worker-resume prompts and the harness-reverse-controls index line — different insights. + +Verdict per candidate: **all 4 new** (no fold, no drop). No sibling PR carries any of these insights. ## Routing decision -- New categories: `infrastructure/agent-orchestration` (5 pages; unified the - competing `orchestration`/`agent-orchestration` names), `databases/data-survey` - (1), `qa/deliverables` (1). All other pages route into existing categories. -- Canonical-path decisions: rate limiting → `backend/common/reliability/` - (sits beside timeouts-and-retries; 6 of 8 variants chose it); stderr - diagnostics → `platforms/processes/` (concern spans beyond shells); leaked - artifacts → `testing/data/artifact-leakage-from-a-suite`; call-site - enumeration → the existing `backend/common/change-impact/` page. -- All 38 new pages listed in their domain indexes (nearest-index rule; backend - routes via its python sub-index for bytecode-cache-staleness); INDEX.md domain - summaries updated for infrastructure/qa/databases. Full-wiki lint: frontmatter, - ids, related-links, index coverage, size, qualifiers, staleness → 0 findings. +| Insight | Target | Rationale | +|---------|--------|-----------| +| 1 headless UA bot-block | `qa/environments/headless-browser-bot-blocking` (new page, existing category) | Harvest hint qa; the failing dimension is the *test client environment* vs a real user's browser — environments category; exploratory/bug-reports rejected (it is a diagnosis-of-environment case, not a session-design or report-format case) | +| 2 SL/TP re-anchor | `backend/common/integrations/estimate-derived-thresholds` (new page, existing category) | Language-agnostic server-side concern; `integrations` already owns consuming external systems' responses — the insight is precisely the estimate/actual gap in an external system's confirmed outcome. No new category needed | +| 3 working-tree assertions | merge into `qa/process/scope-purity-checks` | Merge-before-create: same territory (proving scope purity), complementary directive; harvested "testing" hint re-routed to the page that owns the case (testing index already cross-points release-process → qa) | +| 4 shallow-clone history | `testing/quality/history-dependent-checks-on-shallow-clones` (new page, existing category) | The artifact being written is automated-check code — testing/quality alongside checks-that-cannot-pass/harness-reverse-controls; infrastructure rejected (the fix lives in the check and its workflow stanza, not in pipeline design) | + +One PR, no auto-merge. Queue rows for all 4 candidates will be retired to `.processed.jsonl` after the PR opens. diff --git a/log.md b/log.md index bd066cf..c898f07 100644 --- a/log.md +++ b/log.md @@ -44,3 +44,4 @@ Append-only. Format: `## [YYYY-MM-DD] `; when the environment cannot answer (no commit yet, shallow clone → [testing-quality-history-dependent-checks-on-shallow-clones]), skip explicitly rather than falling back to the tree | + + A permanent suite reading the ambient tree asserts someone else's + in-progress state — tests should access only resources they have a declared + dependency on, or they stop giving historically reproducible results (Bazel + Test Encyclopedia); any uncommitted sibling file fails the suite spuriously. ## Edge cases @@ -51,6 +67,7 @@ scope; or you are writing such a gate for an orchestration/CI workflow. | Paths with whitespace/nonprintable characters | Porcelain v1 quotes them as C string literals, so a plain path prefix no longer matches — use `-z` (NUL-terminated, no quoting) and split on NUL | | Purity must also cover ignored artifacts (build outputs, caches) | `git status` omits ignored files entirely; add `--ignored=matching` to list paths matching ignore patterns | | Gate runs in a fresh worktree/clone | Config differences travel with `$HOME`, not the repo — the explicit `-uall` flag is still required | +| A purity check in a permanent suite went red on a teammate's unrelated WIP file | The failure indicts the runner's tree, not the change under test — move the check to commit-diff evidence (Do #4) or out of the suite into a one-shot workflow gate | ## Instead of @@ -59,8 +76,11 @@ scope; or you are writing such a gate for an orchestration/CI workflow. | Filter default `git status --porcelain` output with per-file path patterns | Add `-uall` first | An entirely-untracked directory collapses to `?? dir/`, which file-level patterns cannot match | | Rely on the repo's ambient untracked-files default | Pass `-uall` explicitly in the gate script | `status.showUntrackedFiles=no` in any user config hides untracked files and turns the gate into a rubber stamp | | Adopt the gate after seeing it fail once on real output | Run known-in-scope and planted-out-of-scope controls | Every mistyped filter also produces a failing run; only the pass/fail pair shows the gate discriminates | +| Assert `git status` output in a permanent test suite | Prove scope from the introducing commit's own diff, or skip when the environment cannot answer | The ambient tree is whoever-runs-it's in-progress state — an undeclared dependency that makes results non-reproducible | ## Sources - https://git-scm.com/docs/git-status — `-u` modes ("normal — Shows untracked files and directories", "all — Also show individual files in untracked directories"), `status.showUntrackedFiles`, porcelain v1 rename format (` -> `), C-string quoting vs `-z`, `--ignored=matching` - Local reproduction (git 2.50.1, 2026-08-05): scratch repo with `qa/cases/x/{a,b}.md`; default porcelain printed the single line `?? qa/`, which a `^\?\? qa/…` per-file filter treated as a violation; `-uall` expanded to three file lines and the filter passed +- https://bazel.build/reference/test-encyclopedia — "Tests should be hermetic: that is, they ought to access only those resources on which they have a declared dependency"; "If tests are not properly hermetic then they do not give historically reproducible results" +- Field reproduction 2026-08-14 (dev-loop, reviews/i83-insight-emission-r1.md): a permanent bats test asserted working-tree purity via `git status` and failed spuriously on an uncommitted sibling file; rewritten to prove scope from the introducing commit's diff → suite back to 521/521 diff --git a/wiki/testing/e2e/e2e-stability.md b/wiki/testing/e2e/e2e-stability.md index 80e7f4a..cb97ad2 100644 --- a/wiki/testing/e2e/e2e-stability.md +++ b/wiki/testing/e2e/e2e-stability.md @@ -11,7 +11,7 @@ sources: - https://playwright.dev/docs/auth - https://testing-library.com/docs/queries/about/ last_verified: 2026-07-10 -related: [testing-flaky-diagnosing-flaky-tests, testing-data-test-data-and-isolation, testing-quality-minimum-case-set, testing-mocking-what-to-mock] +related: [testing-flaky-diagnosing-flaky-tests, testing-data-test-data-and-isolation, testing-quality-minimum-case-set, testing-mocking-what-to-mock, qa-environments-headless-browser-bot-blocking] --- # Keeping Browser E2E Tests Stable and Fast diff --git a/wiki/testing/index.md b/wiki/testing/index.md index 6b26016..0633731 100644 --- a/wiki/testing/index.md +++ b/wiki/testing/index.md @@ -34,6 +34,7 @@ Match your situation to a "load when" line; load only matching pages. | [spec-artifact-checks](quality/spec-artifact-checks.md) | Authoring or reviewing the check itself: that a mapping table covers every rule/field/enum case, that ids resolve across documents; deciding whether a green check earned "verified" or only "present"; designing one negative control per check in a multi-check harness; parsing Markdown table rows programmatically in a doc-as-spec repo (deciding whether a passing gate is enough to *accept the deliverable* → wiki/qa/document-verification/spec-document-gates.md) | | [schema-additions-under-a-golden-gate](quality/schema-additions-under-a-golden-gate.md) | Adding a node kind, variant, discriminator value, or field to a document format (IR, JSON Schema, spec artifact) whose only automated gate builds its negatives by mutating one committed golden example; the gate or the whole suite comes back green right after a schema change; deciding which negative each new schema keyword needs, and whether a green suite that never loads the schema is evidence at all | | [harness-reverse-controls](quality/harness-reverse-controls.md) | You built a harness that scores how well something is verified (mutation run, doc/spec gate suite, CI check matrix) and are about to cite its score in a commit, PR, README, or report; its verdicts come out uniform (every case caught, or every case green); deciding what control run proves the harness discriminates, how to score errored/never-ran cases, and what the harness's isolated working tree must contain | +| [history-dependent-checks-on-shallow-clones](quality/history-dependent-checks-on-shallow-clones.md) | A test or gate resolves git history (`log --diff-filter`, `merge-base`, `rev-list`) and may run under a shallow CI checkout (actions/checkout default `fetch-depth: 1`); a history query reported every tracked file as added by one commit; deciding between a `--is-shallow-repository` skip-guard and deepening the fetch | ## data diff --git a/wiki/testing/quality/checks-that-cannot-pass.md b/wiki/testing/quality/checks-that-cannot-pass.md index bff6252..2aae3e8 100644 --- a/wiki/testing/quality/checks-that-cannot-pass.md +++ b/wiki/testing/quality/checks-that-cannot-pass.md @@ -10,7 +10,7 @@ sources: - https://docs.semgrep.dev/writing-rules/testing-rules - https://docs.pytest.org/en/stable/reference/exit-codes.html last_verified: 2026-08-06 -related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set, backend-common-change-impact-call-site-enumeration, platforms-processes-tool-diagnostics-without-a-failing-exit-code, qa-process-scope-purity-checks] +related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set, backend-common-change-impact-call-site-enumeration, platforms-processes-tool-diagnostics-without-a-failing-exit-code, qa-process-scope-purity-checks, testing-quality-history-dependent-checks-on-shallow-clones] --- # Validating a Check Whose Target Does Not Exist Yet diff --git a/wiki/testing/quality/history-dependent-checks-on-shallow-clones.md b/wiki/testing/quality/history-dependent-checks-on-shallow-clones.md new file mode 100644 index 0000000..715cfd3 --- /dev/null +++ b/wiki/testing/quality/history-dependent-checks-on-shallow-clones.md @@ -0,0 +1,63 @@ +--- +id: testing-quality-history-dependent-checks-on-shallow-clones +domain: testing +category: quality +applies_to: [general] +confidence: verified +sources: + - https://github.com/actions/checkout + - https://git-scm.com/docs/git-rev-parse + - "Local reproduction, git 2.50.1 (Apple Git-155), 2026-08-14: --depth 1 clone of a 306-file repo — the boundary commit reported all 306 tracked files as added" +last_verified: 2026-08-14 +related: [qa-process-scope-purity-checks, testing-quality-checks-that-cannot-pass] +--- + +# History-Dependent Checks Under a Shallow CI Checkout + +## When this applies + +A test or gate resolves git history — `git log --diff-filter`, `merge-base`, +`rev-list`, "which files did this commit add" — and can run where the checkout +is shallow. GitHub Actions is the common case: `actions/checkout` fetches only +a single commit by default (`fetch-depth: 1`). + +## Do this + +1. **Guard before querying:** `git rev-parse --is-shallow-repository` — on + `true`, skip with an explicit message naming the missing capability, or + degrade to a check that needs no history. An honest skip is recoverable; a + false answer is not. +2. **Know the failure shape: shallow history answers falsely, it does not + error.** The depth-1 boundary commit is grafted parentless, so + diff-vs-parent queries attribute the entire tree to it. Reproduction + (git 2.50.1, 2026-08-14): fresh `--depth 1` clone of a 306-file repo; + `git log -1 --diff-filter=A --name-only` on the boundary commit listed all + 306 tracked files as "added". +3. **When the check must run in CI, deepen the fetch in the workflow** instead + of guarding: `fetch-depth: 0` for anything needing full history + (merge-base, tags, blame); `fetch-depth: 2` when only "this commit vs its + parent" is needed. +4. **Validate both branches before trusting the guard**: run the check once on + a full clone (must execute) and once on a `--depth 1` clone (must skip, not + pass) — a guard only ever observed on full clones has not demonstrated it + fires ([testing-quality-checks-that-cannot-pass]). + +## Edge cases + +| Case | Then | +|------|------| +| The check needs `merge-base` between PR head and base | On a shallow clone this can fail loudly or return a graft-truncated wrong answer — run the shallow guard first; do not rely on the command erroring | +| The gate runs locally and in CI, and only CI is shallow | The CI-side skip means CI never exercises the check — print the skip (reason included) in CI output so a permanently-skipped check stays visible, and decide whether that job should deepen its fetch instead | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Assert scope or authorship from `git log --diff-filter` output in a suite that may run shallow | Guard with `--is-shallow-repository` and skip, or deepen the fetch for that job | The boundary commit reports every tracked file as added — the assertion passes or fails on fiction either way | +| Set `fetch-depth: 0` on every job to make the problem go away | Deepen only the jobs whose checks need history | Full-history fetch cost scales with the repo's history and is paid on every run of every job | + +## Sources + +- https://github.com/actions/checkout — README: "Only a single commit is fetched by default, for the ref/SHA that triggered the workflow"; `fetch-depth: 0` documented as "all history for all branches and tags" +- https://git-scm.com/docs/git-rev-parse — `--is-shallow-repository`: "True if this is a shallow repository, otherwise false" +- Local reproduction (git 2.50.1, Apple Git-155, 2026-08-14): `git clone --depth 1 file://…` of a 306-file repo → `git rev-parse --is-shallow-repository` = true; `git log -1 --diff-filter=A --name-only` listed 306/306 tracked files as added by the boundary commit