[Bug] Capsule review fails closed on external Hatch projects without project-runtime handoff #472

Describe the Bug

Signed Code Review 0.49.85 plus SpecFact CLI 0.55.4 can now launch the analyzer capsule in an external consumer repository. After launch, the review still fails closed because the capsule cannot use the consumer's real project environment.
The reporter states that the same files type-check and lint in the consumer Hatch environment. This is evidence of a runtime-isolation mismatch; the individual findings still require validation against the reproduced project environment. The capsule isolates basedpyright, pylint, contracts, and targeted pytest from the consumer dependency graph, then treats that isolation as FAIL/UNKNOWN.
The current C14 path appears validated mainly against nold-ai's own modules repo (dogfood). That repo can supply signed project-runtime-layer-v1 context, src-layout fixtures, and supported pytest policy. Real consumer projects do not. Typical external usage is:

• Hatch or Poetry, not a SpecFact-owned project-runtime OCI layer
• src/ layout plus a detached review environment
• GitHub Actions explicit-file review (assurance_kind: explicit_files)
• pytest config with addopts, plugins, pythonpath, and mixed test/source batches
• third-party packages that are not part of the analyzer capsule (pandas, pyodbc, sqlalchemy, Azure SDK, loguru, icontract, keyring)
  The reported external Hatch project fails when the capsule lacks its dependency environment. Other affected shapes require corpus validation; this report does not establish that every external Python repository fails.

To Reproduce

Reproduced on GitHub-hosted Ubuntu 24.04 against an external FinOps automation repository, not nold-ai/specfact-cli-modules.
Consumer invocation (after installing signed modules into user scope; consumer CI must unset GITHUB_ACTIONS or SpecFact rejects the repo as untrusted_candidate_workflow_context):

hatch run specfact-review:code-review --json \
  --out logs/specfact-review/pr-orchestrator.json -- <changed python files>

The Hatch environment is detached and already contains the consumer packages (pandas, sqlalchemy, pyodbc, icontract, Azure SDK, …). SpecFact still runs analyzers inside the sealed capsule, which does not mount that environment.
Concrete public reproduction:

1. Open PR in origin repo (redacted here)
2. Inspect GitHub Actions run 34817172528 (commit 4966218, 2026-09-14)
3. Download artifact specfact-review-logs / pr-orchestrator.json
4. Observe assurance_kind: explicit_files and the analyzer evidence below
   The consumer already applied every documented host workaround that nold-ai's own CI uses (AppArmor userns, GHCR pull, user-scope signed install). Those were necessary but insufficient. Repository-side capsule monkey patches were then removed after 0.49.85 absorbed cold-cache, mount anchors, config tmpfs, and multithreaded launch isolation. The remaining failure is the missing consumer project-runtime / pytest-policy path.

Expected Behavior

For an external Linux x86_64 consumer with a signed official module:

• Analyzer capsule launches (this now works on 0.49.85).
• basedpyright / pylint / contracts resolve consumer third-party imports from an attested project runtime, or fail with one actionable diagnostic telling the consumer how to generate/attach project-runtime-layer-v1.
• Explicit-file review accepts ordinary Hatch/pytest layouts (testpaths, pythonpath, addopts, asyncio plugin) or reports a precise unsupported field instead of collapsing to pytest_selection_policy_unsupported.
• Independent static analyzers (ruff, radon, semgrep, AST) remain usable, but missing project runtime must not flood the report with cascading reportUnknownMemberType noise that hides real findings.
• Dogfood of nold-ai/specfact-cli-modules is not the only supported customer shape.

Actual Behavior

Capsule identity sha256:972564ab8503bd629b5ff11109fec8f9ee4e4a03baf2749e21453a5b58e5d925, environment linux-x86_64-cp312, report schema 1.6, run id review-capsule-4d3c9276-abc7-44dd-9ff5-c2e791ba14e4.

• Verdict FAIL, score 0, assurance_status: FAIL
• 1514 findings (108 blocking); 1450 from basedpyright
• Scope evidence: {"assurance_kind": "explicit_files", "capsule_execution": "required"}
• No project_runtime descriptor was claimed or materialized (consumer has no generator and no --pr-context-file project-runtime payload)
  Analyzer members:

basedpyright cascade from an empty import graph (not from defective consumer types):

• reportUnknownMemberType 687
• reportUnknownVariableType 343
• reportUnknownArgumentType 180
• reportUnknownParameterType 114
• reportUntypedFunctionDecorator 37
• reportMissingTypeArgument 37
• reportMissingImports 35
• reportUnknownLambdaType 12
• reportMissingModuleSource 3
• reportPrivateUsage 2
  Root missing imports that Hatch already provides:
• pandas, pyodbc, sqlalchemy, icontract, loguru, keyring, azure.core.credentials
  pylint mirrors the same isolation: 34 × E0401 (Unable to import 'pandas', 'pyodbc', 'sqlalchemy', 'loguru', 'keyring', 'azure.core.credentials').
  The consumer's own CI hatch run type-check / hatch run lint on the same commit is green. SpecFact is scoring an empty site-packages, not the project.

Environment

• OS: Linux (GitHub Actions ubuntu-24.04)
• Python Version: 3.12.14 (consumer Hatch specfact-review env; capsule ABI cp312)
• Repo Branch: consumer feature/656018_import_azure_service_health (not a nold-ai checkout)
• Consumer repo:
• Consumer PR:
• SpecFact CLI: 0.55.4 (Hatch pin specfact-cli>=0.55.1)
• Relevant Bundle(s): nold-ai/specfact-code-review 0.49.85 (user scope, official/signed), plus specfact-codebase and specfact-project
• Packaging: Hatchling src/cloud_finops, detached Hatch env specfact-review, PYTHONPATH=src
• Pytest policy that failed inventory: [tool.pytest.ini_options] with addopts = "-ra -v --import-mode=importlib -p pytest_asyncio.plugin", pythonpath = ["src", "tools", "."], testpaths = ["tests"], asyncio auto mode, custom markers

Validation Output

Consumer host gates on the same commit (these are not the capsule):

hatch run format        # pass
hatch run lint          # pass (0 errors)
hatch run type-check    # pass (0 errors, pandas/sqlalchemy warnings only)
hatch run test          # pass (targeted + broader unit suite)

Capsule review summary from CI:

SpecFact review summary
- Verdict: FAIL
- CI exit: 1
- Score: 0
- Reward delta: -80
- Summary: Review completed with 1514 findings (108 blocking).

Sample false findings (imports exist in Hatch, absent in capsule):

error | basedpyright | reportMissingImports | src/cloud_finops/core/fabric.py:18 | Import "pandas" could not be resolved
error | basedpyright | reportMissingImports | src/cloud_finops/core/fabric.py:19 | Import "pyodbc" could not be resolved
error | basedpyright | reportMissingImports | src/cloud_finops/core/fabric.py:20 | Import "sqlalchemy" could not be resolved
error | pylint | E0401 | src/cloud_finops/core/fabric.py:18 | Unable to import 'pandas'
error | pylint | E0401 | src/cloud_finops/local/azure_service_health.py:15 | Unable to import 'keyring'

Independent static members did run and produced real, reviewable findings (example: radon CC18 on consumer transform functions). Those should remain. The import/type cascade and incomplete contracts/pytest members currently dominate the score and block merge.

Additional Context

Why dogfooding hides this

require_project_runtime / _materialize_claimed_project_runtime only activate when claimed PR context already contains schema: project-runtime-layer-v1. nold-ai's modules CI can mint that descriptor and the matching OCI layer. External Hatch/Poetry repos cannot:

• There is no published consumer command to emit project-runtime-layer-v1 from pyproject.toml / a Hatch env
• Explicit-file review never receives --pr-context-file, so project runtime stays unset
• _PROJECT_RUNTIME_MEMBERS (basedpyright, contracts, pylint, pytest coverage) then type-check against the sealed analyzer image only
• Consumer PYTHONPATH=src and Hatch site-packages are invisible by design
  That is correct isolation if the attested consumer layer exists. It is a product gap if the only way to obtain that layer is to be the SpecFact modules repository.

Real-world requirements the module should support

1. Generate a project-runtime layer from a consumer lock/environment (Hatch, pip-tools, Poetry) without requiring the consumer to vendor nold-ai OCI build scripts.
2. Attach that layer on local/CI explicit-file and worktree reviews, not only protected pr_range with claimed context.
3. Fail closed with one diagnostic (project_runtime_required + how to produce the descriptor) instead of 1,400 cascading unknown-type findings.
4. Accept ordinary pytest config used outside nold-ai (addopts, -p plugin, pythonpath, import-mode=importlib) or document the exact forbidden keys.
5. Keep consumer GitHub Actions from being treated as nold-ai/specfact-cli-modules candidate checkouts (untrusted_candidate_workflow_context). User-scope install + dropping GITHUB_ACTIONS is a fragile consumer workaround.
6. Do not require consumers to patch compose_post_base_capsule or bwrap argv. 0.49.85 already absorbed those; remaining work is the runtime handoff.

Related

• Completed capsule launch work in 0.49.85 (cold cache, mount anchors, config tmpfs, isolated traced launch): necessary, not sufficient for external projects
• Related layout/runtime issues: [Bug] C14 rejects verified modules installed with src layout #459, [Change] Dedicated macOS ARM64 Code Review capsule #460, C14 [Change] Implement truthful Code Review scope and differential enforcement #416
• Parent feature framing used by prior bugs: Validation Evidence And Governance Runtime Follow-Ups #163 under epic specfact code / AI-bloat defense #162
• Blocks merge of an external production PR that already has green host format/lint/type-check/tests and a successful Azure Service Health import pipeline
  This issue should stay open until an external Hatch src/ repository can run specfact code review run on changed files and get import-aware basedpyright/pylint/contracts/pytest evidence without copying nold-ai's internal runtime-build pipeline.

Implementation investigation — 2026-09-14 (Europe/Berlin)

Tracked by User Story #473 and OpenSpec code-review-16-portable-project-runtime, explicitly authorized for implementation. Native parent: #163 under #162; resolution is blocked by #473.

Verified by static inspection at modules dev 976460c0541b685f5fae256faef4cec32fe9ba7e: local capsule review does not prepare or attach a project runtime; the v1 path requires claimed PR context and OCI/GitHub-shaped provenance; reserved import policy includes ordinary project packages such as requests, httpx, icontract and pydantic; pytest policy rejects -p and common hooks. Discovery alone will not repair the full path. Candidate selection already distinguishes source checkout layout, so further GitHub Actions changes require a reproducing test rather than assuming every consumer takes that branch.

The linked customer PR returns HTTP 404 through both connected GitHub and authenticated gh access. Its log counts and green host gates remain reporter-provided evidence, not independently reproduced results. Preserve real findings; do not classify every unknown-type diagnostic as false solely from the counts.

Acceptance requires the pinned external-repository matrix and published signed installation in #473. Keep this bug open until the original customer reproduction can also be verified; a synthetic Hatch reconstruction is labelled separately.