Skip to content

[User Story] Customer-owned standards and compatible review checks #514

Description

@djm81

Outcome

Native project standards and plugins, explicit profile contributions, separate analyzer environments, actual contracts, and explicit pytest selection.

OpenSpec change: code-review-18-project-standards-and-checks

Confirmed design

The owner approved the capsule hard cut on 2026-10-16, Europe/Berlin. Customers review trusted repositories. Dependency installation, build hooks, imports, and configured checks may execute locally. SpecFact stays outside the customer environment and respects customer dependencies, locks, coding standards, plugins, and entrypoints. No hostile-code containment is claimed.

This issue tracks future implementation. The current delivery creates architecture/specification artifacts and aligns GitHub metadata only. Implementation tasks remain unchecked. Planning validation is not runtime acceptance or RED/GREEN evidence.

Acceptance

  • Preserve customer dependency declarations and locks; install no SpecFact dependencies into customer environments.
  • Report unavailable required checks as incomplete, with actionable diagnostics.
  • Keep actual findings, source identity, analyzer versions, and execution completeness distinct.
  • Use spec -> tests -> genuine failing evidence -> implementation -> passing evidence in follow-up sessions.
  • Pass applicable quality/security/signature gates and exact installed acceptance before claiming delivery.

Relationships

Parent: #163, within Epic #162.

Direct implementation blockers: #513.

Canonical architecture: nold-ai/specfact-cli-modules/openspec/changes/code-review-17-project-runtime-bootstrap/design.md.

Capsule-only #460, #488, #495, #512 and modules PR #506, #509, #510 are superseded as recorded by the replacement plan. Preserve historical releases and independent signing/security work. C15 and protected C14 assurance remain separate; no local report is protected pr_range evidence. Optional preflight and architecture/requirements add-ons are future consumers, not prerequisites.

Planned requirement scenarios

ADDED Requirements

Requirement: Customer standards remain authoritative

Customer-standard checks SHALL use customer-selected tools, versions, native configuration and plugins. Selection SHALL prefer explicit invocation or configuration, then an explicitly selected organizational profile, then native project discovery. SpecFact supplemental analysis SHALL remain separately attributed without rewriting project lint policy.

Scenario: Project standards disagree with SpecFact development defaults

  • GIVEN the customer configures different lint rules, exclusions, formatting or coverage thresholds
  • WHEN customer-standard checks run
  • THEN their effective configuration follows the customer project and no universal SpecFact coverage floor or style requirement is injected

Scenario: All selected diagnostic identifiers survive

  • GIVEN Ruff or another supported analyzer emits a customer-selected rule outside a historical hardcoded family
  • WHEN the adapter normalizes its result
  • THEN the report retains the rule identifier and diagnostic instead of dropping it

Scenario: Unknown custom check output

  • GIVEN a named customer check has no structured diagnostic adapter
  • WHEN its configured argv runs
  • THEN the report retains command completion, exit status and bounded output without inventing source positions or discarding a nonzero result

Requirement: Explicit declarative profile contribution

ProjectProfileProvider SHALL be a narrow explicitly selected entrypoint in specfact.project_profiles installed in the external CLI environment. It SHALL return declarative selection preferences, configuration references, and named check commands. Runtime ownership and finding severity SHALL remain with the runtime service and review respectively.

Scenario: Explicit organizational profile

  • GIVEN a compatible provider is installed outside the customer repository and explicitly selected
  • WHEN review resolves its contribution
  • THEN the contribution is validated once and native project tools retain their own plugin and configuration semantics

Scenario: Unavailable or incompatible provider

  • GIVEN the selected provider is missing or its contribution version is unsupported
  • WHEN review loads the profile
  • THEN it reports an actionable profile failure without importing arbitrary project providers into the controller or silently choosing another profile

Requirement: Analyzer dependencies do not alter customer environments

Project-defined checks SHALL run in the selected project environment. Source-only supplemental tools SHALL run in separate external tool environments. Import-dependent supplemental checks SHALL use a compatible project tool or an independently prepared same-interpreter environment preserving the complete selected project dependency graph and origins.

Scenario: Compatible project-installed tool

  • GIVEN the selected environment already supplies the configured analyzer and its plugins
  • WHEN the relevant check runs
  • THEN it uses that installation and records its version without installing or upgrading packages

Scenario: Conflicting supplemental analyzer

  • GIVEN an import-dependent analyzer conflicts with a project dependency, package origin, Python version or build requirement
  • WHEN its compatible environment cannot be prepared faithfully
  • THEN the affected check is unavailable with a remedy, other checks continue, and customer dependencies are neither replaced nor added to the controller import path

Requirement: Existing contracts are checked without mandatory decoration

The initial contract adapter SHALL analyze existing supported icontract contracts through CrossHair. Absence of such contracts SHALL be not applicable. A violated contract, an unsupported requested mechanism, and an execution failure SHALL remain distinct outcomes. It SHALL not require missing decorators or synthesize customer contracts.

Scenario: Actual contract violation

  • GIVEN a project uses a supported icontract contract that the configured analysis can falsify
  • WHEN the contract check executes in faithful project context
  • THEN the violation becomes an attributed finding with actual analyzer provenance

Scenario: No contracts or unsupported contract mechanism

  • GIVEN the project has no supported contracts or explicitly requests an unsupported mechanism
  • WHEN applicability is resolved
  • THEN absence is not applicable and unsupported requested analysis is incomplete rather than a missing-icontract finding or clean pass

Requirement: Explicit pytest targets retain native semantics

Explicit test-file selection SHALL remain distinct from automatic source-to-test discovery. The selected pytest file SHALL not be silently replaced or omitted because its basename fails automatic python_files patterns.

Scenario: Nonstandard failing test filename

  • GIVEN tests/checks.py contains a failing test and tests/test_checks.py passes
  • WHEN the customer explicitly selects tests/checks.py
  • THEN that file is collected and executed and its failure appears in the review result

Scenario: Custom patterns and mixed sources

  • GIVEN the project has custom python_files patterns and the review selects both production source and an explicit test file
  • WHEN pytest selection is assembled
  • THEN the explicit test target is preserved and production-source mapping does not indiscriminately execute source files as tests

Implementation design

Design for customer-owned standards and compatible review checks

Canonical architecture

Change 17 design (nold-ai/specfact-cli-modules/openspec/changes/code-review-17-project-runtime-bootstrap/design.md) owns the architecture, API sketch, trust decisions, alternatives, acceptance matrix and synthesis. This document records only this change's implementation responsibility. No runtime implementation or supported distribution is established here.

Responsibility and data flow

Run native customer standards with their configured versions and plugins. Add the explicitly selected declarative profile extension, faithful supplemental analyzer environments, existing icontract analysis, and native explicit pytest selection.

Project selection/configuration is resolved before check execution. Explicit invocation/configuration wins over an explicitly selected organizational profile, which wins over native discovery. Native tools retain their own configuration and plugin semantics. Review owns findings/severity; the profile contributes declarative data only. Missing/incompatible selected providers are errors rather than an automatic fallback.

Use the single entrypoint namespace specfact.project_profiles for explicitly selected external CLI providers. A contribution carries its supported version, selection preferences, configuration paths and named argv commands. It does not create environments, define controller-bound worker objects or load arbitrary project providers into the CLI.

Project checks use project installations. Source-only supplemental tools use external tool environments. Import-dependent tools prefer a compatible project installation; otherwise prepare the exact same-interpreter project graph and origins with compatible analyzer dependencies externally. Never splice paths or upgrade a customer package to make a tool work. Record incompatible checks and remedies. Preserve unknown custom-check exits/output without invented finding positions.

The initial contract mechanism is existing icontract checked through CrossHair. Missing contracts are not applicable. Unsupported requested mechanisms and failures remain incomplete. Remove the implicit 80-percent customer floor and mandatory missing-icontract findings. SpecFact's own repository policy remains configured and enforced.

Issue #480 remains open until its actual behavior is demonstrated. Add native pytest regression cases for default/custom patterns, tests/checks.py, no conventional match, competing passing conventional tests, and mixed source/test selection. A selected failing test is not replaceable by a discovered passing test.

Implementation reconciliation and first step

All implementation remains pending. Accepted interface or responsibility deviations must update the canonical design and this delta before handoff. Start by adding scenario-mapped behavior tests and capturing actual failing evidence. Preserve existing applicable governance while implementing the approved replacement.

Risks and rollback

Wrong interpreter/source, dependency drift and overstated completion are the primary risks. Verify actual executable/source identities, native configuration and failure behavior. Keep an immutable compatible released pair for rollback. No permanent capsule backend remains in the replacement.

Metadata

Metadata

Assignees

Labels

architectureArchitecture-related topicchange-proposalProposal for a new changecodebaseSpecfact codebase related topicenhancementNew feature or requestopenspecopenspec change

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions