Establish an evidence-driven Workshop conformance corpus and seasonal feature census

[Teakowa]

Goal

Establish a durable Workshop conformance corpus that turns real projects, minimized regressions, a canonical full-feature Workshop census, and periodic live-client observations into executable evidence for workshop-rs and downstream language providers.

Context

Current workshop-rs work already targets corpus-tested parsing, validation, emission, locale conversion, catalog identity, and reproducible provenance. The next testing layer should make correctness expectations independent from the implementation under test and make failures attributable to concrete Workshop capabilities.

This corpus should also provide the canonical Workshop-side evidence later consumed by opy-rs and del-rs. Source-language repositories remain responsible for their own syntax and semantic representations; workshop-rs owns the canonical Workshop capability identities and Workshop-observable expectations.

Scope

• Define a provenance-aware fixture model that distinguishes at least:
  • small synthetic/unit or property cases;
  • minimized regressions extracted from real projects;
  • complete real-world project/corpus cases;
  • live-client/seasonal observations.
• Require correctness expectations to identify an independent evidence source such as a Workshop client observation, pinned external oracle, documented semantic contract, or preserved real-project regression. Do not treat implementation output changed in the same patch as independent correctness evidence.
• When a real project exposes a semantic or compatibility defect, preserve a minimized reproduction where practical while retaining repository/revision/path provenance and the original full-project evidence.
• Define stable Workshop feature identities for conformance results, covering the declared surface such as events, actions, values, enums/content IDs, settings, variables, subroutines, control flow, strings/localization, and other catalog-backed constructs.
• Build a canonical Workshop feature census fixture/mode that intentionally exercises the declared Workshop surface and can report failures by feature identity rather than only as a monolithic script diff.
• Keep semantic/normalized equivalence as the correctness target where text identity is not part of the contract.
• Add machine-readable conformance output that distinguishes matched behavior, unsupported surface, known gaps, unexpected regressions/divergence, and inconclusive evidence without counting known failures as passing compatibility.
• Define a repeatable seasonal validation workflow in which the census can be imported/exported through the live Overwatch client and the resulting artifacts recorded with game/season/client-locale/catalog provenance.
• Use English client output as the primary semantic baseline where appropriate, with explicit locale sweeps for localization/catalog drift as separately recorded evidence.
• Make the corpus layout and result contract reusable by downstream opy-rs / del-rs integration tests without duplicating canonical Workshop definitions into those repositories.

Non-goals

• Claiming exhaustive live-runtime/gameplay correctness solely from import/export evidence.
• Requiring byte-identical or formatting-identical output.
• Building a network service or automatically downloading unreviewed catalog data at runtime.
• Moving OPY or DEL/OSTW source-language fixtures or semantics into workshop-rs.
• Treating one giant census script as a replacement for focused unit/property tests or real-world projects.
• Freezing a separate cross-repository test framework before concrete corpus integration requires one.

Acceptance criteria

• The repository documents and enforces an evidence model in which correctness expectations have traceable provenance independent from the implementation being changed.
• At least one real-project failure is represented both by preserved project-level evidence and a minimized provenance-linked regression fixture.
• A canonical feature-census fixture covers the currently declared Workshop semantic/catalog surface with stable machine-readable feature identities.
• Census failures identify affected feature(s) rather than reducing the result to a single whole-file pass/fail.
• parse -> canonical semantics -> emit and supported locale conversion are gated against the corpus using semantic/normalized contracts.
• Known gaps are represented explicitly and are not reported as successful parity/conformance.
• A documented seasonal client-validation procedure records client/game/catalog/locale provenance and supports structured comparison with prior captures.
• The resulting corpus/result contracts can be consumed by opy-rs and del-rs without transferring canonical Workshop ownership out of this repository.

Relationships

• Extends Establish the canonical multi-locale Workshop core #2 rather than replacing its canonical multi-locale Workshop core scope.
• Builds on Define Workshop catalog, locale, provenance, and version boundaries #1 catalog/locale/provenance/version boundaries.
• Downstream source-language adoption should be tracked in the owning opy-rs and del-rs repositories.