diff --git a/compatibility/README.md b/compatibility/README.md index df2e3d9..dc0235d 100644 --- a/compatibility/README.md +++ b/compatibility/README.md @@ -45,7 +45,7 @@ compatibility/fixtures/// Imported fixtures should also record an immutable `sourceCommit`, a direct `sourceUrl`, a `licenseUrl`, and whether the source was modified. The corpus -contains 26 fixtures: 13 original WrightKit-authored synthetic cases and 13 +contains 27 fixtures: 14 original WrightKit-authored synthetic/census cases and 13 real-world projects (11 derived from the pinned OverPy `examples/` tree, GPL-3.0-only, provenance-recorded evidence, plus the independent BSD-2-Clause projects `real-world/ow1-emulator` and `real-world/6v6-adjustments`, full @@ -57,6 +57,24 @@ exit code, normalized diagnostics, normalized Workshop text, and normalized output hash. The runner normalizes line endings, trailing whitespace, and final newline presentation only; it does not remove Workshop operations or values. +`differential-expectations.json` is the independent native-side expectation +record. Each fixture has a native outcome, an expected relationship to the +oracle (`match`, `known-gap`, or `unsupported`), a rationale, and evidence. +The differential runner derives `unexpected-divergence`, `regression`, and +`inconclusive` results from those records; it never treats a reference-success +/ native-failure case as a match. + +Real-world fixtures retain the complete project as the integration case. A +fixture may also declare `regressions` pointing to minimized snippets under +the same directory. Each snippet records its source path, parent source, +reference status, and oracle provenance; the full project is not replaced by +the minimized case. + +The `census/workshop-feature-census` fixture is the OPY-side representation +of the future `workshop-rs#10` conformance boundary. Its feature IDs are +opaque consumer references only. Canonical Workshop identities, catalog +definitions, and validation remain owned by `workshop-rs`. + ## Commands ```sh @@ -140,6 +158,10 @@ The report separates these stages: * `normalized-output`, using the versioned snapshot normalization; and * `semantic`, when both producers provide semantic evidence. +It also separates relationship outcomes: `match`, `known-gap`, `unsupported`, +`unexpected-divergence`, `regression`, and `inconclusive`. Known gaps are +reviewable evidence and are not counted as successful parity. + An exact-output difference with a normalized-output match is reported as a presentation difference. A normalized-output or semantic regression exits 1. Missing producer results or unavailable semantic evidence are `inconclusive` @@ -147,7 +169,7 @@ and exit 2 by default, so a CI job cannot silently pass without a producer. Use `--allow-inconclusive` only for local contract checks. The opy-rs producer side of the differential contract is the native Rust -suite (`crates/opy-frontend/tests/differential.rs`, merged in PR #13), which -runs in `cargo test` with no Node or OverPy installed; `diff.py` remains the +suite (`crates/opy-frontend/tests/differential.rs`), which runs in `cargo test` +with no Node or OverPy installed; `diff.py` remains the generic external-producer contract for other producers, exercised locally via `run_oracle.py` and the corpus snapshots. diff --git a/compatibility/diff.py b/compatibility/diff.py index de0aedd..ec5eeb0 100644 --- a/compatibility/diff.py +++ b/compatibility/diff.py @@ -16,6 +16,10 @@ ROOT = Path(__file__).resolve().parents[1] DEFAULT_FIXTURES = ROOT / "compatibility" / "fixtures" DEFAULT_REPORT = ROOT / "compatibility" / "report.json" +DEFAULT_EXPECTATIONS = ROOT / "compatibility" / "differential-expectations.json" + +EXPECTED_NATIVE_STATUSES = {"success", "failure"} +EXPECTED_CLASSIFICATIONS = {"match", "known-gap", "unsupported"} class DiffError(RuntimeError): @@ -64,6 +68,45 @@ def fixture_ids(fixtures_root: Path) -> list[str]: return ids +def load_expectations(path: Path = DEFAULT_EXPECTATIONS) -> dict[str, dict[str, Any]]: + data = load_json(path) + if data.get("schemaVersion") != 1: + raise DiffError(f"unsupported differential expectation schema: {path}") + cases = data.get("cases") + if not isinstance(cases, list) or not cases: + raise DiffError(f"differential expectations must contain cases: {path}") + + by_fixture: dict[str, dict[str, Any]] = {} + for case in cases: + if not isinstance(case, dict): + raise DiffError(f"differential expectation must be an object: {path}") + fixture = case.get("fixture") + if not isinstance(fixture, str) or not fixture: + raise DiffError(f"differential expectation fixture is invalid: {path}") + if fixture in by_fixture: + raise DiffError(f"duplicate differential expectation: {fixture}") + native_status = case.get("nativeStatus") + if native_status not in EXPECTED_NATIVE_STATUSES: + raise DiffError(f"{fixture}: nativeStatus must be success or failure") + classification = case.get("classification") + if classification not in EXPECTED_CLASSIFICATIONS: + raise DiffError( + f"{fixture}: classification must be match, known-gap, or unsupported" + ) + evidence = case.get("evidence") + if not isinstance(evidence, list) or not evidence or not all( + isinstance(item, str) and item for item in evidence + ): + raise DiffError(f"{fixture}: evidence must be a non-empty string array") + note = case.get("note") + if not isinstance(note, str) or not note: + raise DiffError(f"{fixture}: note must be a non-empty string") + if not isinstance(case.get("ruleNames"), bool): + raise DiffError(f"{fixture}: ruleNames must be boolean") + by_fixture[fixture] = case + return by_fixture + + def require_result_shape(result: dict[str, Any], label: str) -> None: if result.get("schemaVersion") != 1: raise DiffError(f"{label}: unsupported or missing schemaVersion") @@ -202,6 +245,13 @@ def compare_fixture( metadata_path = fixtures_root / fixture_id / "fixture.json" oracle_path = fixtures_root / fixture_id / "oracle.json" metadata = load_json(metadata_path) + expectations_path = fixtures_root.parent / "differential-expectations.json" + expectations = load_expectations( + expectations_path if expectations_path.is_file() else DEFAULT_EXPECTATIONS + ) + expectation = expectations.get(fixture_id) + if expectation is None: + raise DiffError(f"missing differential expectation: {fixture_id}") oracle = load_json(oracle_path) require_result_shape(oracle, f"oracle {fixture_id}") if oracle["fixture"] != fixture_id: @@ -220,6 +270,7 @@ def compare_fixture( "fixture": fixture_id, "category": metadata.get("category", "unknown"), "status": "inconclusive", + "expectedNativeStatus": expectation["nativeStatus"], "reason": f"missing producer result: {path}", "stages": [], } @@ -229,6 +280,7 @@ def compare_fixture( "fixture": fixture_id, "category": metadata.get("category", "unknown"), "status": "inconclusive", + "expectedNativeStatus": expectation["nativeStatus"], "reason": "no producer result root or producer command was provided", "stages": [], } @@ -247,7 +299,22 @@ def compare_fixture( regression_stages = [item["name"] for item in stages if item["outcome"] == "regression"] differences = [item["name"] for item in stages if item["outcome"] == "difference"] inconclusive = [item["name"] for item in stages if item["outcome"] == "inconclusive"] - if regression_stages: + native_status = producer["compile"]["status"] + oracle_status = oracle["compile"]["status"] + native_status_mismatch = native_status != expectation["nativeStatus"] + reference_gap = oracle_status != native_status + declared_reference_gap = oracle_status != expectation["nativeStatus"] + if native_status_mismatch: + status = "unexpected-divergence" + elif expectation["classification"] in {"known-gap", "unsupported"}: + if not declared_reference_gap: + raise DiffError( + f"{fixture_id}: {expectation['classification']} must differ from oracle status" + ) + status = expectation["classification"] + elif oracle_status != native_status: + status = "unexpected-divergence" + elif regression_stages: status = "regression" elif inconclusive: status = "inconclusive" @@ -257,6 +324,13 @@ def compare_fixture( "fixture": fixture_id, "category": metadata.get("category", "unknown"), "status": status, + "expectedClassification": expectation["classification"], + "expectedNativeStatus": expectation["nativeStatus"], + "referenceStatus": oracle_status, + "referenceGap": reference_gap, + "declaredReferenceGap": declared_reference_gap, + "evidence": expectation["evidence"], + "note": expectation["note"], "regressionStages": regression_stages, "differenceStages": differences, "inconclusiveStages": inconclusive, @@ -310,6 +384,19 @@ def run( allow_inconclusive: bool, ) -> int: all_ids = fixture_ids(fixtures_root) + expectations_path = fixtures_root.parent / "differential-expectations.json" + expectations = load_expectations( + expectations_path if expectations_path.is_file() else DEFAULT_EXPECTATIONS + ) + missing = sorted(set(all_ids) - set(expectations)) + extra = sorted(set(expectations) - set(all_ids)) + if missing or extra: + detail = [] + if missing: + detail.append(f"missing expectations: {', '.join(missing)}") + if extra: + detail.append(f"expectations for unknown fixtures: {', '.join(extra)}") + raise DiffError("; ".join(detail)) ids = [fixture_id for fixture_id in all_ids if not selected_ids or fixture_id in selected_ids] unknown = sorted(selected_ids - set(all_ids)) if unknown: @@ -324,15 +411,16 @@ def run( report = build_report(results) write_json(report_path, report) print(json.dumps(report["summary"], indent=2, sort_keys=True)) - regressions = [result for result in results if result["status"] == "regression"] + regressions = [ + result + for result in results + if result["status"] in {"regression", "unexpected-divergence"} + ] inconclusive = [result for result in results if result["status"] == "inconclusive"] if regressions: for result in regressions: - print( - f"REGRESSION {result['fixture']}: " - f"{', '.join(result['regressionStages'])}", - file=sys.stderr, - ) + stages = ", ".join(result.get("regressionStages", [])) or "native outcome" + print(f"REGRESSION {result['fixture']}: {stages}", file=sys.stderr) return 1 if inconclusive and not allow_inconclusive: for result in inconclusive: diff --git a/compatibility/differential-expectations.json b/compatibility/differential-expectations.json new file mode 100644 index 0000000..c97a44b --- /dev/null +++ b/compatibility/differential-expectations.json @@ -0,0 +1,38 @@ +{ + "schemaVersion": 1, + "description": "Independent native outcome expectations for the OPY differential corpus.", + "evidencePolicy": { + "reference": "fixture.json expectedStatus plus the pinned oracle.json snapshot", + "native": "An implementation-local invariant or an explicitly recorded known gap; native behavior is never used to create reference evidence.", + "classification": "match, known-gap, and unsupported are expected relationships. unexpected-divergence, regression, and inconclusive are produced by the runner." + }, + "cases": [ + {"fixture": "synthetic/basic-rule", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/basic-rule/oracle.json", "implementation-invariant:frontend-resolves-basic-rule"], "note": "Minimal rule is part of the Workshop-independent frontend contract."}, + {"fixture": "synthetic/control-flow", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/control-flow/oracle.json", "implementation-invariant:frontend-resolves-control-flow"], "note": "If/elif/else, for-in-range, while, and pass resolve in the OPY semantic model."}, + {"fixture": "synthetic/declarations-numbers", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/declarations-numbers/oracle.json", "implementation-invariant:frontend-resolves-declarations"], "note": "Numeric literals and variable-index declarations resolve in the OPY semantic model."}, + {"fixture": "synthetic/declarations-rules", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/declarations-rules/oracle.json", "implementation-invariant:frontend-resolves-rule-declarations"], "note": "globalvar, playervar, subroutine, def, enum, and rule declarations resolve."}, + {"fixture": "synthetic/expressions-values", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/expressions-values/oracle.json", "implementation-invariant:frontend-resolves-expressions"], "note": "Expressions, arrays, strings, vectors, calls, and format expressions resolve."}, + {"fixture": "synthetic/preprocessing", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/preprocessing/oracle.json", "implementation-invariant:preprocessor-include-define-undef"], "note": "Include, object-like and function-like defines, and undef are preserved through preprocessing."}, + {"fixture": "synthetic/settings", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/settings/oracle.json", "implementation-invariant:settings-structure-only"], "note": "Settings are structurally represented; Workshop key and leaf validation remains lowering-dependent."}, + {"fixture": "synthetic/receiver-calls", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/receiver-calls/oracle.json", "implementation-invariant:receiver-call-resolution"], "note": "The exercised receiver/member forms resolve through the OPY semantic model."}, + {"fixture": "synthetic/chase-enums", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/chase-enums/oracle.json", "implementation-invariant:opaque-workshop-enum-identities"], "note": "Declared chase enum identities resolve without duplicating Workshop catalog validation."}, + {"fixture": "synthetic/chase-condition-agentlab", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/chase-condition-agentlab/oracle.json", "regression:agent-lab-chase-condition-shape"], "note": "The agent-lab chase condition shape is retained as an OPY frontend regression."}, + {"fixture": "synthetic/chase-keywords", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/chase-keywords/oracle.json", "implementation-invariant:keyword-binding"], "note": "Generic keyword binding and the chase contextual form resolve."}, + {"fixture": "synthetic/for-range-agentlab", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/for-range-agentlab/oracle.json", "regression:agent-lab-for-range-binder"], "note": "Implicit for-range binders are retained as an OPY frontend regression."}, + {"fixture": "synthetic/diagnostics", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/diagnostics/oracle.json", "implementation-invariant:parse-error-code"], "note": "Malformed source must produce the stable parse-error diagnostic."}, + {"fixture": "real-world/overpy-cake", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:real-world/overpy-cake/oracle.json", "provenance:real-world/overpy-cake/fixture.json"], "note": "Pinned OverPy example resolves through the native OPY semantic model."}, + {"fixture": "real-world/overpy-pixelart", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:real-world/overpy-pixelart/oracle.json", "provenance:real-world/overpy-pixelart/fixture.json"], "note": "Pinned OverPy example resolves; emitted rule shape is not the contract."}, + {"fixture": "real-world/overpy-santa", "nativeStatus": "failure", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/overpy-santa/oracle.json", "regression:real-world/overpy-santa/fixture.json"], "note": "The pinned oracle accepts the do-while surface; native rejection remains an explicit syntax gap."}, + {"fixture": "real-world/overpy-cronch", "nativeStatus": "failure", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/overpy-cronch/oracle.json", "regression:real-world/overpy-cronch/fixture.json"], "note": "The pinned oracle accepts postfix increment; native rejection remains an explicit syntax gap."}, + {"fixture": "real-world/overpy-broken-weapons", "nativeStatus": "failure", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/overpy-broken-weapons/oracle.json", "regression:real-world/overpy-broken-weapons/fixture.json"], "note": "The pinned oracle accepts numeric-range settings types; native rejection remains a known gap."}, + {"fixture": "real-world/overpy-client-to-server", "nativeStatus": "failure", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/overpy-client-to-server/oracle.json", "regression:real-world/overpy-client-to-server/fixture.json"], "note": "The pinned oracle accepts chained ternaries; native rejection remains an explicit syntax gap."}, + {"fixture": "real-world/overpy-crosshair", "nativeStatus": "failure", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/overpy-crosshair/oracle.json", "regression:real-world/overpy-crosshair/fixture.json"], "note": "The pinned oracle accepts the byte-string modifier; native rejection remains a documented legacy gap."}, + {"fixture": "real-world/overpy-inputhud", "nativeStatus": "failure", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/overpy-inputhud/oracle.json", "regression:real-world/overpy-inputhud/fixture.json"], "note": "The pinned oracle accepts implicit concatenation in parenthesized expressions; native rejection remains a known gap."}, + {"fixture": "real-world/overpy-parabola", "nativeStatus": "failure", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/overpy-parabola/oracle.json", "regression:real-world/overpy-parabola/fixture.json"], "note": "The pinned oracle accepts numeric enum members such as Team.2; native rejection remains a known gap."}, + {"fixture": "real-world/overpy-meipocalypse", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:real-world/overpy-meipocalypse/oracle.json", "provenance:real-world/overpy-meipocalypse/fixture.json"], "note": "The reference rejects the missing script hook; native rejection is retained with a different earlier diagnostic."}, + {"fixture": "real-world/overpy-zencopter", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:real-world/overpy-zencopter/oracle.json", "provenance:real-world/overpy-zencopter/fixture.json"], "note": "Both sides reject the pinned upstream example; native diagnostic wording is not used as semantic evidence."}, + {"fixture": "real-world/ow1-emulator", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:real-world/ow1-emulator/oracle.json", "provenance:real-world/ow1-emulator/fixture.json"], "note": "The full project remains preserved as a failure corpus case with recorded provenance."}, + {"fixture": "real-world/6v6-adjustments", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:real-world/6v6-adjustments/oracle.json", "provenance:real-world/6v6-adjustments/fixture.json"], "note": "The full project remains preserved as a failure corpus case with recorded provenance."}, + {"fixture": "census/workshop-feature-census", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:census/workshop-feature-census/oracle.json", "contract:workshop-rs#10-feature-census"], "note": "OPy source records opaque Workshop feature identities for the future workshop-rs lowering boundary."} + ] +} diff --git a/compatibility/fixtures/README.md b/compatibility/fixtures/README.md index 8c300a6..fdfd0b9 100644 --- a/compatibility/fixtures/README.md +++ b/compatibility/fixtures/README.md @@ -3,7 +3,7 @@ This directory is the opy-rs compatibility corpus: OPY sources with their pinned-oracle snapshots (`oracle.json`), ported from the WrightKit project's evidence base (wright `compatibility/fixtures/`) and re-verified against the -pinned OverPy 9.7.10 oracle on 2026-08-16 (all 26 snapshots match). +pinned OverPy 9.7.10 oracle on 2026-08-17 (all 27 snapshots match). Corpus policy: every fixture records provenance in its `fixture.json` (`kind`, `origin`, `license`, `redistributable`, and — for imported @@ -82,6 +82,15 @@ oracle behavior (accept/reject, diagnostics, normalized Workshop text) for compatibility evidence. See the clean-room policy in `docs/compatibility/upstream-references.md`. +The seven current real-world reference-success/native-gap cases also keep a +minimized regression snippet in the parent fixture's `regressions` metadata. +Those snippets retain a link to the full-project oracle evidence; they are not +standalone replacement expectations. + +`census/workshop-feature-census` is the OPY consumer-side census fixture. Its +`workshopFeatureIds` are opaque IDs reserved for the `workshop-rs#10` contract; +this repository does not copy Workshop catalog definitions or signatures. + ### Independent third-party projects (BSD-2-Clause) | Fixture | Origin | `expectedStatus` | @@ -104,7 +113,7 @@ reference diagnostics, exactly like the pinned oracle behaves. ## Not ported / dropped -* **No fixture was dropped for provenance reasons**: all 26 fixtures in the +* **No fixture was dropped for provenance reasons**: all 27 fixtures in the WrightKit corpus carried complete, reviewed provenance and are ported. * Upstream `examples/` not ported (candidates for later expansion once a demonstrated need exists): `lucioball_all_heroes.opy`, `skirmish_elim.opy`, diff --git a/compatibility/fixtures/census/workshop-feature-census/fixture.json b/compatibility/fixtures/census/workshop-feature-census/fixture.json new file mode 100644 index 0000000..8ad7427 --- /dev/null +++ b/compatibility/fixtures/census/workshop-feature-census/fixture.json @@ -0,0 +1,29 @@ +{ + "schemaVersion": 1, + "id": "census/workshop-feature-census", + "category": "workshop-census", + "features": ["events", "actions", "values", "control flow"], + "workshopFeatureIds": [ + "event.global", + "action.disableInspector", + "action.wait", + "value.compare", + "value.add", + "structural.if" + ], + "source": "workshop-feature-census.opy", + "expectedStatus": "success", + "provenance": { + "kind": "synthetic-original", + "origin": "WrightKit OPY census boundary fixture", + "license": "AGPL-3.0-or-later", + "redistributable": true, + "modifications": "none" + }, + "censusContract": { + "owner": "workshop-rs", + "issue": "https://github.com/wrightkit/workshop-rs/issues/10", + "status": "pending", + "note": "Feature IDs are opaque consumer references; canonical definitions and validation remain owned by workshop-rs." + } +} diff --git a/compatibility/fixtures/census/workshop-feature-census/oracle.json b/compatibility/fixtures/census/workshop-feature-census/oracle.json new file mode 100644 index 0000000..b67173f --- /dev/null +++ b/compatibility/fixtures/census/workshop-feature-census/oracle.json @@ -0,0 +1,27 @@ +{ + "compile": { + "diagnostics": [], + "exitCode": 0, + "status": "success", + "stdout": "", + "workshop": "variables {\n global:\n 0: counter\n}\n\nrule (\"workshop feature census\") {\n event {\n Ongoing - Global;\n }\n actions {\n If(Compare(Global.counter, ==, 0));\n Modify Global Variable(counter, Add, 1);\n Disable Inspector Recording;\n Wait(0.016, Ignore Condition);\n }\n}\n", + "workshopExact": "variables {\n global:\n 0: counter\n}\n\nrule (\"workshop feature census\") {\n event {\n Ongoing - Global;\n }\n actions {\n If(Compare(Global.counter, ==, 0));\n Modify Global Variable(counter, Add, 1);\n Disable Inspector Recording;\n Wait(0.016, Ignore Condition);\n }\n}\n\n", + "workshopSha256": "7be1ad80116c186f8cb40698f1a517f546dc50e5cb746ff8b6b7e77fff5a78e9" + }, + "fixture": "census/workshop-feature-census", + "input": { + "sha256": "0d05e6fe81bc4a4759f833b9dd22bfe76892bbb0a1ab24b23107c85528f09619", + "source": "workshop-feature-census.opy" + }, + "oracle": { + "gitHead": "1e2688954302a402d076944b46db07efb14d7b61", + "integrity": "sha512-oX17nauJcPTaKIrRFY/rD0Rl8atqFUVv9Hg2TKH+A68/fC8+ZO344Mkd1A/Y0oOVp1hr5tktMBjzMEDDnMEYUw==", + "language": "en-US", + "license": "GPL-3.0-only", + "name": "overpy", + "registryTarball": "https://registry.npmjs.org/overpy/-/overpy-9.7.10.tgz", + "repository": "https://github.com/Zezombye/overpy", + "version": "9.7.10" + }, + "schemaVersion": 1 +} diff --git a/compatibility/fixtures/census/workshop-feature-census/workshop-feature-census.opy b/compatibility/fixtures/census/workshop-feature-census/workshop-feature-census.opy new file mode 100644 index 0000000..a3da813 --- /dev/null +++ b/compatibility/fixtures/census/workshop-feature-census/workshop-feature-census.opy @@ -0,0 +1,8 @@ +globalvar counter = 0 + +rule "workshop feature census": + @Event global + if counter == 0: + counter += 1 + disableInspector() + wait() diff --git a/compatibility/fixtures/real-world/overpy-broken-weapons/fixture.json b/compatibility/fixtures/real-world/overpy-broken-weapons/fixture.json index 9ebcea0..529eea3 100644 --- a/compatibility/fixtures/real-world/overpy-broken-weapons/fixture.json +++ b/compatibility/fixtures/real-world/overpy-broken-weapons/fixture.json @@ -24,5 +24,15 @@ }, "runtimeSeconds": 4.4, "schemaVersion": 1, - "source": "broken_weapons.opy" + "source": "broken_weapons.opy", + "regressions": [ + { + "id": "real-world/overpy-broken-weapons/numeric-range-setting", + "source": "regressions/numeric-range-setting.opy", + "derivedFrom": "broken_weapons.opy", + "expectedReferenceStatus": "success", + "kind": "minimized-regression", + "provenance": "oracle:real-world/overpy-broken-weapons/oracle.json; source path 53" + } + ] } diff --git a/compatibility/fixtures/real-world/overpy-broken-weapons/regressions/numeric-range-setting.opy b/compatibility/fixtures/real-world/overpy-broken-weapons/regressions/numeric-range-setting.opy new file mode 100644 index 0000000..d6753cc --- /dev/null +++ b/compatibility/fixtures/real-world/overpy-broken-weapons/regressions/numeric-range-setting.opy @@ -0,0 +1 @@ +globalvar baseChance = createWorkshopSetting(float[0.5:10], "", "Base chance", 4, 0) diff --git a/compatibility/fixtures/real-world/overpy-client-to-server/fixture.json b/compatibility/fixtures/real-world/overpy-client-to-server/fixture.json index 1e4d5f7..b3eb43a 100644 --- a/compatibility/fixtures/real-world/overpy-client-to-server/fixture.json +++ b/compatibility/fixtures/real-world/overpy-client-to-server/fixture.json @@ -25,5 +25,15 @@ }, "runtimeSeconds": 6.4, "schemaVersion": 1, - "source": "clientToServer.opy" + "source": "clientToServer.opy", + "regressions": [ + { + "id": "real-world/overpy-client-to-server/chained-ternary", + "source": "regressions/chained-ternary.opy", + "derivedFrom": "clientToServer.opy", + "expectedReferenceStatus": "success", + "kind": "minimized-regression", + "provenance": "oracle:real-world/overpy-client-to-server/oracle.json; source path 55" + } + ] } diff --git a/compatibility/fixtures/real-world/overpy-client-to-server/regressions/chained-ternary.opy b/compatibility/fixtures/real-world/overpy-client-to-server/regressions/chained-ternary.opy new file mode 100644 index 0000000..94fc27f --- /dev/null +++ b/compatibility/fixtures/real-world/overpy-client-to-server/regressions/chained-ternary.opy @@ -0,0 +1,3 @@ +rule "chained ternary regression": + @Event global + debug(1 if true else 2 if false else 3) diff --git a/compatibility/fixtures/real-world/overpy-cronch/fixture.json b/compatibility/fixtures/real-world/overpy-cronch/fixture.json index a7987a6..dde78d4 100644 --- a/compatibility/fixtures/real-world/overpy-cronch/fixture.json +++ b/compatibility/fixtures/real-world/overpy-cronch/fixture.json @@ -25,5 +25,15 @@ }, "runtimeSeconds": 6.1, "schemaVersion": 1, - "source": "cronch.opy" + "source": "cronch.opy", + "regressions": [ + { + "id": "real-world/overpy-cronch/postfix-increment", + "source": "regressions/postfix-increment.opy", + "derivedFrom": "cronch.opy", + "expectedReferenceStatus": "success", + "kind": "minimized-regression", + "provenance": "oracle:real-world/overpy-cronch/oracle.json; source path 31-34" + } + ] } diff --git a/compatibility/fixtures/real-world/overpy-cronch/regressions/postfix-increment.opy b/compatibility/fixtures/real-world/overpy-cronch/regressions/postfix-increment.opy new file mode 100644 index 0000000..867506f --- /dev/null +++ b/compatibility/fixtures/real-world/overpy-cronch/regressions/postfix-increment.opy @@ -0,0 +1,5 @@ +globalvar counter = 0 + +rule "postfix increment regression": + @Event global + counter++ diff --git a/compatibility/fixtures/real-world/overpy-crosshair/fixture.json b/compatibility/fixtures/real-world/overpy-crosshair/fixture.json index ef0db3d..95de876 100644 --- a/compatibility/fixtures/real-world/overpy-crosshair/fixture.json +++ b/compatibility/fixtures/real-world/overpy-crosshair/fixture.json @@ -25,5 +25,15 @@ }, "runtimeSeconds": 3.2, "schemaVersion": 1, - "source": "crosshair.opy" + "source": "crosshair.opy", + "regressions": [ + { + "id": "real-world/overpy-crosshair/byte-string-modifier", + "source": "regressions/byte-string-modifier.opy", + "derivedFrom": "crosshair.opy", + "expectedReferenceStatus": "success", + "kind": "minimized-regression", + "provenance": "oracle:real-world/overpy-crosshair/oracle.json; source path 31" + } + ] } diff --git a/compatibility/fixtures/real-world/overpy-crosshair/regressions/byte-string-modifier.opy b/compatibility/fixtures/real-world/overpy-crosshair/regressions/byte-string-modifier.opy new file mode 100644 index 0000000..dce1b43 --- /dev/null +++ b/compatibility/fixtures/real-world/overpy-crosshair/regressions/byte-string-modifier.opy @@ -0,0 +1,3 @@ +rule "byte string regression": + @Event global + debug(b"text") diff --git a/compatibility/fixtures/real-world/overpy-inputhud/fixture.json b/compatibility/fixtures/real-world/overpy-inputhud/fixture.json index 35d13de..9092df9 100644 --- a/compatibility/fixtures/real-world/overpy-inputhud/fixture.json +++ b/compatibility/fixtures/real-world/overpy-inputhud/fixture.json @@ -26,5 +26,15 @@ }, "runtimeSeconds": 3.2, "schemaVersion": 1, - "source": "inputhud.opy" + "source": "inputhud.opy", + "regressions": [ + { + "id": "real-world/overpy-inputhud/implicit-string-concatenation", + "source": "regressions/implicit-string-concatenation.opy", + "derivedFrom": "inputhud.opy", + "expectedReferenceStatus": "success", + "kind": "minimized-regression", + "provenance": "oracle:real-world/overpy-inputhud/oracle.json; source path 40-44" + } + ] } diff --git a/compatibility/fixtures/real-world/overpy-inputhud/regressions/implicit-string-concatenation.opy b/compatibility/fixtures/real-world/overpy-inputhud/regressions/implicit-string-concatenation.opy new file mode 100644 index 0000000..9c76f86 --- /dev/null +++ b/compatibility/fixtures/real-world/overpy-inputhud/regressions/implicit-string-concatenation.opy @@ -0,0 +1,3 @@ +rule "implicit concatenation regression": + @Event global + debug(("one" "two")) diff --git a/compatibility/fixtures/real-world/overpy-parabola/fixture.json b/compatibility/fixtures/real-world/overpy-parabola/fixture.json index fe6a76b..4e6bc5b 100644 --- a/compatibility/fixtures/real-world/overpy-parabola/fixture.json +++ b/compatibility/fixtures/real-world/overpy-parabola/fixture.json @@ -25,5 +25,15 @@ }, "runtimeSeconds": 3.0, "schemaVersion": 1, - "source": "parabola.opy" + "source": "parabola.opy", + "regressions": [ + { + "id": "real-world/overpy-parabola/numeric-enum-member", + "source": "regressions/numeric-enum-member.opy", + "derivedFrom": "parabola.opy", + "expectedReferenceStatus": "success", + "kind": "minimized-regression", + "provenance": "oracle:real-world/overpy-parabola/oracle.json; source path 35" + } + ] } diff --git a/compatibility/fixtures/real-world/overpy-parabola/regressions/numeric-enum-member.opy b/compatibility/fixtures/real-world/overpy-parabola/regressions/numeric-enum-member.opy new file mode 100644 index 0000000..1d25923 --- /dev/null +++ b/compatibility/fixtures/real-world/overpy-parabola/regressions/numeric-enum-member.opy @@ -0,0 +1,3 @@ +rule "numeric enum regression": + @Event global + debug(Team.2) diff --git a/compatibility/fixtures/real-world/overpy-santa/fixture.json b/compatibility/fixtures/real-world/overpy-santa/fixture.json index 4720b14..bd20f1e 100644 --- a/compatibility/fixtures/real-world/overpy-santa/fixture.json +++ b/compatibility/fixtures/real-world/overpy-santa/fixture.json @@ -25,5 +25,15 @@ }, "runtimeSeconds": 4.1, "schemaVersion": 1, - "source": "santa.opy" + "source": "santa.opy", + "regressions": [ + { + "id": "real-world/overpy-santa/do-while", + "source": "regressions/do-while.opy", + "derivedFrom": "santa.opy", + "expectedReferenceStatus": "success", + "kind": "minimized-regression", + "provenance": "oracle:real-world/overpy-santa/oracle.json; source path 204-208" + } + ] } diff --git a/compatibility/fixtures/real-world/overpy-santa/regressions/do-while.opy b/compatibility/fixtures/real-world/overpy-santa/regressions/do-while.opy new file mode 100644 index 0000000..8d021fa --- /dev/null +++ b/compatibility/fixtures/real-world/overpy-santa/regressions/do-while.opy @@ -0,0 +1,5 @@ +rule "do while regression": + @Event global + do: + wait() + while true diff --git a/compatibility/support-matrix.json b/compatibility/support-matrix.json index e5ad706..6681d10 100644 --- a/compatibility/support-matrix.json +++ b/compatibility/support-matrix.json @@ -48,7 +48,7 @@ "upstream:src/tests/strings.opy", "upstream:src/tests/arrays.opy" ], - "notes": "Corpus-evidenced: differential suite runs 26 fixtures; upstream tokenizer surface probed." + "notes": "Corpus-evidenced: differential suite runs 27 fixtures; upstream tokenizer surface probed." }, { "id": "syntax/expressions", @@ -466,7 +466,7 @@ "docs:docs/hir/opy-hir-v1.md", "test:opy-frontend-differential" ], - "notes": "Issues #3-#7. Fully Workshop-independent; the whole corpus is the acceptance corpus. Differential harness wired (issue #7): every fixture runs through the native pipeline in cargo test with structural self-checks (HIR validation, wire round-trip, deterministic dump), status/rule-name parity against the recorded oracle.json snapshots, and a machine-readable report (target/opy-differential-report.json). 14/26 corpus fixtures resolve, 12/26 produce expected diagnostics with documented codes; 7 documented reference gaps (reference accepts, native rejects declared-rejected or not-yet-declared surface)." + "notes": "Issues #3-#7 and #25. Fully Workshop-independent; the whole corpus is the acceptance corpus. Differential harness runs every fixture through the native pipeline in cargo test with structural self-checks (HIR validation, wire round-trip, deterministic dump), status/rule-name parity against recorded oracle.json snapshots, explicit native evidence expectations, and a machine-readable report (target/opy-differential-report.json). 15/27 corpus fixtures resolve, 12/27 produce expected diagnostics with documented codes, and 7 reference-success/native-failure cases are reported as known gaps rather than matches." }, { "id": "compilation/workshop-lowering", @@ -486,9 +486,10 @@ "state": "lowering-dependent", "evidence": [ "fixtures:synthetic", - "fixtures:real-world" + "fixtures:real-world", + "fixtures:census/workshop-feature-census" ], - "notes": "Becomes end-to-end-supported only after the workshop-rs integration stage (#8); differential harness contract is ready (compatibility/diff.py, issue #7 wiring)." + "notes": "Becomes end-to-end-supported only after the workshop-rs integration stage (#8); the OPY-side feature census records opaque workshop-rs#10 identities without duplicating catalog definitions. Differential harness contract is ready (compatibility/diff.py, issue #25)." }, { "id": "decompilation/workshop-to-opy", @@ -533,4 +534,4 @@ "decompilation": 2 } } -} \ No newline at end of file +} diff --git a/compatibility/tests/test_diff.py b/compatibility/tests/test_diff.py index bbda7f3..dc27461 100644 --- a/compatibility/tests/test_diff.py +++ b/compatibility/tests/test_diff.py @@ -25,6 +25,14 @@ def setUpClass(cls): ) cls.oracle = json.loads(snapshot.read_text(encoding="utf-8")) + def test_expectations_cover_every_fixture_with_evidence(self): + expectations = diff.load_expectations() + fixtures = set(diff.fixture_ids(COMPATIBILITY_DIR / "fixtures")) + self.assertEqual(set(expectations), fixtures) + for fixture, expectation in expectations.items(): + self.assertTrue(expectation["evidence"], fixture) + self.assertTrue(expectation["note"], fixture) + def write_result(self, root: Path, result: dict): path = root / result["fixture"] / "result.json" path.parent.mkdir(parents=True) @@ -80,6 +88,52 @@ def test_normalized_difference_is_regression(self): self.assertEqual(report_result["status"], "regression") self.assertIn("normalized-output", report_result["regressionStages"]) + def test_reference_success_native_failure_is_not_match(self): + result = copy.deepcopy(self.oracle) + result["compile"]["status"] = "failure" + result["compile"]["exitCode"] = 1 + result["compile"]["diagnostics"] = [{"severity": "error", "text": "Error: gap"}] + result["compile"]["workshopExact"] = "" + result["compile"]["workshop"] = "" + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self.write_result(root, result) + report_result = diff.compare_fixture( + COMPATIBILITY_DIR / "fixtures", + "synthetic/basic-rule", + root, + None, + ) + self.assertEqual(report_result["status"], "unexpected-divergence") + self.assertTrue(report_result["referenceGap"]) + + def test_declared_reference_gap_is_reported_as_known_gap(self): + oracle_path = ( + COMPATIBILITY_DIR + / "fixtures" + / "real-world" + / "overpy-santa" + / "oracle.json" + ) + result = json.loads(oracle_path.read_text(encoding="utf-8")) + result["compile"]["status"] = "failure" + result["compile"]["exitCode"] = 1 + result["compile"]["diagnostics"] = [{"severity": "error", "text": "Error: do"}] + result["compile"]["workshopExact"] = "" + result["compile"]["workshop"] = "" + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self.write_result(root, result) + report_result = diff.compare_fixture( + COMPATIBILITY_DIR / "fixtures", + "real-world/overpy-santa", + root, + None, + ) + self.assertEqual(report_result["status"], "known-gap") + self.assertTrue(report_result["referenceGap"]) + self.assertIn("normalized-output", report_result["regressionStages"]) + def test_diagnostic_difference_is_regression(self): result = copy.deepcopy(self.oracle) result["compile"]["diagnostics"] = [ diff --git a/compatibility/tests/test_runner.py b/compatibility/tests/test_runner.py index 9f7c4fb..a24edc0 100644 --- a/compatibility/tests/test_runner.py +++ b/compatibility/tests/test_runner.py @@ -36,7 +36,7 @@ def test_repository_fixture_metadata_and_snapshots_are_valid(self): fixtures = run_oracle.discover_fixtures( COMPATIBILITY_DIR / "fixtures" ) - self.assertEqual(len(fixtures), 26) + self.assertEqual(len(fixtures), 27) for fixture_path, fixture in fixtures: snapshot = fixture_path.parent / "oracle.json" self.assertTrue(snapshot.is_file(), fixture["id"]) @@ -60,6 +60,46 @@ def test_repository_fixture_metadata_and_snapshots_are_valid(self): ) self.assertEqual(overpy_cake["provenance"]["kind"], "imported-example") + def test_real_world_gaps_have_minimized_provenance_linked_regressions(self): + fixtures = run_oracle.discover_fixtures(COMPATIBILITY_DIR / "fixtures") + gaps = [ + fixture + for _, fixture in fixtures + if fixture["id"] + in { + "real-world/overpy-santa", + "real-world/overpy-cronch", + "real-world/overpy-broken-weapons", + "real-world/overpy-client-to-server", + "real-world/overpy-crosshair", + "real-world/overpy-inputhud", + "real-world/overpy-parabola", + } + ] + self.assertEqual(len(gaps), 7) + for fixture in gaps: + regressions = fixture.get("regressions") + self.assertIsInstance(regressions, list, fixture["id"]) + self.assertGreaterEqual(len(regressions), 1, fixture["id"]) + for regression in regressions: + source = (COMPATIBILITY_DIR / "fixtures" / fixture["id"] / regression["source"]).resolve() + self.assertTrue(source.is_file(), regression["id"]) + self.assertEqual(regression["derivedFrom"], fixture["source"]) + self.assertEqual(regression["expectedReferenceStatus"], fixture["expectedStatus"]) + self.assertEqual(regression["kind"], "minimized-regression") + self.assertIn("oracle:", regression["provenance"]) + + def test_census_uses_opaque_workshop_owned_feature_ids(self): + _, census = next( + fixture + for fixture in run_oracle.discover_fixtures(COMPATIBILITY_DIR / "fixtures") + if fixture[1]["id"] == "census/workshop-feature-census" + ) + self.assertEqual(census["censusContract"]["owner"], "workshop-rs") + self.assertEqual(census["censusContract"]["status"], "pending") + self.assertTrue(census["workshopFeatureIds"]) + self.assertTrue(all(isinstance(item, str) for item in census["workshopFeatureIds"])) + if __name__ == "__main__": unittest.main() diff --git a/crates/opy-frontend/tests/differential.rs b/crates/opy-frontend/tests/differential.rs index 3b05359..130f055 100644 --- a/crates/opy-frontend/tests/differential.rs +++ b/crates/opy-frontend/tests/differential.rs @@ -58,14 +58,16 @@ //! # Report //! //! A machine-readable report is written to -//! `target/opy-differential-report.json` listing per-fixture status -//! (`resolve` / `expected-diagnostic` / `divergence`), the native diagnostic -//! code, the reference status, rule-name comparison, and the support-matrix -//! feature ids the fixture evidences. +//! `target/opy-differential-report.json` listing per-fixture native status and +//! relationship classification (`match` / `known-gap` / +//! `unexpected-divergence` / `inconclusive`), the native diagnostic code, the +//! reference status, rule-name comparison, and the support-matrix feature ids +//! the fixture evidences. A reference-success/native-failure case is never +//! classified as a match. //! //! # Current corpus state //! -//! All 26 declared fixtures run (0 skips, 0 divergences): **14 resolve** and +//! All declared fixtures run (0 skips, 0 divergences): **15 resolve** and //! **12 produce expected diagnostics** with pinned codes; 7 fixtures are //! documented reference gaps (the oracle accepts a surface the native //! frontend deliberately rejects). Settings key-existence/leaf-kind @@ -89,6 +91,25 @@ fn fixtures_root() -> PathBuf { workspace_root().join("compatibility").join("fixtures") } +fn differential_expectations() -> Value { + serde_json::from_str( + &std::fs::read_to_string( + workspace_root().join("compatibility/differential-expectations.json"), + ) + .expect("differential-expectations.json must be readable"), + ) + .expect("differential-expectations.json must parse") +} + +fn expectation_for<'a>(expectations: &'a Value, id: &str) -> &'a Value { + expectations["cases"] + .as_array() + .expect("differential expectations must contain cases") + .iter() + .find(|case| case["fixture"].as_str() == Some(id)) + .unwrap_or_else(|| panic!("fixture '{id}' is missing from differential expectations")) +} + /// What the native frontend is expected to do for a fixture on this branch. #[derive(Clone, Copy, PartialEq, Eq, Debug)] enum Expect { @@ -208,6 +229,12 @@ fn declared_corpus() -> BTreeMap<&'static str, Case> { Some("parse-error"), "expected-failure fixture: the native frontend rejects missing-colon with parse-error; oracle status failure (parity).", ); + resolve( + &mut cases, + "census/workshop-feature-census", + true, + "OPy-side Workshop feature census boundary; canonical feature identities remain owned by workshop-rs#10.", + ); // Real-world fixtures derived from upstream OverPy examples (GPL-3.0-only, // provenance-recorded evidence; oracle status success). @@ -432,6 +459,7 @@ fn run_native( #[test] fn native_and_reference_agree_on_the_declared_corpus() { let corpus = declared_corpus(); + let expectations = differential_expectations(); let matrix: Value = serde_json::from_str( &std::fs::read_to_string(workspace_root().join("compatibility/support-matrix.json")) .unwrap(), @@ -440,9 +468,19 @@ fn native_and_reference_agree_on_the_declared_corpus() { let mut fixtures = BTreeMap::::new(); let mut divergences: Vec = Vec::new(); + let mut known_gaps: Vec = Vec::new(); let mut rule_name_mismatches = 0usize; - let mut counts = - json!({ "total": 0, "resolve": 0, "expectedDiagnostic": 0, "divergence": 0, "skipped": 0 }); + let mut counts = json!({ + "total": 0, + "resolve": 0, + "expectedDiagnostic": 0, + "divergence": 0, + "match": 0, + "knownGap": 0, + "unexpectedDivergence": 0, + "inconclusive": 0, + "skipped": 0 + }); let mut seen: Vec = Vec::new(); for manifest_path in discover_fixtures(&fixtures_root()) { @@ -454,6 +492,29 @@ fn native_and_reference_agree_on_the_declared_corpus() { add an explicit resolve/diagnostic entry with a note" ) }); + let expectation = expectation_for(&expectations, &id); + let expected_native_status = expectation["nativeStatus"] + .as_str() + .unwrap_or_else(|| panic!("{id}: nativeStatus is required")); + let expected_classification = expectation["classification"] + .as_str() + .unwrap_or_else(|| panic!("{id}: classification is required")); + let expected_evidence = expectation["evidence"] + .as_array() + .unwrap_or_else(|| panic!("{id}: evidence is required")); + assert!( + !expected_evidence.is_empty(), + "{id}: differential expectation evidence cannot be empty" + ); + assert_eq!( + expected_native_status, + if matches!(case.expect, Expect::Resolve) { + "success" + } else { + "failure" + }, + "{id}: differential expectation disagrees with native expectation table" + ); let fixture_dir = manifest_path.parent().unwrap().to_path_buf(); let source_path = fixture_dir.join(&source_name); let source = std::fs::read_to_string(&source_path) @@ -490,7 +551,7 @@ fn native_and_reference_agree_on_the_declared_corpus() { }); // Status determination against the expectation table. - let expect_resolve = matches!(case.expect, Expect::Resolve); + let expect_resolve = expected_native_status == "success"; let status = if native_ok == expect_resolve { if expect_resolve { "resolve" @@ -549,6 +610,21 @@ fn native_and_reference_agree_on_the_declared_corpus() { } } + let relationship_holds = match expected_classification { + "match" => !reference_gap, + "known-gap" | "unsupported" => reference_gap, + other => panic!("{id}: unsupported expectation classification '{other}'"), + }; + let classification = if skipped { + "inconclusive" + } else if status == "divergence" || !relationship_holds { + "unexpected-divergence" + } else if reference_gap { + expected_classification + } else { + "match" + }; + // Rule-name parity (informational, opt-in per fixture). let rule_names_entry = if case.rule_names && snapshot_present { match &native { @@ -586,12 +662,17 @@ fn native_and_reference_agree_on_the_declared_corpus() { "detail": detail, "referenceGap": reference_gap, "skip": skipped, + "classification": classification, + "expectedClassification": expected_classification, + "evidence": expected_evidence, }); let code = entry["native"].get("code").and_then(Value::as_str); let label = if skipped { "SKIP" } else if status == "divergence" { "FAIL" + } else if reference_gap { + "KNOWN GAP" } else { "PASS" }; @@ -616,6 +697,15 @@ fn native_and_reference_agree_on_the_declared_corpus() { "detail": detail, })); } + if classification == "known-gap" { + known_gaps.push(json!({ + "fixture": id, + "native": entry["native"], + "reference": entry["reference"], + "note": case.note, + "detail": detail, + })); + } if let Value::Number(count) = &mut counts[status_key] { *count = serde_json::Number::from(count.as_u64().expect("status counts start as u64") + 1); @@ -631,6 +721,16 @@ fn native_and_reference_agree_on_the_declared_corpus() { ); } } + let classification_key = match classification { + "known-gap" => "knownGap", + "unexpected-divergence" => "unexpectedDivergence", + other => other, + }; + if let Value::Number(count) = &mut counts[classification_key] { + *count = serde_json::Number::from( + count.as_u64().expect("classification counts start as u64") + 1, + ); + } fixtures.insert(id.clone(), entry); } @@ -644,10 +744,21 @@ fn native_and_reference_agree_on_the_declared_corpus() { missing.is_empty(), "declared corpus entries missing from compatibility/fixtures: {missing:?}" ); + let extra_expectations: Vec<&str> = expectations["cases"] + .as_array() + .expect("differential expectations must contain cases") + .iter() + .filter_map(|case| case["fixture"].as_str()) + .filter(|id| !seen.iter().any(|seen_id| seen_id == id)) + .collect(); + assert!( + extra_expectations.is_empty(), + "differential expectations reference missing fixtures: {extra_expectations:?}" + ); let report = json!({ "schemaVersion": 1, - "artifact": "opy-rs native-vs-reference differential report (issue #7, part B)", + "artifact": "opy-rs native-vs-reference differential report (issue #25)", "generatedBy": "crates/opy-frontend/tests/differential.rs", "frontend": { "name": FRONTEND_NAME, "version": FRONTEND_VERSION }, "reference": matrix["reference"], @@ -657,9 +768,14 @@ fn native_and_reference_agree_on_the_declared_corpus() { "expectedDiagnostic": counts["expectedDiagnostic"], "divergence": counts["divergence"], "skipped": counts["skipped"], + "match": counts["match"], + "knownGap": counts["knownGap"], + "unexpectedDivergence": counts["unexpectedDivergence"], + "inconclusive": counts["inconclusive"], "ruleNameMismatches": rule_name_mismatches, }, "divergences": divergences, + "knownGaps": known_gaps, "fixtures": Value::Object(fixtures.into_iter().collect()), }); let report_path = workspace_root().join("target/opy-differential-report.json"); diff --git a/docs/compatibility/upstream-references.md b/docs/compatibility/upstream-references.md index 6c0dbbb..a377a51 100644 --- a/docs/compatibility/upstream-references.md +++ b/docs/compatibility/upstream-references.md @@ -49,7 +49,7 @@ never `latest` or a range (see the pinning policy below). pinned `pnpm-lock.yaml`; `pnpm install` resolves `overpy@9.7.10` by its integrity hash. * The compatibility corpus was re-run against a fresh install of the pinned - package on 2026-08-16: all 26 fixture snapshots (`compatibility/fixtures/**/oracle.json`) + package on 2026-08-17: all 27 fixture snapshots (`compatibility/fixtures/**/oracle.json`) match byte-for-byte (run `python3 compatibility/run_oracle.py`). * The imported example fixtures were verified byte-identical to the pinned tree's `examples/` content (see `compatibility/fixtures/README.md`). diff --git a/docs/opy/compatibility-baseline.md b/docs/opy/compatibility-baseline.md index f921c84..4c3012b 100644 --- a/docs/opy/compatibility-baseline.md +++ b/docs/opy/compatibility-baseline.md @@ -17,7 +17,7 @@ The reference identity is the pinned OverPy 9.7.10 content (`889d9749d1def17f146548cbddb94ea1ab015847`); see [`docs/compatibility/upstream-references.md`](../compatibility/upstream-references.md) for provenance. Evidence claims in this document were verified against the -pinned oracle (all 26 corpus snapshots match on 2026-08-16). The opy-rs +pinned oracle (all 27 corpus snapshots match on 2026-08-17). The opy-rs frontend foundation is implemented and merged on `main` (issues #3–#7 partially delivered via PRs #9–#14); the category table below is the **tier assignment contract** for the remaining surface. The state column of