Skip to content

Add reproducible Context Fabric text benchmark corpus - #2804

Merged
Chris0Jeky merged 2 commits into
mainfrom
issue-2319/benchmark-corpus
Sep 8, 2026
Merged

Add reproducible Context Fabric text benchmark corpus#2804
Chris0Jeky merged 2 commits into
mainfrom
issue-2319/benchmark-corpus

Conversation

@Chris0Jeky

Copy link
Copy Markdown
Owner

Refs #2319

Context Fabric processors need shared, reproducible fixtures before their quality can be compared. Add the first CF-24A corpus slice: nine synthetic text/transcript fixtures, exact source hashes, reference outputs, licence/method/date metadata, and explicit hostile-injection controls.

The local command validates the corpus and its 16 KiB source-byte budget, then computes precision/recall/F1 by candidate kind when supplied predictions cover every fixture. Without predictions, metrics are unavailable. The example predictions are deliberately imperfect; no processor is executed or measured. Audio, image, PDF and runtime benchmark acceptance remains on #2319.

Validation at 6c9bec284b1feaade2cd49ce3bf9219c0b1d4320:

  • py -3 -B -m unittest discover -s scripts/context_fabric -p "test_*.py": 21 passed.
  • Corpus validation: 9 fixtures, 934 source bytes; example scoring and schema validation passed.
  • node scripts/check-doc-links.mjs: 664 Markdown files, zero broken links.
  • node scripts/check-docs-governance.mjs and git diff --check: passed.

The testing guide documents the command and its limits. Review-first execution is unchanged. Hosted CI and independent Terra review are separate pending gates. No live provider, external media, or real user data was used.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-08T01:09:12.815784Z 6c9bec2 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Chris0Jeky

Copy link
Copy Markdown
Owner Author

Independent Terra review round 1 at 6c9bec284b1feaade2cd49ce3bf9219c0b1d4320 against db102dcba25cc5f347f22b38ce13d00502948ed9: no confirmed CRITICAL/HIGH findings.

The reviewer independently recomputed all nine source hashes and reference IDs (zero mismatches, 934 bytes), checked byte-preserving attributes, per-kind exact-set scoring, incomplete-prediction rejection, and unavailable metrics when no predictions are provided. No processor or media/runtime measurements are claimed.

One nonblocking documentation/validation parity finding is retained on #2319: the README describes validation against the JSON Schema, while the command uses manual validation and the schema test validates the schema itself. Numeric optional notes illustrates a schema/CLI difference; the committed corpus is unaffected. Disposition: tracked on the existing corpus issue for its next validation slice, no extra fix cascade in this PR.

Local 21-test, corpus, link and governance evidence remains as recorded. Hosted CI is a separate merge gate.

@Chris0Jeky Chris0Jeky moved this from Pending to Review in Taskdeck Execution Sep 8, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6c9bec284b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/context_fabric/benchmark_fixtures.py
Comment thread docs/TESTING_GUIDE.md
@Chris0Jeky
Chris0Jeky merged commit 72a181c into main Sep 8, 2026
35 checks passed
@Chris0Jeky
Chris0Jeky deleted the issue-2319/benchmark-corpus branch September 8, 2026 01:29
@github-project-automation github-project-automation Bot moved this from Review to Done in Taskdeck Execution Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant