Skip to content

Enforce retention maturity and join safety - #6

Merged
parasdoshicom merged 1 commit into
mainfrom
codex/enforcement-evals
Sep 8, 2026
Merged

parasdoshicom merged 1 commit into
mainfrom
codex/enforcement-evals

Conversation

@parasdoshicom

Copy link
Copy Markdown
Owner

Claude's review identified a real gap: ChatData encoded strong analytical judgment, but too much of that judgment still depended on prose. This release turns two common failure modes into runnable checks and adds public evidence for what the package can actually enforce.

  • add exact calendar-period retention with fixed denominators, duplicate handling, explicit maturity, and fail-closed input checks
  • add join-cardinality and measure-reconciliation auditing that blocks fanout inflation
  • add an optional guarded read-only DuckDB adapter for an existing local database
  • add one worked failure to every skill
  • add a reproducible five-case planted-defect evaluation and blinded paired-comparison harness
  • bump the public plugin to 1.7.0 and update capability, privacy, and installation docs

Validation:

  • python3 -m unittest discover -s tests -p 'test_*.py' — 189 passed, 2 optional DuckDB tests skipped in base Python
  • /tmp/chatdata-duckdb-test/bin/python -m unittest tests.test_duckdb_query — 5 passed with DuckDB installed
  • python3 scripts/validate.py — passed
  • claude plugin validate plugins/chatdata — passed
  • enforcement evaluation reproduced the checked-in 5/5 result byte for byte
  • live Claude Code and Codex synthetic doctor runs loaded ChatData 1.7.0 and passed all five checks
  • Cursor project installation passed; its CLI model run is blocked on this machine because Cursor CLI is not authenticated

No completed head-to-head model comparison is claimed. The release ships the blinded harness needed to run and publish one honestly.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-08T19:38:34.642596Z 21d0d98 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@parasdoshicom
parasdoshicom merged commit c99bf8f into main Sep 8, 2026
6 checks passed
@parasdoshicom
parasdoshicom deleted the codex/enforcement-evals branch September 8, 2026 19:33

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 21d0d98700

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +24 to +28
if ";" in stripped:
raise ValueError("Only one SQL statement is allowed")
if not _ALLOWED_START.match(stripped):
raise ValueError("Only read-only SELECT or WITH statements are allowed")
match = _FORBIDDEN.search(stripped)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Tokenize SQL before applying the read-only filter

For otherwise valid read-only queries containing a semicolon or forbidden-looking word inside a string literal or comment, the raw-text checks reject the query—for example, SELECT 'delete' AS action, SELECT 'a;b' AS value, and SELECT 1 -- update stats. This contradicts the adapter's promise to accept one read-only SELECT or WITH statement and will block common predicates on action/status text; inspect SQL tokens so only actual statement separators and executable keywords are rejected.

Useful? React with 👍 / 👎.

Comment on lines +16 to +17
ids = [case.get("id") for case in value]
if any(not isinstance(item, str) or not item for item in ids) or len(ids) != len(set(ids)):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restrict case IDs to safe filename stems

When a custom --cases file supplies an absolute or traversing ID, this validation accepts it and the later path joins can escape the response directories. For example, an ID derived from /tmp/shared.md makes both the ChatData and unguided conditions read that same absolute file, so prepare succeeds with two identical responses while silently ignoring both supplied condition directories and invalidating the comparison. Require IDs to be simple relative filename stems without separators or traversal components.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant