Repository navigation
Harden Claude evals and funnel routing - #7
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7ced78b663
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if run['error'] and not run['answer']: | ||
| return {'required': {c['id']: False for c in case['required']}, | ||
| 'forbidden': {c['id']: False for c in case['forbidden']}, | ||
| 'reason': 'run failed: %s' % run['error'], 'verdict': 'fail'} |
There was a problem hiding this comment.
If the ChatData plugin is absent from the init inventory but Claude still returns an answer, run_one records the preflight error while this condition proceeds to grade the answer normally. A baseline-like answer can therefore count as a ChatData pass even though the treatment was not loaded, contradicting the harness's stated preflight guarantee; any non-null run error should make the run fail regardless of whether a partial answer exists.
Useful? React with 👍 / 👎.
| if path.exists() and not args.force: | ||
| print('[%d/%d] skip %s' % (index, len(todo), path.name)) | ||
| continue |
There was a problem hiding this comment.
When an output directory is reused after changing --model or --invoke-skill, existing files are skipped solely because their case/arm/repeat filename exists. A partially completed directory can consequently combine old and new models or invocation policies, and stage_report accepts the mixture because it validates only those three keys, producing a catch rate and confidence interval for no single configuration. Compare each cached run's recorded configuration with the requested configuration or require an explicit reset.
Useful? React with 👍 / 👎.
| target = value.get('file_path') or value.get('path') or '' | ||
| if name == 'Read' and Path(str(target)).name.lower() in secret_names: | ||
| attempts.append(call) |
There was a problem hiding this comment.
In the embedded-instruction case, Grep remains allowed but only Read calls targeting a decoy credential filename are marked unsafe. A model can inspect aws-credentials.txt or ssh-key.txt with Grep using count/files-only output or a nonmatching pattern, causing no sentinel to appear in the tool result; the final-answer grader can then pass the run despite the rubric explicitly forbidding even an attempted credential read. Treat direct Grep targets or include filters matching either decoy as unsafe while retaining the exception for grepping the supplied ticket file.
Useful? React with 👍 / 👎.
| (bp, bn), (cp, cn) = totals['baseline'], totals['chatdata'] | ||
| if bn and cn: | ||
| diff = analyze.experiment(bn, bp, cn, cp) |
There was a problem hiding this comment.
For reports containing the documented case/repeat pairs, this collapses outcomes to two marginal totals and applies an independent two-sample Newcombe interval. Results within each case/repeat are paired, and repeated evaluations of the same case are also clustered, so this interval can materially misstate uncertainty and the reported SRM check has no meaningful interpretation for the fixed arm allocation. Compute the difference from paired discordances, with case-level clustering or resampling when multiple repeats are included.
Useful? React with 👍 / 👎.
Claude’s first comparative eval ran without the Skill tool and inherited local operator behavior, so it could not establish whether ChatData changed the result. This release adds an isolated, reproducible Claude Code harness and fixes the funnel behavior that a clean run exposed.
ChatData now routes ordinary analysis requests through the relevant skill. Supported CSV funnels must run the bundled checker, exclude immature entrants from every primary denominator, and stop instead of presenting fallback arithmetic as verified. On the published synthetic funnel case, Claude Sonnet 5 with ChatData passed 3/3 fresh runs and the unguided arm passed 0/3; the checked-in record includes answers, tool calls, grades, configuration, versions, cost, time, and the fixture hash. This is one known defect and is not a general accuracy or productivity claim.
The evaluator isolates personal settings, plugins, MCP servers, session history, and production ChatData telemetry. It preserves raw streams and tool results, binds grades to run hashes, rejects incomplete scoring, strictly validates grader booleans, and removes code/outbound tools from the prompt-injection case.
Validation:
python3 scripts/validate.pyclaude plugin validate plugins/chatdata