Repository navigation
Fix causal rollback guidance in ChatData 1.7.2 - #8
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bfaa04b5ae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| Try to disprove the leading explanation with a negative control, unchanged segment, independent source, or timing check. Observational contribution is not causation. Deliver the arithmetic bridge, ranked hypotheses, ruled-out explanations, uncertainty, and an action tied to the next test. If evidence only supports where the change occurred, say that. | ||
|
|
||
| A decomposition cannot identify what an intervention caused or predict what reversing it would do. Unless the evidence includes a design that identifies the intervention's effect, state that both the intervention effect and the rollback effect are unknown. Do not claim that a release helped, harmed, had no effect, or that rollback would or would not reverse the movement. Do not use the direction of an observed within-segment change to argue for or against the intervention; report it only as an association. When asked for an action, say that this arithmetic alone neither supports nor rules out rollback. Do not speculate that rollback could remove a gain, restore a rate, prevent harm, or have any other directional outcome, even conditionally. Keep mechanism hypotheses separate from the action and name the comparison or exposure evidence needed to test them. |
There was a problem hiding this comment.
When an investigation includes a randomized holdout, valid staged rollout, or even a rollback experiment, the preceding exception allows the design to identify an effect, but the following sentences categorically prohibit reporting that the release helped, harmed, or had no effect and forbid any directional rollback conclusion. This forces the skill to suppress valid causal evidence; make these prohibitions conditional on the absence of an identifying design, and distinguish identification of the original intervention from identification of its reversal.
Useful? React with 👍 / 👎.
Claude could correctly decompose an activation drop while still turning the observed segment movement into an unsupported claim about what a release caused and what rollback would do. This release makes the root-cause workflow keep arithmetic attribution separate from causal and rollback conclusions, and makes ordinary Claude analysis invoke the analytical skill and its standalone helper before answering.
The new synthetic regression uses the same prompt in three fresh baseline and three fresh ChatData sessions. Baseline Claude passed 0/3; ChatData 1.7.2 passed 3/3 after the repair. The checked-in result includes prompts, outputs, tool traces, costs, grading criteria, and limitations. This is one repaired synthetic case, not a general performance claim.
Validation:
python3 scripts/validate.pyclaude plugin validate plugins/chatdata