Code review comment posting is nondeterministic — recipe prompt should explicitly require gh pr comment #25

Summary

The shared claude-code-review.yml reusable workflow posts the review comment only intermittently. The review almost always runs to completion, but whether the result actually lands as a PR comment is nondeterministic — it depends on whether Claude chooses to call gh pr comment itself, which is driven by an implicit instruction in the recipe's default prompt.

We hit this in DostEducation/dost-ai-voicebot: the workflow ran green, but no review comment appeared on the PR. Meanwhile other repos on the same recipe (DostEducation/admindashboard, etc.) do get the comment — which initially looked like a per-repo misconfiguration but is actually the same latent flakiness.

Root cause

The recipe runs anthropics/[email protected] in agent mode (auto-detected for pull_request events) with track_progress: false (it's commented out in the recipe because it's incompatible with the labeled trigger consumers use).

In agent mode with track_progress off, the action does not auto-post Claude's output as a PR comment. Posting becomes Claude's own responsibility — it must run gh pr comment (allowlisted via the default Bash(gh pr comment:*) tool).

The default prompt only says:

Review Output Format - IMPORTANT
Create ONE single comment with your complete review using this structure: ...
Do NOT create multiple inline comments. Put everything in ONE comment.

This describes the format of the comment but never explicitly tells Claude to run gh pr comment to post it. So Claude sometimes interprets "create one comment" as "actually post via the tool" and sometimes as "produce one block of text" and stops. The result is flaky.

Evidence

Working run — DostEducation/admindashboard PR #1935 (run 28353530110, agent mode, default recipe prompt):

• Log contains: "text": "Now let me post my review as a single comment:" followed by a Bash tool call.
• PR #1935 has claude-authored ## Code Review Summary comments. ✅ Posted.

Failing run — DostEducation/dost-ai-voicebot PR #22 (agent mode, identical recipe prompt):

• Claude emitted the full review as its final assistant text and never called gh pr comment.
• No review comment on the PR (only the unrelated pytest-coverage comment). ❌ Not posted.
• Likely aggravated by an unusually large/truncated diff that nudged Claude to "wrap up" by printing rather than posting — but the underlying cause is the implicit prompt either way.

Same recipe (@main), same [email protected], same agent mode, same model (claude-sonnet-4-5), byte-identical consumer workflow files. The only difference is whether Claude decided to post.

Affected repos

All consumers of ColoredCow/engineering-recipes/.github/workflows/claude-code-review.yml@main, e.g.:

• DostEducation/dost-ai-voicebot
• DostEducation/admindashboard
• ColoredCow/megafitmeals
• ColoredCow/goonj

The ones that "work today" are getting lucky on the flaky prompt and can silently miss reviews too.

Proposed fix (minimal, correct)

Make the posting step explicit and mandatory in the recipe's default prompt. Append a step like:

6. **Post the Review (REQUIRED — do not skip)**
   You are running headless: your text output is NOT posted automatically.
   You MUST publish the review as a single PR comment yourself:
   - Write the full review markdown to /tmp/review.md (Write tool), then run:
     gh pr comment <PR_NUMBER> --body-file /tmp/review.md
   The task is not complete until the comment is posted.

And add Write to the default allowed_tools fallback so --body-file works:

'Read,Write,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)'

(Inline --body works too and needs no Write, but --body-file avoids shell-escaping large markdown reviews.)

Because the recipe is pinned @main, this fixes every consumer with zero change to any consumer repo — and therefore no claude-code-action OIDC "Workflow validation failed" friction (that error only triggers when a consumer edits its own code-review.yml).

More robust alternative (larger change)

Set track_progress: true so the action creates/updates a sticky review comment deterministically instead of relying on Claude. This requires switching consumers off the labeled trigger to a supported one (opened / synchronize / ready_for_review / reopened), since track_progress doesn't support labeled — that's why it was disabled in the first place. Worth considering as a follow-up, but the explicit-prompt fix above resolves the immediate flakiness with the least disruption.

Notes / red herring

While debugging we briefly tried fixing this per-repo by overriding the prompt input in the consumer's code-review.yml. That works functionally, but editing the consumer workflow trips claude-code-action's OIDC token exchange (401 Workflow validation failed: the workflow file must … have identical content to the version on the repository's default branch) on the PR that introduces the edit — expected behavior, self-resolves on merge. Fixing the recipe upstream avoids this entirely since consumer workflows stay unchanged.