Skip to content

fix(evals): admit re-signed ripgrep 15.2.0 binary - #307

Merged
mdsmithaustin merged 1 commit into
mainfrom
fix/rg-pin-15.2.0-resign
Oct 24, 2026
Merged

mdsmithaustin merged 1 commit into
mainfrom
fix/rg-pin-15.2.0-resign

Conversation

@mdsmithaustin

@mdsmithaustin mdsmithaustin commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Why

The Claude Code evaluation adapter rejects the re-signed ripgrep 15.2.0 binary bundled with ChatGPT because its digest differs from the admitted builds. Add its reviewed digest so evaluation runs can use that binary.

What changed

  • Admit SHA-256 1507d45d9b1f6e7a3b0daaa788554c327e95e040fa7048d93d19ba97f71fa073 alongside the existing ripgrep fingerprints.
  • Extend the runtime admission test to cover all three accepted digests.
  • Document the re-signed build and its signature verification provenance.

Scope

Covers ripgrep admission in the Claude Code promises evaluation adapter. Other executable pins and isolation rules remain unchanged.

Blast Radius

Touches Claude Code evaluation runs that use ChatGPT's bundled ripgrep. Admission still requires an explicitly listed digest, and later checks still reject a binary change during a run.

Verification

Inspected the full branch diff. Tests and live evaluation runs were not executed in this PR drafting phase.

✅ Low: The change adds one reviewed SHA-256 to the ripgrep allowlist tuple. The earlier digests are kept, the test and README are extended to match, and the new value is 64 hex characters and equals shasum -a 256 of the on-host ChatGPT.app rg binary.

Updates from git push no-mistakes

@mdsmithaustin

Copy link
Copy Markdown
Owner Author

Verdict: PASS+NOTES
Head: 0005966
Patch-id: 7c18052ccb3b0243522dbb93e56f4691eb15ba28
Docs: pass

Independent check on the measured host (Darwin 25.6.0 arm64) against base 0bc7cf93.

  1. Digest: shasum -a 256 of /Applications/ChatGPT.app/Contents/Resources/codex-cli/codex-path/rg gives 1507d45d9b1f6e7a3b0daaa788554c327e95e040fa7048d93d19ba97f71fa073. ChatGPT.app's sealed CodeResources lists the same value as that file's hash2, and codesign --verify --strict --deep accepts the whole app for team 2DC432GLL2. The digest names bytes that OpenAI ships in its sealed app bundle.
  2. Signature: the requested codesign --verify --strict -R='identifier "com.openai.codex.rg" and certificate leaf[subject.OU] = "2DC432GLL2"' exits 0. The binary's full designated requirement (anchor apple generic plus the Developer ID OIDs) also passes. Signer: Developer ID Application: OpenAI OpCo, LLC, timestamp 2026-10-21.
  3. Version: rg --version prints ripgrep 15.2.0 (rev e89fff89ac).
  4. Tests: python3.14 -m unittest -v test_claude_code from evals/promises ran 49 tests on each side. The parent had 4 errors: NativeClaude admission and three NativeGit tests, each IsolationUnavailable: unsupported executable: .../codex-path/rg. The head passed all 49. A per-test diff shows only those four going from ERROR to ok, plus the renamed ripgrep digest test, which passes on both sides. No skips.
  5. Lint: tools/gate-check.sh lint at the head passed all eight checks with exit 0 and left the tree clean. PR CI is green.
  6. Preflight: live.py has no dry-run flag, and live.py run calls the adapter's preflight() right before running cases. I called live.preflight("claude-code") directly. It admits the host runtime and runs claude auth status, which uses no model. At the head it passed, admitting Claude 2.1.307 with rg fingerprint 1507d45d.... At the parent it refused with the error recorded in results/2026-10-22-sync-eb4db2b.md. No billed run was started.
  7. Safety: the adapter admits rg only by exact digest and never runs codesign on it, so the signature check is review provenance, which items 1 and 2 supply. Seatbelt grants, the child PATH, and the per-run digest hold are unchanged. The two older digests stay admissible, but neither build is on this host now. They matter only if an older ChatGPT.app comes back, and then that build would be admitted without notice. Both were reviewed before, and every admitted build runs under the same Seatbelt grants, so the boundary does not widen. Pruning them is optional.
  8. Docs: the README sentence matches PINNED_TOOLS: the original fingerprint plus two 15.2.0 builds, exact-digest admission, and refusal of any other bytes. The version check above confirms "same release". Whether the code is otherwise byte-identical to the e97d666f... build can't be checked here, because those bytes are gone. The sync receipt above is a dated record and should stay as written. No other doc names these digests.
  9. Gap, not new in this PR: run evidence does not say which rg build a run used. launch.json and x_binary record only the Claude binary. With three admissible builds, record the rg digest if a result ever depends on it.
  10. The PR body's Verification section says no tests or live runs were executed. Items 4 to 6 cover the tests and the preflight.
  11. Main moved to 8846421a (fix(poteto-mode): exclude PR recaps from review acknowledgments #305) during this review. That commit touches only watch-pr, pr-recap, babysit, and PORTING.md, nothing under evals/ or tools/. The PR merges cleanly onto it, and the patch-id against it is unchanged.

…de runs

ChatGPT.app re-signed its bundled ripgrep 15.2.0 on 2026-10-21, which
changed the binary's bytes, so the Claude Code adapter refused it and every
Claude Code live run and four native tests failed. Admit the new digest
beside the earlier ones; it passes strict code-signature verification for
OpenAI team 2DC432GLL2.
@mdsmithaustin
mdsmithaustin force-pushed the fix/rg-pin-15.2.0-resign branch from 0005966 to 4801731 Compare October 24, 2026 18:10
@mdsmithaustin
mdsmithaustin merged commit c5019db into main Oct 24, 2026
7 checks passed
@mdsmithaustin
mdsmithaustin deleted the fix/rg-pin-15.2.0-resign branch October 24, 2026 18:21
mdsmithaustin added a commit that referenced this pull request Oct 24, 2026
 live promises (#310)

Record the Claude Code live promise runs owed since the ripgrep pin
blocked them, now unblocked by #307. Eight bound cases ran at d7ae919 and
cover the promises whose owners changed in #297, #300, and #304: 9 PASS,
1 INCONCLUSIVE handed to a judge, and no FAIL. Two turns hit the
300-second cap after the routed skill had loaded. Swarm workers were all
BLOCKED because the sandbox cannot read dirname; #309 tracks that.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant