ci(root): add recovery-mode to beta publish for auto-retry on Rekor conflicts

[claude]

Requested by Zahin Mohammad · Slack thread

Description

The beta publish workflow (.github/workflows/publish.yml) can get stuck in a state where retrying it fails with TLOG_CREATE_ENTRY_ERROR (409) sigstore/Rekor collisions for packages whose npm registry write failed on a previous attempt, even though nothing else about the release changed.

What broke: npm publish with provenance (NPM_CONFIG_PROVENANCE: true) logs a Sigstore/Rekor transparency-log entry for the published tarball before/alongside writing to the npm registry. If the registry write itself fails (network blip, registry error, etc.) after the Rekor entry was already created, a naive retry of the workflow fails again with a 409 from Rekor's TLOG_CREATE_ENTRY_ERROR.

Why: Rekor entries are keyed on the tarball's sha256 digest, and that digest is derived from the packed tarball contents, which include the version in package.json. The live npm dist-tag for a stuck package never advances (its registry write failed), so simply re-running the workflow recomputes the exact same version -> the exact same tarball -> the exact same digest -> the same already-logged Rekor entry -> 409 on retry. There was previously no way to get past this short of manually bumping versions by hand.

What this PR changes:

• .github/workflows/publish.yml: adds a recovery-mode boolean workflow_dispatch input (default false). When set, the Lerna Publish step is marked continue-on-error: ${{ inputs.recovery-mode }} so a Rekor conflict there doesn't hard-fail the job, and the Verify Publish step now receives a RECOVERY_MODE: ${{ inputs.recovery-mode }} env var.
• scripts/verify-release.ts: adds a publishWithRecovery() helper used by verifyPackage() in place of the old direct npm publish call. When RECOVERY_MODE is set, it detects a Rekor conflict in the publish output (TLOG_CREATE_ENTRY_ERROR, or a (409) mentioning the transparency log), bumps that package's version in package.json with semver.inc(..., 'prerelease', ..., preid), and retries — up to MAX_RECOVERY_ATTEMPTS (5) attempts per package — until it lands on a version whose tarball digest hasn't already been logged. verify() also no longer aborts on the first stuck package when recovery mode is on; it keeps going through the rest of the packages and reports overall failure via process.exitCode at the end.
• scripts/prepare-release.ts is untouched — no CLI flags were added there, and the fix lives entirely in the publish/verify steps that run after version prep.
• With recovery-mode left at its default (false), behavior is unchanged: publishWithRecovery() makes exactly one attempt and rethrows on failure, same as the previous direct npm publish call.

How to use it (runbook):

1. When a beta publish run fails with TLOG_CREATE_ENTRY_ERROR (409) on retry, re-trigger Publish @bitgo-beta via workflow_dispatch with recovery-mode: true.
2. No manual bump count is needed — verify-release.ts detects the Rekor conflict per package and bumps+retries automatically (up to 5 attempts per package) until each stuck package clears.
3. recovery-mode: false (the default) preserves today's behavior exactly.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

• Reviewed the diff to confirm recovery-mode: false (the default) is a no-op: continue-on-error evaluates to false and publishWithRecovery()'s loop bound collapses to a single attempt that rethrows immediately, identical to the previous direct npm publish call.
• Manually traced the recovery path: a TLOG_CREATE_ENTRY_ERROR/409 in the publish output is caught, the package's package.json version is bumped with semver.inc(..., 'prerelease', ..., preid) and rewritten to disk, and the publish is retried, up to 5 attempts before giving up with an explicit error.
• Did not run the actual publish.yml workflow (no credentials, and it is a live release workflow that should never be triggered from this kind of change).

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• My commits follow Conventional Commits
• The ticket or github issue was included in the commit message as a reference (no Linear ticket exists for this CI-only infra change; used the INF-000 junk-drawer reference, following the WEB-000 / BTC-000 / CSHLD-000 convention documented in CLAUDE.md)
• I have added tests that prove my fix is effective (this is a release-workflow-only change; there is no automated test harness for publish.yml itself, and verify-release.ts has no existing unit test suite to extend)

---

Generated by Claude Code

[claude]

The "Validate Humans In The Loop" check failure is expected here — the commit is signed with an SSH key (this session's default), not GPG, and isn't in the registered-engineer key allowlist. That check exists specifically to require human sign-off on AI-authored commits, so this isn't something to "fix" by re-signing — it needs 2 approvals from the CODEOWNERS group (@bitgo/btc-team, @bitgo/ecosystem, @bitgo/wallet-core, @bitgo/wallet-core-india) to pass, same as it would for any AI-authored commit.

---

Generated by Claude Code

[claude]

Pushed 9005101 addressing review feedback: replaced the two-input design (recovery-mode + recovery-bump-count) with a single recovery-mode boolean. scripts/prepare-release.ts is back to being byte-identical to master; the auto-bump-on-Rekor-conflict retry (capped at 5 attempts per package) now lives in scripts/verify-release.ts, gated behind RECOVERY_MODE, with no behavior change when it's off. Also added continue-on-error to the "Lerna Publish" step when recovery-mode is enabled, since a stuck package's Rekor conflict fails that step outright and would otherwise skip the "Verify Publish" step where the retry logic runs.

---

Generated by Claude Code

[zahin-mohammad]

lgtm

causes failing HITL checks

[gokulhost]

stamp