Skip to content

Mint GitHub tokens for publish through octo-sts instead of a stored PAT - #373

Draft
wallrj wants to merge 1 commit into
cert-manager:masterfrom
wallrj:octo-sts-publish
Draft

Mint GitHub tokens for publish through octo-sts instead of a stored PAT#373
wallrj wants to merge 1 commit into
cert-manager:masterfrom
wallrj:octo-sts-publish

Conversation

@wallrj

@wallrj wallrj commented Sep 11, 2026

Copy link
Copy Markdown
Member

cmrel gcb publish no longer needs a stored GitHub token. When GITHUB_TOKEN is unset and --octo-sts-identity is given, it exchanges the Cloud Build service account's Google ID token with octo-sts for a short-lived GitHub token scoped to the target repository. The KMS-encrypted GITHUB_TOKEN is removed from gcb/publish/cloudbuild.yaml.

Why

The v1.21.2 publish failed with 401 from GitHub. The encrypted GITHUB_TOKEN was a personal access token of the jetstack-release-bot user, created in January 2020 and unchanged since. It was revoked or expired at some point after v1.21.1 on 2026-07-29. Nobody on the project holds that account's login, so the token could not be rotated (cert-manager/cert-manager#9237).

A token minted per publish by a service the org already installed removes the secret and the orphaned bot account. The octo-sts GitHub App is installed on every repository in the cert-manager org, but no repository uses it yet.

How it works

  1. Cloud Build runs the publish step as cert-manager-release-gcb@cert-manager-release.iam.gserviceaccount.com. The metadata server issues a Google ID token for it with audience octo-sts.dev.
  2. cmrel sends that token to https://octo-sts.dev/sts/exchange?scope=<owner>/<repo>&identity=cmrel-publish.
  3. octo-sts reads .github/chainguard/cmrel-publish.sts.yaml from the target repository. If the policy admits the service account, it returns a GitHub installation token with the policy's permissions.

GitHubClient now takes the owner and repository it is for, because each token is scoped to one repository. GITHUB_TOKEN still wins when set.

Before this can merge

  • Add octo-sts trust policy for the release pipeline cert-manager#9334 adding the trust policy is merged.
  • fix(gcb): rotate the publish GITHUB_TOKEN #372 (rotated PAT) is merged and v1.21.2 is published with it, since this PR removes GITHUB_TOKEN from the build. This PR then needs a rebase: both touch the same line.
  • Decide what to do about the Helm chart PR to jetstack/jetstack-charts. That repository is in the jetstack org, where octo-sts is not installed. Until it is, publish with --publish-actions '*,-helmchartpr' and open the chart PR by hand, or keep setting GITHUB_TOKEN for that repository only.

Test plan

  • go build ./..., go vet ./cmd/cmrel/..., go test ./cmd/cmrel/cmd/ pass. The new test covers the request shape, the bearer header, the error body, and an empty response.
  • End to end, without touching a real release: install octo-sts on a personal fork of cert-manager, add the same policy there, then run cmrel publish with --published-github-org=<fork owner> --publish-actions=githubrelease against a staged release.

Not verified: that the hosted octo-sts instance's domain is exactly octo-sts.dev. The code falls back to the domain as the required audience when the policy sets none, so --octo-sts-domain exists to correct it without a code change.

[Claude Fable 5.1]

The v1.21.2 publish failed with 401 from GitHub because the KMS-encrypted
GITHUB_TOKEN in gcb/publish/cloudbuild.yaml, a personal access token of the
jetstack-release-bot user created in January 2020, had been revoked or had
expired. Nobody on the project holds that account's login, so the token could
not be rotated.

Replace the long-lived token with a short-lived one minted at publish time.
When GITHUB_TOKEN is unset and --octo-sts-identity is set, cmrel obtains the
Google ID token of the Cloud Build service account from the metadata server,
with the octo-sts domain as audience, and exchanges it with octo-sts for a
GitHub installation token scoped to the target repository. The trust policy
that admits the service account lives in the target repository at
.github/chainguard/<identity>.sts.yaml, so each repository controls what the
release pipeline may do to it. The octo-sts GitHub App is already installed
on every cert-manager org repository.

GitHubClient now takes the owner and repo it is for, because octo-sts tokens
are scoped to one repository. GITHUB_TOKEN still wins when set, so the
existing flow keeps working for anyone who sets it.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
@cert-manager-prow

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@cert-manager-prow cert-manager-prow Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. labels Sep 11, 2026
@cert-manager-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign thatsmrtalbot for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@cert-manager-prow cert-manager-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 11, 2026
wallrj added a commit to wallrj/cert-manager that referenced this pull request Sep 11, 2026
Lets cmrel gcb publish create the draft GitHub release for a tag with a
short-lived token minted by octo-sts from the Cloud Build service account
identity, instead of the stored personal access token of jetstack-release-bot
that stopped working before v1.21.2 (cert-manager#9237). The octo-sts GitHub App is
already installed on this repository.

The cmrel side is cert-manager/release#373.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: Richard Wall <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant