feat(secrets): add Bitwarden/GSM secret management

[zdrapela]

Summary

• Publish the rhdh-e2e-secrets CLI and the @red-hat-developer-hub/e2e-test-utils/secrets API.
• Load profile-selected Bitwarden secrets into child environments through exec.
• Add paired Bitwarden and Google Secret Manager (GSM) create, update, and delete commands.
• Add GSM describe, list, gsm-login, and gsm-clean commands.
• Add opt-in --stream-secrets execution for passing sorted secret entries over file descriptor 3 instead of the child environment.

Human comment

• Follow-up to feat(secrets): add unified Bitwarden secret runner #157
• I tested locally on macOS, and it works well. Only each operation takes maybe 10 seconds, but that's a limitation of the CLIs and the logic that verifies what's written.
• I tried to avoid writing secrets into temporary files, and that's why I came up with streaming the secrets over FD 3. It should be safer, but I haven't used that much before. The construction is a little more complicated than it would otherwise be.
• It's not possible to place bw CLI inside the container, because there will be a lengthy login process each time.
• This PR includes many unit tests that the agent requires to verify its work. Discussed for the previous version (feat(secrets): add unified Bitwarden secret runner #157) with @subhashkhileri and kept them. Not all of them are needed, but they already helped to find an error that would prevent Linux users from using this tool with the rhdh repo.
• Counterparts in other repos:
  • feat(e2e): use unified Bitwarden secrets runner rhdh-plugin-export-overlays#3565 (need to be updated to 2.1.15, but no breaking change)
  • feat(e2e): use Bitwarden for local secrets rhdh#5404
  • They will require backports to maintain the ability of local runs for release branches.

How to test

Run the automated checks:

yarn check
yarn build && yarn test

To interact with the CLI directly, set BW_SESSION and use an existing profile:

export BW_SESSION="$(bw unlock --raw)"
node dist/secrets/cli.js exec --profile path/to/profile.json -- your-command

Authenticate the CLI's Google Secret Manager (GSM) wrapper before reading or changing GSM secrets:

node dist/secrets/cli.js gsm-login

List the paired collections, list paths in one collection, or inspect a secret's GSM metadata:

node dist/secrets/cli.js list
node dist/secrets/cli.js list --collection rhdh-qe
node dist/secrets/cli.js describe --collection rhdh-qe rhdh/test --output json

Mutation commands write Bitwarden first and GSM second. Add --dry-run to validate both providers and print a value-free plan without changing either provider.

Create an attachment-backed secret from a file:

node dist/secrets/cli.js create \
  --collection rhdh-qe \
  rhdh/test \
  --from-file ./secret-value \
  --dry-run

node dist/secrets/cli.js create \
  --collection rhdh-qe \
  rhdh/test \
  --from-file ./secret-value

Use --from-stdin instead of --from-file to create a note-backed Bitwarden item. GSM prompts for non-secret metadata during create.

Rotate a secret with update. The command requires the secret in both providers and preserves its current Bitwarden storage form:

node dist/secrets/cli.js update \
  --collection rhdh-qe \
  rhdh/test \
  --from-stdin < ./replacement-value \
  --dry-run

node dist/secrets/cli.js update \
  --collection rhdh-qe \
  rhdh/test \
  --from-stdin < ./replacement-value

Delete the secret from both providers:

node dist/secrets/cli.js delete \
  --collection rhdh-qe \
  rhdh/test \
  --dry-run

node dist/secrets/cli.js delete \
  --collection rhdh-qe \
  rhdh/test

Use --force with create or delete to reconcile a secret that exists in only one provider. update does not support --force. If an empty value is intentional, use --allow-empty with create or update. Run node dist/secrets/cli.js gsm-clean to remove the CLI's cached GSM authentication.

Use RHDH PR #5404 to test FD 3 integration. Build this PR in an adjacent rhdh-e2e-test-utils checkout, then run the host-side test wrapper from the RHDH checkout:

# In rhdh-e2e-test-utils
yarn build

# In rhdh/e2e-tests
export BW_SESSION="$(bw unlock --raw)"
export BASE_URL="https://your-rhdh-deployment"
./local-test.sh -- --project=showcase --headed

local-test.sh invokes rhdh-e2e-secrets exec --stream-secrets, reads the selected secrets from FD 3, and passes the remaining arguments to Playwright. Set K8S_CLUSTER_URL and K8S_CLUSTER_TOKEN for cluster-aware tests.

Architecture

• Profiles contain selectors and approved collection mappings, but never secret values.
• Mutations take a per-secret lock, inspect both providers, and build a value-free plan before writing.
• Mutations write Bitwarden first and GSM second. --dry-run validates the preflight plan without writing either provider.
• The mutation flow stores no journal or secret value. Failures use explicit retry and reconciliation guidance instead.
• GSM operations use OpenShift CI's secret-manager.sh with a validated local wrapper cache and a CLI-owned Google ADC directory.
• GSM and Bitwarden attachment operations use private temporary directories with mode 0700 and files with mode 0600. Normal cleanup runs after each mutation. Before another temporary secret is created, the CLI removes directories owned by dead processes; operating-system pruning is the final fallback. SIGKILL, power loss, or a runtime crash can leave files until recovery. The CLI never places GSM values in process arguments through --from-literal.
• Stream mode uses a bounded, length-prefixed binary protocol with headers and a footer. The decoder rejects malformed, truncated, duplicate, or oversized data.
• Child execution validates all selected secrets before spawning, scrubs provider credentials, forwards signals, and terminates stalled stream consumers.

Limitations

• The CLI supports only the configured paired collections. Bitwarden operations explicitly deny rhdh-aws-credentials.
• Bitwarden requires bw on PATH and an already unlocked BW_SESSION. The CLI does not manage the Bitwarden session.
• GSM commands use their own wrapper and ADC cache. The CLI does not share credentials from another secret-manager.sh copy.
• GSM create requires a controlling terminal for metadata prompts, including when the secret value comes from piped stdin.
• Paired provider mutations are not atomic. A GSM failure after a Bitwarden write can leave mixed state.
• Retry the same command after a partial failure. Use --force to reconcile create or delete. update does not support --force.
• Normal exec mode still places selected values in the child environment. Stream mode requires explicit opt-in and targets a trusted local parent-child boundary.
• Stream consumers must read and validate the complete FD 3 payload, then close the descriptor. The protocol is transport framing, not encryption or authentication.
• The GSM wrapper tracks OpenShift CI's main branch. Validation checks its expected structure rather than a signature or pinned checksum.
• Tests mock the external providers and do not exercise live Bitwarden, Google ADC, Podman, or network behavior.

[albarbaro]

/lgtm
Tested locally and it's working very well for me! Thanks!

[subhashkhileri]

Thanks for the work here. I reviewed the full PR branch (39 files, ~6.8k net additions); the build passes and the test suite passes (270/270).

I have two security concerns:

1. High — mutable remote code execution: src/secrets/gsm-wrapper.ts downloads secret-manager.sh from the mutable openshift/release/main branch and executes it with bash. The validation only checks for a few strings, and the cached SHA is calculated from the downloaded content itself, so it does not authenticate the script. Please pin the wrapper to a reviewed commit/release and verify a trusted checksum or signature before execution.

2. Medium — plaintext secret snapshots: src/secrets/mutation.ts:createSnapshot() and src/secrets/bitwarden.ts:createAttachment() write secret values to /tmp. The files are mode 0600, but a crash or SIGKILL before cleanup can leave the values behind. Please avoid disk when possible, or add a documented cleanup/recovery strategy and consider a safer provider/API path.

There is also a design concern for future maintainability: runtime secret consumption, child-process transport, and two-provider create/update/delete reconciliation are all bundled into the same package. That is workable for E2E tooling, but a typical Node application would use a provider interface for reads, keep administration/rotation in a separate CLI or CI job, and use one authoritative secret store. The custom FD3 stream should remain an explicit internal adapter rather than the primary application API.

Finally, the PR is much larger than its final Linux-process-limit title suggests because it includes the entire Bitwarden/GSM lifecycle feature, the FD3 streaming protocol, rollback/process-lifecycle hardening, tests, and documentation across 19 commits. Splitting the provider lifecycle, stream transport, and final Linux-limit fix into separate PRs would make review easier.

[zdrapela]

Hi @subhashkhileri,

High — mutable remote code execution

I agree it's not ideal, but I believe we should continue to use the newest version of the secret-manager.sh script from openshift/release/main and treat it as a trusted source.
The script uses the image quay.io/openshift/ci-public:ci_secret-manager_latest, and there is only the latest tag, so we cannot pin it to use a specific version of the image that is aligned with the specific version of the secret-manager.sh script. As a result, pinning the script without the ability to pin the image could create misalignment and cause errors.
I added hardening that verifies that secret-manager.sh contains the correct image from the trusted Quay repository. If you insist, let me know, but I believe this is the best we can do, and it follows the principles that we use across other repositories, where we treat a repository as a trusted source and believe the script is trustworthy.

Medium — plaintext secret snapshots

As a security measure, I added hardening that unifies the folder being used. I also added cleanup guidance for cases where it's required.
I tried to avoid temporary files wherever possible, but there is a hard limitation on secret length that Bitwarden allows for a secret note of 10000 characters after encryption, and a file attachment is the only reasonable way around. Another hard limitation is that the bw CLI only allows attachments from a file, so we cannot really avoid the use of a temporary file.

Having a CI job for synchronization is an interesting idea. I'm still leaning towards a local CLI, because any errors would surface early, with an immediate response, so they can be resolved. But I agree that separating it into two CLIs can make sense here.

[zdrapela]

I thought about having two CLIs - one for local execution and the other for secrets administration, but I believe it will just complicate things and add extra overhead, and make it harder to understand for users.

[subhashkhileri]

/lgtm