DNM: swap to rhel 8 for testing - #82644
Conversation
WalkthroughOMR v2 provisioning now selects official RHEL 8 hourly AMIs and uses an older-curl-compatible health check. OMR v3 upgrades now use a root-based migration flow with ChangesQuay OMR compatibility and migration
Estimated code review effort: 3 (Moderate) | ~20 minutes Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant UserServices
participant MigrateToV3
participant QuayInstaller
participant RootData
UserServices->>MigrateToV3: stop quay services
MigrateToV3->>QuayInstaller: run sudo migration
QuayInstaller->>RootData: write migration data and logs
MigrateToV3->>RootData: check health and install certificates
🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/quay/omr-v2/host-provision/quay-omr-v2-host-provision-commands.sh`:
- Line 105: Update the documentation in the host-provision step reference for
OMR_V2_AMI_ID and the provisioned host to state RHEL 8 instead of RHEL 9,
ensuring explicit AMI inputs and generated guidance match the RHEL 8 discovery
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 345f9648-bfcf-4c21-b89b-929a94396aa2
📒 Files selected for processing (1)
ci-operator/step-registry/quay/omr-v2/host-provision/quay-omr-v2-host-provision-commands.sh
| --owners 309956199498 \ | ||
| --filters \ | ||
| 'Name=name,Values=RHEL-9.*_HVM-*-x86_64-*-Hourly2-GP3' \ | ||
| 'Name=name,Values=RHEL-8.*_HVM-*-x86_64-*-Hourly2-GP3' \ |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Update the step contract to RHEL 8.
The script now discovers RHEL 8 AMIs, but ci-operator/step-registry/quay/omr-v2/host-provision/quay-omr-v2-host-provision-ref.yaml still documents OMR_V2_AMI_ID and the provisioned host as RHEL 9. Update that file’s documentation to RHEL 8 so explicit AMI inputs and generated guidance match the new behavior.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/quay/omr-v2/host-provision/quay-omr-v2-host-provision-commands.sh`
at line 105, Update the documentation in the host-provision step reference for
OMR_V2_AMI_ID and the provisioned host to state RHEL 8 instead of RHEL 9,
ensuring explicit AMI inputs and generated guidance match the RHEL 8 discovery
behavior.
|
/label wip |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jbpratt The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
@jbpratt: The label(s) DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/hold |
|
/label do-not-merge/work-in-progress |
openshift-ci
Bot
added
the
do-not-merge/hold
|
@jbpratt: The label(s) DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Signed-off-by: Brady Pratt <[email protected]>
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Signed-off-by: Brady Pratt <[email protected]>
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/quay/omr-v3/upgrade-from-v2/quay-omr-v3-upgrade-from-v2-commands.sh`:
- Around line 154-162: Update the certificate installation flow after the health
check to use /var/lib/quay/ssl.cert as the source and copy it to ca.crt in the
existing cert_dir, replacing the rootCA.pem source path. Keep the migrated
root-owned registry paths used by the health check and podman authentication
consistent.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 1248d4fd-34b1-4756-9ca1-cfb084e38a36
📒 Files selected for processing (1)
ci-operator/step-registry/quay/omr-v3/upgrade-from-v2/quay-omr-v3-upgrade-from-v2-commands.sh
| sudo curl --retry 20 --retry-delay 3 --retry-all-errors \ | ||
| --silent --show-error --fail \ | ||
| --cacert "/home/$(id -un)/quay-install/quay-rootCA/rootCA.pem" \ | ||
| --cacert "${data_dir}/rootCA.pem" \ | ||
| "https://${registry_endpoint}/healthz" >/dev/null | ||
| cert_dir="${work_dir}/certs" | ||
| auth_file="${work_dir}/registry-auth.json" | ||
| install -d -m 0700 "${cert_dir}" | ||
| install -m 0600 \ | ||
| "/home/$(id -un)/quay-install/quay-rootCA/rootCA.pem" \ | ||
| sudo install -m 0644 \ | ||
| "${data_dir}/rootCA.pem" \ |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check for cert filename conventions used across the quay step-registry scripts,
# and whether rootCA.pem appears anywhere else (e.g. already-established for `migrate`).
rg -n 'rootCA\.pem|ssl\.cert' ci-operator/step-registry/quay --glob '*.sh'Repository: openshift/release
Length of output: 2100
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect relevant script sections and related files to understand data_dir definition
# and whether rootCA.pem is produced elsewhere in this path before being read.
for f in \
ci-operator/step-registry/quay/omr-v3/upgrade-from-v2/quay-omr-v3-upgrade-from-v2-commands.sh \
ci-operator/step-registry/quay/omr-v3/gather/quay-omr-v3-gather-commands.sh \
ci-operator/step-registry/quay/omr-v3/install/quay-omr-v3-install-commands.sh \
ci-operator/step-registry/quay/omr-v3/install/quay-omr-v3-setup-commands.sh \
ci-operator/step-registry/quay/omr-v3/install/quay-omr-v3-pre-commands.sh
do
if [ -f "$f" ]; then
echo "===== $f ====="
wc -l "$f"
rg -n 'data_dir|migrate|ssl\.cert|rootCA\.pem|quay-rootCA|migrate|healthz|\$curl|SUDO|sudo' "$f" || true
fi
done
echo "===== outline ====="
ast-grep outline ci-operator/step-registry/quay/omr-v3/upgrade-from-v2/quay-omr-v3-upgrade-from-v2-commands.sh || true
echo "===== relevant upgrade excerpts ====="
sed -n '1,230p' ci-operator/quay/step-registry/quay/omr-v3/upgrade-from-v2/quay-omr-v3-upgrade-from-v2-commands.sh 2>/dev/null || \
sed -n '1,230p' ci-operator/step-registry/quay/omr-v3/upgrade-from-v2/quay-omr-v3-upgrade-from-v2-commands.shRepository: openshift/release
Length of output: 10567
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the upgrade script around the certificate usage and nearby migration/setup steps.
file="ci-operator/step-registry/quay/omr-v3/upgrade-from-v2/quay-omr-v3-upgrade-from-v2-commands.sh"
echo "===== $file line count ====="
wc -l "$file"
echo "===== $file relevant excerpts ====="
sed -n '1,230p' "$file"
echo "===== search for data_dir and migrate in upgrade ===="
rg -n 'data_dir|quay migrate|migrate|quay/ssl\.cert|quay-rootCA/rootCA\.pem' "$file"Repository: openshift/release
Length of output: 8411
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "===== repository files mentioning ssl.cert ====="
rg -n 'ssl\.cert|rootCA\.pem|quay-rootCA' . --glob '*.go' --glob '*.py' --glob '*.sh' --glob '*.yaml' --glob '*.yml' || true
echo "===== step registry quay files ====="
git ls-files 'ci-operator/step-registry/quay/**/*'Repository: openshift/release
Length of output: 13574
🌐 Web query:
Quay OMR migration -data-dir rootCA.pem ssl.cert
💡 Result:
When working with Red Hat Quay, including migrations from mirror-registry (OMR) or standalone deployments, the rootCA.pem and ssl.cert files are central to configuring secure TLS communication [1][2]. In typical standalone and OMR configurations, the following approach is used: 1. SSL/TLS Certificate Files: Red Hat Quay expects the server certificate (ssl.cert) and private key (ssl.key) to be present in the configuration directory [1][3][4]. 2. Root CA Handling: The rootCA.pem file represents your Certificate Authority [1][2]. To ensure full certificate chain validation, it is common practice to append the contents of rootCA.pem to ssl.cert [1][2][4]: cat rootCA.pem >> ssl.cert 3. Deployment Contexts: - Standalone/OMR: Certificates are manually managed in the configuration directory and referenced by the config.yaml file [2][4]. - OpenShift (Operator): Custom certificates are typically applied by creating a configBundleSecret in OpenShift and referencing it within the QuayRegistry custom resource, often using the extra_ca_cert field to include additional CAs if necessary [5]. - Migrations: If you are migrating an OMR installation to a Go-based Quay binary (via the quay migrate command), the migration process is designed to checkpoint and copy existing TLS certificates and storage from the old environment to the new deployment [6]. Always ensure that any client or host needing to access the registry has the rootCA.pem imported into its system-wide trust store (e.g., /etc/pki/ca-trust/source/anchors/ followed by update-ca-trust) to prevent SSL verification errors [7][4].
Citations:
- 1: https://docs.redhat.com/en/documentation/red_hat_quay/3.13/html-single/securing_red_hat_quay/index
- 2: https://docs.redhat.com/en/documentation/red_hat_quay/3.15/html-single/securing_red_hat_quay/index
- 3: https://docs.redhat.com/en/documentation/red_hat_quay/3.17/html/deploy_red_hat_quay_-_high_availability/deploying_red_hat_quay
- 4: https://docs.redhat.com/en/documentation/red_hat_quay/3.15/html/proof_of_concept_-_deploying_red_hat_quay/advanced-quay-poc-deployment
- 5: https://docs.redhat.com/en/documentation/red_hat_quay/3.12/html/deploying_the_red_hat_quay_operator_on_openshift_container_platform/operator-custom-ssl-certs-config-bundle
- 6: quay/quay@789fb06
- 7: https://github.com/openshift/microshift/blob/main/docs/contributor/howto_quay_mirror.md
Use /var/lib/quay/ssl.cert for the migrated root-owned registry.
The migration runs quay migrate -data-dir /var/lib/quay, but the same root-owned install/gather flows read /var/lib/quay/ssl.cert for the registry CA. Expect this path to copy ssl.cert to ca.crt instead of a rootCA.pem file in /var/lib/quay; otherwise the health check and podman login rely on a missing file.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/quay/omr-v3/upgrade-from-v2/quay-omr-v3-upgrade-from-v2-commands.sh`
around lines 154 - 162, Update the certificate installation flow after the
health check to use /var/lib/quay/ssl.cert as the source and copy it to ca.crt
in the existing cert_dir, replacing the rootCA.pem source path. Keep the
migrated root-owned registry paths used by the health check and podman
authentication consistent.
|
@jbpratt: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
Summary by CodeRabbit
Updates OpenShift CI’s Quay OMR disconnected install/upgrade testing scripts to support a RHEL 8-based (DNM) environment:
--retry-all-errorswhile preserving the existing retry behavior and health endpoint.migrate-to-v3helper from a rootless migration to a root / cgroups-v1 experiment flow—using/var/lib/quay, stopping user-levelquay-*services, running the migration viasudo, and ensuring health checking and certificate installation use root-owned paths (withsudofor service status and/healthzcalls).