Skip to content

ci: improve image-based e2e step reliability with retries and validation - #82628

Open
sebrandon1 wants to merge 1 commit into
openshift:mainfrom
sebrandon1:fix/image-based-ci-reliability
Open

ci: improve image-based e2e step reliability with retries and validation#82628
sebrandon1 wants to merge 1 commit into
openshift:mainfrom
sebrandon1:fix/image-based-ci-reliability

Conversation

@sebrandon1

@sebrandon1 sebrandon1 commented Jul 29, 2026

Copy link
Copy Markdown
Member

Jira: CNF-26211

Summary

Adds retry logic, pre-condition validation, trap handlers, and configurable timeouts to the lifecycle-agent image-based CI step scripts (ci-operator/step-registry/openshift/image-based/). These changes target the root causes of flaky e2e failures observed across 9 open lifecycle-agent PRs, where dual-stack IPC tests had 100% failure rates and other e2e jobs failed 40-75% of the time.

Motivation

Analysis of 15 failed Prow job logs across lifecycle-agent PRs revealed that non-infrastructure failures fell into preventable categories:

  • IPC dual-stack no route to host timeouts — the IP_STACK env var was silently dropped because it wasn't declared in the IPC configure ref YAML, so make ipc never received the stack mode
  • SCC UID-range race condition — namespace not fully initialized before operator bundle deployment
  • IBU stage transition rejected — test tried to transition before the controller reconciled validNextStages
  • Zero retry logic on SSH/SCP — transient connection failures to freshly provisioned EC2 instances caused immediate step failure
  • No diagnostics on failure — 5 of 6 step scripts had no trap handlers, making failures hard to debug

Changes

SSH retry with exponential backoff (all 5 step scripts)

Every scp call is now wrapped in ssh_retry() with 3 attempts and exponential backoff (10s/20s/40s). The final long-running ssh command that executes the actual test is intentionally not wrapped — these remote scripts are not idempotent, so retrying a partial run could leave the system in a broken state. Only the file transfer operations (which are safe to retry) get the wrapper.

Trap handlers for failure diagnostics (all 5 step scripts)

Added trap 'echo "ERROR: Step failed at line $LINENO with exit code $?"' EXIT to all scripts. The trap is cleared with trap - EXIT before the final SSH command so test exit codes propagate cleanly. This matches the pattern already established in the EC2 provisioning script.

IPC dual-stack fix (network-configuration configure)

  • Added IP_STACK to the ref YAML — previously, the ci-operator config set IP_STACK: v4v6 for dual-stack jobs but the ref didn't declare it, so it was silently dropped
  • Passes IP_STACK to make ipc — the orchestration Makefile needs this to configure dual-stack networking
  • Pre-condition validation — if IP_STACK is v4v6 or v6v4, validates that IPC_IPV6_ADDRESS, IPC_IPV6_MACHINE_NETWORK, and IPC_IPV6_GATEWAY are non-empty before proceeding

DNF and container image pull retries (install + metal-config)

  • Added dnf_install_retry to the IBI install script (reusing the pattern from metal-config)
  • Added podman_retry with 3 attempts and exponential backoff (15s/30s/60s) for container image pulls in both install and metal-config scripts

API call retries and validation (seed-create)

  • Added --retry 3 --retry-delay 5 to curl calls in findImage() that query OCP release APIs
  • Added validation that base_info is non-empty after API calls — previously, a failed API response would silently produce empty SEED_VERSION and RELEASE_IMAGE values
  • Reduced initial seed wait from sleep 5m to sleep 5s — the retry loop already handles polling
  • Added max_attempts=20 counter to the oc wait retry loop to prevent unbounded looping (previously infinite)

Configurable upgrade timeout (IBU target)

  • Changed hardcoded UPGRADE_TIMEOUT="60m" to UPGRADE_TIMEOUT="${UPGRADE_TIMEOUT:-60m}"
  • Added UPGRADE_TIMEOUT to the ref YAML so it can be overridden per-job

SSH connectivity probe (EC2 provisioning)

Added a post-provisioning SSH connectivity verification (10 attempts, 5s apart) after aws ec2 wait instance-status-ok. The AWS waiter confirms the instance is healthy, but doesn't guarantee sshd is listening or that cloud-init has written the SSH key. This prevents the metal-config step from failing on a not-yet-ready sshd.

Bug fixes (pre-existing)

  • Fixed duplicate heredoc preamble in metal-config's nsswitch.conf generation — 3 junk lines (including a literal cat <<EOF > ... string) were being written into the nsswitch.conf file
  • Fixed non-idempotent ssh_retry — removed ssh_retry from metal-config's install.sh execution (not safe to retry) and sudo mv of nsswitch.conf (replaced with cp + rm so retries don't fail on missing source
  • Fixed trailing whitespace in IBU target ref YAML
  • Fixed typo Transfering → Transferring in 4 scripts

Files Modified

All under ci-operator/step-registry/openshift/image-based/:

File Changes
infra/aws/ec2/*-commands.sh SSH connectivity probe after instance-ready
install/*-commands.sh SSH retry, trap, dnf retry, podman retry
network-configuration/configure/*-commands.sh SSH retry, trap, IP_STACK passthrough, IPv6 validation
network-configuration/configure/*-ref.yaml Added IP_STACK env var
upgrade/metal/config/*-commands.sh SSH retry, trap, podman retry, nsswitch.conf heredoc fix
upgrade/seed/create/*-commands.sh SSH retry, trap, curl retry, findImage validation, bounded wait loop
upgrade/target/*-commands.sh SSH retry, trap, configurable UPGRADE_TIMEOUT
upgrade/target/*-ref.yaml Added UPGRADE_TIMEOUT env var, fixed trailing whitespace

Summary by CodeRabbit

Improves reliability of OpenShift image-based CI e2e steps (lifecycle-agent) on AWS EC2 by hardening remote execution, reducing transient failure impact, and improving diagnostics across install, network configuration, and upgrade workflows.

Key reliability & diagnostics updates

  • Adds consistent EXIT traps across the relevant scripts to print the failing $LINENO and non-zero exit code on failure.
  • Adds bounded retry helpers (ssh_retry, plus podman_retry where needed) with exponential backoff and attempt/failure logging, and applies them to remote ssh/scp transfers, remote workdir creation, package install, and container image pull/copy steps.
  • After the EC2 instance reaches instance-status-ok, validates SSH readiness by retrying non-interactive SSH connections to the instance public IP (HOST_PUBLIC_IP) before proceeding.

Install / provisioning hardening

  • Wraps local-to-remote scp of image_based_install.sh and pull-secret files with retry logic, and improves transfer/status messaging.
  • Retries sudo dnf install -y (including dnf clean -y all on subsequent attempts).
  • Retries the podman run ... cp /bin/openshift-install /tmp/openshift-install copy step used during install.

Network configuration dual-stack correctness

  • Introduces IP_STACK to the network-configuration configure step (documenting allowed values v4, v4v6, v6v4).
  • Validates IP_STACK and, for dual-stack modes, requires the required IPC IPv6 inputs (IPC_IPV6_ADDRESS, IPC_IPV6_MACHINE_NETWORK, IPC_IPV6_GATEWAY).
  • Propagates IP_STACK into the generated remote IPC generation (make ipc) and uses retry-safe scp/remote execution for transferring and running network-configuration.sh.

Upgrade robustness and timeouts

  • Seed creation:
    • Updates base image metadata lookup to use curl with retry/delay/max-time semantics (for both ci and release) and fails fast if resolved base_info is empty.
    • Replaces the previous wait behavior with a bounded retry loop around oc wait (attempt-counted, with explicit timeout/error behavior).
    • Transfers create_seed.sh using retry-safe scp.
  • Target/IBU upgrade:
    • Makes the upgrade timeout configurable via UPGRADE_TIMEOUT (defaulting to 60m).
    • Uses retry-safe scp when transferring upgrade_from_seed.sh.
  • Metal/upgrade workflow:
    • Uses retry-safe uploads/commands (including mkdir -p workdir creation and staged config/secret transfers).
    • Ensures container pulls are retry-safe via podman_retry.
    • Fixes nsswitch.conf installation by copying the staged file into place and then removing the staged file.

Cleanup

  • Removes trailing whitespace in upgrade-target dependency config and fixes a minor typo in transfer/log messaging.

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Image-based infrastructure, installation, network configuration, and upgrade scripts now include retry handling, failure traps, bounded waits, SSH readiness checks, and configurable IP stack and upgrade timeout parameters.

Changes

Image-based workflow reliability

Layer / File(s) Summary
Infrastructure readiness and installation retries
ci-operator/step-registry/openshift/image-based/infra/aws/ec2/..., ci-operator/step-registry/openshift/image-based/install/...
Checks SSH reachability after EC2 readiness, reports failures, retries package and image operations, and hardens installation transfers.
Network stack validation and deployment
ci-operator/step-registry/openshift/image-based/network-configuration/configure/...
Adds configurable IP stack values, validates dual-stack IPv6 inputs, passes the selected stack to network configuration, and retries script transfer.
Metal upgrade provisioning retries
ci-operator/step-registry/openshift/image-based/upgrade/metal/config/...
Adds failure reporting and retries for remote provisioning, image pulls, configuration transfer, and secret uploads.
Seed image resolution and creation
ci-operator/step-registry/openshift/image-based/upgrade/seed/create/...
Retries image lookups and script transfer, validates resolved image data, and bounds seed creation wait attempts.
Target upgrade timeout and transfer
ci-operator/step-registry/openshift/image-based/upgrade/target/...
Exposes a configurable upgrade timeout, retries script transfer, and reports non-zero script failures.

Estimated code review effort: 3 (Moderate) | ~30 minutes

Suggested reviewers: fonta-rh, javipolo

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main theme of the changes: improving image-based e2e step reliability with retries and validation.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR only changes shell/YAML step-registry files; no Ginkgo tests or test titles were added or modified in the affected subtree.
Test Structure And Quality ✅ Passed This PR only changes shell/YAML step-registry scripts; no Ginkgo test files or test blocks were modified, so the check is not applicable.
Microshift Test Compatibility ✅ Passed No new Ginkgo tests were added; only shell/YAML step-registry files changed, with no MicroShift-sensitive APIs or test labels to review.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS: The PR only modifies shell scripts and YAML; no new Ginkgo test bodies or SNO-sensitive test logic were added.
Topology-Aware Scheduling Compatibility ✅ Passed Only CI step scripts/YAML changed; no manifests/controllers or scheduling primitives (affinity, nodeSelector, spreads, PDBs) were introduced.
Ote Binary Stdout Contract ✅ Passed PR only changes ci-operator shell/YAML step-registry files; no Go binaries or process-level stdout writes are introduced.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS: The PR only changes step-registry shell/YAML files; no new Ginkgo test blocks or test files were added, so this compatibility check is not applicable.
No-Weak-Crypto ✅ Passed Exact scans of all modified scripts/YAMLs found no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB usage, and no secret/token comparison patterns.
Container-Privileges ✅ Passed Diff only changes shell scripts and step-registry refs; searches found no privileged:true, hostNetwork/PID/IPC, allowPrivilegeEscalation, SYS_ADMIN, or root securityContext fields.
No-Sensitive-Data-In-Logs ✅ Passed Added logs are generic status/error messages; no passwords, tokens, API keys, PII, or session IDs are logged in the changed code.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sebrandon1
Once this PR has been reviewed and has the lgtm label, please assign brandisher for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci
openshift-ci Bot requested review from fonta-rh and javipolo July 29, 2026 16:42

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/step-registry/openshift/image-based/infra/aws/ec2/openshift-image-based-infra-aws-ec2-commands.sh`:
- Around line 539-541: Update the SSH readiness probe near the connection
command to remove StrictHostKeyChecking=no and UserKnownHostsFile=/dev/null.
Obtain the expected host key from the trusted provisioning source, store or
reuse it in a persistent known_hosts file, and configure SSH to verify the host
strictly before accepting the probe.

In
`@ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh`:
- Around line 10-19: Update the retry loops in ssh_retry and the corresponding
helper so the failure message, sleep, and delay increase occur only when the
current attempt is less than the retry limit. After the final failed attempt,
report the overall failure immediately without logging “retrying” or sleeping.
- Around line 127-132: Quote the source and destination path arguments in the
scp invocations within the install-script and pull-secrets transfer blocks,
including the ${SHARED_DIR} paths and remote_workdir destination, while
preserving the existing ssh_retry and SSHOPTS usage.
- Around line 67-76: Add bounded backoff to the retry loop in dnf_install_retry:
after a failed dnf install, sleep before the next attempt but skip sleeping
after the third and final attempt. Keep the existing three-attempt limit,
success return, and failure messaging unchanged.
- Around line 67-76: Update dnf_install_retry so each retry iteration guards dnf
clean and dnf install as one combined condition, continuing only when both
commands succeed and returning success only after both complete successfully.
Preserve the three-attempt retry loop and failure return behavior, while
ensuring a failed cleanup does not terminate the caller before later retries.

In
`@ci-operator/step-registry/openshift/image-based/network-configuration/configure/openshift-image-based-network-configuration-configure-commands.sh`:
- Around line 68-69: Remove the trap - EXIT statement before the final ssh
invocation so the existing failure trap remains active and reports remote
command failures. Preserve the current SSH call and its non-zero exit behavior.
- Around line 64-65: Update the scp invocation in the network configuration
transfer step to quote the local script path and remote destination, using
"${SHARED_DIR}/network-configuration.sh" and "${ssh_host_ip}:${remote_workdir}"
while preserving the existing SSH options.
- Around line 32-39: Validate IP_STACK against the documented values before
generating the remote script: accept v4, v4v6, and v6v4, while exiting with an
error for empty or any other value. Update the generated script assignment for
IP_STACK to quote the value, using the existing validation/configuration flow
around the dual-stack checks and make ipc generation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 4d8f09f3-49dd-458c-9b35-19d805f19fff

📥 Commits

Reviewing files that changed from the base of the PR and between 073940e and 020f372.

📒 Files selected for processing (8)
  • ci-operator/step-registry/openshift/image-based/infra/aws/ec2/openshift-image-based-infra-aws-ec2-commands.sh
  • ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh
  • ci-operator/step-registry/openshift/image-based/network-configuration/configure/openshift-image-based-network-configuration-configure-commands.sh
  • ci-operator/step-registry/openshift/image-based/network-configuration/configure/openshift-image-based-network-configuration-configure-ref.yaml
  • ci-operator/step-registry/openshift/image-based/upgrade/metal/config/openshift-image-based-upgrade-metal-config-commands.sh
  • ci-operator/step-registry/openshift/image-based/upgrade/seed/create/openshift-image-based-upgrade-seed-create-commands.sh
  • ci-operator/step-registry/openshift/image-based/upgrade/target/openshift-image-based-upgrade-target-commands.sh
  • ci-operator/step-registry/openshift/image-based/upgrade/target/openshift-image-based-upgrade-target-ref.yaml

@sebrandon1
sebrandon1 force-pushed the fix/image-based-ci-reliability branch from 020f372 to abee46a Compare July 29, 2026 17:04

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh (1)

71-77: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Keep DNF cleanup inside the retry condition.

A failed dnf clean can terminate the generated script under set -e before later attempts. Guard cleanup and installation together.

Proposed fix
-    sudo dnf clean -y all
-    sudo dnf install -y \${packages[*]} && return 0
+    if sudo dnf clean -y all && sudo dnf install -y "\${packages[@]}"; then
+      return 0
+    fi
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh`
around lines 71 - 77, Update dnf_install_retry so each retry attempt guards sudo
dnf clean -y all and the subsequent sudo dnf install command within the same
conditional, allowing a failed cleanup to continue to the next attempt under set
-e while preserving success on a successful installation.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh`:
- Line 8: Make the EXIT traps success-aware by logging the failure message only
when the captured exit status is non-zero, while preserving the exit status and
retaining each trap through remote execution. Apply this in
ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh:8,
ci-operator/step-registry/openshift/image-based/network-configuration/configure/openshift-image-based-network-configuration-configure-commands.sh:6,
ci-operator/step-registry/openshift/image-based/upgrade/metal/config/openshift-image-based-upgrade-metal-config-commands.sh:8,
ci-operator/step-registry/openshift/image-based/upgrade/seed/create/openshift-image-based-upgrade-seed-create-commands.sh:8,
and
ci-operator/step-registry/openshift/image-based/upgrade/target/openshift-image-based-upgrade-target-commands.sh:8;
remove the corresponding trap - EXIT commands at lines 149, 78, 289, 243, and
176 respectively.

---

Duplicate comments:
In
`@ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh`:
- Around line 71-77: Update dnf_install_retry so each retry attempt guards sudo
dnf clean -y all and the subsequent sudo dnf install command within the same
conditional, allowing a failed cleanup to continue to the next attempt under set
-e while preserving success on a successful installation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: ad324729-6736-4d5d-b5f8-d9414211dd7b

📥 Commits

Reviewing files that changed from the base of the PR and between 020f372 and abee46a.

📒 Files selected for processing (8)
  • ci-operator/step-registry/openshift/image-based/infra/aws/ec2/openshift-image-based-infra-aws-ec2-commands.sh
  • ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh
  • ci-operator/step-registry/openshift/image-based/network-configuration/configure/openshift-image-based-network-configuration-configure-commands.sh
  • ci-operator/step-registry/openshift/image-based/network-configuration/configure/openshift-image-based-network-configuration-configure-ref.yaml
  • ci-operator/step-registry/openshift/image-based/upgrade/metal/config/openshift-image-based-upgrade-metal-config-commands.sh
  • ci-operator/step-registry/openshift/image-based/upgrade/seed/create/openshift-image-based-upgrade-seed-create-commands.sh
  • ci-operator/step-registry/openshift/image-based/upgrade/target/openshift-image-based-upgrade-target-commands.sh
  • ci-operator/step-registry/openshift/image-based/upgrade/target/openshift-image-based-upgrade-target-ref.yaml
🚧 Files skipped from review as they are similar to previous changes (2)
  • ci-operator/step-registry/openshift/image-based/network-configuration/configure/openshift-image-based-network-configuration-configure-ref.yaml
  • ci-operator/step-registry/openshift/image-based/upgrade/target/openshift-image-based-upgrade-target-ref.yaml

@sebrandon1
sebrandon1 force-pushed the fix/image-based-ci-reliability branch from abee46a to 20cf3bd Compare July 29, 2026 17:10

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh (1)

76-79: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Keep dnf clean inside the retry condition.

On retry attempts, a failed cleanup exits the strict generated script before another dnf install attempt. Guard cleanup and installation as one retriable operation. As per coding guidelines, step-registry command scripts should default to set -euo pipefail.

Proposed fix
-    if [ "\$attempt" -gt 1 ]; then
-      sudo dnf clean -y all
-    fi
-    sudo dnf install -y \${packages[*]} && return 0
+    if { [ "\$attempt" -eq 1 ] || sudo dnf clean -y all; } && \
+       sudo dnf install -y "\${packages[@]}"; then
+      return 0
+    fi
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh`
around lines 76 - 79, Update the retry logic around the package installation
command so the `dnf clean -y all` cleanup and subsequent `dnf install` are
guarded as one retriable operation, preventing cleanup failure from aborting
retries. Keep cleanup conditional on attempts greater than one, and ensure the
step-registry command script defaults to `set -euo pipefail`.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In
`@ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh`:
- Around line 76-79: Update the retry logic around the package installation
command so the `dnf clean -y all` cleanup and subsequent `dnf install` are
guarded as one retriable operation, preventing cleanup failure from aborting
retries. Keep cleanup conditional on attempts greater than one, and ensure the
step-registry command script defaults to `set -euo pipefail`.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 778ff8bd-7c77-4b9f-b7b0-3a54ad586429

📥 Commits

Reviewing files that changed from the base of the PR and between abee46a and 20cf3bd.

📒 Files selected for processing (8)
  • ci-operator/step-registry/openshift/image-based/infra/aws/ec2/openshift-image-based-infra-aws-ec2-commands.sh
  • ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh
  • ci-operator/step-registry/openshift/image-based/network-configuration/configure/openshift-image-based-network-configuration-configure-commands.sh
  • ci-operator/step-registry/openshift/image-based/network-configuration/configure/openshift-image-based-network-configuration-configure-ref.yaml
  • ci-operator/step-registry/openshift/image-based/upgrade/metal/config/openshift-image-based-upgrade-metal-config-commands.sh
  • ci-operator/step-registry/openshift/image-based/upgrade/seed/create/openshift-image-based-upgrade-seed-create-commands.sh
  • ci-operator/step-registry/openshift/image-based/upgrade/target/openshift-image-based-upgrade-target-commands.sh
  • ci-operator/step-registry/openshift/image-based/upgrade/target/openshift-image-based-upgrade-target-ref.yaml
🚧 Files skipped from review as they are similar to previous changes (2)
  • ci-operator/step-registry/openshift/image-based/network-configuration/configure/openshift-image-based-network-configuration-configure-ref.yaml
  • ci-operator/step-registry/openshift/image-based/upgrade/target/openshift-image-based-upgrade-target-ref.yaml

@sebrandon1
sebrandon1 force-pushed the fix/image-based-ci-reliability branch from 20cf3bd to abff636 Compare July 29, 2026 17:22
@sebrandon1

Copy link
Copy Markdown
Member Author

/pj-rehearse pull-ci-openshift-kni-lifecycle-agent-main-ibu-e2e-flow pull-ci-openshift-kni-lifecycle-agent-main-ibi-e2e-flow pull-ci-openshift-kni-lifecycle-agent-main-ipc-e2e-flow-v4v6

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sebrandon1: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

@sebrandon1

Copy link
Copy Markdown
Member Author

/pj-rehearse pull-ci-openshift-kni-lifecycle-agent-main-ibu-e2e-flow pull-ci-openshift-kni-lifecycle-agent-main-ibi-e2e-flow pull-ci-openshift-kni-lifecycle-agent-main-ipc-e2e-flow-v4v6

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sebrandon1: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Add SSH retry, trap handlers, pre-condition validation, and configurable
timeouts to the lifecycle-agent image-based CI step scripts to reduce
flaky e2e failures.
@sebrandon1
sebrandon1 force-pushed the fix/image-based-ci-reliability branch from abff636 to 88bb9f8 Compare July 29, 2026 19:55

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh (1)

76-80: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Keep dnf clean inside the guarded retry condition.

At Line 78, a transient cleanup failure terminates the generated installer under errexit, so attempts 2 and 3 never run. This is the previously reported issue, still present in this range.

Proposed fix
-    if [ "\$attempt" -gt 1 ]; then
-      sudo dnf clean -y all
-    fi
-    sudo dnf install -y \${packages[*]} && return 0
+    if { [ "\$attempt" -eq 1 ] || sudo dnf clean -y all; } \
+      && sudo dnf install -y "\${packages[@]}"; then
+      return 0
+    fi

As per coding guidelines, step registry command scripts must default to set -euo pipefail.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh`
around lines 76 - 80, Update the retry loop around the package installation
command so the `sudo dnf clean -y all` invocation in the `attempt > 1` branch
cannot terminate the script under `errexit`; guard or otherwise handle cleanup
failure while still allowing attempts 2 and 3 to proceed. Preserve the existing
`set -euo pipefail` default and the successful `dnf install` return path.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In
`@ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh`:
- Around line 76-80: Update the retry loop around the package installation
command so the `sudo dnf clean -y all` invocation in the `attempt > 1` branch
cannot terminate the script under `errexit`; guard or otherwise handle cleanup
failure while still allowing attempts 2 and 3 to proceed. Preserve the existing
`set -euo pipefail` default and the successful `dnf install` return path.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 8146489e-7ae6-4af9-a336-ff10842f9bc5

📥 Commits

Reviewing files that changed from the base of the PR and between abff636 and 88bb9f8.

📒 Files selected for processing (8)
  • ci-operator/step-registry/openshift/image-based/infra/aws/ec2/openshift-image-based-infra-aws-ec2-commands.sh
  • ci-operator/step-registry/openshift/image-based/install/openshift-image-based-install-commands.sh
  • ci-operator/step-registry/openshift/image-based/network-configuration/configure/openshift-image-based-network-configuration-configure-commands.sh
  • ci-operator/step-registry/openshift/image-based/network-configuration/configure/openshift-image-based-network-configuration-configure-ref.yaml
  • ci-operator/step-registry/openshift/image-based/upgrade/metal/config/openshift-image-based-upgrade-metal-config-commands.sh
  • ci-operator/step-registry/openshift/image-based/upgrade/seed/create/openshift-image-based-upgrade-seed-create-commands.sh
  • ci-operator/step-registry/openshift/image-based/upgrade/target/openshift-image-based-upgrade-target-commands.sh
  • ci-operator/step-registry/openshift/image-based/upgrade/target/openshift-image-based-upgrade-target-ref.yaml
🚧 Files skipped from review as they are similar to previous changes (2)
  • ci-operator/step-registry/openshift/image-based/network-configuration/configure/openshift-image-based-network-configuration-configure-ref.yaml
  • ci-operator/step-registry/openshift/image-based/upgrade/target/openshift-image-based-upgrade-target-ref.yaml

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@sebrandon1: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-rh-ecosystem-edge-bip-orchestrate-vm-main-ibi-e2e-flow rh-ecosystem-edge/bip-orchestrate-vm presubmit Registry content changed
pull-ci-rh-ecosystem-edge-bip-orchestrate-vm-main-ibi-e2e-flow-v4v6 rh-ecosystem-edge/bip-orchestrate-vm presubmit Registry content changed
pull-ci-rh-ecosystem-edge-bip-orchestrate-vm-main-ibi-e2e-flow-v6v4 rh-ecosystem-edge/bip-orchestrate-vm presubmit Registry content changed
pull-ci-rh-ecosystem-edge-bip-orchestrate-vm-main-ipc-e2e-flow rh-ecosystem-edge/bip-orchestrate-vm presubmit Registry content changed
pull-ci-rh-ecosystem-edge-bip-orchestrate-vm-main-ipc-e2e-flow-v4v6 rh-ecosystem-edge/bip-orchestrate-vm presubmit Registry content changed
pull-ci-rh-ecosystem-edge-bip-orchestrate-vm-main-ipc-e2e-flow-v6v4 rh-ecosystem-edge/bip-orchestrate-vm presubmit Registry content changed
pull-ci-rh-ecosystem-edge-bip-orchestrate-vm-main-ibu-e2e-flow rh-ecosystem-edge/bip-orchestrate-vm presubmit Registry content changed
pull-ci-rh-ecosystem-edge-bip-orchestrate-vm-main-ibu-e2e-flow-v4v6 rh-ecosystem-edge/bip-orchestrate-vm presubmit Registry content changed
pull-ci-rh-ecosystem-edge-bip-orchestrate-vm-main-ibu-e2e-flow-v6v4 rh-ecosystem-edge/bip-orchestrate-vm presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-main-ibi-e2e-flow rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-main-ibi-e2e-flow-v4v6 rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-main-ibi-e2e-flow-v6v4 rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-release-4.22-ibi-e2e-flow rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-release-4.22-ibi-e2e-flow-v4v6 rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-release-4.22-ibi-e2e-flow-v6v4 rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-release-4.21-ibi-e2e-flow rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-release-4.20-ibi-e2e-flow rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-release-4.19-ibi-e2e-flow rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-release-4.18-ibi-e2e-flow rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-main-ipc-e2e-flow rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-main-ipc-e2e-flow-v4v6 rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-main-ipc-e2e-flow-v6v4 rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-release-4.22-ipc-e2e-flow rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-release-4.22-ipc-e2e-flow-v4v6 rh-ecosystem-edge/recert presubmit Registry content changed
pull-ci-rh-ecosystem-edge-recert-release-4.22-ipc-e2e-flow-v6v4 rh-ecosystem-edge/recert presubmit Registry content changed

A total of 132 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@sebrandon1

Copy link
Copy Markdown
Member Author

/pj-rehearse pull-ci-openshift-kni-lifecycle-agent-main-ibu-e2e-flow pull-ci-openshift-kni-lifecycle-agent-main-ibi-e2e-flow pull-ci-openshift-kni-lifecycle-agent-main-ipc-e2e-flow-v4v6

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@sebrandon1: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci

openshift-ci Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

@sebrandon1: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/openshift-kni/lifecycle-agent/main/ibi-e2e-flow 88bb9f8 link unknown /pj-rehearse pull-ci-openshift-kni-lifecycle-agent-main-ibi-e2e-flow
ci/rehearse/openshift-kni/lifecycle-agent/main/ipc-e2e-flow-v4v6 88bb9f8 link unknown /pj-rehearse pull-ci-openshift-kni-lifecycle-agent-main-ipc-e2e-flow-v4v6
ci/rehearse/openshift-kni/lifecycle-agent/main/ibu-e2e-flow 88bb9f8 link unknown /pj-rehearse pull-ci-openshift-kni-lifecycle-agent-main-ibu-e2e-flow

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant