Skip to content

Skip cyclictest latency test on non-RT kernels - #1635

Closed
tavital wants to merge 1 commit into
openshift:mainfrom
tavital:adjust-latency-tests-for-non-rt-kernel
Closed

tavital wants to merge 1 commit into
openshift:mainfrom
tavital:adjust-latency-tests-for-non-rt-kernel

Conversation

@tavital

@tavital tavital commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

cyclictest measures real-time scheduler wake-up latency, so its pass/fail verdict is only meaningful on a PREEMPT_RT kernel, where the worst-case latency is bounded and repeatable.

On a non-RT kernel the worst-case latency is unbounded and non-deterministic, so no threshold is both correct and stable and raising it does not help. The result is a flaky, meaningless pass/fail. This is the case on aarch64 today, which has no RT kernel available.

Add IsRealTimeKernelEnabled, a profile-package helper that reads the setting directly from the PerformanceProfile spec, and use it to skip cyclictest when the profile does not enable the real-time kernel. The check runs before the pod is created. oslat and hwlatdetect are unaffected: they measure hardware and firmware jitter, which is bounded regardless of the kernel.

Summary by CodeRabbit

  • Bug Fixes

    • Latency testing now skips profiles that do not enable the real-time (PREEMPT_RT) kernel, avoiding inapplicable test failures.
  • Tests

    • Added coverage for detecting whether the real-time kernel is enabled, including unset and explicitly enabled or disabled configurations.

cyclictest measures real-time scheduler wake-up latency, so its pass/fail
verdict is only meaningful on a PREEMPT_RT kernel, where the worst-case
latency is bounded and repeatable.

On a non-RT kernel the worst-case latency is unbounded and non-deterministic,
so no threshold is both correct and stable and raising it does not help. The
result is a flaky, meaningless pass/fail. This is the case on aarch64 today,
which has no RT kernel available.

Add IsRealTimeKernelEnabled, a profile-package helper that reads the setting
directly from the PerformanceProfile spec, and use it to skip cyclictest when
the profile does not enable the real-time kernel. The check runs before the
pod is created. oslat and hwlatdetect are unaffected: they measure hardware
and firmware jitter, which is bounded regardless of the kernel.

Signed-off-by: Tomer Avital <[email protected]>
@openshift-ci

openshift-ci Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@coderabbitai

coderabbitai Bot commented Sep 14, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 679bbc98-f615-428e-9a7b-837e6b51db7d

📥 Commits

Reviewing files that changed from the base of the PR and between 0759b94 and 3e9c009.

📒 Files selected for processing (3)
  • pkg/performanceprofile/controller/performanceprofile/components/profile/profile.go
  • pkg/performanceprofile/controller/performanceprofile/components/profile/profile_test.go
  • test/e2e/performanceprofile/functests/4_latency/latency.go

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.


Walkthrough

The change adds a helper that reports whether a performance profile enables PREEMPT_RT. The cyclictest e2e setup skips execution when PREEMPT_RT is disabled or unspecified. Unit tests cover the helper’s supported configurations.

Changes

Real-time kernel gating

Layer / File(s) Summary
Real-time kernel detection and tests
pkg/performanceprofile/controller/performanceprofile/components/profile/profile.go, pkg/performanceprofile/controller/performanceprofile/components/profile/profile_test.go
Adds IsRealTimeKernelEnabled, which returns false for nil configuration or flag values and otherwise returns the configured boolean. Unit tests cover all four cases.
Cyclictest eligibility gating
test/e2e/performanceprofile/functests/4_latency/latency.go
The cyclictest setup skips when the profile does not enable PREEMPT_RT.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: tal-or

Merge Risk: ⚪ Minimal · up to 3e9c0

The change gates cyclictest to real-time kernel profiles while leaving other latency tests unaffected, with no identified merge-blocking risk.

🚥 Pre-merge checks | ✅ 13 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 3 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality ⚠️ Warning The four new Ginkgo It blocks each use an assertion without a meaningful failure message: Expect(IsRealTimeKernelEnabled(profile)).To(BeFalse()) or To(BeTrue()) in profile_test.go. The tests o… Add a diagnostic message to each new boolean assertion. For example: Expect(IsRealTimeKernelEnabled(profile)).To(BeFalse(), "nil RealTimeKernel section must disable the real-time kernel"), with equivalent messages for a nil Enabled flag…
✅ Passed checks (13 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the primary change: skipping the cyclictest latency test when the kernel is not real-time.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PASS: The pull request adds only static Ginkgo titles: Real Time Kernel, IsRealTimeKernelEnabled, and four fixed behavior descriptions. The cyclictest change adds a dynamic profile name only to a …
Microshift Test Compatibility ✅ Passed PASS: The pull request does not add a new Ginkgo e2e test. It adds unit-level Ginkgo tests in the pkg/.../profile package, and the e2e change only adds a PREEMPT_RT profile check to the existing cyc…
Single Node Openshift (Sno) Test Compatibility ✅ Passed The pull request does not introduce a SNO-incompatible test. The e2e diff only adds a PREEMPT_RT check to the existing cyclictest BeforeEach and skips before pod creation. It adds no new e2e It, Descr…
Topology-Aware Scheduling Compatibility ✅ Passed PASS — The scoped diff changes only a profile-spec helper, unit tests, and a cyclictest skip. It adds no deployment manifest, replica setting, affinity, topology spread constraint, PDB, control-plane …
Ote Binary Stdout Contract ✅ Passed The pull request introduces no process-level stdout write. The changed code adds a pure profile helper, unit tests, and a Skip(fmt.Sprintf(...)) call inside a Ginkgo BeforeEach, which the check ex…
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No failure condition is introduced. The pull request adds Ginkgo cases only in the unit-test package, not in an e2e test. The e2e change only skips cyclictest when PREEMPT_RT is disabled. The added li…
No-Weak-Crypto ✅ Passed The pull request changes only PerformanceProfile inspection, unit tests, and cyclictest skip logic. The authoritative diff adds no MD5, SHA1, DES, 3DES, RC4, Blowfish, or ECB usage. It adds no custom …
Container-Privileges ✅ Passed The pull request adds only a profile helper, tests, and a PREEMPT_RT skip condition. The reviewed diff adds no privileged, host namespace, SYS_ADMIN, allowPrivilegeEscalation, or root configurat…
No-Sensitive-Data-In-Logs ✅ Passed PASS. The pull request adds no password, token, API key, PII, session ID, hostname, or customer-data logging. The only new output is a Ginkgo skip message that includes the PerformanceProfile resource…
Full details: Test Structure And Quality

Explanation

The four new Ginkgo It blocks each use an assertion without a meaningful failure message: Expect(IsRealTimeKernelEnabled(profile)).To(BeFalse()) or To(BeTrue()) in profile_test.go. The tests otherwise have single responsibility, use the existing BeforeEach setup, create no resources, and add no untimed cluster waits. The cyclictest change only adds a pre-pod Skip; existing pod waits and cleanup already use bounded helpers.

Resolution

Add a diagnostic message to each new boolean assertion. For example: Expect(IsRealTimeKernelEnabled(profile)).To(BeFalse(), "nil RealTimeKernel section must disable the real-time kernel"), with equivalent messages for a nil Enabled flag, Enabled=false, and Enabled=true.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

Some tools did not complete. Review the errors below.

🔧 golangci-lint (2.13.2)

level=error msg="Running error: context loading failed: failed to load packages: failed to load packages: failed to load with go/packages: err: exit status 1: stderr: go: inconsistent vendoring in :\n\tgithub.com/RHsyseng/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/coreos/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/coreos/ignition/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/docker/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/go-logr/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/google/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/jaypipes/[email protected]: i

... [truncated 19684 characters] ...

8s.io/kubectl: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/kubelet: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/metrics: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/mount-utils: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/pod-security-admission: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/sample-apiserver: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tgithub.com/onsi/ginkgo/v2: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\n\tTo ignore the vendor directory, use -mod=readonly or -mod=mod.\n\tTo sync the vendor directory, run:\n\t\tgo mod vendor\n"


Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from Tal-or and jmencak September 14, 2026 07:53
@openshift-ci

openshift-ci Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tavital
Once this PR has been reviewed and has the lgtm label, please assign yanirq for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tavital

tavital commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

/retest-required

@jmencak

jmencak commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

okd-scos-images fails with "nevra" issues, see #1633

@yanirq

yanirq commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

/cc @MarSik @mrniranjan

@openshift-ci
openshift-ci Bot requested review from MarSik and mrniranjan September 14, 2026 09:46
@tavital tavital closed this Sep 14, 2026
@tavital

tavital commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

PR closed because customers in fact use non-rt with cyclictest (they should raise the max latency manually for cyclictest if reached).

@tavital tavital reopened this Sep 14, 2026
@tavital tavital closed this Sep 14, 2026
@tavital tavital reopened this Sep 14, 2026
@tavital

tavital commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

PR closed because customers in fact use non-rt with cyclictest (they should raise the max latency manually for cyclictest if reached).

@tavital tavital closed this Sep 14, 2026
@openshift-ci

openshift-ci Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

@tavital: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/okd-scos-images 3e9c009 link true /test okd-scos-images

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants