Skip to content

ci: defer cargo, integration, and coverage lanes to nightly - #2877

Merged
MarcusSorealheis merged 3 commits into
TraceMachina:mainfrom
b7r6:ci/nightly-redundant-lanes
Oct 19, 2026
Merged

MarcusSorealheis merged 3 commits into
TraceMachina:mainfrom
b7r6:ci/nightly-redundant-lanes

Conversation

@b7r6

@b7r6 b7r6 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

What and why

This moves four lanes off pull requests to a nightly run (0 7 * * * UTC) plus workflow_dispatch, to cut PR latency. Each one is either covered by a lane that stays on PRs, or slow and not gating:

  • Cargo Native, Linux and Windows (native-cargo.yaml). PR build and test signal comes from Bazel Dev.
  • Cargo Dev (nix-cargo in nix.yaml).
  • Integration (buildstream, buck2 and mongo *-with-nativelink-test in nix.yaml). These run real clients end to end against NativeLink.
  • Coverage. It still runs on push: main as well, so the Pages trend updates on merge. It needs an xlarge runner, can take up to 180 minutes, and gates nothing.

Staying on every PR:

Jobs that stay on PRs are gated with github.event_name != 'schedule', so the nightly run skips them.

On "we cannot move all of RBE testing to nightly": the RBE toolchain matrix and LRE both stay on PRs. The integration lane also exercises NativeLink as a remote executor, for buck2 and buildstream. If that's the RBE testing you mean, I'll take it out of this PR and leave it on PRs.

CodeQL is a required status check, so moving it needs a branch-protection change. It is left alone here.

How was this verified?

The workflow files parse, and each job's if matches the split above.

Our fork's main has run this cadence, together with #2880, since 2026-10-14. In today's scheduled run there, Cargo Native and Bazel Native passed. The Nix nightly (Cargo Dev and integration) and Coverage had not started yet when I wrote this, because the fork's runners are backed up.

For scale, the slowest checks on #2910's run today were:

Check Duration After this PR
Coverage 41 min moves to nightly
Windows cargo 17 min moves to nightly
fuzz 16 min stays on PRs
mongo integration 14 min moves to nightly
Bazel Dev 13 min stays on PRs

The combined deferral set (#2886, measured 2026-10-04) finished in 14 minutes end to end.

Risk

Low. This changes when these lanes run, not what they run. Windows cargo builds and the end-to-end integration tests would no longer run on PRs, so a break there shows up in the next nightly or on merge to main instead. No source, config or wire-format changes.

AI assistance

Claude drafted these workflow edits and this description under my direction.

@vercel

vercel Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 19, 2026 2:17pm UTC
nativelink-aidm Ready Ready Preview Oct 19, 2026 2:17pm UTC

Request Review

@github-actions

This comment has been minimized.

@b7r6
b7r6 force-pushed the ci/nightly-redundant-lanes branch from 5356c33 to d1a9e1e Compare October 3, 2026 20:32
@b7r6
b7r6 force-pushed the ci/nightly-redundant-lanes branch from d1a9e1e to 49a657f Compare October 3, 2026 20:37
@b7r6 b7r6 changed the title ci: move redundant and expensive lanes to a nightly schedule ci: move redundant and expensive lanes (cargo + coverage) to nightly Oct 3, 2026
@b7r6
b7r6 force-pushed the ci/nightly-redundant-lanes branch from 49a657f to 2265ca7 Compare October 3, 2026 20:48
@b7r6 b7r6 changed the title ci: move redundant and expensive lanes (cargo + coverage) to nightly ci: defer cargo, integration, RBE-toolchain, and coverage lanes to nightly Oct 3, 2026
@b7r6
b7r6 force-pushed the ci/nightly-redundant-lanes branch from 2265ca7 to da552fe Compare October 3, 2026 20:57

@MarcusSorealheis MarcusSorealheis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MarcusSorealheis MarcusSorealheis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We cannot move all of rbe testing to nightly.

@b7r6 b7r6 changed the title ci: defer cargo, integration, RBE-toolchain, and coverage lanes to nightly ci: defer cargo, integration, and coverage lanes to nightly Oct 3, 2026
@b7r6
b7r6 force-pushed the ci/nightly-redundant-lanes branch from da552fe to 7ef505e Compare October 3, 2026 21:05
b7r6 pushed a commit to b7r6/nativelink that referenced this pull request Oct 4, 2026
Combines TraceMachina#2877 + TraceMachina#2878 + TraceMachina#2880 on one branch so a single CI run shows the
true end-state per-PR wall clock. Not for merge: the individual PRs are the
review units; this exists to measure their net effect.
b7r6 pushed a commit to b7r6/nativelink that referenced this pull request Oct 4, 2026
Combines TraceMachina#2877 + TraceMachina#2878 + TraceMachina#2880 on one branch so a single CI run shows the
true end-state per-PR wall clock. Not for merge: the individual PRs are the
review units; this exists to measure their net effect.
@b7r6

b7r6 commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

@MarcusSorealheis I think we got those back in on Saturday, or am I conflating the pictued lanes with the ones you mean? This is the most recent run.

image

b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 7, 2026
…achina#2880) on fork main

Mirrors the intended upstream per-PR layout ahead of land: Bazel Native and
Cargo Native (incl. Windows) move to nightly + merge-to-main; per-PR build/test
signal is carried by the Nix Bazel Dev lane. Keeps the fork representative of
where upstream CI is going, not where main happens to sit today. TraceMachina#2897 retry
logic preserved in native-bazel.yaml.
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 7, 2026
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off
PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's
integration-tests (docker-compose image build + run_integration_tests.sh)
fell between them and still ran on every PR. Defer to nightly + merge-to-main,
matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly
set.
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 14, 2026
…achina#2880) on fork main

Mirrors the intended upstream per-PR layout ahead of land: Bazel Native and
Cargo Native (incl. Windows) move to nightly + merge-to-main; per-PR build/test
signal is carried by the Nix Bazel Dev lane. Keeps the fork representative of
where upstream CI is going, not where main happens to sit today. TraceMachina#2897 retry
logic preserved in native-bazel.yaml.
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 14, 2026
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off
PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's
integration-tests (docker-compose image build + run_integration_tests.sh)
fell between them and still ran on every PR. Defer to nightly + merge-to-main,
matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly
set.
@b7r6
b7r6 force-pushed the ci/nightly-redundant-lanes branch from 4f6fdb3 to cb1df1d Compare October 14, 2026 17:40
Defer CI lanes that are redundant with another per-PR lane, or expensive
and non-gating, to a nightly schedule (0 7 * * * UTC) + workflow_dispatch.
Per-PR keeps Bazel Dev, the install smoke test, and NativeLink's own RBE
toolchain matrix; the rest runs nightly.

- Cargo Native (Linux + Windows): nightly.
- Cargo Dev (nix-cargo): nightly.
- Integration (buildstream/buck2/mongo): nightly; core build/test covered
  per-PR by Bazel Dev.
- Coverage: nightly + push:main (xlarge, 180-min, non-gating).

Kept per-PR: Bazel Dev, installation, RBE toolchain matrix
(generate-rbe-commands + rbe-toolchain), asan, Bazel Native, LRE, lints.
CodeQL is a required check, left for a coordinated branch-protection follow-up.
The comment listed the integration tests among the jobs that stay on the
per-PR path, but the integration job's `if` runs it only on the schedule and
on demand. Name both nightly-only jobs (Cargo Dev and integration) and the
ones that stay per-PR.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

@MarcusSorealheis MarcusSorealheis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MarcusSorealheis
MarcusSorealheis merged commit 34da49a into TraceMachina:main Oct 19, 2026
31 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — 5e3948c3 Deployed Oct 19, 2026 by vercel[bot]
Preview – nativelink-aidm — 5e3948c3 Deployed Oct 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants