Skip to content

ci: measure full Rust coverage nightly#495

Merged
jeremi merged 5 commits into
mainfrom
agent/nightly-rust-coverage
Jul 26, 2026
Merged

ci: measure full Rust coverage nightly#495
jeremi merged 5 commits into
mainfrom
agent/nightly-rust-coverage

Conversation

@jeremi

@jeremi jeremi commented Jul 26, 2026

Copy link
Copy Markdown
Member

Pull Request

Summary

Add comprehensive nightly Rust coverage without extending the pull-request critical path.

  • derive six disk-bounded unit-test shards from the same authoritative inventory as PR CI, covering all 27 workspace members
  • run Platform and Relay shards with all features, plus a separate Notary CEL worker-fixture shard
  • collect Relay PostgreSQL coverage on PostgreSQL 16, 17, and 18
  • invoke the existing Notary PostgreSQL coverage workflow at the same commit through workflow_call
  • make the reusable Notary coverage path explicit and strict for nightly callers
  • upload every explicit LCOV report to Codecov under a bounded flag and retain GitHub artifacts for seven days
  • carry the latest nightly flag measurements forward across intervening commits
  • run at 19:17 UTC nightly or by manual dispatch, with at most four Rust unit shards in parallel

Impact

The workflow has no pull_request, push, or merge-queue trigger and is not a required check. It therefore adds no steady-state PR latency. Coverage jobs use dedicated target directories, registry-only Cargo caches, immutable Action pins, and OIDC authentication. Codecov upload failures fail the nightly workflow so missing measurements are visible without blocking merges.

The platform Codecov flag is scoped only to Platform crates. registry-config-report remains measured under developer-tools, preventing the same carried flag from representing different package sets on PR and nightly commits.

Checks

  • actionlint
  • python3 .github/scripts/test_ci_changes.py: 14 tests passed
  • Codecov configuration validation via POST https://codecov.io/validate: Valid!
  • generated coverage matrix inventory check
    • all 27 workspace packages assigned exactly once across the six standard shards
  • local Manifest shard through cargo llvm-cov
    • tests passed
    • text report succeeded
    • produced a non-empty 8,685-line LCOV report
  • git diff --check
  • complete remote nightly workflow: run 30205531552, passed in 7m44s
    • six standard workspace shards passed
    • Notary CEL shard passed
    • Relay PostgreSQL coverage passed on PostgreSQL 16, 17, and 18
    • Notary PostgreSQL coverage passed on PostgreSQL 16, 17, and 18
    • all 13 LCOV artifacts and Codecov uploads passed
  • Codecov processed the verified commit as 13 sessions, 284 files, 173,047 measured lines, and 85.13% merged line coverage
  • Codex review follow-up
    • scheduled reusable callers opt into Notary coverage through strict_coverage
    • strict callers fail when a Notary Codecov upload fails
    • Platform and developer-tools flag scopes are consistent
    • all commit sign-offs verified by DCO

Initial line-coverage baselines

Flag Line coverage
platform 92.04%
manifest-unit 90.81%
developer-tools 86.09%
registryctl-unit 82.61%
notary-cel 81.62%
notary-unit 81.29%
relay-unit 75.92%
notary-postgres 19.95% to 19.96%
relay-postgres 14.42%

The PostgreSQL percentages measure the focused database conformance paths against their owning crate source, so they are expected to be much lower than the unit-test shards. The merged Codecov result combines all sessions rather than averaging these percentages.

Notes

  • Owning area: CI configuration only.
  • Notary PKCS#11 is not enabled by the generic feature shard because it requires its dedicated external-provider environment. The nightly set mirrors deployable CI feature combinations rather than feature-unifying incompatible environments.
  • Fuzzing remains in the existing nightly security workflow and is not a line-coverage source.
  • GitHub currently emits a non-blocking Node.js 20 deprecation annotation for the pinned actions/upload-artifact release. GitHub runs it on Node.js 24, and the uploads passed.

DCO

  • Every commit includes a Signed-off-by trailer.
  • I reviewed the submitted changes and am responsible for the contribution.

@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment

Thanks for integrating Codecov - We've got you covered ☂️

Base automatically changed from agent/codecov-coverage to main July 26, 2026 14:27
@jeremi
jeremi marked this pull request as ready for review July 26, 2026 14:27
@jeremi
jeremi force-pushed the agent/nightly-rust-coverage branch from f79ca24 to 80bd4c8 Compare July 26, 2026 14:28
@chatgpt-codex-connector

Copy link
Copy Markdown

💡 Codex Review


P1 Badge Add the required DCO sign-off

This commit has no Signed-off-by: trailer, so it violates the repository's DCO policy and may be rejected by DCO enforcement despite the checked DCO statement in the commit message; recreate the commit using git commit -s before merging.

AGENTS.md reference: AGENTS.md:L68-L70



P2 Badge Fail nightly runs when the Notary upload fails

When .github/workflows/nightly-rust-coverage.yml invokes this workflow through its notary-postgres reusable job, an authentication, Codecov, or upload failure is suppressed by fail_ci_if_error: false; all three Notary uploads can therefore be absent while the nightly workflow remains green. Make this input fail the step for workflow_call executions while retaining non-blocking behavior for branch-triggered conformance runs if desired.


- "crates/registry-config-report/"

P2 Badge Keep the platform flag's package scope consistent

The PR platform-coverage job uploads registry-config-report and all platform crates under the platform flag, but the nightly matrix derives its platform package list from SHARDS["platform"], which excludes registry-config-report and instead assigns it to developer-tools. Consequently the same Codecov flag measures different source sets on PR and nightly commits, making its carried-forward project and patch comparisons inconsistent; either include registry-config-report in the nightly platform upload or remove it from this flag's scope.


if: >-
github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
github.event_name == 'workflow_call'

P1 Badge Run Notary coverage for scheduled callers

In a reusable workflow, github.event_name remains the event that triggered the caller, so the nightly cron invokes this workflow with github.event_name == 'schedule', not workflow_call. Because schedule is absent from this condition, all three Notary PostgreSQL coverage jobs are skipped during normal nightly runs; a manual validation passes only because workflow_dispatch is explicitly allowed.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 661597c9ab

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/nightly-rust-coverage.yml
@jeremi
jeremi merged commit a4e5a89 into main Jul 26, 2026
33 checks passed
@jeremi
jeremi deleted the agent/nightly-rust-coverage branch July 26, 2026 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant