Skip to content

CI: line coverage of the host, CPU texture and tool tests, with diff coverage per PR - #439

Closed
v0lumehi wants to merge 1 commit into
seedds:mainfrom
v0lumehi:ci-coverage
Closed

v0lumehi wants to merge 1 commit into
seedds:mainfrom
v0lumehi:ci-coverage

Conversation

@v0lumehi

@v0lumehi v0lumehi commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Update: #412's sanitizers are on main now (ce99604), so this is rebased onto main as one commit, with no merge order any more. The coverage job uses the xcode-27 runner like every other job. make coverage (host 48.2% of regions, CPU textures 5,334/5,334 checks) and the reporter tests pass.

Result: the first CI run is green in every job. The coverage job took 3m09s and measured all three areas: host 56.3%, CPU textures 86.3%, Python tools 36.4%, within a few tenths of the local numbers. The summary is in the job's summary tab, and the raw reports (host.json, textures.json, python.json, summary.md) are in the coverage artifact.

Part of #313 ("Code quality checks in CI": coverage as a job summary, with diff coverage per PR; no external service, so Codecov stays optional).

Problem

Nobody can see which code the tests run. A PR that adds untested code looks the same in CI as one that's fully covered, and it's hard to tell where new tests would help most.

Change

  • C coverage through CI: ASan, UBSan and TSan over the host and CPU texture tests, as a ratchet #412's hook:

    • make coverage in dm/host/tests and dm/gfx/tex/tests rebuilds the tests with -fprofile-instr-generate -fcoverage-mapping into their own folder and runs them. It merges with xcrun llvm-profdata and exports with xcrun llvm-cov (dm/sanitizers/coverage.mk). The plain build is untouched.
    • The host target reuses run, so a new test is covered without a list entry.
    • Raw profiles use continuous mode (%c%p). Without it, children that tests fork and end with _exit write no profile: hle_fpslock.c showed 0%, and it's 62%.
  • Python coverage: coverage.py (pinned in .github/requirements/coverage.txt) over the tool tests, run file by file as the tool-test job runs them. It measures dm/tools, dm/gfx/tools, dm/games and scripts, subprocesses included ([tool.coverage.run] in pyproject.toml).

  • dm/tools/coverage_report.py writes the job summary:

    • totals per area, and the 15 largest files with their %;
    • diff coverage: the changed C and Python lines the tests run vs don't, per file.

    C line counts are rebuilt from llvm-cov's segments; they match llvm-cov show line for line (checked on five host files). test_coverage_report.py has 10 tests on synthetic samples, no llvm or coverage.py needed.

  • CI: a coverage job in tests.yml.

    • Report only: it never fails the PR, and an area that couldn't be measured shows as "not measured".
    • The raw reports are uploaded as an artifact (7 days).
    • For a PR, diff coverage is against the merge commit's first parent, the PR's base.
  • Docs: docs/development.md (Continuous integration, how to run it locally), 06-dev-loop.md.

Coverage today

Area Lines Covered
Host (the 32 files the tests compile) 6,487 56.5%
CPU textures (3 files) 233 86.3%
Python tools (84 files) 11,763 36.2%

The largest gaps: shader_split.py 28% (1,266 lines not run), patch_nso.py 24% (815), dmsh_build.py 4% (810), build_ipa_from_nsp.py 55% (726), hle_movie.c 11% (553). Most of dm/gfx/nvn doesn't appear at all: no CPU test compiles it, and the Metal tests stay local (#112). The summary says so.

Diff coverage on a real change, #351's gyro commit: 186 of the 203 changed lines that the tests measure are run (91.6%).

Checked

  • make -C dm/host/tests GAME=010015100B514000 -j8 coverage: 29 s, no ignored errors. make -C dm/gfx/tex/tests coverage: all passed: 5334/5334, 1 s. Tool tests under coverage.py: 96 s.
  • Tool tests (372), test_docs, Python lint ratchet: OK, 0 new.
  • Not yet run on GitHub: this PR's own run is the first. Expect about 6–8 minutes.

Notes

  • The job runs on macos-26 like the other jobs on this base. If CI builds and tests with the toolchain the project supports (Xcode 27.1) #421 lands first, its toolchain test will ask for this job to move to xcode-27 too; I'll update whichever lands second.
  • Two questions:
    1. Should the translator tests (dm/gfx/tests) join the Python coverage? They'd lift dm/gfx/tools, but need the dotnet / spirv-tools setup in this job.
    2. coverage.mk sits in dm/sanitizers/ next to sanitizers.mk. Rename the folder to something broader?

…coverage per PR (seedds#313)

The roadmap (seedds#313) asks which code the tests reach and which lines of a
change they run. A coverage job now answers that in its job summary,
with no external service and no threshold: it reports, it never fails.

C: `make coverage` in dm/host/tests and dm/gfx/tex/tests builds the same
test rules again through seedds#412's SAN hook with clang's source-based
coverage, into build/cov and cov/ (dm/sanitizers/coverage.mk), so the
plain build stays as it is. Each test process writes its raw profile in
continuous mode (%c), so the children that t_fpslock, t_preempt and
others fork and end with _exit count too (without it hle_fpslock.c read
0%). llvm-profdata merges, llvm-cov exports JSON without the tests,
third-party code and generated headers. The coverage run uses make -i:
it measures, the plain run is the test.

Python: coverage.py over dm/tools/tests, measuring dm/tools,
dm/gfx/tools, dm/games and scripts, subprocesses included (the tests
run many tools with sys.executable); configured in pyproject.toml,
pinned in .github/requirements/coverage.txt.

dm/tools/coverage_report.py writes the Markdown: totals per area, the
15 largest files and, given a base, the changed C, Objective-C and
Python lines the tests run and those they do not, counting only lines
the data knows. llvm-cov lines are rebuilt from the export's segments
as `llvm-cov show` marks them (checked line for line on five host
files), not from its per-function summaries, which count shared lines
twice. In CI the pull request's merge commit gives HEAD^1 as the base,
so fetch-depth 2 is enough. The raw reports are uploaded on every run.

Checked on an M-series Mac: make coverage in both folders (host 29 s,
textures 1 s), the tool tests under coverage (96 s); host 56.5% of
6,487 lines, CPU textures 86.3% of 233, Python tools 36.2% of 11,763;
the reporter against 659c69d~1 (seedds#351: 186 of 203 changed lines run);
unittest discover over dm/tools/tests (372 tests, OK), test_docs, and
lint_python.py (0 new).
seedds added a commit that referenced this pull request Oct 30, 2026
…ge per PR (PR #439)

make coverage in dm/host/tests and dm/gfx/tex/tests rebuilds the tests with clang's
source-based coverage into their own folders (dm/sanitizers/coverage.mk), runs them
with continuous-mode profiles (children that end with _exit still count) and exports
llvm-cov's lcov. coverage.py measures the tool tests over dm/tools, dm/gfx/tools,
dm/games and scripts, subprocesses included. dm/tools/coverage_report.py writes the
job summary: totals per area, the largest files and, on a pull request, the changed
lines the tests run and those they do not. The coverage job in tests.yml is report
only and uploads the raw reports.

Changed from the PR: the C line counts are llvm-cov's own (the DA lines of
-format=lcov, which llvm-cov show marks) instead of a rebuild of its line logic
from the JSON export's segments; both give the same counts on every file of the
host and texture tests (42 files, 0 lines differ).

Today: host 61.7% of 7,883 lines, CPU textures 85.4% of 240, Python tools 51.9%
of 15,023.

Co-authored-by: v0lumehi <[email protected]>
@seedds

seedds commented Oct 30, 2026

Copy link
Copy Markdown
Owner

Thank you! This landed on main as 68db795, with you as co-author. One change from the PR: the C line counts are now llvm-cov's own, the DA lines of llvm-cov export -format=lcov (the lines llvm-cov show marks). Before, they were rebuilt from the JSON export's segments. Both give the same counts on every file of the host and texture tests: 42 files, 0 lines differ. coverage.mk writes coverage.lcov, and coverage_report.py --host/--textures reads it.

Measured locally: host 61.7% of 7,883 lines, CPU textures 85.4% of 240, Python tools 51.9% of 15,023.

On your questions: the translator tests stay out for now (they'd need the dotnet / spirv-tools setup in this job). coverage.mk stays in dm/sanitizers/, since it's the same kind of instrumented test build.

@seedds seedds closed this Oct 30, 2026
Know-It-All-Brother pushed a commit to Know-It-All-Brother/nxnative that referenced this pull request Nov 9, 2026
…ge per PR (PR seedds#439)

make coverage in dm/host/tests and dm/gfx/tex/tests rebuilds the tests with clang's
source-based coverage into their own folders (dm/sanitizers/coverage.mk), runs them
with continuous-mode profiles (children that end with _exit still count) and exports
llvm-cov's lcov. coverage.py measures the tool tests over dm/tools, dm/gfx/tools,
dm/games and scripts, subprocesses included. dm/tools/coverage_report.py writes the
job summary: totals per area, the largest files and, on a pull request, the changed
lines the tests run and those they do not. The coverage job in tests.yml is report
only and uploads the raw reports.

Changed from the PR: the C line counts are llvm-cov's own (the DA lines of
-format=lcov, which llvm-cov show marks) instead of a rebuild of its line logic
from the JSON export's segments; both give the same counts on every file of the
host and texture tests (42 files, 0 lines differ).

Today: host 61.7% of 7,883 lines, CPU textures 85.4% of 240, Python tools 51.9%
of 15,023.

Co-authored-by: v0lumehi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants