Repository navigation
Conversation
1659841 to
436ec4f
Compare
330c500 to
2b5329b
Compare
…coverage per PR (seedds#313) The roadmap (seedds#313) asks which code the tests reach and which lines of a change they run. A coverage job now answers that in its job summary, with no external service and no threshold: it reports, it never fails. C: `make coverage` in dm/host/tests and dm/gfx/tex/tests builds the same test rules again through seedds#412's SAN hook with clang's source-based coverage, into build/cov and cov/ (dm/sanitizers/coverage.mk), so the plain build stays as it is. Each test process writes its raw profile in continuous mode (%c), so the children that t_fpslock, t_preempt and others fork and end with _exit count too (without it hle_fpslock.c read 0%). llvm-profdata merges, llvm-cov exports JSON without the tests, third-party code and generated headers. The coverage run uses make -i: it measures, the plain run is the test. Python: coverage.py over dm/tools/tests, measuring dm/tools, dm/gfx/tools, dm/games and scripts, subprocesses included (the tests run many tools with sys.executable); configured in pyproject.toml, pinned in .github/requirements/coverage.txt. dm/tools/coverage_report.py writes the Markdown: totals per area, the 15 largest files and, given a base, the changed C, Objective-C and Python lines the tests run and those they do not, counting only lines the data knows. llvm-cov lines are rebuilt from the export's segments as `llvm-cov show` marks them (checked line for line on five host files), not from its per-function summaries, which count shared lines twice. In CI the pull request's merge commit gives HEAD^1 as the base, so fetch-depth 2 is enough. The raw reports are uploaded on every run. Checked on an M-series Mac: make coverage in both folders (host 29 s, textures 1 s), the tool tests under coverage (96 s); host 56.5% of 6,487 lines, CPU textures 86.3% of 233, Python tools 36.2% of 11,763; the reporter against 659c69d~1 (seedds#351: 186 of 203 changed lines run); unittest discover over dm/tools/tests (372 tests, OK), test_docs, and lint_python.py (0 new).
…ge per PR (PR #439) make coverage in dm/host/tests and dm/gfx/tex/tests rebuilds the tests with clang's source-based coverage into their own folders (dm/sanitizers/coverage.mk), runs them with continuous-mode profiles (children that end with _exit still count) and exports llvm-cov's lcov. coverage.py measures the tool tests over dm/tools, dm/gfx/tools, dm/games and scripts, subprocesses included. dm/tools/coverage_report.py writes the job summary: totals per area, the largest files and, on a pull request, the changed lines the tests run and those they do not. The coverage job in tests.yml is report only and uploads the raw reports. Changed from the PR: the C line counts are llvm-cov's own (the DA lines of -format=lcov, which llvm-cov show marks) instead of a rebuild of its line logic from the JSON export's segments; both give the same counts on every file of the host and texture tests (42 files, 0 lines differ). Today: host 61.7% of 7,883 lines, CPU textures 85.4% of 240, Python tools 51.9% of 15,023. Co-authored-by: v0lumehi <[email protected]>
|
Thank you! This landed on Measured locally: host 61.7% of 7,883 lines, CPU textures 85.4% of 240, Python tools 51.9% of 15,023. On your questions: the translator tests stay out for now (they'd need the dotnet / spirv-tools setup in this job). |
…ge per PR (PR seedds#439) make coverage in dm/host/tests and dm/gfx/tex/tests rebuilds the tests with clang's source-based coverage into their own folders (dm/sanitizers/coverage.mk), runs them with continuous-mode profiles (children that end with _exit still count) and exports llvm-cov's lcov. coverage.py measures the tool tests over dm/tools, dm/gfx/tools, dm/games and scripts, subprocesses included. dm/tools/coverage_report.py writes the job summary: totals per area, the largest files and, on a pull request, the changed lines the tests run and those they do not. The coverage job in tests.yml is report only and uploads the raw reports. Changed from the PR: the C line counts are llvm-cov's own (the DA lines of -format=lcov, which llvm-cov show marks) instead of a rebuild of its line logic from the JSON export's segments; both give the same counts on every file of the host and texture tests (42 files, 0 lines differ). Today: host 61.7% of 7,883 lines, CPU textures 85.4% of 240, Python tools 51.9% of 15,023. Co-authored-by: v0lumehi <[email protected]>
Update: #412's sanitizers are on
mainnow (ce99604), so this is rebased ontomainas one commit, with no merge order any more. The coverage job uses thexcode-27runner like every other job.make coverage(host 48.2% of regions, CPU textures 5,334/5,334 checks) and the reporter tests pass.Result: the first CI run is green in every job. The coverage job took 3m09s and measured all three areas: host 56.3%, CPU textures 86.3%, Python tools 36.4%, within a few tenths of the local numbers. The summary is in the job's summary tab, and the raw reports (
host.json,textures.json,python.json,summary.md) are in thecoverageartifact.Part of #313 ("Code quality checks in CI": coverage as a job summary, with diff coverage per PR; no external service, so Codecov stays optional).
Problem
Nobody can see which code the tests run. A PR that adds untested code looks the same in CI as one that's fully covered, and it's hard to tell where new tests would help most.
Change
C coverage through CI: ASan, UBSan and TSan over the host and CPU texture tests, as a ratchet #412's hook:
make coverageindm/host/testsanddm/gfx/tex/testsrebuilds the tests with-fprofile-instr-generate -fcoverage-mappinginto their own folder and runs them. It merges withxcrun llvm-profdataand exports withxcrun llvm-cov(dm/sanitizers/coverage.mk). The plain build is untouched.run, so a new test is covered without a list entry.%c%p). Without it, children that tests fork and end with_exitwrite no profile:hle_fpslock.cshowed 0%, and it's 62%.Python coverage: coverage.py (pinned in
.github/requirements/coverage.txt) over the tool tests, run file by file as the tool-test job runs them. It measuresdm/tools,dm/gfx/tools,dm/gamesandscripts, subprocesses included ([tool.coverage.run]inpyproject.toml).dm/tools/coverage_report.pywrites the job summary:C line counts are rebuilt from llvm-cov's segments; they match
llvm-cov showline for line (checked on five host files).test_coverage_report.pyhas 10 tests on synthetic samples, no llvm or coverage.py needed.CI: a
coveragejob intests.yml.Docs:
docs/development.md(Continuous integration, how to run it locally),06-dev-loop.md.Coverage today
The largest gaps:
shader_split.py28% (1,266 lines not run),patch_nso.py24% (815),dmsh_build.py4% (810),build_ipa_from_nsp.py55% (726),hle_movie.c11% (553). Most ofdm/gfx/nvndoesn't appear at all: no CPU test compiles it, and the Metal tests stay local (#112). The summary says so.Diff coverage on a real change, #351's gyro commit: 186 of the 203 changed lines that the tests measure are run (91.6%).
Checked
make -C dm/host/tests GAME=010015100B514000 -j8 coverage: 29 s, no ignored errors.make -C dm/gfx/tex/tests coverage:all passed: 5334/5334, 1 s. Tool tests under coverage.py: 96 s.test_docs, Python lint ratchet: OK, 0 new.Notes
macos-26like the other jobs on this base. If CI builds and tests with the toolchain the project supports (Xcode 27.1) #421 lands first, its toolchain test will ask for this job to move toxcode-27too; I'll update whichever lands second.dm/gfx/tests) join the Python coverage? They'd liftdm/gfx/tools, but need the dotnet / spirv-tools setup in this job.coverage.mksits indm/sanitizers/next tosanitizers.mk. Rename the folder to something broader?