Repository navigation
Test targets omit prerequisites, so stale binaries can pass -- and silently invalidate mutation testing #901
Description
Rationale
A test target that does not declare every source contributing to what it asserts can pass on a stale binary. That is bad on its own, but the serious consequence is that it defeats mutation testing: you inject a defect, re-run the test, see it pass, and conclude the code is fine -- when in fact the test binary was never rebuilt and the test never ran against the change. A mutation test that certifies a test which is not running is worse than no mutation test, because it converts "unverified" into "verified".
Hit concretely while building --diff-roots (PR #897); the class is repo-wide.
Current State
Concrete instance (fixed in #897). tests/Makefile had:
diff_roots_tests: diff_roots_tests.c ../frontier-cli/diff_roots.c ../frontier-cli/diff_roots.hThose tests assert on a table of packed-header field offsets that is defined in Common/source/oppack_v7.c and declared in Common/headers/op.h -- neither listed. Deliberately reintroducing the original off-by-six offset bug in oppack_v7.c and re-running produced 17/17 passing. Forcing a relink (rm -f diff_roots_tests) then failed the two tests that pin those offsets, which is the correct result. Fixed by adding both files as prerequisites.
Why the compile line does not save you. These targets compile a large fixed source list ($(LANG_RUNTIME_SOURCES)) in a single command. The sources are all on the command line, so a from-scratch build is correct -- but make only decides whether to re-run that command by comparing the target against its declared prerequisites. Any source in the command line but not in the prerequisite list is invisible to that decision.
Scope. This is structural, not specific to one target: most test targets in tests/Makefile list only their own .c (plus occasionally the unit under test) while compiling dozens of shared sources. Any test asserting on a constant, offset table, struct layout, or inline function owned by another module has the same exposure.
Desired Outcome
- Audit
tests/Makefilefor targets whose assertions depend on sources not in their prerequisite list; add the missing prerequisites. Priority to tests that pin layout/offset/constant values, since those are exactly the ones that fail silently. - Consider a structural fix so this cannot recur, e.g. a shared variable used for BOTH the prerequisite list and the compile line, so the two cannot drift by construction.
- Document the trap next to the test-authoring guidance: a mutation test is only valid if you have confirmed the binary was rebuilt.
rm -f <target>before a mutation run is a cheap guarantee.
Notes
No functional bug in shipping code -- this is test-infrastructure integrity. Its severity is that it silently weakens the evidence every other check depends on.
Activity
test(odb): commit behavioral fixtures; every compare path proves DETE…
feat(odb): --diff-roots value-level equality walker (root-build Step …