Skip to content

fix: make the functional, performance and simulation test steps able to fail - #7

Draft
srpatcha wants to merge 1 commit into
masterfrom
autofix/ci-test-steps-cannot-fail
Draft

srpatcha wants to merge 1 commit into
masterfrom
autofix/ci-test-steps-cannot-fail

Conversation

@srpatcha

@srpatcha srpatcha commented Sep 14, 2026

Copy link
Copy Markdown
Member

Problem

Three of the four test steps in CI ΓÇö .github cannot fail:

- name: Run functional tests
  run: python3 -m pytest tests/functional/ -v --tb=short 2>/dev/null || echo "Functional tests via run_all_tests.py"
- name: Run performance tests
  run: python3 -m pytest tests/performance/ -v --tb=short 2>/dev/null || echo "Performance tests via run_all_tests.py"
- name: Run simulation tests
  run: python3 -m pytest tests/simulation/ -v --tb=short 2>/dev/null || echo "Simulation tests via run_all_tests.py"

.github/workflows/ci.yml:30-37. Each step's exit status is the echo's, so it
is always 0. The 2>/dev/null in front of it means a collection error ΓÇö a
syntax error, a missing import, a renamed directory ΓÇö leaves no trace in the
log either. A regression in any of those three suites lands on master and the
run still reports success.

Root cause

The || echo "... via run_all_tests.py" fallbacks were written for the case
where pytest is not installed and run_all_tests.py is the real runner. But
the Install dependencies step (:22) does pip install pytest pytest-cov gcovr, so pytest is always present, and the message these steps print is not
even true: run_all_tests.py is invoked only by the Run unit tests step
above them. The fallback protects against a condition that cannot occur, and
pays for it by discarding the result in the condition that can.

.ai/reviewer.md lists this shape explicitly ΓÇö "a || true on a build or test
step, a verification whose result is discarded" ΓÇö and directs that it be
treated as a finding regardless of the reason given.

Fix

Drop the 2>/dev/null || echo ... from the three steps, leaving the bare
pytest invocation. Nothing else changes.

Files changed

  • .github/workflows/ci.yml

Expected impact

Nothing changes on a passing run. The three suites already pass: run
34663355034
on master shows the same three invocations reporting 3 passed, 2 passed
and 2 passed respectively ΓÇö they were green, the workflow simply was not
asking. From here on a failure in tests/functional/, tests/performance/ or
tests/simulation/ turns the job red instead of printing a sentence and
continuing.

Risks and compatibility

The risk is the intended one: these steps can now fail. That is the point of
the change, and the evidence above is that they do not fail today.

Two related items are deliberately not touched here, so this PR stays one
focused change:

  • Run unit tests (:28) still reads
    pytest tests/unit/ ... 2>/dev/null || python3 run_all_tests.py. There is no
    tests/unit/ directory in this repository, so pytest always exits non-zero
    and the fallback always runs. That step can fail ΓÇö run_all_tests.py
    propagates its status ΓÇö so it is not a gate that cannot fail, but the
    arrangement is misleading and the right repair (point it at a directory that
    exists, or drop the dead pytest half) is a separate decision.
  • security-scan (:55) runs
    cppcheck --error-exitcode=0 --quiet . 2>&1 | head -50 || true, which cannot
    fail on three independent grounds. Making a static-analysis gate real is a
    triage commitment ΓÇö someone has to own whatever it then reports ΓÇö so it is
    recorded for a maintainer rather than flipped here.

Both are in the maintenance backlog.

Note on the verification table below: the three pytest-* labels were first
recorded as failures against /usr/bin/python3, which has no pytest module on
this runner; they were re-run against the interpreter that does (Python 3.12)
and passed. CI installs pytest explicitly and runs Python 3.11, so the
python3 -m pytest form in the workflow is correct there. workflow-lint
parses the edited YAML and asserts none of the three steps still discards its
own result.

Verification

Executed in an isolated worktree branched from origin/master:

Check Result Duration Command
pytest-functional pass 0s /home/srpatcha/.local/share/uv/tools/pytest/bin/python -m pytest tests/functional/ -v --tb=short
pytest-performance pass 0s /home/srpatcha/.local/share/uv/tools/pytest/bin/python -m pytest tests/performance/ -v --tb=short
pytest-simulation pass 1s /home/srpatcha/.local/share/uv/tools/pytest/bin/python -m pytest tests/simulation/ -v --tb=short
workflow-lint pass 0s python3 /tmp/check-ci-yaml.py

Opened by the scheduled autoreview pipeline (model claude-opus-5), branched from origin/master. No human has reviewed this yet. Close it freely if the fix is wrong - a bad automated PR is a bug worth reporting.

Fixes #11

…to fail

Opened by the scheduled autoreview pipeline after review of open PRs.
Reviewed against the EmbeddedOS Master Design v2.0.

Files: .github/workflows/ci.yml
@srpatcha

Copy link
Copy Markdown
Member Author

Resolved the contribution-caused linked-issue policy failure without changing the branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Functional, performance, and simulation CI steps swallow pytest failures

1 participant