From 65aa416b9ecf0fde1a0e3078e40801d5fe35201c Mon Sep 17 00:00:00 2001
From: Fred Thiele <8555720+f3rdy@users.noreply.github.com>
Date: Tue, 28 Apr 2026 09:56:08 +0200
Subject: [PATCH] feat(ci): hard-fail CI on test skips (#55)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Backstop for the no-skips-in-CI policy. The conftest hook rewrites any
test reported as skipped into a setup error when GITHUB_ACTIONS=true,
which propagates to a non-zero pytest exit code through normal failure
accounting. Local dev is unchanged — the env var is GH-Actions-specific.

Now if a future test gains a skip marker without its tool being added
to CI alongside (the trap that #54 just plugged for cue), the very next
CI run fails loudly instead of silently passing with the test having
never executed.

Implementation:
- pytest_runtest_makereport hookwrapper in tests/conftest.py
- Watches for `report.skipped and report.when == "setup"` (where skipif
  markers fire) under GITHUB_ACTIONS=true
- Rewrites outcome to "failed" with a CI-policy explanation, leaving
  the original skip reason inline for traceability
- Verified locally: env unset → 380 passed; env set with no skips →
  380 passed; env set with a synthetic skipped test → exit 1 with the
  policy violation rendered as a setup error

Closes #55.
---
 CLAUDE.md         |  6 ++++++
 tests/conftest.py | 33 +++++++++++++++++++++++++++++++++
 2 files changed, 39 insertions(+)

diff --git a/CLAUDE.md b/CLAUDE.md
index cb60119..c17aab7 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -215,6 +215,12 @@ Examples:
 - Mock ansible-vault subprocess calls in unit tests
 - Use real ansible-vault in integration tests (test fixtures)
 - Minimum 70% coverage enforced
+- **No skips in CI**: `tests/conftest.py` rewrites any skipped test into a
+  setup error when `GITHUB_ACTIONS=true`. `requires_cue` /
+  `requires_ansible_vault` markers stay useful for local dev (graceful
+  fallback when the tool is missing) but never fire in CI because both
+  tools are installed there. New skip markers without their corresponding
+  CI tool will fail loudly. See #55 for the rationale.
 
 ### Commit Convention
 
diff --git a/tests/conftest.py b/tests/conftest.py
index 410f3ad..26ffbbd 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -13,6 +13,39 @@
 TEST_PASSWORD = "test-vault-password-12345"
 
 
+# --- CI policy: no skips ---
+#
+# Local dev keeps `requires_cue` / `requires_ansible_vault` markers as graceful
+# fallbacks. In CI we install both tools (#54), so those markers never fire —
+# unless someone adds a new skip marker without its CI tool. The hook below
+# rewrites any skip report into a failure when running under GitHub Actions,
+# so the gap turns into a real test failure rather than silent green.
+
+
+@pytest.hookimpl(hookwrapper=True)
+def pytest_runtest_makereport(item: pytest.Item, call: pytest.CallInfo[None]) -> object:
+    """Convert skip → failure under GITHUB_ACTIONS=true.
+
+    Skip outcomes appear during the "setup" phase (skipif markers fire there).
+    Rewriting the report's outcome makes pytest count the skip as a failed
+    test, which propagates to a non-zero exit code through pytest's normal
+    accounting.
+    """
+    outcome = yield
+    if os.environ.get("GITHUB_ACTIONS") != "true":
+        return outcome
+    report = outcome.get_result()
+    if report.skipped and report.when == "setup":
+        original = report.longrepr
+        report.outcome = "failed"
+        report.longrepr = (
+            f"CI policy: skipped tests are forbidden under GITHUB_ACTIONS=true. "
+            f"Install the required tool in CI instead of skipping. "
+            f"Original skip reason: {original}"
+        )
+    return outcome
+
+
 @pytest.fixture
 def tmp_dir(tmp_path):
     """Return a temporary directory as Path."""