(#6) ci: wire CPU/GPU/multi-GPU/nightly + docs

.github/actions/setup-dmi/action.yml

@@ -0,0 +1,77 @@
+name: "Set up DMI"
+description: >-
+  Install DMI's Python deps + the modified Transformers fork, then build and
+  install DMI via the Stage 3 pip entrypoint (pip install -e . --no-build-isolation).
+  The entrypoint calls setup.py NativeBuildExt which runs cmake + make internally.
+  Pass skip-native-build: "true" on CPU-only runners to bypass nvcc/cmake entirely.
+
+inputs:
+  python-version:
+    description: "Python version to use"
+    required: false
+    default: "3.12"
+  install-vllm:
+    description: "Also install the vLLM fork (heavy; only for vLLM E2E suites)"
+    required: false
+    default: "false"
+  skip-native-build:
+    description: >-
+      Set SKIP_NATIVE_BUILD=1 so NativeBuildExt.run() exits early.
+      Use on CPU-only runners that have no nvcc. The native .so will be absent,
+      and tests that need it must be excluded via marker (not e2e and not gpu).
+    required: false
+    default: "false"
+
+runs:
+  using: "composite"
+  steps:
+    - name: Set up Python ${{ inputs.python-version }}
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ inputs.python-version }}
+
+    - name: Show toolchain
+      shell: bash
+      run: |
+        python --version
+        cmake --version
+        if [[ "${{ inputs.skip-native-build }}" != "true" ]]; then
+          nvcc --version || {
+            echo "::error::nvcc not found — pass skip-native-build: 'true' for CPU-only runners"
+            exit 1
+          }
+        fi
+
+    - name: Install Python dependencies
+      shell: bash
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r requirements.txt
+
+    - name: Install the modified Transformers fork
+      shell: bash
+      run: pip install -e integration/transformers/ --no-deps
+
+    - name: Install vLLM fork (optional)
+      if: ${{ inputs.install-vllm == 'true' }}
+      shell: bash
+      run: pip install -e integration/vllm/
+
+    - name: Build and install DMI (Stage 3 entrypoint)
+      shell: bash
+      # pip install -e . --no-build-isolation calls NativeBuildExt (setup.py):
+      #   1. git submodule update --init libs/clickhouse-cpp  (if needed)
+      #   2. cmake configure + build (libclickhouse-cpp-lib.a)
+      #   3. make -C monitoring  (compile monitoring_native_backend.so via nvcc)
+      # SKIP_NATIVE_BUILD=1 short-circuits all three steps — no nvcc required.
+      env:
+        SKIP_NATIVE_BUILD: ${{ inputs.skip-native-build == 'true' && '1' || '' }}
+      run: pip install -e . --no-build-isolation
+
+    - name: Smoke-check monitoring imports
+      shell: bash
+      run: |
+        python -c "import monitoring; print('monitoring:', monitoring.__file__)"
+        if [[ "${{ inputs.skip-native-build }}" != "true" ]]; then
+          python -c "from monitoring._native_engine import RingConfig; print(RingConfig())"
+        fi

.github/workflows/tests.yml

@@ -0,0 +1,169 @@
+name: tests
+
+# Stage-aware CI test suite aligned with #55 three-stage plan:
+#
+#   Stage 1 — CPU PR gate:
+#     runs-on: ubuntu-latest (GitHub-hosted)
+#     Trigger: every push / pull_request
+#     No nvcc, no self-hosted risk — SKIP_NATIVE_BUILD=1 bypasses the .so build.
+#
+#   Stage 2 — GPU / native regression:
+#     runs-on: [self-hosted, linux, gpu|multi-gpu]
+#     Trigger: nightly schedule, workflow_dispatch, or 'run-gpu' label on
+#              TRUSTED (non-fork) internal PRs only.
+#     Security guard prevents fork PRs from running on Frootlab self-hosted runners.
+#
+#   Stage 3 — Packaging:
+#     Native backend compiled via `pip install -e . --no-build-isolation`
+#     (setup.py NativeBuildExt), not by manually calling cmake/make in CI.
+#     The setup-dmi composite action wraps this entrypoint.
+
+on:
+  push:
+    branches: [main, "**"]
+  pull_request:
+    types: [opened, synchronize, reopened, labeled]
+  schedule:
+    - cron: "0 2 * * *"   # 02:00 UTC nightly
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  # --------------------------------------------------------------------------
+  # Stage 1: CPU PR gate — GitHub-hosted, no GPU, no native build required
+  # --------------------------------------------------------------------------
+  cpu:
+    # Run on every push and PR (opened/sync/reopened/labeled).
+    # Skip on the nightly schedule — the nightly job is the authoritative sweep.
+    if: github.event_name == 'push' || github.event_name == 'pull_request'
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    concurrency:
+      group: cpu-${{ github.ref }}
+      cancel-in-progress: true
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: false   # vLLM fork is 3.7 GB — too heavy for CPU gate
+      - name: Init transformers submodule
+        run: git submodule update --init --depth 1 integration/transformers
+      - uses: ./.github/actions/setup-dmi
+        with:
+          skip-native-build: "true"
+      - name: CPU default suite
+        run: python -m pytest -m "not gpu and not e2e and not manual and not native_backend" -q
+
+  # --------------------------------------------------------------------------
+  # Stage 2: Single-GPU smoke — self-hosted, trusted triggers only
+  #
+  # Security: self-hosted jobs must NEVER run for fork PRs (untrusted code
+  # would execute on the Frootlab runner). Allowed triggers:
+  #   • schedule (nightly)
+  #   • workflow_dispatch (manual)
+  #   • pull_request labeled 'run-gpu' AND head is this repo (non-fork)
+  # --------------------------------------------------------------------------
+  gpu-smoke:
+    if: |
+      github.event_name == 'schedule' ||
+      github.event_name == 'workflow_dispatch' ||
+      (
+        github.event_name == 'pull_request' &&
+        github.event.label.name == 'run-gpu' &&
+        github.event.pull_request.head.repo.full_name == github.repository
+      )
+    runs-on: [self-hosted, linux, gpu]
+    timeout-minutes: 60
+    concurrency:
+      group: gpu-smoke-${{ github.ref }}
+      cancel-in-progress: false   # don't kill a running GPU test mid-flight
+    env:
+      CUDA_VISIBLE_DEVICES: "0"
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: false
+      - name: Init submodules (transformers + clickhouse-cpp)
+        run: |
+          git submodule update --init --recursive integration/transformers
+          git submodule update --init --recursive libs/clickhouse-cpp
+      - uses: ./.github/actions/setup-dmi
+      - name: Single-GPU smoke suite
+        # Exit code 5 ("no tests collected") is tolerated — a hardware tier
+        # with nothing marked yet is not a failure.
+        run: |
+          set +e
+          python -m pytest -m "gpu and not multi_gpu and not slow" -q
+          rc=$?
+          if [ "$rc" -eq 5 ]; then echo "::notice::no single-GPU tests collected"; exit 0; fi
+          exit $rc
+
+  # --------------------------------------------------------------------------
+  # Stage 2: Multi-GPU / TP — self-hosted, same trusted-trigger guard
+  # --------------------------------------------------------------------------
+  multi-gpu:
+    if: |
+      github.event_name == 'schedule' ||
+      github.event_name == 'workflow_dispatch' ||
+      (
+        github.event_name == 'pull_request' &&
+        github.event.label.name == 'run-gpu' &&
+        github.event.pull_request.head.repo.full_name == github.repository
+      )
+    runs-on: [self-hosted, linux, multi-gpu]
+    timeout-minutes: 90
+    concurrency:
+      group: multi-gpu-${{ github.ref }}
+      cancel-in-progress: false
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: false
+      - name: Init submodules (transformers + clickhouse-cpp)
+        run: |
+          git submodule update --init --recursive integration/transformers
+          git submodule update --init --recursive libs/clickhouse-cpp
+      - uses: ./.github/actions/setup-dmi
+      - name: Multi-GPU / TP suite
+        # multi_gpu currently selects zero tests (TP tests land in a later
+        # phase). Exit code 5 is tolerated.
+        run: |
+          set +e
+          python -m pytest -m "multi_gpu" -q
+          rc=$?
+          if [ "$rc" -eq 5 ]; then echo "::notice::no multi_gpu tests collected"; exit 0; fi
+          exit $rc
+
+  # --------------------------------------------------------------------------
+  # Nightly: full sweep — schedule / workflow_dispatch only
+  # --------------------------------------------------------------------------
+  nightly:
+    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+    runs-on: [self-hosted, linux, gpu]
+    timeout-minutes: 180
+    concurrency:
+      group: nightly-${{ github.ref }}
+      cancel-in-progress: false
+    env:
+      CUDA_VISIBLE_DEVICES: "0"
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: false
+      - name: Init submodules (transformers + clickhouse-cpp + vllm)
+        run: |
+          git submodule update --init --recursive integration/transformers
+          git submodule update --init --recursive libs/clickhouse-cpp
+          git submodule update --init --recursive integration/vllm
+      - uses: ./.github/actions/setup-dmi
+        with:
+          install-vllm: "true"
+      - name: Full / nightly sweep
+        # Exit code 5 ("no tests collected") is tolerated for an empty selection.
+        run: |
+          set +e
+          python -m pytest -m "slow or nightly" -q
+          rc=$?
+          if [ "$rc" -eq 5 ]; then echo "::notice::no slow/nightly tests collected"; exit 0; fi
+          exit $rc

conftest.py

@@ -0,0 +1,16 @@
+"""
+Root conftest: compatibility shims loaded before any test module is imported.
+"""
+import huggingface_hub
+
+# huggingface_hub >= 1.0 removed is_offline_mode() as a top-level export;
+# the vendored integration/transformers fork (4.57.0.dev0) still imports it
+# from the package root in ~11 files.  Restore it here so the fork loads
+# cleanly on modern huggingface_hub without requiring a submodule bump.
+if not hasattr(huggingface_hub, "is_offline_mode"):
+    from huggingface_hub import constants as _hf_constants
+
+    def _is_offline_mode() -> bool:
+        return bool(_hf_constants.HF_HUB_OFFLINE)
+
+    huggingface_hub.is_offline_mode = _is_offline_mode

docs/testing.md

@@ -0,0 +1,121 @@
+# Testing
+
+The test suite is split into explicit categories by **pytest markers** so each
+test declares the resources it needs. The default suite is CPU-only; everything
+that needs a GPU, ClickHouse, vLLM, model weights, or the native CUDA build is
+marked and opt-in.
+
+## The four canonical commands
+
+| Suite | When | Command |
+|---|---|---|
+| **CPU default** | every PR / push | `python -m pytest -m "not gpu and not e2e and not manual" -q` |
+| **Single-GPU smoke** | per PR (GPU runner) | `python -m pytest -m "gpu and not multi_gpu and not slow" -q` |
+| **Multi-GPU / TP** | per PR (multi-GPU runner) | `python -m pytest -m "multi_gpu" -q` |
+| **Full / nightly** | nightly schedule | `python -m pytest -m "slow or nightly" -q` |
+
+The CPU default suite is the acceptance gate for every PR: it must pass with no
+CUDA device, no ClickHouse, no vLLM runtime, and no downloaded model weights.
+
+> The native backend `.so` still has to be **built** for the CPU suite, because
+> importing `monitoring` loads it at import time (JIT is disabled for
+> reproducibility). Building needs `nvcc` but not a GPU at runtime. See
+> [install.md](install.md) §5.
+
+## Marker taxonomy
+
+Markers are registered in [`pyproject.toml`](../pyproject.toml). CPU is the
+**default unmarked** suite — a test with no resource marker is assumed CPU-safe;
+GPU/E2E/etc. must be marked explicitly.
+
+| Marker | Meaning |
+|---|---|
+| `cpu` | Pure-CPU contract/unit test; the default suite. No CUDA / ClickHouse / vLLM / weights / native build needed at runtime. |
+| `gpu` | Requires a CUDA device. |
+| `multi_gpu` | Requires ≥ 2 CUDA devices (TP / EP / routing). |
+| `e2e` | End-to-end pipeline through the native backend + host engine. |
+| `clickhouse` | Requires a reachable ClickHouse instance. |
+| `vllm` | Requires the vLLM runtime importable. |
+| `hf` | Requires HuggingFace weights / model cache. |
+| `ring_native` | Native CUDA ring tests built via `tests/ring/Makefile` (needs `nvcc`). |
+| `slow` | > ~30 s (full per-hook sweep, large E2E sweeps). Skipped unless selected. |
+| `nightly` | Scheduled full-sweep tests; run via `-m "slow or nightly"`. |
+| `numeric` | Per-hook numeric-difference study (drift vs the unhooked baseline). |
+| `manual` | Investigation / tooling, **not** a regression gate; not collected by default. |
+
+A test may carry several markers (e.g. `gpu`, `vllm`, `clickhouse`, `e2e`).
+Selection composes them with boolean expressions:
+
+```bash
+python -m pytest -m "gpu and not multi_gpu and not slow" -q
+python -m pytest -m "vllm and clickhouse" -q
+```
+
+`manual` tools and the `tests/tools` / `tests/ring` directories are excluded
+from default collection (`addopts = -ra -m 'not manual'` plus `norecursedirs`).
+
+## Skip-guards
+
+GPU / E2E tests fail **closed with a reason** instead of erroring on a missing
+prerequisite, via the helpers in [`tests/_requirements.py`](../tests/_requirements.py):
+
+| Helper | Skips when |
+|---|---|
+| `require_cuda()` | no CUDA device visible |
+| `require_gpus(n)` | fewer than `n` CUDA devices |
+| `require_clickhouse(host, port)` | the ClickHouse TCP port is unreachable |
+| `require_vllm()` | the vLLM runtime is not importable |
+| `require_model_cache(model)` | the model is not in the local HF cache / path |
+| `require_nvcc()` | `nvcc` is not on `PATH` |
+
+Use them as decorators or in a module-level `pytestmark` list:
+
+```python
+import pytest
+from tests._requirements import require_cuda, require_clickhouse
+
+pytestmark = [pytest.mark.gpu, require_cuda()]
+
+@require_clickhouse()
+def test_rows_land_in_clickhouse():
+    ...
+```
+
+Relevant env vars (defaults match the runners): `DMX_DB_HOST` / `DMX_DB_PORT`
+for the ClickHouse probe, `HF_HOME` / `HF_HUB_CACHE` for the weight-cache check.
+
+## Continuous integration
+
+[`.github/workflows/tests.yml`](../.github/workflows/tests.yml) wires the four
+commands into four jobs using the three-stage plan from #55:
+
+| Job | Stage | Trigger | Runner | Command |
+|---|---|---|---|---|
+| `cpu` | 1 — CPU gate | push / every PR | `ubuntu-latest` (GitHub-hosted) | CPU default |
+| `gpu-smoke` | 2 — GPU regression | nightly / `run-gpu` label / manual | `[self-hosted, linux, gpu]` | single-GPU smoke |
+| `multi-gpu` | 2 — GPU regression | nightly / `run-gpu` label / manual | `[self-hosted, linux, multi-gpu]` | multi-GPU / TP |
+| `nightly` | 2 — GPU regression | `schedule` 02:00 UTC / manual | `[self-hosted, linux, gpu]` | `slow or nightly` |
+
+**Stage 1 — CPU gate** runs on GitHub-hosted `ubuntu-latest` on every push and
+PR. It uses `SKIP_NATIVE_BUILD=1` so no `nvcc` is needed; the native `.so` is
+absent, and tests that need it must carry an `e2e` or `gpu` marker (which the
+CPU selector `-m "not gpu and not e2e and not manual"` already excludes).
+
+**Stage 2 — GPU / native regression** jobs run on Frootlab self-hosted runners
+with CUDA. They are restricted to trusted triggers to prevent fork PRs from
+executing untrusted code on the runner:
+- **`schedule`** — nightly at 02:00 UTC
+- **`workflow_dispatch`** — manual trigger via the GitHub Actions UI
+- **`pull_request` labeled `run-gpu`** — maintainer applies the label to trusted
+  internal PRs; a fork-PR check (`head.repo.full_name == github.repository`)
+  ensures the label cannot be abused by external contributors
+
+**Stage 3 — Packaging**: the [`setup-dmi`](../.github/actions/setup-dmi/action.yml)
+composite action installs DMI via `pip install -e . --no-build-isolation`
+(the Stage 3 entrypoint from `setup.py`), which internally runs cmake for
+`libs/clickhouse-cpp` and `make -C monitoring`. CI no longer calls cmake or
+make directly; the build is owned by `setup.py` `NativeBuildExt`.
+
+Because the GPU suites use the skip-guards in `tests/_requirements.py`, a runner
+missing ClickHouse or model weights **skips** the affected tests with a reason
+rather than failing the job.