[tests] Set up CI/CD pipeline (ROADMAP)

[Samfisheryu]

Background

ROADMAP short-term item #4: "Set up CI/CD pipeline." The repo currently has no
GitHub Actions workflows (.github/ only holds a disabled dependabot.yml);
tests are run by hand via tests/run_regression.sh, and the native backend is
built by hand (make) with prebuilt .so files committed to git. This issue
covers three stages: a CPU PR gate, a GPU regression run, and packaging so the
native backend builds via pip install.

The pipeline builds its own environment from declarative sources — no
preexisting machine setup is assumed:

• Python deps from requirements.txt
• the transformers and vLLM forks via their git submodules
• the native backend compiled from source

Stage 0 (prerequisite): green the test baseline. Tracked in #54. The CPU
gate is only meaningful once the existing tests pass, so #54 should land
before/with Stage 1.

Why split CPU / GPU

The native backend requires CUDA to compile (-lcudart -lcuda -ltorch_cuda
plus nvcc kernels), so it cannot be built on free CPU runners. We run a fast
CPU gate on every PR, and the heavier GPU regression on a self-hosted runner.

Stage	Runner	Trigger	Scope
Stage 1 — CPU PR gate	GitHub-hosted CPU (free)	every PR / push	CPU-only pytest subset + convention guards
Stage 2 — GPU regression	self-hosted GPU	nightly + gpu label	build native + full run_regression.sh
Stage 3 — Packaging	(build step, used by Stage 2 + users)	—	pip install builds the native backend

Stage 1 — CPU-side CI

• Add .github/workflows/ci.yml; setup-python (3.10), then
  pip install -r requirements.txt (no vLLM, no heavy submodules).
• Run the CPU-only pytest subset with -m "not slow" -m "not gpu".
• Smoke check: deps install cleanly and pure-Python modules import.
• Make this gate required via branch protection.

Stage 2 — GPU-side CI

• Add a gpu pytest marker (registered in pyproject) so Stage 1 selects
  -m "not gpu" and Stage 2 selects -m gpu — no hardcoded file lists.
• Register a self-hosted runner on a CUDA-capable machine.
• Workflow: init submodules + install forks → build the native backend via
  the Stage 3 entrypoint (pip install . --no-build-isolation) →
  bash tests/run_regression.sh.
• Triggers: nightly schedule + manual gpu label on PRs.
• Security: never auto-run self-hosted jobs on fork PRs (secret-leak /
  RCE risk); restrict to internal PRs or require manual approval.

Stage 3 — Packaging (native build via pip install)

Today the build lives only in monitoring/Makefile and is not wired into pip;
pip install works only because prebuilt .so files are committed to git.
Wrap the existing build so pip install compiles the native backend on the
user's own torch/CUDA (source-distribution / compile-on-install model).

• Add a setup.py (alongside pyproject.toml) with a custom build_ext
  that shells out to the existing build: submodule update --init →
  build libs/clickhouse-cpp (cmake) → make -C monitoring.
• Document the build-isolation requirement: users install torch first, then
  pip install . --no-build-isolation (torch must be importable at build
  time for the include/lib paths).
• Stop committing .so files to git; ensure the sdist ships the sources
  (csrc/*, Makefile, submodule refs).
• Update install docs to the single pip install flow.

[Samfisheryu]

#61