[tests] Refactor stale tests and introduce a configurable correctness matrix

[XbzOnGit]

Problem

The current test suite mixes several different jobs under tests/:

• CPU-only unit/contract tests for config, hook metadata, shape inference, adapter flow, and source patching.
• GPU/native tests for Ring² transport, CUDA producer behavior, CUDA graphs, and tensor-parallel shapes.
• HF/vLLM E2E correctness tests that need CUDA, model weights, ClickHouse, and sometimes torchrun.
• Manual shell/script entry points that are useful for debugging or release sweeps but are not clean CPU regression gates.

Some tests are also outdated relative to the current runners. For example, tests/test_e2e_correctness_vs_hf.py still contains legacy/skipped sections and old assumptions, while the newer subprocess runners (hf_reference_runner.py, hf_monitored_runner.py, hf_compare_runner.py, vllm_compare_runner.py) are closer to the desired architecture. The result is that it is unclear which tests should run on CPU CI, which ones require a single GPU, which ones require multi-GPU/TP, and which ones are manual studies.

We should refactor the test suite into explicit categories, remove or refresh stale tests, and build a configurable E2E matrix that can cover mode/model/hook combinations without duplicating shell scripts.

Goals

1. Categorize all tests by required resources and intended cadence.
2. Refresh unit tests so they cover current APIs and remove outdated/duplicative assertions.
3. Introduce configurable E2E correctness tests across execution mode, model, backend, hook selection, TP, and comparison standard.
4. Add a numeric-difference study mode: enable one hook at a time, compare logits/logprobs against the original unhooked model, and report alert-worthy drift.

Proposed test categories

CPU-only default suite

Run on every PR and developer preflight. No CUDA, ClickHouse, vLLM runtime, model download, or native extension should be required.

Suggested contents:

• tests/test_config.py
• tests/test_adapter_protocol.py
• tests/test_hook_spec_flags.py
• tests/test_moe_v1_routing_hooks.py, if imports are kept lightweight and do not require model weights
• tests/test_producer_chunked_schema.py
• tests/test_ring_transport_no_framework_strings.py
• tests/test_tp_shapes.py
• tests/test_hf_eos_strip.py
• unit portions of tests/test_per_hook_isolation.py

Expected command:

python -m pytest -m "cpu and not gpu and not e2e" -q

or, if we do not want a positive cpu marker, make this the default unmarked suite and mark everything GPU/E2E explicitly.

Single-GPU smoke suite

Run on GPU CI or before merging changes that touch hooks, adapters, ring transport, HF/vLLM integration, or shape logic.

Suggested contents:

• HF compare runner smoke: gpt2 and qwen3, eager and CUDA graph, TP=1.
• vLLM compare runner smoke: gpt2 and qwen3, eager and CUDA graph, TP=1.
• tests/test_no_graph_breaks.py if model cache and vLLM import prerequisites are available.
• CUDA ring native tests from tests/ring/Makefile, at least producer + ring metadata.

Expected command:

python -m pytest -m "gpu and not multi_gpu and not slow" -q

Multi-GPU / tensor-parallel suite

Run on scheduled CI or before merging TP/EP/routing changes.

Suggested contents:

• HF TP=2 where meaningful: qwen3 and llama. Keep GPT-2 TP=2 omitted because HF auto-TP is structurally not useful for GPT-2 Conv1D.
• vLLM TP=2 for gpt2, qwen3, llama where memory allows.
• Qwen2-MoE / EP smoke cells.
• Ring native tests that require CUDA and exercise multi-rank behavior, if any are added later.

Expected command:

python -m pytest -m "multi_gpu" -q

Full regression / scheduled suite

Run nightly or manually for release candidates.

Suggested contents:

• Full HF/vLLM matrix across models, modes, TP, hook selections, and ring sizes.
• Capacity behavior for small ring sizes, CPU-direct fallback, and partial-ring paths.
• Longer per-hook isolation sweeps.
• Benchmark smoke, if we want performance regressions in scope.

Expected command:

python -m pytest -m "slow or nightly" -q

Manual analysis tools

Move these out of regression semantics or mark them clearly as tools:

• ad hoc debug scripts
• profiler/trace capture scripts
• plotting and perturbation scripts
• release-sweep shell wrappers that should not run during pytest discovery

Suggested location: tools/, experiments/, or tests/tools/ with README documentation. They should not be discovered by pytest unless explicitly requested.

Unit-test cleanup

Audit every tests/test_*.py file and classify it as:

• keep: current API, deterministic, categorized.
• refresh: useful but uses stale env vars, stale comments, old model assumptions, or deprecated helper paths.
• remove: duplicated by newer runner-based tests or permanently skipped legacy code.
• manual: useful investigation code, but not a test gate.

Specific cleanup candidates:

• Split tests/test_e2e_correctness_vs_hf.py into a small pytest wrapper plus shared comparator code. Remove legacy/skipped bodies once the runner-based version covers the same behavior.
• Consolidate HF/vLLM compare logic around shared helpers for reading ClickHouse, loading disk reference tensors, merging segments, and reporting bitwise/numeric failures.
• Normalize env names. Today E2E_HOOKS and DMX_HOOK_SELECTION are both used, and E2E_HOOK_SELECTION appears in tracked test documentation. Pick one public test input and translate internally.
• Add pytest markers in pyproject.toml: cpu, gpu, multi_gpu, e2e, clickhouse, vllm, hf, ring_native, slow, manual, numeric.
• Ensure GPU tests skip on missing prerequisites with precise reasons: no CUDA, insufficient GPU count, missing ClickHouse, missing vLLM import, missing model cache, missing nvcc.
• Avoid tests that mutate vendored source without robust restore. Where source patching is unavoidable, isolate it in a context manager and assert byte-identical restoration.

Configurable E2E harness

Build one matrix-driven harness instead of many hardcoded shell scripts. The harness should be able to select:

• Backend: hf, vllm.
• Execution mode: eager, cuda_graph.
• Standard:
  • bitwise: exact tensor/logit/logprob equality.
  • allclose: bounded numeric tolerance with max/mean diff reporting.
  • row_count: schema and segment-count validation only.
  • transport_bitwise: compare .copy_() reference buffers vs ClickHouse ring output.
• Model: gpt2, qwen3, llama, qwen2_moe, or explicit model path.
• Hook selection: preset (vllm-full, hidden-states) or one hook (q, k, resid_pre, final_logits, etc.).
• TP size: 1, 2, later higher values when supported.
• Ring config: payload/pinned size, chunk size, CPU-direct fallback settings.
• Prompt set: short smoke prompts, math prompts, chat-like prompts, random seeded prompts.
• Dtype and scheduler knobs: bf16/fp16/fp32, max model len, max batched tokens, GPU memory utilization.

Example interface:

python -m tests.e2e_matrix \
  --backend hf,vllm \
  --model gpt2,qwen3 \
  --mode eager,cuda_graph \
  --standard transport_bitwise \
  --hooks vllm-full \
  --tp 1 \
  --prompt-set smoke

The harness should produce a machine-readable result file, for example:

{
  "backend": "vllm",
  "model": "qwen3",
  "mode": "cuda_graph",
  "standard": "transport_bitwise",
  "hook_selection": "vllm-full",
  "tp": 1,
  "passed": true,
  "checks": [
    {"name": "token_ids", "passed": true},
    {"name": "layer.0.q", "passed": true, "max_abs": 0.0}
  ]
}

Standards by mode

Eager mode should be bitwise for transparency checks:

• Original model vs monitored/hooked model logits/logprobs must be bitwise identical when the hook should be observational only.
• .copy_() reference buffers vs ring/ClickHouse output should be bitwise identical.

CUDA graph / compiled mode should have explicit standards:

• Transport correctness remains bitwise: if the same hooked/compare graph emits both .copy_() buffers and ring rows, ClickHouse data must match the reference buffer exactly.
• Model-output transparency may use allclose only when comparing different compiled class hierarchies or compiled vs eager. The threshold must be named and reported, not hidden in the test.
• A separate numeric drift report should record exact max/mean diff even when the test passes.

Numeric-difference study

Add an opt-in study mode that enables one hook at a time and compares against the original model:

1. Run baseline original model with no hooks.
2. For each hook H, run monitored/hooked model with only H enabled.
3. Compare logits/logprobs/token IDs against baseline.
4. Record:
   • bitwise pass/fail
   • max abs diff
   • mean abs diff
   • first differing token position
   • top-k vocab diffs at first differing position
   • whether generated token IDs diverged
5. Emit a summary table and JSON artifact.

This should replace hardcoded per-model/per-hook study scripts with a configurable implementation.

Suggested alert policy:

• Eager mode: any non-bitwise logits/logprobs difference is an alert.
• CUDA graph mode: alert when max diff exceeds the configured per-model/per-dtype threshold, or when greedy token IDs diverge.
• Always alert on missing hook rows, hook identity swaps, shape mismatches, or ClickHouse row-count mismatches.

Acceptance criteria

• pyproject.toml defines markers for CPU, GPU, multi-GPU, E2E, ClickHouse, vLLM, HF, native ring, slow, manual, and numeric tests.
• Every pytest test is categorized and either runs in the default CPU suite or is explicitly marked/skipped.
• Stale skipped/legacy E2E code is removed or moved into a documented manual/debug location.
• A developer can run CPU-only tests without CUDA, ClickHouse, vLLM, or model weights.
• A developer with one GPU and ClickHouse can run a documented smoke suite for HF and vLLM.
• A documented multi-GPU suite covers TP=2 where meaningful.
• The E2E matrix can run at least:
  • backend: HF and vLLM
  • model: gpt2 and qwen3
  • mode: eager and CUDA graph
  • hook selection: full and one individual hook
  • standard: transport bitwise and output numeric comparison
• Eager output transparency is bitwise-gated.
• CUDA graph numeric differences are reported with thresholds and artifacts.
• Per-hook numeric study outputs a JSON summary plus a human-readable table.

Non-goals

• Do not solve every known numeric drift immediately. The first step is to make drift visible, categorized, and reproducible.
• Do not make all GPU tests default. CPU-only PR checks should stay fast and dependency-light.

[SieDeta]

@XbzOnGit @Samfisheryu, sorry for opening a separate PR for this issue. Could you please take a look and let me know your thoughts? I'd really appreciate any comments or feedback.

• marker taxonomy and requirement guards to categorize the entire test suite (CPU/GPU/E2E/manual), enabling fast and dependency-light PR validation (PR (#1) test: marker taxonomy + skip guards + categorize #56).
• environment variable normalization (E2E_HOOK_SELECTION) and hardened source patching (isolated_hook) to simplify E2E configuration and ensure vendored sources are safely restored after tests (PR (#2) test: normalize env vars + harden source-patch #57).
• shared E2E libraries and a matrix-based execution harness (e2e_matrix.py) to eliminate duplicated comparison/reporting logic and provide a scalable E2E testing framework (PR (#3) test: shared E2E lib + configurable matrix #58).
• migration of legacy E2E tests to matrix wrappers to simplify the testing infrastructure, remove obsolete code paths, and preserve compatibility with existing workflows (PR (#4) test: migrate to matrix + remove legacy E2E #59).
• numeric drift study framework to quantify and reproduce the impact of individual hooks on model outputs using metrics such as max/mean differences, token divergence, and top-k vocabulary changes (PR (#5) test: per-hook numeric-difference study #60).
• structured CI pipelines and testing documentation to separate CPU, GPU, multi-GPU, and nightly suites, enabling automated validation aligned with resource requirements (PR (#6) ci: wire CPU/GPU/multi-GPU/nightly + docs #61).

Thanks!