feat: model full debug and add TileIR JIT overrides #1428

Tracks pull request NVlabs/cutile-rs#237 (NVlabs/cutile-rs#237) by @almightychang, opened 2026-08-23: feat: model full debug and add TileIR JIT overrides. As of 2026-10-07 the branch conflicts with cutile-rs main in: cutile-book/guide/debugging-and-profiling.md, cutile-compiler/src/check_optimizations.rs, cutile-compiler/src/cuda_tile_runtime_utils.rs, cutile-compiler/src/hints.rs, cutile/src/tile_kernel.rs.

@almightychang: that repository has moved here and is being archived, so the pull request is closed there, not rejected. If you want to continue the work, please recreate it against NVIDIA/cuda-rust. AGENTS.md in NVIDIA/cuda-rust has a "Transplanting a cutile-rs pull request" section that lets an agent do the replay (it fetches pull/237/head from NVlabs/cutile-rs, keeps your authorship and sign-off, and maps the moved paths).

Original description:

Summary

This PR has been rewritten on top of main after #236 landed with most of the original implementation. The remaining scope is:

• model TileIR optimization as Optimization::Level(u8) | Optimization::FullDebug
• add process-wide environment overrides for optimization, memcheck, and line information

Why the optimization model comes first

On current main, device_debug=true plus an explicit optimized level produces an argv combination that tileiras rejects (optimized debugging is currently not supported; only O0 is accepted with --device-debug). Without environment overrides, both setters come from one call site, so that behavior was a reasonable API choice for the new compile options.

The override changes who produces the combination: an application can set level 3 while a wrapper independently enables debug through the environment. Neither party made an invalid local choice, but the failure would surface later during JIT in a process the wrapper does not own. Making full debug a distinct optimization state prevents that cross-party invalid combination; FullDebug owns both --opt-level 0 and --device-debug.

Only the opt_level / device_debug pair is folded together. lineinfo and sanitize_memcheck remain independent because the measured combinations are accepted by tileiras (-O3 --lineinfo, -O0 -g --sanitize=memcheck, and -O3 --sanitize=memcheck).

This modeling change does not change cache identity. FullDebug still encodes opt level 0 and device-debug in flags bit 0; lineinfo remains bit 1 and memcheck bit 2. The L2 key regression test compares the new representation with the previous encoding across every valid combination. The cache domain, entry format version, and entry header layout are unchanged.

Environment overrides

• CUTILE_JIT_OPTIMIZATION=0|1|2|3|debug
• CUTILE_JIT_SANITIZE=memcheck|none
• CUTILE_JIT_LINEINFO=1|0 (also true/false, yes/no, and on/off)

Each variable has three states: unset preserves the code setting, an explicit value overrides it (including explicit off), and an invalid or non-Unicode value returns a clear JIT error. The values are read once per process with OnceLock in the existing TileirasOptions::from_compile_options resolution point, before the L2 cache key is constructed.

CUTILE_JIT_LINEINFO is included for completeness because #236 added lineinfo to the programmatic surface after the original PR was written. I am happy to remove that override if the additional knob is not wanted.

Validation

Validated in the cutile-ci:13.3 container used to mirror .github/workflows/pr.yml:

• cargo build
• cargo fmt -- --check
• cargo clippy --all-features
• cargo test --no-run
• bash scripts/run_cpu_tests.sh
• TileIR smoke test with --nocapture
• loom model check and miri slot-table checks
• bash scripts/run_all.sh (CPU section passed; GPU-dependent sections failed because libcuda.so was unavailable)

compiler2_e2e was confirmed to execute rather than silently skip: it printed its target and produced non-empty cubins with the real CUDA 13.3 tileiras. In particular, application code specifying opt_level(3) plus CUTILE_JIT_OPTIMIZATION=debug resolved to --opt-level 0 --device-debug and compiled successfully. The real backend also accepted the directly tested default, O0, debug, debug+memcheck, O3+memcheck, and lineinfo argv combinations.

GPU launch paths were not validated on this machine because the container had no CUDA driver (libcuda.so, driver error 303); those paths rely on CI with a compatible GPU and driver. The local wrapper also lacks the workflow's seccomp=unconfined option, so the ThreadSanitizer command stopped at setarch: Operation not permitted; the dedicated CI job remains the verification for that check.