Skip to content

cutile: one CodegenMode for device debug, plus CUTILE_JIT_* run-time overrides - #1600

Open
almightychang wants to merge 2 commits into
NVIDIA:mainfrom
almightychang:feat/tileiras-jit-env-overrides
Open

almightychang wants to merge 2 commits into
NVIDIA:mainfrom
almightychang:feat/tileiras-jit-env-overrides

Conversation

@almightychang

@almightychang almightychang commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Continues NVlabs/cutile-rs#237, tracked in #1428.

Problem

tileiras accepts --device-debug only at --opt-level 0. That rule spans two fields, but CompileOptions stores them independently, and each source writes them field by field:

CompileOptions { opt_level: Option<u8>, device_debug: bool, lineinfo: bool }

 build default ─┐
 code setters  ─┼─ field by field ─▶ TileirasOptions ─────▶ tileiras
 (run-time env)─┘                    can still hold          rejects it
                                     -O3 + device-debug      at JIT time

A field-by-field merge cannot keep a rule that spans two fields. Each writer would need its own copy of the "level 0 if debug" rule. Today the only copy, the default in from_compile_options, applies only when no level is set. The resolved options can still hold the rejected pair, so the only real check is tileiras itself, at JIT time, far from the setter that caused it.

NVlabs/cutile-rs#285 made the build profile another writer. Dev-profile builds now start in full debug, so CompileOptions::new().opt_level(3) alone fails at JIT time, while the same code passes with --release. The run-time overrides that NVlabs/cutile-rs#237 proposed would add a third writer and a third copy of the rule.

Change

Make the rejected pair unrepresentable, and have each writer replace the whole value:

enum CodegenMode { Debug, Optimized { level: u8, lineinfo: bool } }

 build default ─▶ code setters ─▶ CUTILE_JIT_* ─▶ CodegenMode ─┬─▶ frontend (no check hoisting in Debug)
       (each replaces the mode; the last one wins)              └─▶ tileiras flags

The rule is removed instead of copied. The run-time overrides then reduce to the same setters, applied after the code's:

Variable Values Acts like
CUTILE_JIT_DEBUG none, line, full debug_info(...)
CUTILE_JIT_OPT_LEVEL 0–3 opt_level(n)
CUTILE_JIT_SANITIZE memcheck, none sanitize_memcheck(...)

CUTILE_JIT_DEBUG=full together with CUTILE_JIT_OPT_LEVEL, or an invalid value, fails with a JIT error that names the variable. The overrides are applied once before both compile stages, so the frontend and tileiras see the same mode. NVlabs/cutile-rs#237 applied them only to the tileiras flags.

Behavior and API changes

Before After
dev build + opt_level(3) -O3 --device-debug, rejected -O3
.debug_info(Full).opt_level(3) -O3 --device-debug, rejected -O3 (last call wins)
.opt_level(3).debug_info(Full) -O3 --device-debug, rejected -O0 --device-debug
device_debug(bool), lineinfo(bool) setters removed; debug_info() covers them
opt_level, device_debug, lineinfo fields public replaced by codegen

Cache key and entry-header bytes are unchanged for every mode; a test pins them against the previous encoding. DebugInfoLevel and debug_info() from NVlabs/cutile-rs#285 stay. I left CHANGELOG.md for the maintainers.

Validation

Local container from nvidia/cuda:13.3.1-devel-ubuntu24.04 with an RTX 5080 (driver 580, CUDA 13.0):

  • cargo fmt -- --check; cargo clippy --workspace --exclude cuda-tile-rs --all-targets (only the existing rmsnorm.rs unused-unsafe warning)
  • scripts/run_cpu_tests.sh and scripts/run_gpu_tests.sh: all passed
  • scripts/test_debug_defaults.sh: passed
  • Each commit builds and passes the touched suites on its own
  • A logging shim in place of tileiras (CUTILE_TILEIRAS_PATH, empty disk cache) recorded the flags it received:
no env                                 --opt-level 0 --device-debug    (debug-profile default)
CUTILE_JIT_DEBUG=none                  --opt-level 3
CUTILE_JIT_DEBUG=full, SANITIZE        --opt-level 0 --device-debug --sanitize=memcheck
CUTILE_JIT_OPT_LEVEL=1, DEBUG=line     --opt-level 1 --lineinfo
CUTILE_JIT_DEBUG=full, OPT_LEVEL=2     JIT error before tileiras runs

…nMode

`tileiras` accepts `--device-debug` only at optimization level 0, but
`CompileOptions` kept the level and the debug flag as independent fields.
Debug builds default to full debug, so adding only `.opt_level(3)` produced
`--opt-level 3 --device-debug`, which `tileiras` rejects at JIT time.

Replace `opt_level`, `device_debug`, and `lineinfo` with
`codegen: CodegenMode`, where `Debug` owns level 0 and
`Optimized { level, lineinfo }` covers the rest. Every value now maps to
flags `tileiras` accepts. `debug_info()` and `opt_level()` select a mode,
and the last call wins. The level-0 default rule in
`TileirasOptions::from_compile_options` is gone, and the frontend reads the
same mode to decide check hoisting.

Remove the `device_debug(bool)` and `lineinfo(bool)` setters, which
`debug_info()` covers. The cache key and entry header bytes are unchanged
for every mode; a test pins them against the previous encoding.

Signed-off-by: almightychang (Sam, Joochul Chang) <[email protected]>
Nothing could switch an already-built application to a debug or sanitizer
build without changing its code. Add three process-wide variables, read
once per process:

- CUTILE_JIT_DEBUG=none|line|full acts like `debug_info()`
- CUTILE_JIT_OPT_LEVEL=0|1|2|3 acts like `opt_level()`
- CUTILE_JIT_SANITIZE=memcheck|none acts like `sanitize_memcheck()`

Each applies after the code's options, as if its setter were called last.
`CompileOptions::with_jit_env_overrides()` applies them once before both
compile stages, so the frontend's check placement and the `tileiras` flags
always come from the same options. Invalid values, and full debug together
with an explicit level, return a JIT error naming the variable.

This continues NVlabs/cutile-rs#237, which applied the overrides only to
the `tileiras` flags and left the frontend hoisting checks under
`CUTILE_JIT_OPTIMIZATION=debug`.

Signed-off-by: almightychang (Sam, Joochul Chang) <[email protected]>
@copy-pr-bot

copy-pr-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@elibol

elibol commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Reviewing this against #1428. The setter/parser unit tests, subprocess full-debug test, previous-cache-encoding regression, and recorded real tileiras flag matrix provide useful coverage. The important contract is that the build default, explicit options, frontend check placement, assembler flags, and caches all agree on the same effective mode.

Two integration gaps to close:

  • Make jit_env_overrides hermetic: use a dedicated child-process marker and clear all three CUTILE_JIT_* variables before setting the case. Currently an inherited DEBUG value can bypass the parent path, and inherited OPT_LEVEL/SANITIZE can change the result. Test explicit off, line+optimization, invalid/conflicting values, and the read-once behavior in isolated processes.
  • The current cache-key assertion compares two compiler configurations under the same DEBUG=full override. Both would receive the same override even if the resolved mode were omitted from the cache key. Compare independently launched processes with unset/none/line/full settings and a shared temporary disk cache: different effective modes must separate, equivalent programmatic/environment modes must reuse. Exercise the normal launcher as well as KernelCompiler, with an actual kernel result and frontend check-placement counts; retain the real-assembler flag checks already reported.

For the debug-mode claim, retain a source-breakpoint/inline-backtrace smoke test on the produced full-debug cubin. This PR controls compilation mode; it should not be treated as the fix for #1379's trap attribution or #1476's inline-call breakpoint resolver.

Once accepted, #1428 is the issue this PR should close; the other source-location audit items remain independent.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants