[DO NOT REVIEW] Drop null problem-cache sentinels on load, keep in-run dedup (gpuep-rel-2610) - #5207
Closed
danieyan-amd wants to merge 1 commit into
Closed
Conversation
A null mark() sentinel is a transient in-run 'benchmark in progress' signal that compile_ops relies on to benchmark a repeated problem only once. It must not be persisted: a shipped or stale null that is re-loaded makes the op skip forever, leaving it unresolved (0xC0000005 / 'No valid tuned compilation', AIRADSW-871). Drop null sentinels at the load boundary in the json and sqlite backends so a persisted null cannot come back, while the in-run dedup (get()/compile_ops) is unchanged. Also guards is_module_fusible against a null or non-string solution. Updates the backend round-trip tests (a sentinel no longer survives save/load). Co-authored-by: Uros Petkov <[email protected]> Signed-off-by: danieyan-amd <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A null
mark()entry in the problem cache is a transient, in-run "benchmark in progress" sentinel. It was stored in the same map that is persisted to disk, so a null that never got overwritten (a failed or interrupted benchmark) could be persisted, shipped, and re-loaded. On the next runget()returned that stale null and the op was skipped forever, leaving it unresolved →0xC0000005/ "No valid tuned compilation" (AIRADSW-871, BERT pooler on Navi48/gfx1201). Deletingproblem_cache.jsoncleared the nulls and worked around it.Fix
Drop null sentinels at the load boundary in the json and sqlite backends, so a persisted or shipped null can never come back. The in-run dedup is unchanged:
get()still returns the in-memorymark(), andcompile_opsskips on a null so a repeated problem is benchmarked once, not once per instruction. Also guardsis_module_fusibleagainst a null or non-string solution.This is the
gpuep-rel-2610counterpart of develop PR #5204. It supersedes #5201, which treated a null as a cache miss — that fixed the crash but removed the in-run dedup and re-benchmarked a repeated problem once per instruction (raised in review).Tests
Backend round-trip tests updated: a null sentinel no longer survives save/load (real solutions still do).
Note: not built on this box (the 2610 build needs MIOpen, not installed here). The identical change is build- and test-verified on develop #5204 — reduced MLIR-off gfx1100 build, all four problem-cache test binaries pass — and CI validates here.