Skip to content

gfx/shaders: ignore BAHS candidates outside validated binary records - #2943

Closed
gino8080 wants to merge 1 commit into
seedds:mainfrom
gino8080:fix/bahs-binary-record-entries
Closed

gino8080 wants to merge 1 commit into
seedds:mainfrom
gino8080:fix/bahs-binary-record-entries

Conversation

@gino8080

@gino8080 gino8080 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Mario Kart World 1.8.0 contains BAHS payload that looks like a shader descriptor even though it is outside the binary record entry slots. The scan accepts it, pairs unrelated control and code, and sends a nonexistent shader to the translator. The existing exclusion for descriptors inside control sections does not cover this case.

Reuse the validated binary array already read for variation pairing to identify the real entry offsets. Filter the scan only when that array validates; keep the existing candidates for unrecognized or inconsistent arrays. The shared reader continues to supply variation pairs.

On the local 1.8.0 dump, this removes two false shader keys and leaves 11,601 shaders. The synthetic regression fixture returns three candidates on upstream and the expected two with this fix. A second test checks that malformed array headers and record sizes retain the scan fallback.

Validation on the current base

Rebased onto c379a903f47dd6d8cbdc4b2b36e67e7bb8ebc250 on 2026-11-18. The only conflict was the generated dm/gfx/nvn/dmsh_chain.h; regenerating it from the combined sources produced b705e45154b2. The parser and its regression tests are byte-for-byte unchanged from the previous PR head.

  • 124 split tests passed.
  • 42 chain/header, documentation, native-tool and SPIRV-Cross pin tests passed.
  • Pinned Python lint against the new base: 305 existing findings, zero new findings and zero stale entries.
  • git diff --check passed.
  • A fresh full regress.py check compared the current base with 456671e83876 on Mario Kart World 1.8.0: 11,601/11,601 Mac shaders built on both sides, identical shader manifests, zero changed MSL, zero new shader/interface failures. All 9,591 checked pipeline interfaces matched; both pipeline prebuilds compiled 9,591 bases without failure.
  • Both Mac runs still crash in the same Metal driver call before gameplay (exit 139, frame 113). The harness reports PASS because the candidate adds no regression to that failing baseline. This is not a successful gameplay test, and no performance claim is made.
  • The package was already split with the corrected parser, so this full comparison checks translation/output regressions; discovery behavior is covered by the synthetic before/after fixture and the earlier full dump split.
  • Mario Kart World is the only locally available game. The impact report marks broad shader coverage and profiles without usage manifests as unknown. Other BAHS games have not been rebuilt.
  • The full standalone Metal graphics suite is blocked by main's t_geometry.m compile failure below. The broader tool suite was not repeated in full locally.

Earlier device evidence: this same parser implementation built all 11,601 shaders for Mac and iOS on 2925d7af; Mario Kart World 1.8.0 was installed and played on an iPhone Duo with iOS 27.2.1 using Xcode 27.2. The current rebase has not been installed on the phone, and the iOS shader build was not repeated for this base.

Existing failures on main

Verified against the exact base above on 2026-11-18:

  • t_geometry.m:478,488: undeclared g_mdi_v, reproduced locally.
  • test_shader_toolchain_cache.TargetSelection.test_mesh_compiles_only_requested_targets: undefined dmsh_rewrite in all three target subcases, reproduced locally.
  • t_audio_device under TSan: race in mock_aogpd, reproduced locally.
  • Main's test run also fails the host and ASan jobs at t_codec_video.c:111 (vt_session_lifecycle); its tool-test job was cancelled.
  • Main's lint-C run still reports the geometry errors, a potential match leak in haptics_apple.m, and stale baseline entries. These files and the baseline are not changed by this PR.

Docs

The BAHS entry-validation reference and Mario Kart World note describe the cause, fallback and validation limits.

  • Finding documented; symptom index checked by the docs tests.
  • Shader chain header regenerated.
  • Fixtures are synthetic. No game files, keys or extracted game data are included.
Current-base full regression report

regress.py check (full): PASS

Baseline main-20261118 c379a90, candidate HEAD 456671e; Apple M4 Pro, macOS 27.2.1, Xcode 27.2; 971.5 s

game status shaders (base → cand) newly failing newly building changed MSL interface (new / known) boot fps p50 frame p99 ms peak MB prepare s new RED
Mario Kart World 0400C3F00006E000 PASS 11601/11601 → 11601/11601 0 0 0 0 / 0 FAIL 0.0 → 0.0 None → None 2835 → 2618 None → None 0

Mario Kart World 0400C3F00006E000: PASS

  • known (baseline too): crash ([dm] CRASH sig=# addr=# pc=-[AGXG#XFamilyRenderContext_mtlnext setRenderPipelineState:]+# lr=draw_+# sp=#)
  • known (baseline too): frame 6000 not reached by 190 s (max 113)
    • baseline: exit 139, first frame 3.3 s, max frame 113, reached False, prepare None s, prebuild 142.3 s (9591 built, 0 failed), fps 0.0, p50/p99 None/None ms, peak 2835 MB
    • candidate: exit 139, first frame 3.1 s, max frame 113, reached False, prepare None s, prebuild 179.2 s (9591 built, 0 failed), fps 0.0, p50/p99 None/None ms, peak 2618 MB
    • time: chain 637.9 s, smoke 333.2 s

@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers tools Tools and scripts (dm/tools, scripts) docs Documentation shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders game: 0400C3F00006E000 A game profile labels Nov 17, 2026
@gino8080
gino8080 force-pushed the fix/bahs-binary-record-entries branch from 18860b4 to fa03c62 Compare November 17, 2026 17:20
Use validated BAHS binary record offsets to reject descriptor-shaped payload.
Preserve the scan fallback for unknown layouts and share the record reader
with variation pairing. Add synthetic regression and fallback coverage,
regenerate the chain header, and document the game-specific evidence.
@gino8080
gino8080 force-pushed the fix/bahs-binary-record-entries branch from fa03c62 to 456671e Compare November 18, 2026 08:11
seedds added a commit that referenced this pull request Nov 18, 2026
BAHS payload in Mario Kart World and Donkey Kong Bananza can pass the
shader scan even though it occupies no binary record slot. Validate a
nonempty binary array within the table region before filtering, share
its offsets with variation pairing, and keep the scan fallback when the
layout is inconsistent.

Refresh the affected shader maps and shader chain header. Fresh splits
removed two false keys from each tested build while preserving retained
shader data and variation pairs. The full Mario Kart World 1.8.0 Mac
regression passed with 11,601 shaders and 9,591 pipeline interfaces. The
registered v0 Mario Kart World map removes the same agl_common key based
on matching offsets, stages and keys. A full v0 resplit remains open.

Co-authored-by: gino8080 <[email protected]>
@seedds

seedds commented Nov 18, 2026

Copy link
Copy Markdown
Owner

Landed on main in 5df3dbe8, with @gino8080 credited as co-author for the record filter and fixtures.

The landed parser preserves scan candidates when the record array is empty or overlaps shader code. Fresh splits removed two false keys each from Mario Kart World 1.8.0 and Donkey Kong Bananza v458752. The full Mario Kart World Mac regression passed with identical manifests for all 11,601 shaders, 9,591 matching pipeline interfaces, and both revisions completing the Grand Prix route.

A full repack of Mario Kart World v0 remains open. Its registered map was updated using the matching agl_common entries in 1.8.0.

@seedds seedds closed this Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 0400C3F00006E000 A game profile nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants