[RTX] 3DGS ParticleField prims intermittently not rendered at all: missing device-scope memory ordering in the per-prim AABB reduction of "SceneDb Gaussian Instance List Buffer Update" (Vulkan and D3D12) #910

Description

Summary

We stream many 3DGS ParticleField prims in and out at runtime: payload load + activate, often 8–20 prims in the same frame. Occasionally RTX does not render one of these prims at all. The prim is loaded: the log shows Gaussian prim loaded, geometry streaming reports finished loading, and the prim has an instance id. Yet it contributes 0 pixels to every camera, from every view. The missing prim then shows the dome light through a hole in the scene.

Once a prim is missing:

• Reloading it, toggling its visibility, changing its partition or moving its transform does not bring it back.
• Changing its point data does: moving one point by 1 mm makes it render on the next frame.

It happens on both Vulkan and D3D12. The rate depends on the process: some Kit processes almost never hit it, while others hit it on about half of the reload rounds. It also depends on the driver: it is much more frequent on 591.74 than on 595.97, but 595.97 still has it.

Root cause (analysis of the shipped shader)

The compute shader named SceneDb Gaussian Instance List Buffer Update Signature (DXIL entry point updateGaussianInstances) computes each prim's AABB with a "last workgroup finishes" reduction:

1. In each subgroup, the elected lane does 6 atomic SMin/SMax operations into the AABB storage buffer (binding 3). The values are stored as order-preserving ints.
2. Workgroup barrier.
3. Thread 0 decrements a completion counter (binding 2) with atomicAdd(-1). The workgroup that sees the old value 1 is the "last" one.
4. Workgroup barrier. Threads 0..5 of the last workgroup read the 6 AABB words with plain loads, convert ordered-int back to float, and store the results.

The barriers only order workgroup (shared) memory, and all the atomics are relaxed. So nothing makes the other workgroups' AABB atomics visible before the last workgroup does its conversion. A late atomic can land after the conversion and leave a mixed result.

One real bad AABB: min_x = -0.10925779, max_x = -34.5772438, while the expected min_x is -40.41397858. The value -0.10925779 is exactly the ordered-int bit pattern of -40.0600, which is one workgroup's partial minimum that was never converted. The prim's TLAS is then built from an AABB with min > max, and rays never hit it.

SPIR-V (sha256 7680371ed5fe450590ffb0b6480ae9efcff412ed34aac4453fecf8f294dc43df, 26408 bytes, OpMemoryModel Logical GLSL450):

; elected lane of each subgroup, 6 times (x/y/z min and max):
%1267 = OpAtomicSMin %int %aabb_min_x %uint_1 /*Device*/ %uint_0 /*Relaxed*/ %partial
        ...
        OpControlBarrier %uint_2 /*Workgroup*/ %uint_2 /*Workgroup*/ %uint_264 /*AcquireRelease|WorkgroupMemory*/
; if gl_LocalInvocationIndex == 0:
%1299 = OpAtomicIAdd %uint %counter %uint_1 /*Device*/ %uint_0 /*Relaxed*/ %uint_4294967295
%1302 = OpIEqual %bool %old %int_1          ; this is the last workgroup
        OpControlBarrier %uint_2 %uint_2 %uint_264
; if last && gl_LocalInvocationIndex < 6:
%1326 = OpLoad %int %aabb_word               ; plain load
        ; (v < 0 ? v ^ 0x7fffffff : v) -> float bits, OpStore back to %aabb_word

In GLSL terms this is barrier() around atomicAdd(counter, -1) without memoryBarrierBuffer().

DXIL (sha256 e57fc139d46aa62ffb1073fc4e451cf6b611c400583edb20d515ab300c5abec7, 9972 bytes):

%453..%458 = call i32 @dx.op.atomicBinOp.i32(i32 78, %aabb, i32 4 or 5 /*IMin/IMax*/, ...)
call void @dx.op.barrier(i32 80, i32 9)     ; SyncThreadGroup|TGSMFence, no UAVFenceGlobal
%462 = call i32 @dx.op.atomicBinOp.i32(i32 78, %counter, i32 0 /*Add*/, ..., i32 -1)   ; last = (%462 == 1)
call void @dx.op.barrier(i32 80, i32 9)
%470 = call %dx.types.ResRet.i32 @dx.op.rawBufferLoad.i32(i32 139, %aabb, %tid, ...)   ; then convert + rawBufferStore
; UAV metadata: globallycoherent = false on the counter and on the AABB buffer

In HLSL terms this is GroupMemoryBarrierWithGroupSync() where DeviceMemoryBarrierWithGroupSync() plus globallycoherent are needed.

There is a secondary issue: the first barrier, after the shared-header initialization, sits inside if (DTid.x < count). In the tail workgroup, not all invocations reach it, so it is a non-uniform barrier. Fixing only this does not fix the bug (see the table below).

Evidence

Standalone replay (no Kit, no USD, no AS build). We dispatch only the extracted shader on captured inputs for one affected prim (12366 gaussians, 97 workgroups × 128 threads), wait for completion and read back the AABB. Each run is 2000 iterations; the table lists the wrong results of each separate run:

• Every wrong component fits the late-atomic pattern: either an unconverted partial minimum, or a stale partial value. None falls outside it. The completion counter always returns to 0.
• The reference AABB was also computed independently on the CPU from the instance transforms; it matches the correct GPU result within 4.1e-7.

In Kit (Isaac Sim 6.2.0, Vulkan): same camera path and streaming schedule, 470 frames, alternating 10 processes per arm.

• With the original shader, 5 of 10 runs lost a whole prim for 3 or more consecutive frames, while the same prim had 500 or more pixels in the other runs.
• With only the patched SPIR-V substituted (process-local Vulkan layer that matches the shader by exact bytes), 0 of 10 runs did.

D3D12 runs lose prims the same way; for example, one prim was missing for 67 frames in a 1660-frame walk.

--/rtx/scenedb/GPUProceduralAABB/enabled=false does not avoid the problem: the shader is still created and dispatched.

Suggested fix

• GLSL/Slang: release/acquire at device scope for storage buffers around the completion counter. Two ways to write it:
  • memoryBarrierBuffer() before the decrement and after observing "last";
  • or atomicAdd(counter, -1, gl_ScopeDevice, gl_StorageSemanticsBuffer, gl_SemanticsAcquireRelease) with matching controlBarrier(gl_ScopeWorkgroup, gl_ScopeDevice, gl_StorageSemanticsBuffer | gl_StorageSemanticsShared, gl_SemanticsAcquireRelease).
• HLSL: DeviceMemoryBarrierWithGroupSync(), and globallycoherent on the counter and AABB buffers.
• Move the first barrier out of the DTid.x < count branch.

On request we can share privately: the standalone replay programs (Vulkan and D3D12), the captured inputs for the affected prim, the patched SPIR-V/DXIL, and Kit logs.

Isaac Sim version

6.2.0 (Kit 110.4.0+feature.386251, omni.hydra.rtx 1.0.5+187cf21b)

Operating System (OS)

Windows 11 Pro 10.0.26200

GPU Name

RTX 5070 (12 GB)

GPU Driver and CUDA versions

Driver 595.97, CUDA 13.2 (also reproduced on 591.74, more often)

Logs

Not attached; Kit logs from good and bad runs are available on request. No error or warning in the log distinguishes a bad run. The maxInstanceIndex exceeds GPU table message appears in good and bad runs alike.

Additional information

• Tested only on Windows so far, with the Vulkan backend (Isaac Sim's default experience) and the D3D12 backend (Isaac Lab 3.0's experience).
• A Linux machine (L40S + L20, driver 595.58.03) has not been tested yet.