Skip to content

Document the package layout and the load and frame pipeline per backend #46

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Xget7 merged 1 commit into from
Oct 20, 2026
Merged

Document the package layout and the load and frame pipeline per backend #46

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

SDK integration or verification: read [the harness](docs/AGENT_HARNESS.md).
Implementation: follow [CONTRIBUTING.md](CONTRIBUTING.md); terminology lives in [CONTEXT.md](CONTEXT.md).
Architecture (packages, load and frame pipeline on Metal and Vulkan): read [ARCHITECTURE.md](docs/ARCHITECTURE.md) before exploring; a change to a stage, GPU layout or package dependency updates it in the same PR.
Keep documentation terse; link code/contracts instead of duplicating them.
Report skipped checks, approximation limits and emulator/physical-device provenance explicitly.

Expand Down
87 changes: 87 additions & 0 deletions docs/ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# Architecture

How the packages fit together and what each stage of the splat pipeline does on Metal and Vulkan.
Terms follow [CONTEXT.md](../CONTEXT.md); the linked files are the source of truth.

## Packages

`splat-core` -> `splatkit-engine` -> `splatkit-ios` | `splatkit-android` -> `react-native-splatkit` -> `apps/react-native`

[Code layout](../CONTRIBUTING.md#code-layout) says what each package and folder owns.
Around the packages:

- [scripts](../scripts): world preparation (`prepare-world.sh`), iOS packaging and source export, the [SDK harness](AGENT_HARNESS.md), Perfetto and benchmark analysis, and their tests.
- [.github/workflows](../.github/workflows): one workflow per package plus lint, release and the iOS mirror; [ci-filters.yml](../.github/ci-filters.yml) maps changed files to jobs.
- [VALIDATION.md](VALIDATION.md), [BENCHMARKS.md](BENCHMARKS.md), [RELEASING.md](RELEASING.md) and [VULKAN.md](../packages/splatkit-android/docs/VULKAN.md) for checks, measurements, releases and the Vulkan contracts.

## Load (shared, CPU)

Both platforms run the same code up to the upload.

1. **Decode** on a loader thread.
SPZ v1 to v4 goes through Niantic's spz, fetched and pinned by [dependencies.cmake][deps]; [SpzDecoder.cpp][spz] inflates gzip with zlib, and spz handles zstd for v4.
A `.lodsplat` carries its tree and skips steps 3 and 4.
A tileset reads only `tileset.json` at load; two `TileLoader` threads decode its tiles later.
PLY is read only by the offline tools.
Loader threads: iOS serial queue `com.splatkit.loader` ([RenderThread.swift][ios-thread]), React Native iOS `com.splatkit.rn.loader`, Android single-thread executor `SplatKitLoader` ([RenderThread.kt][android-thread]).
2. **Bake** per splat inside the decoder, scalar: covariance `R diag(s)^2 R^T` ([Covariance.cpp][cov]), sigmoid opacity, the band 0 spherical harmonics (SH) folded into colour, SH truncated to `maxShDegree`.
3. **Reorder** by 30-bit Morton code ([SpatialOrder.cpp][morton]), plain SPZ only.
4. **Tree**: with a splat budget above 0, `buildLodTree` builds a size-adaptive grid at load ([LodTree.cpp][tree]); a budget of 0 draws the flat cloud.
5. **Pack** with [GpuLayout.cpp][layout]: `GpuSplat` is 32 bytes (position `f32x3`, `rgba8`, six half covariance terms, `lodAlpha`); SH sits in its own buffer as half pairs, bands 1 to the degree.

The decoded cloud waits in [SplatEngine.cpp][engine] until the render thread takes it and uploads it.

## Frame (per backend)

| Stage | Metal (iOS) | Vulkan (Android) |
|---|---|---|
| Upload | Zero copy: the packers write straight into `Shared` buffers on unified memory; tiles pack into a shared slab ([MetalWorld.mm][mtl-world]) | 2 MiB host-visible staging windows into device-local VMA buffers, each a one-shot copy with a fence wait and `ALL_COMMANDS`/`TRANSFER` barriers ([VulkanBuffer.cpp][vk-buffer], [VulkanWorld.cpp][vk-world]) |
| Selection | GPU frontier traversal of the tree by pixel error within the splat budget, with cluster frustum culling, in its own command buffer; the CPU adapts the pixel threshold from counters read back one to two frames late ([MetalLodSelection.mm][mtl-lod], [SplatLOD.metal][mtl-lod-shader]) | GPU frontier traversal of the tree by pixel error within the splat budget (binary search over prefix costs), with cluster frustum culling, at a fixed 1 px threshold ([lod_selection.comp][vk-lod]) |
| Cull | `visibility` kernel culls alpha, behind camera, near/far, screen edge and subpixel; SIMD prefix compaction; writes keys, values and a 32-byte `ProjectedSplat` with SH evaluated ([SplatVisibility.metal][mtl-vis], [SplatProjection.metalh][mtl-proj]) | `visibility.comp` culls alpha (fixed 1/255), behind camera, frustum with margin, near/far by covariance and subpixel; subgroup compaction; writes indices and depth keys only ([visibility.comp][vk-vis]) |
| Sort | GPU LSD radix, 8-bit digits, 4 passes on 32-bit keys (default) or 2 on 16-bit; histogram and scatter rank with `simd_ballot`; indirect dispatch; needs 32-wide SIMD groups ([SplatRadixSort.metal][mtl-sort]) | GPU stable LSD radix, 8-bit digits, 4 or 2 passes; histogram with workgroup atomics because ballot loses keys on Adreno 640; scatter ranks with an `atomicOr` bitmask; indirect dispatch; 3M capacity ([radix_histogram.comp][vk-hist], [VulkanRadixSort.cpp][vk-sort]) |
| Raster | Instanced quads in 7 indirect draws, front to back with "under" blending, in batches of 1, 1, 2, 4, 8, 16 and 32 sixty-fourths; between batches a mask pass writes `Depth16` where alpha reaches 254/255, so later batches skip saturated pixels; only the mask writes depth ([MetalSplatPipeline.mm][mtl-pipe], [SplatRaster.metal][mtl-raster], [PrepareIndirect.metalh][mtl-indirect]) | One indirect draw of instanced quads, back to front with "over" blending, one subpass with one colour attachment; the vertex shader reprojects and evaluates SH ([VulkanSplatPipeline.cpp][vk-pipe], [splat.vert][vk-vert]) |
| Present | With GPU sort, always an RGBA16F target, then a fullscreen render-pass blit (or MetalFX spatial upscale) into a BGRA8 drawable ([MetalSurface.mm][mtl-surface], [MetalUpscaler.mm][mtl-upscale]) | Straight into the swapchain at render scale 1, otherwise offscreen plus `vkCmdBlitImage`; a UNORM swapchain unless linear blending is on, since an sRGB attachment costs about 40% of the frame on Adreno 640 ([VulkanSwapchain.cpp][vk-swap], [VulkanSurface.cpp][vk-surface]) |
| Submission | Three command buffers per frame (selection; cull, sort and indirect arguments; render and present), 2 frames in flight ([MetalSplatRenderer.mm][mtl-renderer]) | One command buffer per frame (selection, cull, sort, indirect arguments, diagnostics copy, render pass), 2 frames in flight, a swapchain of at least 3 images ([VulkanSplatRenderer.cpp][vk-renderer]) |

Both backends:

- Splats blend in the encoded space by default; linear blending (off by default) applies `pow(rgb, 2.2)` to splat colour and presents to an sRGB surface.
- The render policy applies per backend: Metal honours the pixel error, Vulkan only the sort depth and the subpixel threshold ([VulkanFrameCompute.h][vk-frame]).
- Splats rasterize as hardware quads over the whole screen; `supportsComputeTiles` is false on both.
- [SplatEngine.cpp][engine] draws only when something marks a redraw, so a still view costs no GPU time.
- Collision runs on the CPU in the walk camera and moves only the camera.
- Without GPU sort, Metal falls back to a CPU order with one back-to-front "over" draw; Vulkan needs subgroup basic, ballot and arithmetic support and refuses to load a world without it.

[deps]: ../packages/splat-core/cmake/dependencies.cmake
[spz]: ../packages/splat-core/src/formats/SpzDecoder.cpp
[cov]: ../packages/splat-core/src/math/Covariance.cpp
[morton]: ../packages/splat-core/src/sorting/SpatialOrder.cpp
[tree]: ../packages/splat-core/src/lod/LodTree.cpp
[layout]: ../packages/splatkit-engine/src/rendering/GpuLayout.cpp
[engine]: ../packages/splatkit-engine/src/engine/SplatEngine.cpp
[ios-thread]: ../packages/splatkit-ios/Sources/SplatKit/engine/RenderThread.swift
[android-thread]: ../packages/splatkit-android/src/main/kotlin/com/splatkit/engine/RenderThread.kt
[mtl-world]: ../packages/splatkit-ios/Sources/SplatKitCore/rendering/metal/world/MetalWorld.mm
[mtl-lod]: ../packages/splatkit-ios/Sources/SplatKitCore/rendering/metal/selection/MetalLodSelection.mm
[mtl-lod-shader]: ../packages/splatkit-ios/Sources/SplatKitCore/rendering/metal/selection/SplatLOD.metal
[mtl-vis]: ../packages/splatkit-ios/Sources/SplatKitCore/rendering/metal/cull/SplatVisibility.metal
[mtl-proj]: ../packages/splatkit-ios/Sources/SplatKitCore/rendering/metal/layout/SplatProjection.metalh
[mtl-sort]: ../packages/splatkit-ios/Sources/SplatKitCore/rendering/metal/sort/SplatRadixSort.metal
[mtl-pipe]: ../packages/splatkit-ios/Sources/SplatKitCore/rendering/metal/raster/MetalSplatPipeline.mm
[mtl-raster]: ../packages/splatkit-ios/Sources/SplatKitCore/rendering/metal/raster/SplatRaster.metal
[mtl-indirect]: ../packages/splatkit-ios/Sources/SplatKitCore/rendering/metal/cull/PrepareIndirect.metalh
[mtl-surface]: ../packages/splatkit-ios/Sources/SplatKitCore/rendering/metal/presentation/MetalSurface.mm
[mtl-upscale]: ../packages/splatkit-ios/Sources/SplatKitCore/rendering/metal/presentation/MetalUpscaler.mm
[mtl-renderer]: ../packages/splatkit-ios/Sources/SplatKitCore/rendering/metal/MetalSplatRenderer.mm
[vk-buffer]: ../packages/splatkit-android/src/main/cpp/rendering/vulkan/device/VulkanBuffer.cpp
[vk-world]: ../packages/splatkit-android/src/main/cpp/rendering/vulkan/world/VulkanWorld.cpp
[vk-lod]: ../packages/splatkit-android/src/main/cpp/rendering/vulkan/selection/lod_selection.comp
[vk-vis]: ../packages/splatkit-android/src/main/cpp/rendering/vulkan/cull/visibility.comp
[vk-hist]: ../packages/splatkit-android/src/main/cpp/rendering/vulkan/sort/radix_histogram.comp
[vk-sort]: ../packages/splatkit-android/src/main/cpp/rendering/vulkan/sort/VulkanRadixSort.cpp
[vk-pipe]: ../packages/splatkit-android/src/main/cpp/rendering/vulkan/raster/VulkanSplatPipeline.cpp
[vk-vert]: ../packages/splatkit-android/src/main/cpp/rendering/vulkan/raster/splat.vert
[vk-swap]: ../packages/splatkit-android/src/main/cpp/rendering/vulkan/presentation/VulkanSwapchain.cpp
[vk-surface]: ../packages/splatkit-android/src/main/cpp/rendering/vulkan/presentation/VulkanSurface.cpp
[vk-renderer]: ../packages/splatkit-android/src/main/cpp/rendering/vulkan/VulkanSplatRenderer.cpp
[vk-frame]: ../packages/splatkit-android/src/main/cpp/rendering/vulkan/frame/VulkanFrameCompute.h