Pin stable-diffusion-native v0.3.0-4 and decode full VAEs on iOS Metal (#1047)

CHANGELOG.md

@@ -2,12 +2,13 @@ Releases before 0.10.0 are in the
 [changelog archive](https://github.com/leehack/llamadart/blob/main/doc/changelog_archive.md).
 
 ## Unreleased
+- Adopt stable-diffusion `v0.3.0-4`: on iOS Metal the runtime splits graphs, so a full VAE decodes without tripping the GPU watchdog and `ImageGenerationEngine.load` no longer refuses it there; a runtime without the split keeps the refusal. Run on an iPhone 16 Pro and an iPhone 17 Pro (30 of 30 soak decodes). On iOS 26 the second engine dispose can leave about 2 GB resident ([#1047](https://github.com/leehack/llamadart/issues/1047), [#2004](https://github.com/leehack/llamadart/issues/2004)).
 
 - Chat app microphone: a microphone that disconnects mid-recording now transcribes the audio captured so far with a warning, a missing input device on Web is reported as such instead of as a permission denial, and a failed Web start releases the microphone ([#329](https://github.com/leehack/llamadart/issues/329)).
 - Aligned the default WebGPU bridge assets to `v0.1.85`: a page without cross-origin isolation now fails at once with a `LlamaUnsupportedException` naming COOP/COEP `require-corp` instead of waiting minutes for a generic init timeout, and Firefox CPU runs (`gpuLayers: 0`) no longer create the WebGPU backend (stories15M, 32 tokens: 640 to 762 tok/s against 10 before) ([#1025](https://github.com/leehack/llamadart/issues/1025)).
 - `ComputeDevice.gpu` on Web now names why WebGPU was not used (for example `adapter_missing_shader_f16`: Chrome on Linux with NVIDIA needs driver 615.71 or newer); documented the measured Safari 27, Firefox 160 and NVIDIA L4 results ([#1025](https://github.com/leehack/llamadart/issues/1025)).
 
-Runtime baseline: native llama.cpp `v0.9.0` and Web use upstream `v0.9.0@cd959bae99ff4bed22bef2604f45241c09bbb8c8`; native LiteRT-LM uses `v0.19.1-4` and stable-diffusion uses `v0.3.0-3`. Web LiteRT-LM uses `@litert-lm/[email protected]`. Manifest SHA-256: `6767783edd85e14853a8a79cd6ce53d53f4eb2d99aeb62601014bd68c1b6b7b3`.
+Runtime baseline: native llama.cpp `v0.9.0` and Web use upstream `v0.9.0@cd959bae99ff4bed22bef2604f45241c09bbb8c8`; native LiteRT-LM uses `v0.19.1-4` and stable-diffusion uses `v0.3.0-4`. Web LiteRT-LM uses `@litert-lm/[email protected]`. Manifest SHA-256: `6767783edd85e14853a8a79cd6ce53d53f4eb2d99aeb62601014bd68c1b6b7b3`.
 
 ## 0.19.9
 

README.md

@@ -146,10 +146,9 @@ creation fails, `ComputeDevice.auto` retries on the CPU with a warning;
 Image generation
 is not available on the web. On iOS, use
 `ImageModelPlacement(vaeOnCpu: true)` to keep diffusion on Metal, or CPU
-generation. Metal VAE decoding needs a native context policy that preserves
-an intrinsic SDXS tiny decoder through model refresh. The pinned
-`v0.3.0-3` runtime provides that policy; full VAE and external-decoder
-Metal decoding remain refused on iOS. This is not new physical-device qualification.
+generation. The pinned `v0.3.0-4` runtime splits Metal graphs, so a full VAE
+decodes on iOS Metal; a runtime without the split keeps the intrinsic SDXS
+tiny-decoder policy and refuses full-VAE Metal decoding.
 Per-target support, device limitations and runtime options:
 
 - [Platform and backend support matrix](https://llamadart.leehack.com/docs/platforms/support-matrix), including

doc/cross_platform_validation.md

@@ -919,9 +919,11 @@ a follow-up. Do not insert requested bundle metadata as observed runtime evidenc
 checks: `image-sdxs-cpu` (Android, CPU) and `image-sdxs-metal` (iOS, Metal).
 The stable-diffusion-native `v0.3.0-2` pin provides the intrinsic tiny-decoder
 v1 context policy required by the iOS Metal SDXS profile. Its physical-device
-qualification must still use this exact runtime and consumer build. Full-VAE
-and external-decoder Metal paths remain refused; use CPU VAE placement for
-those models ([#1047](https://github.com/leehack/llamadart/issues/1047)).
+qualification must still use this exact runtime and consumer build. The
+`v0.3.0-4` graph split lifts the full-VAE refusal on iOS Metal (see
+[iOS full-VAE graph split](#ios-full-vae-graph-split-1047-2026-11-13)); a
+runtime without it still refuses, and needs CPU VAE placement
+([#1047](https://github.com/leehack/llamadart/issues/1047)).
 
 They run the chat-app test `integration_test/image_generation_e2e_test.dart`
 (SDXS download through the image screen, one seeded 512x512 image, and a
@@ -2341,3 +2343,29 @@ unqualified, as do accelerators, other models and physical devices.
 
 This covers one device, model and quantization and qualifies no other GPU.
 See [Known platform issues](../website/docs/platforms/known-platform-issues.md).
+
+## iOS full-VAE graph split (#1047, 2026-11-13)
+
+stable-diffusion-native `v0.3.0-4` (native commit `4a84ac24`, manifest SHA-256
+`86c674911194b5b59f2e78f1906052af2df899ca60de56881690efcf0a6833a1`) splits
+Metal graphs on iOS through upstream's eval callback at 2^26 elements and
+exports `sd_dart_graph_split_set_threshold`, `sd_dart_graph_split_threshold`
+and `sd_dart_graph_split_cuts`. The core lifts its full-VAE iOS Metal refusal
+when the threshold getter exists and returns above 0.
+
+DDP ran the XCFramework (zip SHA-256 `0d022638f54be80aa75fd8e73a94d139ad8887f4f3c73084501369ca5e7ac354`)
+with the SD-Turbo full-VAE harness:
+
+| Run | Devices | Result |
+| --- | --- | --- |
+| C1, split arms | iPhone 16 Pro, iOS 18.3 | every arm passed 3 of 3 decodes |
+| C1, split off | iPhone 16 Pro, iOS 18.3 | third decode failed with the GPU command-buffer error (watchdog) |
+| C2b soak | iPhone 16 Pro, iPhone 17 Pro | 30 of 30 decodes passed, one image hash, flat footprint |
+
+Known limit ([#2004](https://github.com/leehack/llamadart/issues/2004)): on
+the iPhone 17 Pro (iOS 26.5.1) the second engine dispose kept about 2 GB
+resident and later loads failed the memory check; the iPhone 16 Pro returned to
+its baseline. Whether it is a leak or delayed reclaim is undecided.
+
+Final qualification of the public API at the PR head and the published pin is a
+separate DDP run (`r1047-final`).

example/chat_app/README.md

@@ -306,7 +306,7 @@ layouts) opens a separate screen for the experimental `ImageGenerationEngine`
   runtime…** meanwhile. With an empty Metal shader cache the first check
   compiles ggml's Metal library, about 16 s on an M4 Max; with the cache warm
   it takes under 0.5 s.
-- On iOS, the pinned SD runtime preserves the intrinsic SDXS tiny decoder across model refresh. Full VAE and external-decoder recipes require CPU VAE placement or CPU generation; this update does not add physical-iOS qualification.
+- On iOS, the pinned SD runtime (`v0.3.0-4`) splits Metal graphs, so full-VAE recipes decode on the GPU; the app has no iOS-specific gating. A runtime without the split refuses them and needs CPU VAE placement or CPU generation. On iOS 26 the second engine dispose can leave about 2 GB resident ([#2004](https://github.com/leehack/llamadart/issues/2004)).
 - Models: **SDXS-512** (683 MB, recommended, fits most phones) and
   **SD-Turbo + TAESD** (2.0 GB). Both download into the app's model cache
   from pinned Hugging Face revisions with exact sizes and SHA-256 checks; an

lib/src/backends/stable_diffusion/stable_diffusion_bindings.dart

@@ -358,6 +358,15 @@ external int sd_dart_gpu_device_memory(
   ffi.Pointer<sd_dart_gpu_device_memory_t> out,
 );
 
+@ffi.Native<ffi.Uint64 Function()>()
+external int sd_dart_graph_split_cuts();
+
+@ffi.Native<ffi.Void Function(ffi.Uint64)>()
+external void sd_dart_graph_split_set_threshold(int elements);
+
+@ffi.Native<ffi.Uint64 Function()>()
+external int sd_dart_graph_split_threshold();
+
 @ffi.Native<ffi.Size Function(ffi.Pointer<ffi.Char>, ffi.Size)>()
 external int sd_dart_last_error(ffi.Pointer<ffi.Char> text, int capacity);
 

lib/src/backends/stable_diffusion/stable_diffusion_calls.dart

@@ -27,6 +27,7 @@ typedef _CancelGenerationNative =
     Void Function(Pointer<sd.sd_ctx_t>, UnsignedInt);
 typedef _ExitFreeNative = Void Function(Pointer<Void>);
 typedef _GpuDeviceCountNative = Int32 Function();
+typedef _GraphSplitThresholdNative = Uint64 Function();
 typedef _GpuDeviceMemoryNative =
     Int32 Function(Int32, Pointer<sd.sd_dart_gpu_device_memory_t>);
 typedef _LogEnableNative = Void Function();
@@ -65,7 +66,8 @@ typedef _FreeImageGenerationResultNative =
 /// added. Each group is bound only when the runtime exports all of it, and is
 /// `null` otherwise: a runtime between the two releases still generates
 /// images, without the runtime's log and its reason for a failed load, and
-/// without the memory check of a Vulkan GPU.
+/// without the memory check of a Vulkan GPU. [graphSplit] is the same for
+/// [graphSplitNativeRelease]'s Metal graph splitting.
 ///
 /// `sd_dart_exit_teardown`, `sd_dart_exit_call_begin`,
 /// `sd_dart_exit_call_end`, `sd_dart_exit_track` and `sd_dart_exit_untrack`
@@ -94,6 +96,7 @@ final class StableDiffusionCalls {
     required this.freeImages,
     this.log,
     this.gpu,
+    this.graphSplit,
     this.loadsFp8AsF16OnMetal = false,
     this.newContextIntrinsicTinyV1,
   });
@@ -131,6 +134,16 @@ final class StableDiffusionCalls {
     'sd_dart_gpu_device_memory',
   ];
 
+  /// The `stable-diffusion-native` release that added [graphSplitSymbols].
+  static const String graphSplitNativeRelease = 'v0.3.0-4';
+
+  /// The `sd_dart_` functions of [graphSplit] that [tryResolve] binds. The
+  /// release also exports `sd_dart_graph_split_set_threshold` and
+  /// `sd_dart_graph_split_cuts`, which Dart does not call.
+  static const List<String> graphSplitSymbols = [
+    'sd_dart_graph_split_threshold',
+  ];
+
   /// The first `stable-diffusion-native` release that loads FP8 weights a
   /// Metal backend computes with as F16 instead of aborting the process
   /// (leehack/stable-diffusion-native#29).
@@ -225,6 +238,10 @@ final class StableDiffusionCalls {
       Native.addressOf<NativeFunction<_GpuDeviceMemoryNative>>(
         sd.sd_dart_gpu_device_memory,
       ),
+    'sd_dart_graph_split_threshold' =>
+      Native.addressOf<NativeFunction<_GraphSplitThresholdNative>>(
+        sd.sd_dart_graph_split_threshold,
+      ),
     'free_sd_image_generation_result' =>
       Native.addressOf<NativeFunction<_FreeImageGenerationResultNative>>(
         sd.free_sd_image_generation_result,
@@ -307,6 +324,13 @@ final class StableDiffusionCalls {
           ).asFunction(),
         ),
       ),
+      graphSplit: optional(
+        () => StableDiffusionGraphSplitCalls(
+          threshold: function<_GraphSplitThresholdNative>(
+            'sd_dart_graph_split_threshold',
+          ).asFunction(),
+        ),
+      ),
       loadsFp8AsF16OnMetal:
           _releaseAtLeast(releaseTag, fp8MetalNativeRelease) &&
           (optional(() => fp8MetalSymbols.map(symbol).toList()) != null),
@@ -398,6 +422,10 @@ final class StableDiffusionCalls {
   /// [optionalNativeRelease].
   final StableDiffusionGpuCalls? gpu;
 
+  /// The runtime's Metal graph-split query, or `null` on a runtime older than
+  /// [graphSplitNativeRelease].
+  final StableDiffusionGraphSplitCalls? graphSplit;
+
   /// Whether the runtime loads FP8 weights a Metal backend computes with as
   /// F16, so FP8 weights can reach a Metal context without aborting the
   /// process.
@@ -435,6 +463,22 @@ final class StableDiffusionGpuCalls {
   deviceMemory;
 }
 
+/// The Metal graph-split query of the stable_diffusion runtime.
+///
+/// On iOS Metal the runtime commits a graph in several command buffers, so
+/// that no single one of a full-VAE decode outlasts the GPU watchdog.
+final class StableDiffusionGraphSplitCalls {
+  /// Creates the calls from functions.
+  const StableDiffusionGraphSplitCalls({required this.threshold});
+
+  /// `sd_dart_graph_split_threshold`: the number of tensor elements after
+  /// which a graph is split, or 0 when it is not split. Returns at once.
+  final int Function() threshold;
+
+  /// Whether the runtime splits graphs.
+  bool get isActive => threshold() > 0;
+}
+
 /// The log recorder of the stable_diffusion runtime: the library copies each
 /// message of stable-diffusion.cpp and ggml into a buffer of its own and Dart
 /// reads them, so Dart gives the runtime no log callback either.

lib/src/backends/stable_diffusion/stable_diffusion_runtime_io.dart

@@ -34,6 +34,10 @@ abstract interface class StableDiffusionNativeApi {
   /// [StableDiffusionCalls.loadsFp8AsF16OnMetal] of the runtime.
   bool loadsFp8AsF16OnMetal();
 
+  /// Whether the runtime exports [StableDiffusionCalls.graphSplit] and
+  /// reports a threshold above 0.
+  bool splitsGraphsOnMetal();
+
   /// Whether the runtime enforces the intrinsic tiny-decoder v1 contract.
   bool supportsIntrinsicTinyDecoderV1();
 }
@@ -182,6 +186,7 @@ StableDiffusionRuntimeStatus probeStableDiffusionRuntime({
       commit: commit,
       devices: devices,
       loadsFp8AsF16OnMetal: api.loadsFp8AsF16OnMetal(),
+      splitsGraphsOnMetal: api.splitsGraphsOnMetal(),
       supportsIntrinsicTinyDecoderV1: api.supportsIntrinsicTinyDecoderV1(),
     );
   } on ArgumentError catch (error) {
@@ -404,6 +409,9 @@ final class BundledStableDiffusionApi implements StableDiffusionNativeApi {
   @override
   bool loadsFp8AsF16OnMetal() => resolveCalls()?.loadsFp8AsF16OnMetal ?? false;
 
+  @override
+  bool splitsGraphsOnMetal() => resolveCalls()?.graphSplit?.isActive ?? false;
+
   @override
   bool supportsIntrinsicTinyDecoderV1() =>
       resolveCalls()?.newContextIntrinsicTinyV1 != null;

lib/src/backends/stable_diffusion/stable_diffusion_runtime_status.dart

@@ -31,6 +31,11 @@ final class StableDiffusionRuntimeStatus {
   /// weights abort the process when a Metal context is created.
   final bool loadsFp8AsF16OnMetal;
 
+  /// Whether the runtime splits large graphs on Metal, as
+  /// `StableDiffusionCalls.graphSplit` reports: a full-VAE decode then stays
+  /// under the iOS GPU watchdog.
+  final bool splitsGraphsOnMetal;
+
   /// Whether a native context can enforce the intrinsic tiny-decoder v1
   /// policy through every automatic model refresh.
   final bool supportsIntrinsicTinyDecoderV1;
@@ -41,6 +46,7 @@ final class StableDiffusionRuntimeStatus {
     required String this.commit,
     required this.devices,
     this.loadsFp8AsF16OnMetal = false,
+    this.splitsGraphsOnMetal = false,
     this.supportsIntrinsicTinyDecoderV1 = false,
   }) : unavailableReason = null;
 
@@ -52,6 +58,7 @@ final class StableDiffusionRuntimeStatus {
       commit = null,
       devices = const [],
       loadsFp8AsF16OnMetal = false,
+      splitsGraphsOnMetal = false,
       supportsIntrinsicTinyDecoderV1 = false;
 
   /// Whether the runtime loaded and answered the probe.

lib/src/core/image/image_generation_engine.dart

@@ -417,8 +417,9 @@ class ImageGenerationEngine
   ///   unused here. Also when
   ///   [ImageModelParams.vaeDirectConvolution] is `true` on iOS with the
   ///   Metal backend, before any file is resolved or loaded. Full-VAE Metal
-  ///   decoding on iOS is also refused. Known full-VAEs are rejected before
-  ///   native creation; tiny candidates require a native context policy that
+  ///   decoding on iOS is also refused unless the runtime splits graphs
+  ///   (stable-diffusion-native v0.3.0-4 or later). Known full-VAEs are
+  ///   rejected before native creation; tiny candidates require a native context policy that
   ///   preserves the intrinsic decoder through automatic source refreshes. Use
   ///   [ImageModelPlacement.vaeOnCpu] or [ComputeDevice.cpu] for other recipes,
   ///   including external tiny decoders. Tiling does not bypass this restriction.
@@ -645,12 +646,15 @@ class ImageGenerationEngine
     final requiresIosTinyDecoder =
         driver.operatingSystem == 'ios' &&
         _isMetal(backendName) &&
-        !placement.vaeOnCpu;
+        !placement.vaeOnCpu &&
+        !runtime.splitsGraphsOnMetal;
     LlamaUnsupportedException
     unsupportedIosDecoder() => LlamaUnsupportedException(
       'Full-VAE decoding on iOS with the Metal backend ($backendName) is '
-      'not supported: repeated decodes can trip the GPU watchdog '
-      '(https://github.com/leehack/llamadart/issues/1047). GPU decoding '
+      'not supported on this runtime: repeated decodes can trip the GPU '
+      'watchdog (https://github.com/leehack/llamadart/issues/1047). '
+      'stable-diffusion-native v0.3.0-4 or later splits the graph and '
+      'decodes a full VAE; this runtime does not. Otherwise GPU decoding '
       'requires an intrinsic SDXS tiny decoder and a selected runtime with '
       'the intrinsic tiny-decoder v1 context policy, so model refresh cannot '
       'switch to a full VAE. External tiny decoders can '