diff --git a/README.md b/README.md index cc17f1f..35a471e 100644 --- a/README.md +++ b/README.md @@ -326,7 +326,9 @@ v0.18 this includes the iOS `LiteRtMetalAccelerator` and `LiteRtTopKMetalSampler` modules; their framework-relative loader paths avoid flat dylibs that App Store bundles cannot ship. v0.19 `LiteRtLm` links its GPU accelerator and sampler, so neither is needed; releases after `v0.19.1-2` ship -no Metal module. Releases after `v0.19.1-3` also drop the macOS `libLiteRt` and +no Metal module. Android v0.19 runtimes after `v0.19.1-4` link both the OpenCL +and WebGPU samplers, because the linked Android GPU accelerator can serve either +backend. Releases after `v0.19.1-3` also drop the macOS `libLiteRt` and `libLiteRtWebGpuAccelerator` dylibs, the iOS loose `libLiteRt` and the matching `LiteRt` and `LiteRtWebGpuAccelerator` XCFramework zips: v0.19 `LiteRtLm` embeds LiteRT and never opens them. diff --git a/docs/android_gpu_qualification.md b/docs/android_gpu_qualification.md index 9f8e7b3..a4bd768 100644 --- a/docs/android_gpu_qualification.md +++ b/docs/android_gpu_qualification.md @@ -55,6 +55,18 @@ immutable wrapper version, then qualify its exact packaged artifact before updating downstream pins. The local single-library candidate does not qualify a future release or authorize consumer PR #503 to merge unchanged. +## WebGPU sampler (v0.19.0 through v0.19.1-4) + +Android runtimes `v0.19.0` through `v0.19.1-4` define the OpenCL sampler +functions but not the WebGPU ones. On a device or app without loadable OpenCL, +the linked accelerator selects WebGPU, the engine loads, and the first +`SampleLogits` returns `Can not find OpenCL library on this device` because the +OpenCL sampler is the only one linked and does not report itself unavailable +(Galaxy S24, Gemma 4 E2B, +[litert-lm-native#120](https://github.com/leehack/litert-lm-native/issues/120)). +Later runtimes link both samplers and the build rejects a candidate missing +either. Device result on the fixed runtime: pending. + ## GPU environment teardown patch LiteRT `9fe5be45`, pinned by LiteRT-LM 0.17.x, loads OpenCL before it records diff --git a/docs/v019_qualification.md b/docs/v019_qualification.md index d5d3c7b..3b9c0ee 100644 --- a/docs/v019_qualification.md +++ b/docs/v019_qualification.md @@ -33,6 +33,7 @@ The exact-tag verification gate remains enabled. | ASR | `omni/asr` and the legacy engine are gone; `omni/speech2text` factories route through `OmniRegistry` and require a container | C ASR ABI v1 is unchanged. Containers use the registry; a `.tflite` plus tokenizer uses a bridge factory (`litert_lm_asr_omni_engine.cc`). Container metadata and Parakeet sections pick the registry name, so file names, free-form `model_name`s and `decoder_type` do not change routing, and a `model_name` naming another family fails at engine creation. Other LLM containers lost the generic LLM route and need a registered name. Start tokens are dropped for LM-backed families, whose decoders reject them. | | Protos | `Asr*` became `Speech2Text*` | Internal to the adapter. | | Packaging | No `LiteRtTopK*Sampler` library; `libLiteRtLm` links Dawn | Validators and manifests key this on `has_prebuilt_topk_samplers`. | +| Android GPU samplers | `libLiteRtLm` links `ml_drift_gpu_accelerator`, which serves OpenCL and WebGPU, but upstream links only the OpenCL sampler unless built with `--copt=-DLITERT_USE_WEBGPU_ACCELERATOR`. Without OpenCL the WebGPU route loads, then fails at the first token with `Can not find OpenCL library on this device` ([litert-lm-native#120](https://github.com/leehack/litert-lm-native/issues/120), [llamadart#1998](https://github.com/leehack/llamadart/issues/1998)) | The bridge links `//runtime/components:top_k_webgpu_sampler_static` on Android (`--define=litert_lm_android_webgpu_sampler=true`). Build and package validators compare exact dynamic exports and reject a candidate missing either sampler set (`android_static_gpu_sampler_symbols`). Re-audit on the next upstream bump: upstream `main` already splits the Android accelerator by `gpu_accelerator`. | | Apple GPU | `LiteRtLm` links its GPU accelerator: ml_drift Metal on iOS, WebGPU on macOS | LiteRT registers it before searching for a shared one, so the prebuilt Metal accelerator is never opened ([llamadart#1798](https://github.com/leehack/llamadart/issues/1798)). Packaging after `v0.19.1-2` skips it on iOS and macOS, requires the static accelerator symbol in `LiteRtLm`, and rejects a candidate that still ships it (`has_prebuilt_apple_metal_accelerator`). | | Apple LiteRT companions | `LiteRtLm` embeds LiteRT and its GPU accelerator | dyld traces on macOS (CPU, GPU, vision, audio, MTP, embeddings, ASR) and an iOS simulator probe load neither `libLiteRt` nor `libLiteRtWebGpuAccelerator` ([llamadart#1911](https://github.com/leehack/llamadart/issues/1911)). Packaging after `v0.19.1-3` skips the macOS `libLiteRt` and `libLiteRtWebGpuAccelerator` and the iOS loose `libLiteRt`, and rejects a candidate that still ships them (`has_prebuilt_apple_litert_companions`). | diff --git a/native/bridge/BUILD.bazel b/native/bridge/BUILD.bazel index 60a2870..332cae4 100644 --- a/native/bridge/BUILD.bazel +++ b/native/bridge/BUILD.bazel @@ -89,6 +89,14 @@ config_setting( define_values = {"litert_lm_c_api_v1": "true"}, ) +# Upstream links only the OpenCL TopK sampler on Android, but its GPU +# accelerator runs on WebGPU when OpenCL cannot be loaded. +config_setting( + name = "android_webgpu_sampler", + constraint_values = ["@platforms//os:android"], + define_values = {"litert_lm_android_webgpu_sampler": "true"}, +) + # v0.17 moved the capabilities C implementation into //c:engine. config_setting( name = "capabilities_in_c_engine", @@ -278,7 +286,10 @@ cc_binary( ":litert_lm_bridge", ":litert_lm_asr_bridge", "//c:engine", - ] + CAPABILITIES_DEPS, + ] + CAPABILITIES_DEPS + select({ + ":android_webgpu_sampler": ["//runtime/components:top_k_webgpu_sampler_static"], + "//conditions:default": [], + }), ) cc_binary( diff --git a/tools/build_upstream_runtime.py b/tools/build_upstream_runtime.py index ad51652..008b98b 100644 --- a/tools/build_upstream_runtime.py +++ b/tools/build_upstream_runtime.py @@ -26,8 +26,10 @@ verify_dynamic_runtime, ) from litert_lm_symbols import ( + android_static_gpu_sampler_symbols, has_asr_bridge, has_omni_asr_engine, + has_prebuilt_topk_samplers, in_process_litert_abi_version, is_at_least, uses_c_api_v1, @@ -39,6 +41,7 @@ from macos_runtime_policy import UNSUPPORTED_MACOS_ARCH_REASON from package_upstream_prebuilts import apply_prebuilt_overrides from runtime_dependency_utils import ( + elf_exported_symbols, elf_has_global_flag, elf_needed_libraries, is_elf, @@ -585,6 +588,8 @@ def build_runtime( command.append("--define=litert_lm_c_api_v1=true") if in_process_litert_abi_version(upstream_tag) == 1: command.append("--define=litert_lm_in_process_litert=true") + if platform == "android" and not has_prebuilt_topk_samplers(upstream_tag): + command.append("--define=litert_lm_android_webgpu_sampler=true") if require_finish_reason_patch_decision(upstream_tag): command.append(FINISH_REASON_DEFINE) if platform == "macos": @@ -899,6 +904,19 @@ def validate_android_global_visibility(output: Path, platform: str) -> None: print(f"Validated Android global symbol visibility in {output}", flush=True) +def validate_android_gpu_samplers(output: Path, platform: str, upstream_tag: str) -> None: + if platform != "android": + return + missing = sorted( + set(android_static_gpu_sampler_symbols(upstream_tag)) - elf_exported_symbols(output)) + if missing: + raise RuntimeError( + f"{output} does not define the GPU sampler functions " + f"{', '.join(missing)}; generation fails on the GPU backend they " + "serve (#120)") + print(f"Validated linked OpenCL and WebGPU samplers in {output}", flush=True) + + def validate_android_gpu_environment_teardown(output: Path, platform: str) -> None: if platform != "android": return @@ -911,6 +929,7 @@ def validate_runtime(output: Path, platform: str, upstream_tag: str) -> None: validate_in_process_litert_linkage(output, platform, upstream_tag) validate_android_global_visibility(output, platform) validate_android_protobuf_isolation(output, platform, upstream_tag) + validate_android_gpu_samplers(output, platform, upstream_tag) validate_android_gpu_environment_teardown(output, platform) diff --git a/tools/litert_lm_symbols.py b/tools/litert_lm_symbols.py index b609743..c27f1c0 100644 --- a/tools/litert_lm_symbols.py +++ b/tools/litert_lm_symbols.py @@ -81,6 +81,16 @@ "LiteRtTopKOpenClSampler_SetInferenceFuncAndInputTensors", ] +ANDROID_WEBGPU_SAMPLER_SYMBOLS = [ + "LiteRtTopKWebGpuSampler_Create", + "LiteRtTopKWebGpuSampler_Destroy", + "LiteRtTopKWebGpuSampler_SampleToIdAndScoreBuffer", + "LiteRtTopKWebGpuSampler_UpdateConfig", + "LiteRtTopKWebGpuSampler_CanHandleInput", + "LiteRtTopKWebGpuSampler_HandlesInput", + "LiteRtTopKWebGpuSampler_SetInferenceFuncAndInputTensors", +] + APPLE_METAL_SAMPLER_SYMBOLS = [ "LiteRtTopKMetalSampler_Create", "LiteRtTopKMetalSampler_Destroy", @@ -158,6 +168,17 @@ def has_prebuilt_topk_samplers(upstream_tag: str) -> bool: return not is_at_least(upstream_tag, (0, 19, 0)) +def android_static_gpu_sampler_symbols(upstream_tag: str) -> list[str]: + """Sampler functions an Android libLiteRtLm.so must define itself. + + The v0.19 Android GPU accelerator runs on OpenCL, or on WebGPU when OpenCL + cannot be loaded, and the sampler is created for whichever it chose (#120). + """ + if has_prebuilt_topk_samplers(upstream_tag): + return [] + return ANDROID_OPENCL_SAMPLER_SYMBOLS + ANDROID_WEBGPU_SAMPLER_SYMBOLS + + def has_prebuilt_apple_metal_accelerator(upstream_tag: str) -> bool: """v0.19 Apple runtimes link their GPU accelerator and never open the prebuilt one.""" return not is_at_least(upstream_tag, (0, 19, 0)) diff --git a/tools/test_android_protobuf_policy.py b/tools/test_android_protobuf_policy.py index b242a87..b6661c8 100644 --- a/tools/test_android_protobuf_policy.py +++ b/tools/test_android_protobuf_policy.py @@ -12,7 +12,7 @@ import android_protobuf_policy as policy from litert_lm_symbols import ( - ANDROID_OPENCL_SAMPLER_SYMBOLS, + android_static_gpu_sampler_symbols, required_bridge_symbols, required_c_api_symbols, ) @@ -151,7 +151,7 @@ def setUpClass(cls): def compile(self, directory, arch, *, private=True, global_flag=True, missing=None, alignment=16384, dependency=None): directory.mkdir(parents=True, exist_ok=True) required = {symbol.decode() for symbol in required_c_api_symbols('v0.19.0') + required_bridge_symbols('v0.19.0')} - public = required | set(ANDROID_OPENCL_SAMPLER_SYMBOLS) | {'LiteRtAcceleratorImpl'} + public = required | set(android_static_gpu_sampler_symbols('v0.19.0')) | {'LiteRtAcceleratorImpl'} self.assertFalse(any('protobuf' in symbol for symbol in public)) source = directory / 'fixture.cc' source.write_text('''namespace google { namespace protobuf { namespace internal { diff --git a/tools/test_build_upstream_runtime.py b/tools/test_build_upstream_runtime.py index af39d66..db336cc 100644 --- a/tools/test_build_upstream_runtime.py +++ b/tools/test_build_upstream_runtime.py @@ -11,6 +11,8 @@ import build_upstream_runtime import package_upstream_prebuilts +from litert_lm_symbols import ANDROID_OPENCL_SAMPLER_SYMBOLS, ANDROID_WEBGPU_SAMPLER_SYMBOLS +from test_runtime_dependency_utils import write_elf_with_exports class BuildUpstreamRuntimeTest(unittest.TestCase): @@ -161,6 +163,43 @@ def test_desktop_build_selects_shared_runtime_only_for_v016_to_v018(self) -> Non self.assertEqual("--define=litert_lm_in_process_litert=true" in command, tag in {"v0.19.0", "v0.19.1"}) self.assertIn("--define=resolve_symbols_in_exec=false", command) + self.assertEqual("--define=litert_lm_android_webgpu_sampler=true" in command, + platform == "android" and tag in {"v0.19.0", "v0.19.1"}) + + def test_android_webgpu_sampler_define_links_the_static_sampler(self) -> None: + bazel = (build_upstream_runtime.REPO_ROOT / "native/bridge/BUILD.bazel").read_text() + setting = re.search( + r'name = "android_webgpu_sampler",\s*constraint_values = \["@platforms//os:android"\],' + r'\s*define_values = \{"([^"]+)": "true"\}', bazel) + self.assertIsNotNone(setting) + script = Path(build_upstream_runtime.__file__).read_text() + self.assertIn(f'"--define={setting.group(1)}=true"', script) + self.assertEqual(len(re.findall( + r'":android_webgpu_sampler": \["//runtime/components:top_k_webgpu_sampler_static"\]', + bazel)), 1) + + def test_android_v019_build_rejects_a_runtime_without_both_gpu_samplers(self) -> None: + both = ANDROID_OPENCL_SAMPLER_SYMBOLS + ANDROID_WEBGPU_SAMPLER_SYMBOLS + published = ANDROID_OPENCL_SAMPLER_SYMBOLS + [ + f"{symbol}_Static" for symbol in ANDROID_WEBGPU_SAMPLER_SYMBOLS] + with tempfile.TemporaryDirectory() as temp: + output = Path(temp) / "libLiteRtLm.so" + write_elf_with_exports(output, published) + with self.assertRaisesRegex(RuntimeError, r"LiteRtTopKWebGpuSampler_Create,.*#120"): + build_upstream_runtime.validate_android_gpu_samplers(output, "android", "v0.19.1") + build_upstream_runtime.validate_android_gpu_samplers(output, "android", "v0.18.1") + build_upstream_runtime.validate_android_gpu_samplers(output, "linux", "v0.19.1") + write_elf_with_exports(output, both) + build_upstream_runtime.validate_android_gpu_samplers(output, "android", "v0.19.1") + with patch.object(build_upstream_runtime, "validate_exported_symbols"), \ + patch.object(build_upstream_runtime, "validate_in_process_litert_linkage"), \ + patch.object(build_upstream_runtime, "validate_android_global_visibility"), \ + patch.object(build_upstream_runtime, "validate_android_protobuf_isolation"), \ + patch.object(build_upstream_runtime, "validate_android_gpu_environment_teardown"): + build_upstream_runtime.validate_runtime(output, "android", "v0.19.1") + write_elf_with_exports(output, published) + with self.assertRaisesRegex(RuntimeError, "#120"): + build_upstream_runtime.validate_runtime(output, "android", "v0.19.1") def test_in_process_litert_define_matches_bazel_config_setting(self) -> None: bazel = (build_upstream_runtime.REPO_ROOT / "native/bridge/BUILD.bazel").read_text() diff --git a/tools/test_runtime_dependency_utils.py b/tools/test_runtime_dependency_utils.py index ffac24b..09ee6d9 100644 --- a/tools/test_runtime_dependency_utils.py +++ b/tools/test_runtime_dependency_utils.py @@ -10,7 +10,9 @@ import runtime_dependency_utils from litert_lm_symbols import ( ANDROID_OPENCL_SAMPLER_SYMBOLS, + ANDROID_WEBGPU_SAMPLER_SYMBOLS, APPLE_METAL_SAMPLER_SYMBOLS, + android_static_gpu_sampler_symbols, ) from validate_runtime_dependencies import ( newest_glibc_requirement, @@ -225,6 +227,45 @@ def test_android_opencl_sampler_accepts_complete_plugin_api(self) -> None: self.assertEqual(validate_elf_dependencies(root, "v0.18.0"), 1) + def test_android_v019_runtime_requires_both_linked_gpu_samplers(self) -> None: + # sampler_factory.cc resolves the same seven functions for each family. + self.assertEqual( + ANDROID_WEBGPU_SAMPLER_SYMBOLS, + [symbol.replace("OpenCl", "WebGpu") for symbol in ANDROID_OPENCL_SAMPLER_SYMBOLS], + ) + self.assertEqual(android_static_gpu_sampler_symbols("v0.18.1"), []) + both = ANDROID_OPENCL_SAMPLER_SYMBOLS + ANDROID_WEBGPU_SAMPLER_SYMBOLS + for tag in ("v0.19.0", "v0.19.1", "v0.20.0"): + self.assertEqual(android_static_gpu_sampler_symbols(tag), both) + # The published v0.19.1-4 shape: OpenCL functions plus the WebGPU + # function pointers sampler_factory.cc always defines. + published = ANDROID_OPENCL_SAMPLER_SYMBOLS + [ + f"{symbol}_Static" for symbol in ANDROID_WEBGPU_SAMPLER_SYMBOLS + ] + for arch in ("arm64", "x64"): + with self.subTest(arch=arch), tempfile.TemporaryDirectory() as temp: + root = Path(temp) + library = root / "bin" / "android" / arch / "libLiteRtLm.so" + library.parent.mkdir(parents=True) + write_elf_with_exports(library, published) + with self.assertRaises(SystemExit) as raised: + validate_elf_dependencies(root, "v0.19.1") + message = str(raised.exception) + for symbol in ANDROID_WEBGPU_SAMPLER_SYMBOLS: + self.assertRegex(message, rf"{symbol}(,|$|\n)") + self.assertNotIn("LiteRtTopKOpenClSampler", message) + write_elf_with_exports(library, both) + with self.assertRaises(SystemExit) as raised: + validate_elf_dependencies(root, "v0.19.1") + self.assertNotIn("Sampler", str(raised.exception)) + write_elf_with_exports(library, ANDROID_WEBGPU_SAMPLER_SYMBOLS) + with self.assertRaisesRegex(SystemExit, "LiteRtTopKOpenClSampler_Create"): + validate_elf_dependencies(root, "v0.19.1") + write_elf_with_exports(library, published) + with self.assertRaises(SystemExit) as raised: + validate_elf_dependencies(root, "v0.18.1") + self.assertNotIn("Sampler", str(raised.exception)) + def test_ios_metal_sampler_requires_complete_plugin_api(self) -> None: with tempfile.TemporaryDirectory() as temp: root = Path(temp) diff --git a/tools/validate_runtime_dependencies.py b/tools/validate_runtime_dependencies.py index 56b5520..e2c68da 100644 --- a/tools/validate_runtime_dependencies.py +++ b/tools/validate_runtime_dependencies.py @@ -20,6 +20,7 @@ ) from litert_lm_symbols import ( ANDROID_OPENCL_SAMPLER_SYMBOLS, + android_static_gpu_sampler_symbols, APPLE_METAL_ACCELERATOR_SYMBOLS, APPLE_METAL_SAMPLER_SYMBOLS, ) @@ -118,6 +119,15 @@ def validate_elf_dependencies(root: Path, upstream_tag: str) -> int: validate_android_protobuf_isolation(library, platform, upstream_tag) except RuntimeError as error: errors.append(str(error)) + missing = sorted( + set(android_static_gpu_sampler_symbols(upstream_tag)) + - elf_exported_symbols(library) + ) + if missing: + errors.append( + f"{library.relative_to(root).as_posix()} does not " + "define the GPU sampler functions: " + ", ".join(missing) + ) if library.name == "libLiteRtTopKOpenClSampler.so": exported = elf_exported_symbols(library) missing = sorted(set(ANDROID_OPENCL_SAMPLER_SYMBOLS) - exported)