Run the chat-app Web smoke in Safari and Firefox and add a WebGPU generation E2E (#1025)

doc/testing_matrix.md

@@ -161,7 +161,7 @@ Pick targeted rows based on the touched surface:
 | Tool-result media (`LlamaToolResultContent.content`, `LlamaToolOutput`) | `root-vm`, `tool-result-image-smoke`, validation `qwen3vl-gguf-cpu` / `-metal` (`M01`) |
 | LiteRT-LM embedding engine | `litert-lm-multimodal-embedding`; includes the documented local-only upstream WebGPU parity diagnostic |
 | LiteRT-LM native backend | `litert-lm-lifecycle`, `litert-lm-engine-smoke`, `litert-lm-chat-features-smoke`, `litert-lm-json-schema`, `litert-lm-asr-smoke`, `litert-lm-asr-preservation`, `litert-lm-media-preservation`, `litert-lm-utility-preservation` |
-| Web bridge bootstrap or interop | `web-bridge-smoke`, `web-mock-chat-smoke`, `web-real-model-smoke`, `web-tool-penalty-smoke`, `web-lazy-grammar`, `web-tool-choice-grammar` |
+| Web bridge bootstrap or interop | `web-bridge-smoke`, `web-mock-chat-smoke`, `web-real-model-smoke`, `web-real-model-smoke-safari`, `web-real-model-smoke-firefox`, `web-tool-penalty-smoke`, `web-lazy-grammar`, `web-webgpu-generation`, `web-tool-choice-grammar` |
 | WebGPU multimodal | `webgpu-multimodal-regression`, `web-speech-task-cancel`, plus `web-speech-to-text-smoke` for typed Qwen3-ASR and `web-text-to-speech-smoke` for typed Qwen3-TTS |
 | Web microphone recorder (`record` bump, permission or interruption handling) | `web-microphone-lifecycle`, `chat-app-web-microphone-device-smoke` for real hardware |
 | Large WebGPU GGUF / wasm64 selection | `gemma4-webgpu-mem64` |
@@ -233,7 +233,7 @@ The matrix is designed to cover these essential axes:
 | Axis | Covered by |
 | --- | --- |
 | llama.cpp native GGUF | `root-vm`, `native-prompt-reuse-parity`, `native-inference-benchmark`, `gguf-chat-features-smoke`, `gguf-audio-chat-smoke`, `decision-model-smoke` |
-| llama.cpp WebGPU GGUF | `web-bridge-smoke`, `web-mock-chat-smoke`, `web-real-model-smoke`, `web-tool-penalty-smoke`, `web-lazy-grammar`, `web-tool-choice-grammar`, `webgpu-multimodal-regression`, `web-speech-to-text-smoke`, `web-microphone-lifecycle`, `web-text-to-speech-smoke`, `web-speech-task-cancel`, `gemma4-webgpu-mem64`, `decision-model-web-smoke` |
+| llama.cpp WebGPU GGUF | `web-bridge-smoke`, `web-mock-chat-smoke`, `web-real-model-smoke`, `web-real-model-smoke-safari`, `web-real-model-smoke-firefox`, `web-tool-penalty-smoke`, `web-lazy-grammar`, `web-webgpu-generation`, `web-tool-choice-grammar`, `webgpu-multimodal-regression`, `web-speech-to-text-smoke`, `web-microphone-lifecycle`, `web-text-to-speech-smoke`, `web-speech-task-cancel`, `gemma4-webgpu-mem64`, `decision-model-web-smoke` |
 | LiteRT-LM native `.litertlm` | `litert-lm-engine-smoke`, `litert-lm-chat-features-smoke`, `native-hook-bundles` |
 | LiteRT-LM web `.litertlm` | `gemma4-litert-web` |
 | Model families | Qwen 2.5 prompt reuse, Qwen 3/3.5 chat/multimodal, Gemma 4 tool/thinking/mem64/LiteRT-LM/audio, Laya ModernBERT decision encoder and head |
@@ -1480,6 +1480,46 @@ This qualifies the Web grammar plumbing on the selected weights, not hardware
 WebGPU or LFM2 auto accuracy/enforcement (#1715). The latter still needs its
 separate template contract and false-probe restriction.
 
+## Safari and Firefox chat-app smoke
+
+`web-real-model-smoke-safari` and `web-real-model-smoke-firefox` run
+`chat-app-web-real-model-smoke --browser safari|firefox` through
+`tool/testing/safari_webdriver.py`. The runner serves the repo with
+`--coep require-corp`, because both browsers expose `SharedArrayBuffer` only to
+cross-origin isolated pages. `--backend webgpu` adds `--backend-index 0
+--gpu-layers 99 --expect-gpu-active true`, so a silent CPU fallback fails; the
+default stays CPU Wasm. It requests 99 layers, but llamadart caps some models
+(Qwen3.5-0.8B at 2 layers): use an uncapped model such as stories15M with
+`--allow-any-response` to claim full offload. Run each with `--dry-run` first.
+
+- Safari needs Settings > Developer > Allow Remote Automation and an unlocked
+  screen; a hidden page fails before the model loads.
+- Firefox runs the binary in `FIREFOX_EXECUTABLE` through the geckodriver in
+  `GECKODRIVER` (default: `PATH`). geckodriver's script sandbox sees page
+  objects through wrappers, so the helper installs its capture hooks and runs
+  its page-state reads as page `<script>` elements. `web-real-model-smoke-firefox`
+  is unqualified until a real Firefox run passes.
+- Both cover plain chat only: no tool-call, speech or mmproj checks.
+- The Dart web scenarios (`web-lazy-grammar`, `web-webgpu-generation` and the
+  other `dart test -p chrome*` scenarios) reject `--browser safari|firefox`:
+  `package:test` pages are never cross-origin isolated.
+
+## Web WebGPU generation
+
+`web-webgpu-generation` loads one GGUF at `gpuLayers: 99` and at `gpuLayers: 0`
+through the pinned bridge in direct and worker runtimes, and requires the first
+eight greedy tokens to match with no replacement characters. Later divergence is
+printed as `WEBGPU_GENERATION_MATCH`, not failed. Use a small model that
+llamadart does not cap on WebGPU (not Qwen3.5-0.8B). Stage
+`build/webgpu_generation_e2e/config.json`:
+
+```json
+{"assets":"bridge/","model":"models/stories15M.gguf"}
+```
+
+Copy the model beside it; the runner fetches the pinned assets into `bridge/`
+and uses `chrome-webgpu`. Without a WebGPU adapter the suite fails.
+
 ## Web tool-choice grammar
 
 `web-tool-choice-grammar` ([#1673](https://github.com/leehack/llamadart/issues/1673))

test/e2e/webgpu/webgpu_generation_e2e_test.dart

@@ -0,0 +1,154 @@
+/// Real pinned-asset text generation at full WebGPU offload, compared with the
+/// CPU Wasm runtime, in direct and worker runtimes (#1025).
+///
+/// Stage build/webgpu_generation_e2e/config.json with {"assets":"bridge/",
+/// "model":"models/stories15M.gguf"} beside the pinned bridge files and the
+/// model, then run the web-webgpu-generation scenario. The model must not be
+/// one llamadart caps on WebGPU (Qwen3.5-0.8B).
+@TestOn('browser')
+@Tags(['local-only', 'e2e'])
+@Timeout(Duration(minutes: 10))
+library;
+
+import 'dart:convert';
+import 'dart:js_interop';
+import 'dart:js_interop_unsafe';
+
+import 'package:llamadart/llamadart.dart';
+import 'package:llamadart/src/backends/webgpu/interop.dart';
+import 'package:llamadart/src/backends/webgpu/webgpu_backend.dart';
+import 'package:test/test.dart';
+import 'package:web/web.dart' show window;
+
+const _greedy = GenerationParams(
+  maxTokens: 32,
+  temp: 0,
+  topK: 1,
+  topP: 1,
+  penalty: 1,
+  seed: 7,
+);
+const _prompt = 'Once upon a time';
+
+void main() {
+  final configUrl = Uri.base.resolve(
+    '../../../build/webgpu_generation_e2e/config.json',
+  );
+  late Map<String, dynamic> config;
+  late JSFunction constructor;
+  String asset(String name) =>
+      configUrl.resolve(config['assets'] as String).resolve(name).toString();
+
+  LlamaEngine newEngine(bool worker) => LlamaEngine(
+    WebGpuLlamaBackend(
+      bridgeFactory: ([_]) => constructor.callAsConstructor<LlamaWebGpuBridge>(
+        JSObject()
+          ..['disableWorker'] = (!worker).toJS
+          ..['workerUrl'] = asset('llama_webgpu_bridge_worker.js').toJS
+          ..['coreModuleUrl'] = asset('llama_webgpu_core.js').toJS
+          ..['wasmUrl'] = asset('llama_webgpu_core.wasm').toJS,
+      ),
+    ),
+  );
+
+  Future<String> generate(bool worker, ModelParams params, int tokens) async {
+    final engine = newEngine(worker);
+    try {
+      await engine.loadModelFromUrl(
+        configUrl.resolve(config['model'] as String).toString(),
+        modelParams: params,
+      );
+      final metadata = await engine.getMetadata();
+      expect(
+        metadata['llamadart.webgpu.execution'],
+        worker ? 'worker' : 'main-thread',
+      );
+      if (params.device == ComputeDevice.gpu) {
+        expect(await engine.getBackendName(), contains('WebGPU'));
+        expect(metadata['llamadart.webgpu.n_gpu_layers'], '99');
+      }
+      final watch = Stopwatch()..start();
+      final text = await engine
+          .generate(_prompt, params: _greedy.copyWith(maxTokens: tokens))
+          .join();
+      print(
+        'WEBGPU_GENERATION ${jsonEncode({'runtime': worker ? 'worker' : 'direct', 'device': params.device.name, 'backend': await engine.getBackendName(), 'tokens': tokens, 'ms': watch.elapsedMilliseconds, 'text': text})}',
+      );
+      return text;
+    } finally {
+      await engine.dispose();
+    }
+  }
+
+  setUpAll(() async {
+    final response = await window.fetch(configUrl.toString().toJS).toDart;
+    expect(response.ok, isTrue, reason: 'Missing $configUrl');
+    config =
+        jsonDecode((await response.text().toDart).toDart)
+            as Map<String, dynamic>;
+    final module = await importModule(
+      asset('llama_webgpu_bridge.js').toJS,
+    ).toDart;
+    constructor = module['LlamaWebGpuBridge']! as JSFunction;
+    final adapter = await _adapterInfo();
+    print('WEBGPU_ADAPTER ${jsonEncode(adapter)}');
+    expect(
+      adapter['error'],
+      isNull,
+      reason: 'This test needs a WebGPU adapter.',
+    );
+  });
+
+  for (final worker in [false, true]) {
+    test('${worker ? 'worker' : 'direct'} full WebGPU offload generates the '
+        'CPU runtime text', () async {
+      const cpu = ModelParams(
+        contextSize: 512,
+        gpuLayers: 0,
+        device: ComputeDevice.cpu,
+        numberOfThreads: 2,
+        numberOfThreadsBatch: 2,
+      );
+      const gpu = ModelParams(
+        contextSize: 512,
+        gpuLayers: 99,
+        device: ComputeDevice.gpu,
+        numberOfThreads: 2,
+        numberOfThreadsBatch: 2,
+      );
+      // Greedy text can diverge late across GPU drivers, so the gate is the
+      // first tokens; the full runs are recorded for the qualification table.
+      final cpuHead = await generate(worker, cpu, 8);
+      final gpuHead = await generate(worker, gpu, 8);
+      expect(gpuHead, isNotEmpty);
+      expect(gpuHead, isNot(contains('�')));
+      expect(gpuHead, cpuHead);
+      final cpuText = await generate(worker, cpu, 32);
+      final gpuText = await generate(worker, gpu, 32);
+      expect(gpuText, isNot(contains('�')));
+      print(
+        'WEBGPU_GENERATION_MATCH ${jsonEncode({'runtime': worker ? 'worker' : 'direct', 'fullTextEqual': gpuText == cpuText})}',
+      );
+    });
+  }
+}
+
+Future<Map<String, Object?>> _adapterInfo() async {
+  final gpu = (window as JSObject)
+      .getProperty<JSObject?>('navigator'.toJS)
+      ?.getProperty<JSAny?>('gpu'.toJS);
+  if (gpu == null || !gpu.isA<JSObject>()) return {'error': 'no navigator.gpu'};
+  final adapter = await (gpu as JSObject)
+      .callMethod<JSPromise<JSAny?>>('requestAdapter'.toJS)
+      .toDart;
+  if (adapter == null) return {'error': 'no adapter'};
+  final info = (adapter as JSObject).getProperty<JSObject?>('info'.toJS);
+  String? field(String name) =>
+      (info?.getProperty<JSAny?>(name.toJS) as JSString?)?.toDart;
+  return {
+    'vendor': field('vendor'),
+    'architecture': field('architecture'),
+    'device': field('device'),
+    'description': field('description'),
+  };
+}