diff --git a/CHANGELOG.md b/CHANGELOG.md index bdb0cab..17d0cab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## Unreleased +- A load on a page that is not cross-origin isolated rejects at once with `CrossOriginIsolationError`, which names `Cross-Origin-Embedder-Policy: require-corp` and that `credentialless` does not isolate Safari, instead of hanging until the core init timeout. The `threads_capped_no_coi` note now appears only for a core loaded with `allowSingleThreadCore`. +- A load with `nGpuLayers: 0` keeps WebGPU out of the model and context backends, so CPU decoding on Firefox no longer waits on its 100 ms WebGPU queue ticks at every sync (about 50x faster on a small model). `splitMode: 3` at 0 layers still loads. +- A load with `nGpuLayers > 0` that ends up on the CPU adds the runtime note `gpu_unavailable:` and one warning, such as `adapter_missing_shader_f16` for Chrome on NVIDIA drivers older than 615.71. +- `allowSingleThreadCore: true` lets a `coreModuleUrl` core built without pthreads load on a page that is not cross-origin isolated, capped to one thread. - `CompletionOptions.thinkingBudget.pauseGrammar: false` keeps a grammar applied inside a reasoning block, so a grammar that takes the block and its end tag itself, such as one for a prompt that ends inside the block, sees the budget's forced ``. It needs the new `thinkingBudgetPauseGrammar` capability; omitted, the grammar pauses as before. - WebGPU text-to-speech applies the GGUF's suppress tokens in the talker sampler, as upstream does. Qwen3-TTS no longer fails with `code out of range` when a codec control token is sampled. diff --git a/docs/api.md b/docs/api.md index 879666f..990d47c 100644 --- a/docs/api.md +++ b/docs/api.md @@ -154,9 +154,10 @@ it. | `coreModuleUrlMem64` | `string` | URL for the optional wasm64 Emscripten JS loader. | | `wasmUrl` | `string` | URL for the wasm32 `llama_webgpu_core.wasm` binary. | | `wasmUrlMem64` | `string` | URL for the optional wasm64 wasm binary. | -| `coreModuleFactory` | `function \| Promise` | Preloaded Emscripten factory. Supplying this disables worker mode because factories cannot be transferred to the worker. | +| `coreModuleFactory` | `LlamaCoreModuleFactory \| Promise` | Preloaded Emscripten factory. Supplying this disables worker mode because factories cannot be transferred to the worker. It also skips the cross-origin isolation check, so a custom pthread core on a page that is not isolated still waits for `coreInitTimeoutMs`. | | `workerUrl` | `string` | URL for `llama_webgpu_bridge_worker.js`. Defaults to a sibling of the bridge module when possible. | | `disableWorker` | `boolean` | Force direct runtime mode. | +| `allowSingleThreadCore` | `boolean` | Set for a `coreModuleUrl` core built without pthreads (`WEBGPU_BRIDGE_PTHREADS=0`). It then loads on a page that is not cross-origin isolated, capped to one thread (`threads_capped_no_coi`), instead of rejecting with `CrossOriginIsolationError`. | | `preferMemory64` | `boolean` | Prefer the wasm64 core when available. Defaults to `true`; the runtime can still fall back to wasm32. | | `workerRequestTimeoutMs` | `number` | Default timeout for worker RPC calls, 2 minutes unless a method has its own default. It also sets the `embed`/`embedBatch` budget, which defaults to 10 minutes when an input has image or audio parts. | | `workerInitTimeoutMs` | `number` | Timeout for worker initialization. | @@ -167,7 +168,7 @@ it. | `workerGenerationStallTimeoutMs` | `number` | Stall timeout for worker generation after no token events arrive; clamped by the bridge. | | `coreInitTimeoutMs` | `number` | Timeout while initializing the Emscripten core module. | | `cacheName` | `string` | Cache Storage name used by model prefetch/load helpers. | -| `threads` | `number` | Requested llama.cpp thread count. The bridge caps this to the compiled pthread pool and runtime isolation support. | +| `threads` | `number` | Requested llama.cpp thread count. The bridge caps this to the compiled pthread pool, and to one thread for a single-thread core on a page that is not isolated. | | `threadsBatch` | `number` | Requested batch thread count; defaults to `threads`. | | `threadPoolSize` | `number` | Hint for the compiled pthread pool size, used before the core can report it. | | `nBatch` | `number` | llama.cpp batch size override. | @@ -222,7 +223,7 @@ preserves the model's resolved native load settings, including sequence count, batch sizes, memory mapping/locking, KV cache types, flash attention, RoPE and device split settings. It changes GPU placement to CPU and retains the existing memory32 preference and transfer safeguards for recovery. Thread counts remain -subject to the replacement core's pool and browser isolation limits. Recovery +subject to the replacement core's pool. Recovery continues to refuse active LoRA adapters and speculative generation. ### `loadModelFromUrl(url, options?)` @@ -1153,7 +1154,7 @@ include: | `llamadart.webgpu.core_variant` | `wasm32`, `wasm64`, or initialization state. | | `llamadart.webgpu.model_source` | Model load source, such as network/cache/remote fetch. | | `llamadart.webgpu.model_cache_state` | Cache Storage state for the loaded model. | -| `llamadart.webgpu.runtime_notes` | Semicolon-separated bridge notes such as thread caps or fallback reasons. | +| `llamadart.webgpu.runtime_notes` | Semicolon-separated bridge notes such as thread caps or fallback reasons. `gpu_unavailable:` (`no_webgpu`, `no_adapter`, `adapter_missing_shader_f16`, `unknown`) says why a load with `nGpuLayers > 0` runs on the CPU; the bridge also logs one warning. Chrome hides `shader-f16` on NVIDIA drivers older than 615.71, and ggml-webgpu then registers no device. | | `llamadart.webgpu.mmproj_loaded` | `1` when a projector is loaded. | | `llamadart.webgpu.supports_vision` | `1` when the active projector supports vision. | | `llamadart.webgpu.supports_audio` | `1` when the active projector supports audio. | @@ -1242,7 +1243,10 @@ this function themselves. - Large single-file model loads and pthread-backed runtime paths require cross-origin isolation (`Cross-Origin-Opener-Policy: same-origin` and `Cross-Origin-Embedder-Policy: require-corp`) so `SharedArrayBuffer` is - available. + available. `credentialless` does not isolate Safari. On a page that cannot + share memory with workers, a load rejects immediately with + `CrossOriginIsolationError` (`code: 'CROSS_ORIGIN_ISOLATION_REQUIRED'`) + instead of waiting for the core init timeout. - Serve `llama_webgpu_core.js` and `llama_webgpu_core.wasm` from URLs reachable by both the page and worker. If using wasm64, also serve the `_mem64` files. - Keep `llama_webgpu_bridge.js`, `llama_webgpu_bridge_worker.js`, the core JS, diff --git a/js/llama_webgpu_bridge.d.ts b/js/llama_webgpu_bridge.d.ts index 7b269ae..319c0b6 100644 --- a/js/llama_webgpu_bridge.d.ts +++ b/js/llama_webgpu_bridge.d.ts @@ -7,7 +7,10 @@ export interface LlamaWebGpuBridgeConfig { coreModuleUrlMem64?: string; wasmUrl?: string; wasmUrlMem64?: string; + /** A custom factory skips the cross-origin isolation check, so a custom pthread core on a page that is not isolated still waits for `coreInitTimeoutMs`. */ coreModuleFactory?: LlamaCoreModuleFactory | Promise; + /** Set for a `coreModuleUrl` core built without pthreads (`WEBGPU_BRIDGE_PTHREADS=0`): it loads on a page that is not cross-origin isolated, capped to one thread, instead of rejecting with `CrossOriginIsolationError`. */ + allowSingleThreadCore?: boolean; workerUrl?: string; disableWorker?: boolean; preferMemory64?: boolean; @@ -298,6 +301,13 @@ export class EmbeddingError extends Error { constructor(code: EmbeddingErrorCode, message: string); } +export type CrossOriginIsolationErrorCode = 'CROSS_ORIGIN_ISOLATION_REQUIRED'; +/** Thrown by a load or core init on a page that cannot share memory with workers, before the core starts. */ +export class CrossOriginIsolationError extends Error { + code: CrossOriginIsolationErrorCode; + constructor(message?: string); +} + export interface NextTokenScoreOptions { /** Token ids to score, in the order the result lists them. */ candidates?: ArrayLike; diff --git a/js/llama_webgpu_bridge.js b/js/llama_webgpu_bridge.js index 102b988..8ec34e3 100644 --- a/js/llama_webgpu_bridge.js +++ b/js/llama_webgpu_bridge.js @@ -1402,16 +1402,53 @@ function isSafariUserAgent(userAgent) { const hasOtherBrowserToken = /(Chrome|Chromium|CriOS|Edg|OPR|Firefox|FxiOS)\//.test(userAgent); return hasSafariToken && !hasOtherBrowserToken; } -function isCrossOriginIsolatedRuntime() { +function canPostSharedMemory() { + if (typeof MessageChannel !== "function" || typeof WebAssembly === "undefined") { + return true; + } + let channel = null; try { - if (typeof globalThis.crossOriginIsolated === "boolean") { - return globalThis.crossOriginIsolated; - } + channel = new MessageChannel(); + channel.port1.postMessage(new WebAssembly.Memory({ initial: 1, maximum: 1, shared: true })); + return true; + } catch (_) { + return false; + } finally { + channel?.port1.close(); + channel?.port2.close(); + } +} +async function describeWebGpuUnavailable() { + const gpu = globalThis.navigator?.gpu; + if (!gpu || typeof gpu.requestAdapter !== "function") { + return { reason: "no_webgpu", adapter: "" }; + } + let adapter = null; + try { + adapter = await gpu.requestAdapter(); } catch (_) { + adapter = null; } - return true; + if (!adapter) { + return { reason: "no_adapter", adapter: "" }; + } + return { + reason: adapter.features?.has?.("shader-f16") ? "unknown" : "adapter_missing_shader_f16", + adapter: [adapter.info?.vendor, adapter.info?.architecture].filter(Boolean).join("/") + }; } +// js/src/internal/isolation.ts +var CROSS_ORIGIN_ISOLATION_REQUIRED = "CROSS_ORIGIN_ISOLATION_REQUIRED"; +var CrossOriginIsolationError = class extends Error { + constructor(message = CROSS_ORIGIN_ISOLATION_MESSAGE) { + super(message); + this.name = "CrossOriginIsolationError"; + this.code = CROSS_ORIGIN_ISOLATION_REQUIRED; + } +}; +var CROSS_ORIGIN_ISOLATION_MESSAGE = 'The llama.cpp web core needs shared memory, which this page cannot use because it is not cross-origin isolated. Serve the page with "Cross-Origin-Opener-Policy: same-origin" and "Cross-Origin-Embedder-Policy: require-corp". "credentialless" does not isolate Safari.'; + // js/src/internal/image.ts async function decodeImageBytesToRgb(bytes, options = {}) { const sourceBytes = toUint8Array(bytes); @@ -2751,6 +2788,9 @@ var LlamaWebGpuBridgeRuntime = class { this._applyCoreLogLevel(); return this._core; } + if (!this._config.coreModuleFactory && this._config.allowSingleThreadCore !== true && !canPostSharedMemory()) { + throw new CrossOriginIsolationError(); + } const candidates = []; if (this._config.coreModuleFactory) { candidates.push({ @@ -2942,6 +2982,18 @@ var LlamaWebGpuBridgeRuntime = class { if (Number.isFinite(requestedGpuLayers)) { this._nGpuLayers = Math.trunc(requestedGpuLayers); } + if (this._nGpuLayers > 0 && !this._gpuActive) { + const unavailable = await describeWebGpuUnavailable(); + throwIfAborted(operationSignal, abortMessage); + this._pushRuntimeNote(`gpu_unavailable:${unavailable.reason}`); + const detail = { + no_webgpu: "this browser has no navigator.gpu", + no_adapter: "the browser returned no WebGPU adapter", + adapter_missing_shader_f16: `the ${unavailable.adapter || "WebGPU"} adapter lacks shader-f16`, + unknown: "the core registered no WebGPU device" + }[unavailable.reason]; + this._emitLogger("warn", `WebGPU is unavailable (${detail}); loading on the WebAssembly CPU runtime.`); + } const isCpuModelMode = this._nGpuLayers === 0; const requestedBatch = Number(options.nBatch); this._nBatch = Number.isFinite(requestedBatch) && requestedBatch > 0 ? Math.max(32, Math.trunc(requestedBatch)) : isCpuModelMode ? Math.min(this._nCtx, 512) : 0; @@ -2963,14 +3015,10 @@ var LlamaWebGpuBridgeRuntime = class { if (Number.isFinite(this._threadPoolSizeHint) && this._threadPoolSizeHint > 0) { this._pushRuntimeNote(`thread_pool_size:${this._threadPoolSizeHint}`); } - if (!isCrossOriginIsolatedRuntime()) { + if (!canPostSharedMemory()) { this._runtimeNotes.push("threads_capped_no_coi"); - if (this._threads > 1) { - this._threads = 1; - } - if (this._threadsBatch > 1) { - this._threadsBatch = 1; - } + this._threads = Math.min(this._threads, 1); + this._threadsBatch = Math.min(this._threadsBatch, 1); } this._pushRuntimeNote(`threads_batch:${this._threadsBatch}`); if (this._nBatch > 0) { @@ -5346,7 +5394,7 @@ var BridgeWorkerProxy = class { this._pending.delete(id); if (type === "error") { const errorMessage = String(message.message || "Worker request failed"); - const workerError = message.embeddingErrorCode ? new EmbeddingError(message.embeddingErrorCode, errorMessage) : new Error(errorMessage); + const workerError = message.embeddingErrorCode ? new EmbeddingError(message.embeddingErrorCode, errorMessage) : message.errorCode === CROSS_ORIGIN_ISOLATION_REQUIRED ? new CrossOriginIsolationError(errorMessage) : new Error(errorMessage); if (isDecisionErrorCode(message.decisionErrorCode)) { withDecisionErrorCode(workerError, message.decisionErrorCode); } @@ -6472,6 +6520,7 @@ var LlamaWebGpuBridge = class { coreModuleUrl: typeof config.coreModuleUrl === "string" ? config.coreModuleUrl : void 0, coreModuleUrlMem64: typeof config.coreModuleUrlMem64 === "string" ? config.coreModuleUrlMem64 : void 0, preferMemory64: typeof config.preferMemory64 === "boolean" ? config.preferMemory64 : void 0, + allowSingleThreadCore: config.allowSingleThreadCore === true ? true : void 0, threads: Number(config.threads) > 0 ? Number(config.threads) : void 0, threadsBatch: Number(config.threadsBatch) > 0 ? Number(config.threadsBatch) : void 0, threadPoolSize: Number(config.threadPoolSize) > 0 ? Number(config.threadPoolSize) : void 0, @@ -8261,6 +8310,7 @@ function installBridgeWorkerHost() { id, message: serializeWorkerError(error), ...error instanceof EmbeddingError ? { embeddingErrorCode: error.code } : {}, + ...error instanceof CrossOriginIsolationError ? { errorCode: CROSS_ORIGIN_ISOLATION_REQUIRED } : {}, ...decisionErrorCode ? { decisionErrorCode } : {}, state }); @@ -8628,6 +8678,7 @@ if (typeof window !== "undefined") { } } export { + CrossOriginIsolationError, EmbeddingError, LlamaWebGpuBridge, enableBridgeWorkerHost diff --git a/js/src/bridge.ts b/js/src/bridge.ts index 83a763e..865de7d 100644 --- a/js/src/bridge.ts +++ b/js/src/bridge.ts @@ -1460,6 +1460,7 @@ export class LlamaWebGpuBridge implements PublicLlamaWebGpuBridge { preferMemory64: typeof config.preferMemory64 === 'boolean' ? config.preferMemory64 : undefined, + allowSingleThreadCore: config.allowSingleThreadCore === true ? true : undefined, threads: Number(config.threads) > 0 ? Number(config.threads) : undefined, threadsBatch: Number(config.threadsBatch) > 0 ? Number(config.threadsBatch) diff --git a/js/src/internal/environment.ts b/js/src/internal/environment.ts index c373d6b..2395086 100644 --- a/js/src/internal/environment.ts +++ b/js/src/internal/environment.ts @@ -10,15 +10,55 @@ export function isSafariUserAgent(userAgent: unknown): boolean { return hasSafariToken && !hasOtherBrowserToken; } -export function isCrossOriginIsolatedRuntime(): boolean { +// A pthread core preallocates workers and posts its shared memory to them. +// Browsers refuse that post (DataCloneError) unless the page is cross-origin +// isolated, and core init then never settles. Probing the post itself, rather +// than `crossOriginIsolated`, also accepts pages that enable shared memory +// another way, such as Chrome's SharedArrayBufferUnrestrictedAccessAllowed. +export function canPostSharedMemory(): boolean { + if (typeof MessageChannel !== 'function' || typeof WebAssembly === 'undefined') { + return true; + } + + let channel: MessageChannel | null = null; + try { + channel = new MessageChannel(); + channel.port1.postMessage(new WebAssembly.Memory({ initial: 1, maximum: 1, shared: true })); + return true; + } catch (_) { + return false; + } finally { + channel?.port1.close(); + channel?.port2.close(); + } +} + +export type WebGpuUnavailable = { + reason: 'no_webgpu' | 'no_adapter' | 'adapter_missing_shader_f16' | 'unknown'; + adapter: string; +}; + +// ggml-webgpu registers its device only for an adapter with shader-f16 and +// logs nothing otherwise, so the page-side adapter is the only place that +// can say why a GPU load runs on the CPU. +export async function describeWebGpuUnavailable(): Promise { + const gpu = (globalThis.navigator as { gpu?: { requestAdapter?: () => Promise } } | undefined)?.gpu; + if (!gpu || typeof gpu.requestAdapter !== 'function') { + return { reason: 'no_webgpu', adapter: '' }; + } + + let adapter = null; try { - if (typeof globalThis.crossOriginIsolated === 'boolean') { - return globalThis.crossOriginIsolated; - } + adapter = await gpu.requestAdapter(); } catch (_) { - // ignore environment probing failures + adapter = null; + } + if (!adapter) { + return { reason: 'no_adapter', adapter: '' }; } - // Assume isolated in runtimes that do not expose the signal. - return true; + return { + reason: adapter.features?.has?.('shader-f16') ? 'unknown' : 'adapter_missing_shader_f16', + adapter: [adapter.info?.vendor, adapter.info?.architecture].filter(Boolean).join('/'), + }; } diff --git a/js/src/internal/isolation.ts b/js/src/internal/isolation.ts new file mode 100644 index 0000000..709a391 --- /dev/null +++ b/js/src/internal/isolation.ts @@ -0,0 +1,16 @@ +// Raised before core init when the page cannot share memory with workers. +export const CROSS_ORIGIN_ISOLATION_REQUIRED = 'CROSS_ORIGIN_ISOLATION_REQUIRED'; + +export class CrossOriginIsolationError extends Error { + declare code: typeof CROSS_ORIGIN_ISOLATION_REQUIRED; + constructor(message: string = CROSS_ORIGIN_ISOLATION_MESSAGE) { + super(message); + this.name = 'CrossOriginIsolationError'; + this.code = CROSS_ORIGIN_ISOLATION_REQUIRED; + } +} + +export const CROSS_ORIGIN_ISOLATION_MESSAGE = + 'The llama.cpp web core needs shared memory, which this page cannot use because it is not ' + + 'cross-origin isolated. Serve the page with "Cross-Origin-Opener-Policy: same-origin" and ' + + '"Cross-Origin-Embedder-Policy: require-corp". "credentialless" does not isolate Safari.'; diff --git a/js/src/llama_webgpu_bridge.d.ts b/js/src/llama_webgpu_bridge.d.ts index 2162de0..0657150 100644 --- a/js/src/llama_webgpu_bridge.d.ts +++ b/js/src/llama_webgpu_bridge.d.ts @@ -5,7 +5,10 @@ export interface LlamaWebGpuBridgeConfig { coreModuleUrlMem64?: string; wasmUrl?: string; wasmUrlMem64?: string; + /** A custom factory skips the cross-origin isolation check, so a custom pthread core on a page that is not isolated still waits for `coreInitTimeoutMs`. */ coreModuleFactory?: LlamaCoreModuleFactory | Promise; + /** Set for a `coreModuleUrl` core built without pthreads (`WEBGPU_BRIDGE_PTHREADS=0`): it loads on a page that is not cross-origin isolated, capped to one thread, instead of rejecting with `CrossOriginIsolationError`. */ + allowSingleThreadCore?: boolean; workerUrl?: string; disableWorker?: boolean; preferMemory64?: boolean; @@ -296,6 +299,13 @@ export class EmbeddingError extends Error { constructor(code: EmbeddingErrorCode, message: string); } +export type CrossOriginIsolationErrorCode = 'CROSS_ORIGIN_ISOLATION_REQUIRED'; +/** Thrown by a load or core init on a page that cannot share memory with workers, before the core starts. */ +export class CrossOriginIsolationError extends Error { + code: CrossOriginIsolationErrorCode; + constructor(message?: string); +} + export interface NextTokenScoreOptions { /** Token ids to score, in the order the result lists them. */ candidates?: ArrayLike; diff --git a/js/src/llama_webgpu_bridge.js b/js/src/llama_webgpu_bridge.js index ab9b854..05d4085 100644 --- a/js/src/llama_webgpu_bridge.js +++ b/js/src/llama_webgpu_bridge.js @@ -21,3 +21,4 @@ if (typeof window !== 'undefined') { export { LlamaWebGpuBridge, enableBridgeWorkerHost }; export { EmbeddingError } from './internal/embedding.ts'; +export { CrossOriginIsolationError } from './internal/isolation.ts'; diff --git a/js/src/runtime.ts b/js/src/runtime.ts index 1964aab..5cf8506 100644 --- a/js/src/runtime.ts +++ b/js/src/runtime.ts @@ -34,7 +34,8 @@ import { unlinkFsFile, writeResponseToFsFileWithProgress, } from './internal/download.ts'; -import { isCrossOriginIsolatedRuntime, isSafariUserAgent } from './internal/environment.ts'; +import { canPostSharedMemory, describeWebGpuUnavailable, isSafariUserAgent } from './internal/environment.ts'; +import { CrossOriginIsolationError } from './internal/isolation.ts'; import { decodeImageBytesToRgb } from './internal/image.ts'; import { EmbeddingError, EMPTY_EMBEDDING_CAPABILITIES, EMBEDDING_MEDIA_MARKER, @@ -1777,6 +1778,10 @@ export class LlamaWebGpuBridgeRuntime { return this._core; } + if (!this._config.coreModuleFactory && this._config.allowSingleThreadCore !== true && !canPostSharedMemory()) { + throw new CrossOriginIsolationError(); + } + const candidates = []; if (this._config.coreModuleFactory) { candidates.push({ @@ -2016,6 +2021,19 @@ export class LlamaWebGpuBridgeRuntime { this._nGpuLayers = Math.trunc(requestedGpuLayers); } + if (this._nGpuLayers > 0 && !this._gpuActive) { + const unavailable = await describeWebGpuUnavailable(); + throwIfAborted(operationSignal, abortMessage); + this._pushRuntimeNote(`gpu_unavailable:${unavailable.reason}`); + const detail = { + no_webgpu: 'this browser has no navigator.gpu', + no_adapter: 'the browser returned no WebGPU adapter', + adapter_missing_shader_f16: `the ${unavailable.adapter || 'WebGPU'} adapter lacks shader-f16`, + unknown: 'the core registered no WebGPU device', + }[unavailable.reason]; + this._emitLogger('warn', `WebGPU is unavailable (${detail}); loading on the WebAssembly CPU runtime.`); + } + const isCpuModelMode = this._nGpuLayers === 0; const requestedBatch = Number(options.nBatch); @@ -2048,14 +2066,11 @@ export class LlamaWebGpuBridgeRuntime { this._pushRuntimeNote(`thread_pool_size:${this._threadPoolSizeHint}`); } - if (!isCrossOriginIsolatedRuntime()) { + // Only a core that never posts shared memory gets here without isolation. + if (!canPostSharedMemory()) { this._runtimeNotes.push('threads_capped_no_coi'); - if (this._threads > 1) { - this._threads = 1; - } - if (this._threadsBatch > 1) { - this._threadsBatch = 1; - } + this._threads = Math.min(this._threads, 1); + this._threadsBatch = Math.min(this._threadsBatch, 1); } this._pushRuntimeNote(`threads_batch:${this._threadsBatch}`); diff --git a/js/src/worker_host.ts b/js/src/worker_host.ts index 440dd81..0fa2853 100644 --- a/js/src/worker_host.ts +++ b/js/src/worker_host.ts @@ -3,6 +3,7 @@ import { LlamaWebGpuBridge } from './bridge.ts'; import { decisionErrorCodeOf } from './internal/decision.ts'; import { EMBED_BATCH_PROGRESS_OPTION, EmbeddingError } from './internal/embedding.ts'; +import { CROSS_ORIGIN_ISOLATION_REQUIRED, CrossOriginIsolationError } from './internal/isolation.ts'; import type { EmbedBatchProgressHook } from './internal/embedding.ts'; import { toUint8Array } from './internal/typed_values.ts'; import { @@ -55,6 +56,7 @@ export function installBridgeWorkerHost() { id, message: serializeWorkerError(error), ...(error instanceof EmbeddingError ? { embeddingErrorCode: error.code } : {}), + ...(error instanceof CrossOriginIsolationError ? { errorCode: CROSS_ORIGIN_ISOLATION_REQUIRED } : {}), ...(decisionErrorCode ? { decisionErrorCode } : {}), state, }); diff --git a/js/src/worker_protocol.ts b/js/src/worker_protocol.ts index 6f4a01a..d32f755 100644 --- a/js/src/worker_protocol.ts +++ b/js/src/worker_protocol.ts @@ -21,6 +21,7 @@ export interface WorkerResponse { message?: string; embeddingErrorCode?: EmbeddingErrorCode; decisionErrorCode?: DecisionErrorCode; + errorCode?: 'CROSS_ORIGIN_ISOLATION_REQUIRED'; event?: string; payload?: unknown; } diff --git a/js/src/worker_proxy.ts b/js/src/worker_proxy.ts index e2d784e..8e1aae6 100644 --- a/js/src/worker_proxy.ts +++ b/js/src/worker_proxy.ts @@ -6,6 +6,7 @@ import { withDecisionErrorCode, } from './internal/decision.ts'; import { EMBEDDING_MEDIA_WORKER_TIMEOUT_MS, EmbeddingError, hasEmbeddingMediaParts } from './internal/embedding.ts'; +import { CROSS_ORIGIN_ISOLATION_REQUIRED, CrossOriginIsolationError } from './internal/isolation.ts'; import type { LlamaWebGpuBridgeConfig } from './llama_webgpu_bridge.d.ts'; import { bridgeWorkerModeParam } from './worker_protocol.ts'; import type { WorkerEventHandler, WorkerResponse } from './worker_protocol.ts'; @@ -160,7 +161,9 @@ export class BridgeWorkerProxy { if (type === 'error') { const errorMessage = String(message.message || 'Worker request failed'); const workerError: Error & { state?: unknown } = message.embeddingErrorCode - ? new EmbeddingError(message.embeddingErrorCode, errorMessage) : new Error(errorMessage); + ? new EmbeddingError(message.embeddingErrorCode, errorMessage) + : message.errorCode === CROSS_ORIGIN_ISOLATION_REQUIRED + ? new CrossOriginIsolationError(errorMessage) : new Error(errorMessage); if (isDecisionErrorCode(message.decisionErrorCode)) { withDecisionErrorCode(workerError, message.decisionErrorCode); } diff --git a/src/core/model_load.inc b/src/core/model_load.inc index 1b40ddd..09d3707 100644 --- a/src/core/model_load.inc +++ b/src/core/model_load.inc @@ -59,6 +59,18 @@ int32_t load_model_internal( mparams.main_gpu = main_gpu; } + // With no layer offloaded, keep WebGPU out of the model and context backends: + // ggml-webgpu waits on queue work at every sync, and Firefox resolves those + // waits on a 100 ms tick. Static because draft models reuse g_model_params. + static ggml_backend_dev_t cpu_only_devices[] = {nullptr}; + if (n_gpu_layers == 0) { + mparams.devices = cpu_only_devices; + // Tensor split needs at least one device, and the empty list has none. + if (mparams.split_mode == LLAMA_SPLIT_MODE_TENSOR) { + mparams.split_mode = LLAMA_SPLIT_MODE_LAYER; + } + } + g_state.model = llama_model_load_from_file(model_path, mparams); if (g_state.model == nullptr) { set_error("llama_model_load_from_file failed"); diff --git a/tests/bridge/cpu_only_devices_native_test.mjs b/tests/bridge/cpu_only_devices_native_test.mjs new file mode 100644 index 0000000..3fb30bc --- /dev/null +++ b/tests/bridge/cpu_only_devices_native_test.mjs @@ -0,0 +1,28 @@ +import assert from 'node:assert/strict'; + +import { readNativeCoreSource } from './native_core_source.mjs'; + +// With no layer offloaded the model and context must not list the WebGPU +// device: Firefox resolves the queue waits ggml-webgpu blocks on every sync +// on a 100 ms tick, which cuts CPU decoding by roughly 50x. +const source = readNativeCoreSource(); +const loader = source.slice(source.indexOf('int32_t load_model_internal(')); +const body = loader.slice(0, loader.indexOf('llama_model_load_from_file(')); + +assert.match( + body, + /if \(n_gpu_layers == 0\) \{\s*mparams\.devices = [A-Za-z_]+;/, + 'a zero-layer load must pass an empty device list', +); +assert.match( + body, + /static ggml_backend_dev_t [A-Za-z_]+\[\] = \{nullptr\};/, + 'the empty list outlives the call: draft models reuse g_model_params', +); + +// Tensor split needs a device, and an empty list has none. +assert.match( + body, + /if \(n_gpu_layers == 0\) \{[^}]*LLAMA_SPLIT_MODE_TENSOR[^}]*LLAMA_SPLIT_MODE_LAYER/, + 'a zero-layer load must not ask for tensor split', +); diff --git a/tests/bridge/cross_origin_isolation_test.mjs b/tests/bridge/cross_origin_isolation_test.mjs new file mode 100644 index 0000000..3a7d2ac --- /dev/null +++ b/tests/bridge/cross_origin_isolation_test.mjs @@ -0,0 +1,174 @@ +import assert from 'node:assert/strict'; + +import { LlamaWebGpuBridge } from '../../js/src/llama_webgpu_bridge.js'; +import { CrossOriginIsolationError } from '../../js/src/llama_webgpu_bridge.js'; +import { installBridgeWorkerHost } from '../../js/src/worker_host.ts'; +import { + createRealWorkerBridge, + withStubWorkerEnvironment, + workerDriver, +} from './bridge_operation_queue_fixtures.mjs'; + +// A core module whose factory never settles, as the pthread core does when +// posting its shared memory to a worker throws DataCloneError. +const HANGING_CORE = `data:text/javascript,export default () => new Promise(() => {});`; +const REACHED_CORE = `data:text/javascript,export default async () => { throw new Error('core factory reached'); };`; + +function withUnpostableSharedMemory(run) { + const original = globalThis.MessageChannel; + globalThis.MessageChannel = class { + constructor() { + this.port1 = { + postMessage() { + throw new DOMException('The object can not be cloned.', 'DataCloneError'); + }, + close() {}, + }; + this.port2 = { close() {} }; + } + }; + return Promise.resolve(run()).finally(() => { + globalThis.MessageChannel = original; + }); +} + +function rejectionWithin(promise, ms) { + let timer; + const timeout = new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error(`did not settle within ${ms}ms`)), ms); + }); + return Promise.race([promise, timeout]).finally(() => clearTimeout(timer)); +} + +function directBridge(coreModuleUrl, extra = {}) { + return new LlamaWebGpuBridge({ disableWorker: true, coreModuleUrl, ...extra }); +} + +// Non-isolated page: shared memory cannot be posted, so core init fails fast. +await withUnpostableSharedMemory(async () => { + for (const extra of [{}, { coreModuleUrlMem64: HANGING_CORE, preferMemory64: true }]) { + const bridge = directBridge(HANGING_CORE, extra); + const started = Date.now(); + const error = await rejectionWithin( + bridge.loadModelFromUrl('https://models.test/m.gguf', { nGpuLayers: 0 }).then( + () => assert.fail('load must reject'), + (rejected) => rejected, + ), + 2000, + ); + assert.ok(Date.now() - started < 2000); + assert.ok(error instanceof CrossOriginIsolationError, String(error)); + assert.equal(error.name, 'CrossOriginIsolationError'); + assert.equal(error.code, 'CROSS_ORIGIN_ISOLATION_REQUIRED'); + assert.match(error.message, /cross-origin isolat/i); + assert.match(error.message, /Cross-Origin-Embedder-Policy: require-corp/); + assert.match(error.message, /credentialless/); + assert.match(error.message, /Safari/); + await bridge.dispose(); + } +}); + +// Isolated page: the same configuration reaches the core factory. +{ + const bridge = directBridge(REACHED_CORE); + await assert.rejects( + bridge.loadModelFromUrl('https://models.test/m.gguf', { nGpuLayers: 0 }), + /core factory reached/, + ); + await bridge.dispose(); +} + +// A custom core factory is not the pthread core and is never gated. +await withUnpostableSharedMemory(async () => { + const bridge = new LlamaWebGpuBridge({ + disableWorker: true, + coreModuleFactory: async () => { throw new Error('custom factory reached'); }, + }); + await assert.rejects( + bridge.loadModelFromUrl('https://models.test/m.gguf', { nGpuLayers: 0 }), + /custom factory reached/, + ); + await bridge.dispose(); +}); + +// The worker host reports the typed error and the proxy restores it. +await withUnpostableSharedMemory(async () => { + const replies = []; + const oldSelf = globalThis.self; + globalThis.self = { postMessage: (message) => replies.push(message) }; + try { + installBridgeWorkerHost(); + await self.onmessage({ data: { type: 'init', config: { coreModuleUrl: HANGING_CORE } } }); + await self.onmessage({ data: { type: 'call', id: 1, method: 'loadModelFromUrl', args: [ + 'https://models.test/m.gguf', { nGpuLayers: 0 }, + ] } }); + const reply = replies.find((message) => message.id === 1); + assert.equal(reply.type, 'error'); + assert.equal(reply.errorCode, 'CROSS_ORIGIN_ISOLATION_REQUIRED'); + } finally { + if (oldSelf === undefined) delete globalThis.self; + else globalThis.self = oldSelf; + } +}); + +await withStubWorkerEnvironment(async ({ workers }) => { + const bridge = createRealWorkerBridge(); + const proxy = bridge._workerProxy; + const driver = workerDriver(proxy, workers[0]); + driver.ready(); + const call = proxy.call('loadModelFromUrl', ['https://models.test/m.gguf', {}]); + await driver.settle(); + proxy._worker.onmessage({ data: { + type: 'error', + id: driver.calls('loadModelFromUrl')[0].id, + message: 'cross-origin isolation required', + errorCode: 'CROSS_ORIGIN_ISOLATION_REQUIRED', + } }); + const error = await call.catch((rejected) => rejected); + assert.ok(error instanceof CrossOriginIsolationError, String(error)); + proxy._worker.onmessage({ data: { type: 'result', id: 0 } }); + await bridge.dispose().catch(() => {}); +}); + +// A core built without pthreads never posts shared memory, so a page that +// opts out loads it on a non-isolated page, capped to one thread. +await withUnpostableSharedMemory(async () => { + const bridge = directBridge(REACHED_CORE, { allowSingleThreadCore: true }); + await assert.rejects( + bridge.loadModelFromUrl('https://models.test/m.gguf', { nGpuLayers: 0 }), + /core factory reached/, + ); + await bridge.dispose(); + + const { createDirectBridge } = await import('./bridge_operation_queue_fixtures.mjs'); + const { bridge: loaded, core } = createDirectBridge(); + const runtime = loaded._runtime; + runtime._modelBytes = 0; + const loads = []; + const ccall = core.ccall.bind(core); + core.ccall = (name, type, types, args = []) => { + if (name === 'llamadart_webgpu_load_model') { loads.push([...args]); return 0; } + if (name === 'llamadart_webgpu_free_model') return 0; + if (name === 'llamadart_webgpu_get_context_size') return 128; + return ccall(name, type, types, args); + }; + runtime._probeBackends = async () => {}; + runtime._ensureCore = async () => core; + runtime._coreSupportsPthreads = () => true; + runtime._syncThreadPoolSizeHintFromCore = () => { runtime._threadPoolSizeHint = 4; }; + runtime._getCachedModelResponse = async () => new Response(new Uint8Array([1, 2, 3])); + runtime._emitLogger = () => {}; + await loaded.loadModelFromUrl('https://models.test/m.gguf', { + nCtx: 128, nThreads: 4, nThreadsBatch: 4, nGpuLayers: 0, forceRemoteFetchBackend: false, useCache: false, + }); + assert.equal(loads[0][2], 1); + assert.equal(loads[0][3], 1); + assert.match(loaded.getModelMetadata()['llamadart.webgpu.runtime_notes'], /threads_capped_no_coi/); +}); + +// The opt-out reaches the worker host. +{ + const bridge = directBridge(REACHED_CORE, { allowSingleThreadCore: true }); + assert.equal(bridge._workerConfig().allowSingleThreadCore, true); + assert.equal(directBridge(REACHED_CORE)._workerConfig().allowSingleThreadCore, undefined); +} diff --git a/tests/bridge/generation_recovery_load_options_test.mjs b/tests/bridge/generation_recovery_load_options_test.mjs index 6818714..1933e82 100644 --- a/tests/bridge/generation_recovery_load_options_test.mjs +++ b/tests/bridge/generation_recovery_load_options_test.mjs @@ -32,9 +32,7 @@ function fixture() { return { bridge, runtime, core, loads }; } -const oldIsolation = Object.getOwnPropertyDescriptor(globalThis, 'crossOriginIsolated'); -Object.defineProperty(globalThis, 'crossOriginIsolated', { configurable: true, value: true }); -try { +{ for (const custom of [ {}, { nSeqMax: 3, useMmap: true, useMlock: true, flashAttention: 1, cacheTypeK: 8, cacheTypeV: 2, kvUnified: false, @@ -59,29 +57,22 @@ try { assert.deepEqual(loads[1], expected, 'recovery must preserve every resolved native load argument'); assert.equal(runtime._preferMemory64, preferMemory64); } - for (const loseIsolation of [false, true]) { + { const { bridge, runtime, loads } = fixture(); runtime._syncThreadPoolSizeHintFromCore = () => { runtime._threadPoolSizeHint = loads.length === 0 ? 4 : 2; - if (loseIsolation && loads.length > 0) { - Object.defineProperty(globalThis, 'crossOriginIsolated', { configurable: true, value: false }); - } }; - try { - await bridge.loadModelFromUrl('https://models.test/pool.gguf', { - nCtx: 128, nGpuLayers: 99, nThreads: 4, nThreadsBatch: 3, - nBatch: 96, nUbatch: 64, ropeFrequencyScale: 0.5, forceRemoteFetchBackend: false, - }); - assert.equal(await bridge.createCompletion('hello', { nPredict: 2 }), 'OK'); - assert.equal(loads.length, 2); - const expected = [...loads[0]]; - expected[2] = loseIsolation ? 1 : 2; - expected[3] = loseIsolation ? 1 : 2; - expected[6] = 0; - assert.deepEqual(loads[1], expected, 'recovery must respect replacement pool and isolation limits'); - } finally { - Object.defineProperty(globalThis, 'crossOriginIsolated', { configurable: true, value: true }); - } + await bridge.loadModelFromUrl('https://models.test/pool.gguf', { + nCtx: 128, nGpuLayers: 99, nThreads: 4, nThreadsBatch: 3, + nBatch: 96, nUbatch: 64, ropeFrequencyScale: 0.5, forceRemoteFetchBackend: false, + }); + assert.equal(await bridge.createCompletion('hello', { nPredict: 2 }), 'OK'); + assert.equal(loads.length, 2); + const expected = [...loads[0]]; + expected[2] = 2; + expected[3] = 2; + expected[6] = 0; + assert.deepEqual(loads[1], expected, 'recovery must respect the replacement pool limit'); } const { bridge, runtime, loads } = fixture(); await bridge.loadModelFromUrl('https://models.test/model.gguf', { nGpuLayers: 99, forceRemoteFetchBackend: false }); @@ -133,9 +124,6 @@ try { if (oldSelf === undefined) delete globalThis.self; else globalThis.self = oldSelf; } -} finally { - if (oldIsolation) Object.defineProperty(globalThis, 'crossOriginIsolated', oldIsolation); - else delete globalThis.crossOriginIsolated; } console.log('Generation recovery load option tests passed'); diff --git a/tests/bridge/gpu_unavailable_reason_test.mjs b/tests/bridge/gpu_unavailable_reason_test.mjs new file mode 100644 index 0000000..e02f345 --- /dev/null +++ b/tests/bridge/gpu_unavailable_reason_test.mjs @@ -0,0 +1,74 @@ +import assert from 'node:assert/strict'; +import { createDirectBridge } from './bridge_operation_queue_fixtures.mjs'; + +// Real facade, load path and backend probe. Only the core, the network and +// navigator.gpu are doubles. +async function load({ gpu, coreHasWebGpu = false, nGpuLayers = 99 }) { + const { bridge, core } = createDirectBridge(); + const runtime = bridge._runtime; + const ccall = core.ccall.bind(core); + core.ccall = (name, type, types, args = []) => { + if (name === 'llamadart_webgpu_probe') return coreHasWebGpu ? 1 : 0; + if (name === 'llamadart_webgpu_backends_json') return coreHasWebGpu ? '["WebGPU","CPU"]' : '["CPU"]'; + if (name === 'llamadart_webgpu_load_model' || name === 'llamadart_webgpu_free_model') return 0; + if (name === 'llamadart_webgpu_get_context_size') return 128; + return ccall(name, type, types, args); + }; + runtime._modelBytes = 0; + runtime._ensureCore = async () => core; + runtime._coreSupportsPthreads = () => true; + runtime._syncThreadPoolSizeHintFromCore = () => {}; + runtime._getCachedModelResponse = async () => new Response(new Uint8Array([1, 2, 3])); + const warnings = []; + runtime._emitLogger = (level, message) => { + if (level === 'warn') warnings.push(String(message)); + }; + + const oldNavigator = Object.getOwnPropertyDescriptor(globalThis, 'navigator'); + Object.defineProperty(globalThis, 'navigator', { configurable: true, value: { userAgent: 'Chrome', gpu } }); + try { + await bridge.loadModelFromUrl('https://models.test/model.gguf', { + nCtx: 128, nThreads: 1, nGpuLayers, forceRemoteFetchBackend: false, useCache: false, + }); + } finally { + Object.defineProperty(globalThis, 'navigator', oldNavigator); + } + const notes = bridge.getModelMetadata()['llamadart.webgpu.runtime_notes'].split(';'); + return { bridge, warnings, reasons: notes.filter((note) => note.startsWith('gpu_unavailable:')) }; +} + +function fakeGpu(features) { + const gpu = { requests: 0 }; + gpu.requestAdapter = async () => { + gpu.requests += 1; + return features && { features: new Set(features), info: { vendor: 'nvidia', architecture: 'lovelace' } }; + }; + return gpu; +} + +// Chrome on Linux hides shader-f16 on NVIDIA drivers older than 615.71, and +// ggml-webgpu then registers no device: the load must say why it is on CPU. +const hidden = await load({ gpu: fakeGpu(['subgroups', 'timestamp-query']) }); +assert.equal(hidden.bridge.isGpuActive(), false); +assert.equal(hidden.bridge.getBackendName(), 'CPU'); +assert.deepEqual(hidden.reasons, ['gpu_unavailable:adapter_missing_shader_f16']); +assert.equal(hidden.warnings.length, 1); +assert.match(hidden.warnings[0], /shader-f16/); +assert.match(hidden.warnings[0], /nvidia/); + +assert.deepEqual((await load({ gpu: fakeGpu(null) })).reasons, ['gpu_unavailable:no_adapter']); +assert.deepEqual((await load({ gpu: undefined })).reasons, ['gpu_unavailable:no_webgpu']); +const throwing = { requestAdapter: async () => { throw new Error('lost'); } }; +assert.deepEqual((await load({ gpu: throwing })).reasons, ['gpu_unavailable:no_adapter']); +// An adapter that qualifies while the core has no WebGPU device is not explained. +assert.deepEqual((await load({ gpu: fakeGpu(['shader-f16']) })).reasons, ['gpu_unavailable:unknown']); + +// A working WebGPU load and a CPU-only request stay silent and never ask for an adapter. +for (const quiet of [{ coreHasWebGpu: true }, { nGpuLayers: 0 }]) { + const gpu = fakeGpu(['subgroups']); + const loaded = await load({ gpu, ...quiet }); + assert.deepEqual(loaded.reasons, []); + assert.deepEqual(loaded.warnings, []); + assert.equal(gpu.requests, 0); + assert.equal(loaded.bridge.isGpuActive(), quiet.coreHasWebGpu === true); +}