Frigate

A fully self-contained Swift package for on-device text embeddings and LLM inference using MLX.

All fork sources are vendored directly — no external URLs for patched libraries appear in Package.swift. The only external dependencies are three standard Apple packages (swift-numerics, swift-collections, swift-crypto).

What's inside

API	Default model	Notes
`FrigateEmbedder`	`rao-studios/voyage-4-nano-mlx-8bit` (voyage-4-nano, 1024 dims)	Returns `[[Float]]`; query/document roles
`FrigateLLM`	`mlx-community/Qwen3-0.6B-4bit`	Returns `AsyncStream<String>`
`FrigateBoost`	local `.json` file	XGBoost tree-ensemble inference, zero runtime deps
`FrigateVisionAX`	bundled ONNX region classifier	Pixel perception — the VisionAX runtime, hosted here — macOS only
`SinatraHarness`	any MLXLLM chat model, e.g. `mlx-community/Mistral-Small-3.2-24B-Instruct-2506-4bit`	The on-device harness: a side model biases the logits toward the retrieved context and learns from the grounding it measures — macOS only, GPL-3.0-only

HuggingFace models are downloaded on first use and cached at ~/.cache/huggingface/. FrigateBoost loads a model exported with booster.save_model("model.json") — no libxgboost required at runtime.

FrigateVisionAX — pixel perception

The FrigateVisionAX product and module is the vision runtime, hosted here: the OpenCV region detector (C++ CVisionAX), Apple's text recognition, the ONNX role classifier and the page map. The module re-exports VisionAXCore from the VisionAX repository — the AX/A11Y data structures, the dataset schema and the role vocabulary — so one import carries both.

import FrigateVisionAX

let engine = try VisionEngine()
let scene = try engine.perceive(
    image: capture,
    projection: ScreenProjection(origin: pageOrigin, pixelsPerPoint: 2),
    classifier: try? RegionClassifier.bundled(),
    lanes: [.regions, .text])
let map = scene.pageMap()          // rows, what each affords, where its name came from

The runtime's own documentation — how a region becomes a node, the classifier, the media lane, the page map, benchmarks — is Sources/FrigateVisionAX/README.md. The VisionAX repository keeps what is not runtime: VisionAXCore, the training pipeline that exports Sources/FrigateVisionAX/Resources/Models, the harvester that builds its data and the bench that tunes the detector.

Things worth knowing:

The product and the module share one name. A consumer's manifest names FrigateVisionAX and its sources say import FrigateVisionAX. (Until 2026-09-10 the product was FrigateVision and the module VisionAX.)
macOS only. OpenCV and ONNX Runtime arrive as xcframeworks, which Linux cannot resolve, so the dependencies, the targets and the product all live inside #if !os(Linux) in Package.swift. A Linux build of Frigate never learns they exist, and ../VisionAX need not be checked out on a Linux box. ManifestPlatformTests fails on macOS the moment something escapes that guard, and scripts/check-linux-manifest.sh evaluates the manifest under Linux for real.
It is not in the Frigate umbrella. import Frigate does not carry it, deliberately: VisionAXCore declares AXNodeSnapshot, AXScreenElement and AXNodeCategory under the same names an accessibility layer already uses, and every consumer of the umbrella would inherit that ambiguity at the use site.
The classifier's backbone runs on MLX when it can. On ONNX Runtime's CPU path it is ~40% of a page read. It runs on Metal when mlx.metallib sits beside the binary and the model ships MLX weights converted from its own ONNX backbone; otherwise it stays on the CPU path and RegionClassifier.backboneDescription says why. FRIGATE_VISION_BACKBONE=onnx forces the CPU path for an A/B. So taking FrigateVisionAX links MLX's core — not the transformer stack — and a binary that wants the GPU path needs the metallib: scripts/build-metallib.sh <config> --package <consumer> beside a SwiftPM-built binary, --app <App.app> inside an app bundle.
The model and the real-capture fixtures are git-lfs (see .gitattributes): run git lfs pull after a clone, or RegionClassifier.bundled() reports a pointer file.
An app bundle copies Frigate_FrigateVisionAX.bundle into Contents/Resources — SwiftPM names resource bundles <package>_<target>.

Two consequences of the dependencies, both benign:

On macOS, every consumer's Package.resolved carries an opencv-spm pin and their next resolve fetches the OpenCV and ONNX Runtime artifacts, whether or not they link FrigateVisionAX.
Frigate's own Package.resolved differs by host (macOS has opencv-spm; Linux does not). A plain swift build on Linux drops the extra pin with a warning and rewrites the file — which is why the Linux scripts do not pass --force-resolved-versions.

VisionAX is taken by URL (rao-studios/VisionAX, branch main), so Frigate carries no path dependency of its own and a consumer may take Frigate by URL — .package(url: "https://github.com/rao-studios/Frigate.git", branch: "main"), as AII does — or by path while iterating on it.

SinatraHarness — the on-device harness (GPL-3.0-only)

The SinatraHarness product and module is an on-device LLM harness for Apple silicon with an injection layer between the transformer and the decoder. When a turn is answered with retrieved context, only that context is encoded with the model's own embedding table and read by a small time-series side model; its weights become a bias added to the logits right before sampling. After each turn the output is re-scored with and without its context, and that grounding measurement labels the turn and trains the side model. One model stays resident and one generation runs at a time.

import SinatraHarness

let harness = Harness(storeDirectory: storeURL)
try await harness.load(modelID: "mlx-community/Mistral-Small-3.2-24B-Instruct-2506-4bit")

let handle = try await harness.generate(GenerateRequest(
    input: UserInput(chat: [.system(systemWithContext), .user(userText)]),
    parameters: GenerateParameters(maxTokens: 400, temperature: 0.4, topP: 0.9),
    turn: TurnInput(owner: OwnerID(userId), retrieved: partitions),
    bareInput: UserInput(chat: [.system(systemWithoutContext), .user(userText)])))
for await event in handle.stream { /* .chunk / .toolCall / .info, as MLXLMCommon.generate */ }
let completion = await handle.completion.value

The theory, the integration guide (Sewn's local provider in AII is the reference) and the CLI are in Sources/SinatraHarness/README.md.

Things worth knowing:

It is GPL-3.0-only, unlike the rest of this package (MIT). The text is Sources/SinatraHarness/LICENSE; NOTICE names the three directories it covers. Naming the SinatraHarness product links GPL code, and the linking work is subject to the GPL. Nothing else here depends on it.
macOS only. The targets and the product share one #if !os(Linux) block in Package.swift (sinatraTargets), the FrigateVisionAX treatment; ManifestPlatformTests and scripts/check-linux-manifest.sh pin it.
Not in the Frigate umbrella, and the module keeps its name. import Frigate never carries Harness, Partition or OwnerID; and the module stays SinatraHarness because Sewn has a class named Sinatra, and a module of that name would break its qualified lookups.
It moved here from the SinatraHarness repository on 2026-10-21. A consumer that took .product(name: "SinatraHarness", package: "SinatraHarness") now writes .product(name: "SinatraHarness", package: "Frigate") and drops the second package.
The CLI builds with swift build --target sinatra-harness (no product; like flux2-cli), needs ./scripts/build-metallib.sh debug once, and runs as .build/debug/sinatra-harness ….
Tests: swift test --filter SinatraHarnessTests runs the pure-Swift suites; ./scripts/test-mlx.sh --filter SinatraHarnessTests adds the MLX-gated ones.

Requirements

	Version
Swift	6.3+
Ubuntu	24.04 Noble (Linux)
CUDA	12.x — GPU sm_86+ recommended (e.g. RTX 3090)
macOS	15+ — Metal, no CUDA needed

Setup — Ubuntu 24.04 (fresh machine)

Step 1 — Clone the repo

git clone <this-repo> Frigate
cd Frigate

Step 2 — Run the setup script

./setup-frigate-ubuntu.sh

The script installs everything in order and then builds Frigate:

Swift 6.3.2 via swiftly — placed in ~/.local/share/swiftly/
CUDA 12.9 toolkit — adds the NVIDIA apt repo and installs cuda-toolkit-12-9
BLAS / LAPACK / gfortran — libopenblas-dev, liblapacke-dev
cudnn-frontend v1.16.0 — clones and cmake-installs headers to /usr/local/cudnn-frontend/
huggingface_hub — pip3 install huggingface_hub for model downloads
~/.bashrc — exports SWIFTLY_HOME, CUDA paths, and SPM_CUDA=1
swift build -c release --jobs 2 — compiles all targets (~20 min first time)

The script is idempotent — safe to re-run if any step failed.

CPU-only (no GPU required):

./setup-frigate-ubuntu.sh --cpu

Install deps only, build later:

./setup-frigate-ubuntu.sh --skip-build

Step 3 — Download a model

# Embedding model (~370 MB). FrigateEmbedder downloads it on first use into
# $HF_HOME/snapshots (without HF_HOME, ~/.cache/huggingface/snapshots). An app with its own
# models folder passes HubDownloader(home:) — Rao's apps pass ~/.rao/models/huggingface.
HF_HOME=~/.rao/models/huggingface hf download rao-studios/voyage-4-nano-mlx-8bit \
    --local-dir ~/.rao/models/huggingface/snapshots/models/rao-studios/voyage-4-nano-mlx-8bit

# LLM (~400 MB)
hf download mlx-community/Qwen3-0.6B-4bit

HuggingFace Hub caches models at ~/.cache/huggingface/hub/. Downloads happen automatically at first use if you skip this step.

Step 4 — Use from Swift

Add Frigate as a local package dependency in your Package.swift:

.package(path: "/path/to/Frigate"),

Then import and use:

import Frigate

// XGBoost — load model, predict P(class=1) for a batch of feature vectors
let boost = try FrigateBoost(modelURL: URL(fileURLWithPath: "model.json"))
let probs: [Float] = await boost.predict(features: [
    [0.12, 0.003, 0.47, 0.06, 0.31, 0.84, 0.002, 0.001, 0.51, 0.29],
])
print(probs) // e.g. [0.731]

// Embeddings — voyage-4-nano is asymmetric: say whether a text is a query or a document
let embedder = FrigateEmbedder()
let passages: [[Float]] = try await embedder.embed([
    "hello world",
    "machine learning on GPU",
], role: .document)
let query = try await embedder.embed(["what runs models on the GPU?"], role: .query)

// LLM
let llm = FrigateLLM()
for await token in try await llm.generate(prompt: "Explain MLX in one sentence.") {
    print(token, terminator: "")
}

Manual build (after setup script, without --skip-build)

# CUDA bin must be in PATH — the system /usr/bin/nvcc stub does not include CUDA headers
source ~/.bashrc   # loads SWIFTLY_HOME, CUDA PATH, SPM_CUDA=1
swift build -c release --jobs 2

Or inline:

PATH="/usr/local/cuda/bin:$PATH" SPM_CUDA=1 CUDA_ARCH=sm_86 swift build -c release --jobs 2

Setup — macOS

git clone <this-repo> Frigate
cd Frigate
swift build -c release
./scripts/build-metallib.sh release     # required — see below

Swift 6.3+ required (brew install swiftly && swiftly install latest).

The metallib step is not optional on macOS. swift build has no Metal stage — SwiftPM never compiles Sources/Cmlx/mlx-generated/metal/*.metal — so a fresh checkout has no mlx.metallib and the first GPU op dies with "Failed to load the default metallib". Nothing warns you at build time; it looks fine until inference runs. scripts/build-metallib.sh compiles the shaders and installs the result beside every binary in .build, including inside .xctest bundles, which is what lets the FRIGATE_MLX_TESTS=1 suites run at all.

It is a no-op on Linux (no Metal backend), so it is safe to put in a build script that runs on both. Consumers should call it rather than keeping their own copy — Bonnie, Fleet, Sewn, Thread and Zehn each ship a one-line build-metallib.sh that delegates here:

./scripts/build-metallib.sh release --package ../Thread   # or, from Thread: ./build-metallib.sh release

scripts/test-mlx.sh runs the tests with the MLX-gated suites on: it strips each test bundle's metallib (a copy made before the build breaks the bundle's signature), builds, installs a fresh copy beside every binary and bundle, and runs swift test --skip-build with FRIGATE_MLX_TESTS=1.

API reference

FrigateEmbedder

public actor FrigateEmbedder {
    public init(profile: Profile = .voyage4Nano)
    public init(modelId: String)            // "org/repo", "org/repo@<sha>", or a directory
    public nonisolated var status: Status   // idle · downloading(fraction) · loading · ready · failed
    public func embed(_ texts: [String], role: Role = .document) async throws -> [[Float]]
    public func embedWithUsage(_ texts: [String], role: Role = .document) async throws -> (embeddings: [[Float]], promptTokens: Int)
    public func warmup() async throws       // download, load, one short pass
}

A Profile names the model (a Hub repo pinned to a commit, or a directory), the prompt per role, the Matryoshka width, the token budget and the vector space it writes (voyage-4@1024), which hosts stamp beside an index. Profile.resolve maps a repo id to its known profile; an unknown repo takes its prompts from the snapshot's config_sentence_transformers.json.

Snapshots load through HubDownloader from one local folder: a pinned commit must match the copy on disk, HF_HOME is honoured, and with it set ~/Documents is not searched. Texts over the budget are cut inside the tokenizer's special-token frame, so a trailing token a model pools on (Qwen3-Embedding's <|endoftext|>) survives truncation. Non-finite output is an error.

scripts/embedders/ holds the conversion that builds the voyage-4-nano snapshot and the parity and speed checks it has to pass.

FrigateLLM

public actor FrigateLLM {
    public init(modelId: String = "mlx-community/Qwen3-0.6B-4bit")
    public func generate(prompt: String, maxTokens: Int = 512) async throws -> AsyncStream<String>
    public func warmup() async throws
}

Both actors deduplicate concurrent model loads — calling embed or generate from multiple tasks concurrently is safe.

FrigateBoost

public actor FrigateBoost {
    /// Load an XGBoost model exported with `booster.save_model("model.json")`.
    public init(modelURL: URL) throws
    /// Predict P(class=1) for a batch of feature vectors.
    public func predict(features: [[Float]]) async -> [Float]
}

FrigateBoost parses the XGBoost v2 JSON format in pure Swift and walks the tree ensemble directly — no libxgboost binary required. The binary:logistic objective is supported; leaf values are summed across all trees and sigmoid is applied to produce final probabilities.

Vendored sources

All fork sources are copied directly into Sources/. No git submodules, no external URLs for patched code.

Record the tag, not a branch — the absence of that record is what made the 2026-09 audit necessary. Last synced 2026-09-08.

Directory	From	Version	Local patches
`Sources/Cmlx/mlx/`	`ml-explore/mlx`	v0.31.1	4 files in `backend/cuda/` + 3 added `no_*_impl.cpp` (CUTLASS-disabled fallbacks, sm_86); `io/safetensors.cpp` lays tensors out in key order, so the same tensors always save to the same bytes (`SafetensorsLayoutTests`; upstream follows the map's insertion order, which Swift seeds per process); `io/gguf.cpp` writes its metadata, tensor infos and tensor data in key order the same way. That file is not built here: `Package.swift` excludes it for `no_gguf.cpp`, gguflib is not vendored, and mlx-c has no GGUF binding. It was checked on 2026-10-19 against gguflib `8fa6eb6`, upstream's pin, in a scratch harness linked to Frigate's Cmlx objects
`Sources/Cmlx/mlx-c/`	`ml-explore/mlx-c`	v0.6.0	none
`Sources/Cmlx/{json,fmt,metal-cpp}/`	pinned by mlx-swift	json 3.11.3, fmt 12.1.0	none — these are mlx's own pins, not drift
`Sources/MLX/` … `Sources/MLXOptimizers/`	`ml-explore/mlx-swift`	0.31.6	none
`Sources/Jinja/`	`huggingface/swift-jinja`	2.5.0	none
`Sources/Hub/` … `Sources/Models/`	`huggingface/swift-transformers`	1.3.4	upstream adopted the fork's Linux patches; only `LocalizedStringLinuxShim.swift` added
`Sources/HuggingFace/`	`huggingface/swift-huggingface`	0.10.1	none (Xet trait off)
`Sources/EventSource/`	`mattt/EventSource`	1.5.1	`EventSource+AsyncHTTPClient.swift` dropped (AsyncHTTPClient trait off — keeps SwiftNIO out)
`Sources/MLXLMCommon/` … `Sources/MLXEmbedders/`	`ml-explore/mlx-swift-lm`	3.31.4	`LinuxCompat.swift` added; Linux `canImport` guards on `UserInput` (incl. `AudioProcessing.audioFormat`), `UserInput+Audio`, `ChatSession`, `ParoQuant/ParoQuantLoader`; `MLXEmbedders/Models/Bert.swift` sanitize chain made table-driven
`Sources/mlx_embeddings/`	`mzbac/mlx.embeddings`	v0.1.3	`Bert.swift`; `Qwen3.swift` adds `Qwen3BidirectionalModel` (voyage-4-nano: padding-only mask, `linear` head, mean pooling), chosen in the `qwen3` creator by `use_bidirectional_attention`
`Sources/FrigateBridge/`	This package — concrete `Downloader` / `TokenizerLoader` for mlx-swift-lm 3.x
`Sources/MLXAccelerate/`	This package — Linux-compatible Accelerate ops via MLX (`gaussianBlur`, `sobelGradients`, `filter2D`, `perspectiveWarp`, `spectralDistance`)
`Sources/Frigate/`	This package — `FrigateEmbedder`, `FrigateLLM`, `FrigateBoost`
`Sources/FrigateVisionAX/`, `Sources/CVisionAX/`	This package — the vision runtime (moved in from the VisionAX repository, which keeps `VisionAXCore` and training); see `Sources/FrigateVisionAX/README.md`
`Sources/SinatraHarness/`, `Sources/sinatra-harness/`	This package — the on-device harness and its CLI (moved in from the SinatraHarness repository, 2026-10); GPL-3.0-only, see `Sources/SinatraHarness/LICENSE` and `NOTICE`

mlx C++ is deliberately held at 0.31.1. v0.32.2 exists, but every mlx-swift release through 0.31.6 defines MLX_VERSION as "0.31.1" (see Package.swift), so bumping the C++ core would outrun the mlx-c / Swift bindings. Revisit when mlx-swift 0.32.x ships.

mlx-swift-lm 3.x takes the downloader and tokenizer as parameters. Upstream supplies that wiring through its MLXHuggingFace macro target, which needs swift-syntax; this package wires it by hand in Sources/FrigateBridge/ instead, so no compiler plugin enters the Linux build.

yyjson is an external dependency, not vendored — deliberately. swift-transformers 1.3.x needs it, and WhisperKit brings its own swift-transformers into the same link in at least one consumer (Bonnie). Two copies of yyjson.c define the same C symbols and nothing in SwiftPM separates them: module aliasing explicitly does not cover non-Swift symbols, renaming the target does not rename its symbols, and even hidden (private external) visibility still trips ld64's duplicate check — all three were tried. Depending on the same package URL makes SwiftPM dedupe by identity, so every graph gets exactly one copy. The vendoring rule above is about patched fork sources; this is stock upstream C at the pin swift-transformers itself uses.

String(localized:) does not exist on Linux. swift-corelibs-foundation ships no localized: overload (checked on Swift 6.3 / Ubuntu 24.04, with and without FoundationEssentials), and swift-transformers 1.3.x and mlx-swift-lm 3.x both use it for error text. Each affected vendored module carries an internal LocalizedStringLinuxShim.swift that returns the already-interpolated literal: Hub, Tokenizers, Models, MLXLMCommon, MLXLLM, MLXVLM.

Three targets leave the Linux graph entirely, alongside FrigateVisionAX: ObscurKit, FluxKit and flux2-cli. ObscurDinov2 decodes a CGImage through CGContext/CGColorSpace and the other two follow it in; no consumer takes them. See imageGenTargets in Package.swift. SinatraHarness, sinatra-harness and SinatraHarnessTests leave it too (sinatraTargets): the harness has never been built on Linux.

The CPU-only Linux build (SPM_CUDA=0) was broken before the 2026-09 audit and is now fixed. Its exclude lists had drifted: GPU+CUDA.swift (a portable stub that is the only definition of GPU), MLXFast.swift/MLXFastKernel.swift (which the MLXFast shim target dereferences), and mlx-c/mlx/c/fast.cpp (leaving mlx_fast_* undefined — invisible to a library build, caught only when Thread linked an executable). All four are now built. MLXAccelerate/BatchedImageOps.swift also needed nonisolated(unsafe) on two pointers, because Linux's libdispatch declares concurrentPerform's closure @Sendable where Darwin's does not.

Known GPU constraints

container.perform is a pure inference zone. Never call MLX.Memory.*, Stream.*, or any CommandEncoder API from inside a container.perform closure. The CUDA allocator is active during the closure; re-entry causes SIGSEGV (address ~0x6529) or crash at cudaGraphLaunch. All memory management runs after perform returns.

SDPA cache size. MLX_CUDA_SDPA_CACHE_SIZE=2048 is set in FrigateEmbedder.init. The default of 256 triggers a fatal error after 512 cache misses when sequence lengths vary across sub-batches.

Batch and token limits. FrigateEmbedder uses 8 inputs per sub-batch and caps sequences at 512 tokens. Larger values cause cudaMallocAsync OOM on 24 GB cards because encoder temporary buffers accumulate until CommandEncoder::commit() fires.

GPU architecture. Default is sm_86 (RTX 3090). Override before building: export CUDA_ARCH=sm_89 for RTX 4090. CUTLASS is disabled; GPU fallback uses affine_dequantize + CublasGemm (works on any sm_80+ without CUTLASS).

TODO — MLXVLM Linux port

Vision-language models (Gemma3, Qwen2-VL, Qwen3-VL, PaliGemma, Pixtral, SmolVLM2, etc.) are fully implemented in Sources/MLXVLM/ but excluded on Linux because they depend on Apple-only frameworks: AVFoundation, CoreImage, CoreGraphics.

The exclusion is in Package.swift via vlmExcludes — removing those excludes and providing Linux-compatible replacements is all that is needed to unlock VLM on Linux.

What is blocked and why

File	Dependency	Used for
`MediaProcessing.swift`	`AVFoundation`, `CoreImage`	Image resize, pixel buffer extraction, video frame decoding
`Models/Qwen2VL.swift` and similar	`CoreGraphics` (`CGSize`, `CGFloat`)	Bounding-box coordinates in vision encoders
`Models/Paligemma.swift` etc.	`CoreImage.CIFilterBuiltins`	Image preprocessing (normalise, crop)
`VLMModelFactory.swift`	Depends on all model types above	Registers all VLM model constructors

Sources/MLXLMCommon/LinuxCompat.swift already provides CGSize and CGFloat stubs, so coordinate types compile. The primary blocker is image I/O and pixel manipulation in MediaProcessing.swift.

Prescribed path to completion

1. Replace MediaProcessing.swift with a Linux-compatible image backend.

The file needs to:

Load an image from a file path or Data blob into a float tensor (MLXArray of shape [H, W, 3])
Resize to a target CGSize
Normalise pixel values (mean/std per channel)
Return an MLXArray directly (no CIImage, no CGImage)

Candidate backends (add as a vendored source or SPM dependency):

stb_image (C, single-header) — simplest, handles JPEG/PNG/BMP, link via a small Sources/CStbImage/ C target
libjpeg-turbo + libpng via system libraries (apt install libjpeg-turbo8-dev libpng-dev) — more deps but battle-tested
swift-image or Swim — pure Swift, no system deps, covers common formats

On Apple platforms keep the existing CoreImage path using #if canImport(CoreImage).

2. Audit each model file for remaining Apple API calls.

After MediaProcessing is replaced, compile with:

PATH="/usr/local/cuda/bin:$PATH" SPM_CUDA=1 swift build -c release --jobs 2 2>&1 | grep "error:"

Known locations to check:

Models/Qwen2VL.swift, Qwen3VL.swift — use CGSize for patch grid calculations (LinuxCompat stub should cover these)
Models/FastVLM.swift — uses CoreGraphics for tile sizing
Models/Gemma3.swift — CoreImage for image normalisation

Wrap any remaining calls with #if canImport(CoreImage) ... #else ... #endif.

3. Remove the Linux excludes from Package.swift.

// Before
#if os(Linux)
let vlmExcludes: [String] = [
    "README.md", "MediaProcessing.swift", "Models", "VLMModelFactory.swift",
    "Gemma4AssistantRegistration.swift",
]
#else
let vlmExcludes: [String] = ["README.md"]
#endif

// After (once Linux-compatible image backend exists)
let vlmExcludes: [String] = ["README.md"]

4. Expose FrigateVLM in Sources/Frigate/.

public actor FrigateVLM {
    public init(modelId: String = "mlx-community/Qwen2-VL-2B-Instruct-4bit")
    public func generate(prompt: String, imageData: Data, maxTokens: Int = 512) async throws -> AsyncStream<String>
    public func warmup() async throws
}

Wire it to VLMModelFactory.shared.loadContainer(configuration:) following the same pattern as FrigateLLM, ensuring all MLX.Memory.* calls stay outside container.perform.

5. Add MLXVLM to the Frigate target's dependencies in Package.swift.

.target(
    name: "Frigate",
    dependencies: [
        "MLX", "MLXNN", "Tokenizers",
        "MLXLMCommon", "MLXLLM", "MLXVLM", "mlx_embeddings",  // add MLXVLM
    ],
    ...
)

Estimated scope

Task	Effort
Implement `MediaProcessing.swift` Linux backend with `stb_image`	~2–4 hours
Fix remaining `CoreGraphics` / `CoreImage` calls in model files	~1–2 hours
Write `FrigateVLM` actor	~1 hour
Integration test with Qwen2-VL-2B on RTX 3090	~1 hour

Navigation Menu

Repository navigation

About

Repository files navigation

Frigate

What's inside

FrigateVisionAX — pixel perception

SinatraHarness — the on-device harness (GPL-3.0-only)

Requirements

Setup — Ubuntu 24.04 (fresh machine)

Step 1 — Clone the repo

Step 2 — Run the setup script

Step 3 — Download a model

Step 4 — Use from Swift

Manual build (after setup script, without --skip-build)

Setup — macOS

API reference

FrigateEmbedder

FrigateLLM

FrigateBoost

Vendored sources

Known GPU constraints

TODO — MLXVLM Linux port

What is blocked and why

Prescribed path to completion

Estimated scope

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.vscode		.vscode
Plugins/CudaBuild		Plugins/CudaBuild
Sources		Sources
Tests		Tests
scripts		scripts
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
NOTICE		NOTICE
Package.resolved		Package.resolved
Package.swift		Package.swift
README.md		README.md
setup-frigate-ubuntu.sh		setup-frigate-ubuntu.sh

Latest commit

History

Folders and files

Repository files navigation

Frigate

What's inside

FrigateVisionAX — pixel perception

SinatraHarness — the on-device harness (GPL-3.0-only)

Requirements

Setup — Ubuntu 24.04 (fresh machine)

Step 1 — Clone the repo

Step 2 — Run the setup script

Step 3 — Download a model

Step 4 — Use from Swift

Manual build (after setup script, without --skip-build)

Setup — macOS

API reference

FrigateEmbedder

FrigateLLM

FrigateBoost

Vendored sources

Known GPU constraints

TODO — MLXVLM Linux port

What is blocked and why

Prescribed path to completion

Estimated scope

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages