ReframeWeb

ReframeWeb is an experimental agentic workflow environment for replacing the legacy browser mental model.

It is not a conventional web browser with agents layered on top. Instead, ReframeWeb treats interactive work as a combination of agent-readable semantic capabilities, user-visible visual panels, persistent memory, and deterministic compute tools.

The project starts from the user interaction loop: Python receives audio, BAML drives the agentic flow, and the rest of the system is called from there. Native visual panels, a transport layer, WebAssembly Stores, Data Lenses, Compute Modules, and memory hang off that host-driven flow.

Core Premise

The legacy browser model is built around pages, tabs, browser chrome, DOM state, and visual interaction as the primary interface. ReframeWeb starts from a different assumption: agents and users should share a richer interaction surface where semantic capabilities are first-class.

In ReframeWeb, the first runtime surface is the Agent Host: a Python process that receives audio, coordinates BAML, and routes work through native windows, transport calls, Stores, Lenses, Compute Modules, and memory.

The shared substrate for those components is a typed local protocol. A Semantic Store is an installed WebAssembly Component that exposes resources, functions, schemas, and focused usage guidance through progressive discovery rather than existing as a normal application library. The reusable Rust host and conformance CLI live in semantic-store.

CEF is used as a rendering and native windowing foundation for Visual Panels. It is not the product's central abstraction. The goal is to build a new agent-native workflow layer, not to preserve the browser as the underlying concept.

What This Is Not

• Not a browser extension.
• Not Selenium-style browser automation.
• Not a legacy browser with an assistant bolted onto it.
• Not a DOM-first control system where agents primarily click their way through pages.

Main Components

• Agent Host: A Python manager that handles audio input, transcription, typed data and side-effect boundaries, memory persistence, and TTS playback.
• BAML Flow: The complete routed voice-turn orchestrator. Audio is transcribed and handed to one top-level BAML flow that owns agentic decisions, branches, retries, and task reselection.
• Transport Layer: Local typed routing, framing, calls, streaming, and cancellation. The Semantic Store protocol and platform IPC are implemented; integration with Lenses, Views, and Compute remains future work.
• Semantic Store: A locally hosted WebAssembly Component whose typed resources and functions agents can progressively discover, inspect, and call.
• Visual Panel: A React display surface shown in a native CEF-backed panel window. Users can see state and occasionally click or scroll, but required functionality should be exposed for agent-driven control.
• Data Lens: An optional Rust support layer between a Store and a Visual Panel. It is for cases where behavior needs to differ substantially from the original site or application intent, or where the Store lacks a sensible API surface for React to parse directly.
• Compute Module: A short Rust script for specialized, complex, repetitive deterministic tasks. Compute Modules are not the default path for ordinary repeated behavior, because overusing them would slow down the agentic flow.
• Memory Graph: A graph-backed memory system for preferences, task context, user habits, and recent relevant memories.
• Agent Workspace: A transparent projected filesystem for Codex, OpenCode, and their child processes. It combines memory-derived files, ephemeral working state, explicit native placement, and policy-controlled retained snapshots.

Underlying Technology

ReframeWeb is driven by a small set of deliberate technology choices:

More detail is tracked in Technology.

The projected agent workspace is specified separately in Agent Workspace Architecture Decisions, with a phased delivery plan in the Agent Workspace Implementation Design.

• Rust for native CEF/window agentic bindings, the projected workspace daemon, Data Lenses, Compute Modules, memory-related runtime components, and likely Store implementations compiled to WebAssembly.
• CEF as the embedded rendering and native windowing foundation for Visual Panels. CEF is infrastructure here, not the conceptual model of the product.
• React for Visual Panel content, display state, Store-backed data fetching, and any user-visible controls.
• Python for the Agent Host, which owns setup, audio processing, transport and memory I/O, persistence, and TTS playback through typed BAML boundaries.
• BAML for the complete routed voice-turn flow and all agentic decisions, branches, loops, retries, and task reselection.
• WebAssembly Components for portable Semantic Stores, hosted by an asynchronous Rust/Wasmtime runtime with WASI HTTP.
• Graph database storage for memory, including tagged memory nodes, descriptions, created/read/modified timestamps, relationships, and recency-aware retrieval.
• sounddevice for microphone input and audio playback control.
• pocketsphinx for local keyphrase spotting, including commands such as "jarvis do x" and a "conversation on" mode trigger.
• silero-vad for voice activity detection.
• faster-whisper and whisper.cpp for transcribing spoken prompts before they are passed into the BAML-driven agentic flow.
• kokoro-onnx for TTS playback, using the af_heart voice.

The audio layer should support cancelling spoken playback when the user starts talking without destroying the underlying work the agent was already performing.

Current Status

This repository now contains a working Agent Host prototype. The Python voice loop, graph-backed memory flow, the cross-platform Agent Workspace, and the presentation-independent Semantic Store host are active code. The CEF Visual Panel system is also implemented and active. The trusted React layer, Data Lens runtime, Compute Module runtime, and remaining application-level integration between these subsystems remain future-facing architecture.

Implemented pieces include:

1. A uv-managed Python Agent Host with CLI commands for setup, checks, voice turns, memory seeding, and benchmarks.
2. Local wake/phrase detection, VAD, local Whisper transcription, and turn recording into the memory graph.
3. BAML stages for task choice, conversation memory-search hint generation, and per-domain search-depth selection.
4. A SurrealDB-backed memory graph with roots for providers, tasks, sessions, conversations, session memories, task-choice memories, conversation evaluation memories, and search-depth memories.
5. Graph-based memory retrieval that starts from existing roots and relations, applies search hints and timestamp breadth to candidate nodes, and hydrates parent wrappers needed to explain valid child matches.
6. Benchmark harnesses for task choice, conversation evaluation, and control flow/search-depth behavior.
7. A Rust Agent Workspace daemon and uv-integrated Agent Host surface with Windows ProjFS plus Linux/macOS FUSE mounts, resident deduplicated file bytes, SQLite metadata, BLAKE3 retained blobs, SurrealDB-backed filesystem memory nodes, native scratch paths, journals, checkpoints, and remount support. The daemon is installed by uv sync and controlled through the existing reframe-agent-host workspace CLI over user-local IPC. See Agent Workspace usage.
8. A cross-platform Rust Semantic Store workspace with a strict typed package format, bounded progressive catalog, isolated Wasmtime Component execution, WASI HTTP, streaming and cancellation, secure user-local IPC, a guest SDK, reference Store, and conformance CLI. See the Semantic Store host guide.
9. A production Rust CEF Visual Panel subsystem with native and offscreen presentation, browser/profile lifecycle management, replacement and crash recovery, media coordination, inspection, policy enforcement, and local IPC. See the Panel Manager guide.

Agent Host Setup

For the configured Windows development checkout, run uv run --project agent-host reframe-agent-host voice-turn from the repository. The CLI requests Windows permission for WinFsp when needed, then starts the Host, saved startup Stores and default panels before listening for voice input. No launcher script or environment-variable setup is required. The selection Store creates a fresh session and conversation each launch; existing ones remain in the selector. See Host configuration.

Install BAML 0.17.0 before generating the Agent Host SDK. The project pins that toolchain in agent-host/baml.toml and the matching baml-bridge==0.17.0 runtime in pyproject.toml; the older local-dev toolchain and Git fork are no longer used.

cd agent-host
uv sync
uv run reframe-generate-baml
uv run reframe-check-baml
uv run reframe-agent-host doctor

Use reframe-generate-baml after editing agent-host/baml_src. It checks the BAML project, runs the deterministic workspace BAML tests, generates the Python SDK in a staging directory, verifies it loads in Python, and replaces obsolete generated modules. Python text is normalized, but compiler bytecode and its embedded toolchain metadata are preserved intact: rewriting strings inside 0.17 bytecode can corrupt it. Bytecode may contain compiler-generated checkout paths, so regeneration in another checkout can change the binary artifact. reframe-check-baml runs the same pipeline and fails if it changes a generated file in the current checkout; use it before committing. Both commands resolve the developer checkout from the current directory and fail clearly outside one; BAML sources are not part of the runtime wheel. Do not invoke baml generate directly; the project command also extracts the bytecode into _inlinedbaml.bin, verifies runtime compatibility, and removes stale SDK files.

The Call* BAML entry points expose typed client overrides to Python; Build*Request entry points render requests for debugging without sending them. The routed voice-turn flow still owns decisions and retries in BAML. Its model clients preserve the existing two retries with 300ms and 450ms delays.

Run offline checks without model calls:

baml test -x '*opencode*smoke' -x '*action summary preserves a formatted payload as one value'
uv run python -m unittest discover -s tests -p 'test_*.py'

The Agent Host uses OpenCode Go through its OpenAI-compatible endpoint and reads the API key from OPENCODE_GO_API_KEY.

Model use is a BAML choice, not user-selected global configuration and not Python-side model routing. Each agentic task should have an explicit model assignment through a memory Provider node that points at a BAML surface. The current model-backed prompt layers use gpt-5.6-luna with high reasoning effort. This includes routing, conversation evaluation, memory search and selection, prompt construction, task execution, action-history summarization, task completion, and request completion.

Current benchmarked OpenCode Go model IDs include:

• kimi-k2.7-code
• kimi-k2.6
• kimi-k2.5
• glm-5.1
• gpt-5.6-luna
• glm-5
• deepseek-v4-pro
• deepseek-v4-flash
• mimo-v2.5-pro
• mimo-v2.5

First Voice Pipeline

The first runnable microphone path is intentionally narrow and testable:

cd agent-host
uv run reframe-agent-host transcription-check
uv run reframe-agent-host audio-devices
uv run reframe-agent-host voice-turn --device 1

The project config sets uv's package install link mode to copy, which avoids Windows hardlink warnings when uv's cache and this repository live on different drives.

If uv is not on PATH on Windows, use the local runner instead:

cd agent-host
.\reframe-agent-host.cmd transcription-check
.\reframe-agent-host.cmd audio-devices
.\reframe-agent-host.cmd voice-turn --device 1

voice-turn listens for one utterance, detects the speech boundary, transcribes it with the configured local Whisper backend, records the current turn when session and conversation IDs are available, sends the transcript through the BAML control flow, retrieves graph memory context, and prints concise per-stage summaries and latencies. When memory retrieval runs, the CLI prints the retrieved memories directly instead of dumping the full turn result as JSON.

The current top-level BAML voice-turn path is:

1. Choose an initial task from the task catalog.
2. Generate memory search hints from the conversation and selected task.
3. Choose timestamp breadth for each search domain.
4. Retrieve graph memories through a typed host boundary.
5. Select relevant memories and compose the task prompt.
6. Execute the task through a typed host boundary and summarize its actions.
7. Run the existing completion pass/fail check.
8. On failure, generate and persist a validation_reply, then either retry the same task with stacked task-session replies or return to task selection.
9. On success, return the turn result without retaining task-local retry data.

Python supplies data and performs external side effects at the named boundaries. It does not decide which BAML path runs. The complete agentic turn is kept in a single BAML graph so the operational flow remains inspectable.

Memory retrieval is relation-first rather than table-wide. The task catalog is searched from the task root. Past conversation context is searched through session, conversation, message, and session-memory relations. Search hints are alternatives, so any positive tag or string hint can match a candidate. Timestamp breadth is restrictive: candidate nodes must pass created_at, updated_at, and, when present, read_at cutoffs. Parent wrappers are included when a valid child match needs them for context. Ordinary current-session memories are always included. Saved/verified project locators require relevance selection even in their originating session; unselected paths and Store bindings are not injected.

The default memory database is anchored at agent-host/.reframe-memory, regardless of the launch directory. REFRAME_MEMORY_URL still overrides it. For installations with the earlier repository-root database, the guarded, dry-run-first migration is documented in Lifecycle corrections.

By default, WakeCommand mode is wake-gated locally with a rolling PocketSphinx phrase recognizer. It does not require an account, network call, or paid wake-word service. Say the single-word trigger "jarvis" followed by the prompt. The phrase "conversation on" switches the host into continuous conversation mode. If more speech follows in the same utterance, such as "conversation on this is a test", the trigger audio is trimmed away and the remaining command audio is sent through VAD and local transcription.

uv run reframe-agent-host voice-turn --device 1 --no-task-choice

On Windows, Linux, and macOS, the default transcription backend tries CUDA faster-whisper first, then whisper.cpp, then CPU faster-whisper. CUDA is still supported when available. Non-CUDA GPU support comes through a whisper.cpp binary compiled for the target backend, such as Metal/Core ML on macOS, Vulkan or OpenVINO on Windows/Linux, and ROCm on Linux.

Examples:

uv run reframe-agent-host transcription-check --transcriber whisper-cpp --transcriber-device metal --whisper-cpp-bin /path/to/whisper-cli --whisper-cpp-model /path/to/ggml-model.bin
uv run reframe-agent-host voice-turn --transcriber whisper-cpp --transcriber-device vulkan --whisper-cpp-model /path/to/ggml-model.bin
uv run reframe-agent-host voice-turn --transcriber faster-whisper --transcriber-device cpu --whisper-cpu-compute-type int8

Windows runner equivalent:

.\reframe-agent-host.cmd voice-turn --device 1 --no-task-choice

Useful tuning flags:

• --wake-keyword jarvis to configure local wake keyphrases.
• --conversation-on-phrase "conversation on" to configure the local conversation-mode trigger. The phrase is treated as control input only, not as a user request.
• --conversation-on-confirm-window-ms 2000 to tune how much recent audio is sent to Whisper for the second-layer confirmation.
• --wake-gain 1.0 to tune local phrase detector gain without changing the audio sent to Whisper.
• --wake-threshold 1e-30 to tune local PocketSphinx KWS confirmation for wake-keyword candidates.
• --wake-replay-pre-ms 0 starts command replay at the confirmed wake boundary so Whisper does not need to transcribe the wake word.
• --debug-audio-dir .debug-audio to opt in to saving local WAV clips and JSON sidecars for missed wake timeouts and detected keyphrases.
• --debug-audio-seconds 8 to tune how much rolling microphone audio is kept for those debug clips.
• --debug-audio-period-seconds 5 to opt in to saving rolling clips every few seconds while waiting for a wake phrase.
• --vad silero to require Silero VAD, or --vad energy to use the simple RMS fallback.
• --vad-threshold 0.35 to tune Silero sensitivity for quieter microphone input.
• --min-silence-ms 0 is the default. Increase it only if the utterance cuts off too early.
• --final-silence-ms 1450 controls how long a provisional endpoint can be cancelled if speech resumes after Whisper starts.
• --pre-speech-ms 320 keeps audio just before VAD start so first syllables are not cut off.
• --wake-carry-ms 220 keeps audio around wake detection so commands that start immediately after the wake word are not clipped.
• --energy-start-threshold 0.02 if the fallback detector starts too easily.
• --transcriber faster-whisper or --transcriber whisper-cpp to choose a backend explicitly.
• --transcriber-device cuda, cpu, metal, coreml, vulkan, openvino, or rocm to choose the preferred local runtime.
• --whisper-model large-v3 to choose a faster-whisper model name or local faster-whisper model path.
• --whisper-cpp-model /path/to/ggml-model.bin to use a local whisper.cpp ggml model.
• --whisper-cpp-bin /path/to/whisper-cli when the whisper.cpp binary is not on PATH.
• --whisper-compute-type int8_float16 to test lower CUDA memory use than the default float16.
• --whisper-cpu-compute-type int8 to choose the CPU fallback compute type.
• --no-cpu-fallback to fail fast when the requested GPU backend is unavailable.

On Windows, the CUDA faster-whisper path checks for CUDA 12 cuBLAS DLLs before listening, including project-local agent-host\.cuda\bin, normal CUDA Toolkit installs, and NVIDIA Python wheel locations such as nvidia-cublas-cu12.

Saved debug WAVs can be replayed through the local wake recognizer without calling Whisper:

uv run reframe-agent-host debug-wake-audio ".debug-audio/*.wav"

Development Philosophy

ReframeWeb is being designed around a few working principles:

• Agent-native semantic interfaces should be the primary interaction layer.
• The first implementation path starts where the user interacts: audio into the Python Agent Host, then BAML-driven agentic flow.
• Visual Panels are React display outputs for showing useful state and controls. Users may click or scroll when they want to, but core functionality should be available to the agent rather than requiring manual UI operation.
• Compute Modules are for specialized, complex, repetitive deterministic behavior where a module genuinely improves reliability or clarity. Ordinary repeated actions should stay in the Store and agent flow.
• Personalization should usually happen through Visual Panels and Store-backed presentation. Data Lenses are for larger behavioral changes or for supporting cases where the Store does not expose a sensible API surface for React.
• Incremental development should build the actual project architecture rather than a substantially cut-down or throwaway version of it.

Current project contracts

This project is unreleased and supports one current contract per subsystem. Workspace policies separate import exclusions, checkpoint exclusions, and native placement; exclusions never imply disk storage. Store packages require canonical annotations. Filesystem memories require their persisted canonical identity. DRM accepts only the current signed evidence contract, with service acceptance checked separately. Obsolete data is rejected without automatic conversion; see cleanup verification and existing data.