HERMIT

A voice-first personal agent that lives on a 1 GB Raspberry Pi.

One Rust binary · five tools · four-layer memory · hardware echo cancellation

by Sudo Automations

---

HERMIT is the reference build of the Sudo agent harness: a headless voice assistant engineered around one number — the silence between "you stop talking" and "it starts answering". Every architectural decision in this repository is downstream of that latency budget.

It runs on a Raspberry Pi 4 with 1 GB of RAM, a reSpeaker XVF3800 mic array, and a 3 W speaker. It wakes on "Hey Sudo", answers in Hindi, Odia, or English, plays All India Radio and Spotify, and remembers what matters — while fitting the entire agent, wake word included, in a single 9 MB binary.

The shape of the system, in one sentence: hardware does the signal processing, one Rust binary does the agent, rented cloud does the intelligence — and everything between microphone and speaker streams.

Measured performance

hermit/scripts/bench.sh gates every release; it exits non-zero if any p50 misses. Numbers below are measured on the Pi over Wi-Fi against live services, not estimated.

Gate	Target	Measured
Local harness overhead (route + recall + assemble)	≤ 15 ms	1.1 ms
Fast-path device command (pause / volume / next)	< 50 ms, no LLM	1.0 ms
Text: first token, no tools	< 700 ms	366 ms
Voice: first audio, no tools	< 1.2 s	gated on TTS key
Voice: first audio, one web search	< 2.0 s	gated on TTS + search keys

Connection pre-warming alone is worth ~130 ms: a cold TLS handshake reaches first byte in 497 ms p50, the daemon's pooled warm connection in 366 ms.

What's in the box

hermit/src/router.rs	Regex fast path: pause, volume, named radio — answered in ~1 ms with no LLM call
hermit/src/orchestrator.rs	The turn loop, with a hard two-round tool cap (schema-starved final round, stray calls dropped)
hermit/src/memory/	SQLite + FTS5 (BM25, no embedding model) behind a structural write firewall
hermit/src/speech/	Streaming STT/TTS over persistent websockets, sentence chunker, and the project's own "Hey Sudo" wake word — a 3-graph ONNX pipeline ported from the Python reference and verified numerically against it
hermit/src/audio/	Single-card ALSA path with instant barge-in flush and AEC keepalive
hermit/src/music/	mpv IPC for internet radio (232-station Akashvani catalogue) + Spotify via librespot
hermit/tools/sudo-console.py	Dependency-free curses operator console: live meters, transcripts, mute leases, volume
hermit/deploy/	provision.sh, systemd units, scoped sudoers, ALSA config — the full device story

Quick start

Target hardware: a Raspberry Pi 4 (1 GB is enough), a reSpeaker XVF3800 USB 4-mic array, and any small speaker on its 3.5 mm out. Never compile on the Pi; the 1 GB of RAM is for inference-adjacent work, not rustc. Builds run in an arm64 container on your dev machine.

# on your dev machine (Docker or Colima running)
git clone https://github.com/heysudo/sudo-agent-harness && cd sudo-agent-harness/hermit
cargo test                         # 241 tests; no network, no API keys
scripts/build-pi.sh                # arm64 binary via rust:1-bookworm container

# on the device (fresh Raspberry Pi OS / Debian 12+, SSH enabled)
sudo deploy/provision.sh           # users, dirs, systemd units, sudoers, ALSA
sudoedit /etc/hermit/hermit.env    # keys: start from .env.example (Cerebras required)

# back on your dev machine
scripts/deploy.sh <pi-host>        # rsync binary + config + firmware, restart

Then say "Hey Sudo". hermit/docs/BRINGUP.md walks the hardware phase (mic-array firmware, ALSA verification, latency gates); hermit/deploy/README.md covers provisioning in detail. The only required key is CEREBRAS_API_KEY; everything else degrades gracefully — no TTS key means text-only answers, no Spotify means radio still works.

Security posture

The daemon assumes it will one day sit on a hostile network and eat hostile web pages. The interesting defenses, all acceptance-tested:

• WS gateway is auth-gated and capped. Loopback by default; a non-loopback bind refuses to start without HERMIT_WS_TOKEN (bearer header or /auth first message), connections are capped, frames are limited to 64 KiB, and remote mic-arming (/listen) is a separate opt-in (server.ws_allow_listen).
• Memory firewall. There is no API to write raw text into memory; tool output structurally cannot reach the reflection transcript (tests/memory_firewall.rs proves it with a poisoned page).
• Skill quarantine. Model-distilled skills land in /var/lib/hermit/skills-pending/, which is never indexed into prompts. A human moves a reviewed draft into the root-owned config dir to activate it (tests/skill_quarantine.rs proves drafts can't reach recall) — closing the second-order injection path where a poisoned page transits the model's own answer into persistent instructions.
• No shell anywhere. The one child process (piper TTS) gets text via stdin. Music is mpv JSON-IPC and the Spotify Web API. Fetch URLs are scheme-whitelisted with credentials-in-URL rejected.
• Hardened unit. ProtectSystem=strict, closed device policy with an explicit ALSA exception, seccomp, memory cgroups, watchdog; secrets only via root-owned EnvironmentFile, never in config or repo.

Design positions

Latency is a budget, not a hope. Persistent TTS websockets (a cold handshake costs 100–400 ms against a 1.2 s first-audio budget), HTTP pre-warmed at boot and re-warmed every 4 minutes, and a sentence chunker that starts speaking the first clause while the model is still generating.

The memory firewall is structural. There is no API to insert raw text as a fact. The only write path is a typed ReflectionBatch built from the reflection model's own JSON; tool output can never enter the transcript that reflection reads. tests/memory_firewall.rs proves it with a poisoned page.

Device control never waits for a model. "Pause", "volume up", and "play akashvani cuttack" resolve by regex against configured stations in about a millisecond. The LLM is for questions, not for switches.

One sound card, on purpose. Every audio source — speech, radio, Spotify — sums through the XVF3800, so the hardware echo canceller always has its loopback reference. That is what lets the wake word interrupt music. The console's mutes are TTL leases (a crashed console can never leave the device deaf or silent); its volume is an edge-triggered command that coexists with voice control instead of fighting it.

No embedding model. BM25 over FTS5 costs zero RAM and, on a personal corpus, retrieves as well as a small embedder would.

Nothing heavy in the hot path. No Python or Node between microphone and speaker, no PipeWire/PulseAudio (plain ALSA), no Docker on the Pi, no local models beyond the wake word, and no compiling on the Pi — its 1 GB is spoken for. These are locked positions, not defaults to tune.

Hermit and Hermes Agent

Hermit is not a fork of Hermes Agent — it is the same agent idea rebuilt for a device that must answer out loud in under a second. Hermes Agent is the general brain; Hermit is the ears and mouth.

	Hermit	Hermes Agent
Form	one ~9 MB Rust binary, single process	Python framework, many subsystems
Home	a passively cooled 1 GB Pi, always listening	desktop or server, invoked on demand
Optimized for	latency — wake word to first audio under 1.2 s	capability — depth, breadth, correctness
Interface	microphone and speaker: wake word, earcons, streamed TTS	chat, files, editors, GUI panes
Tools	five, behind a regex fast path and a two-round cap	hundreds, plus MCP servers, subagents, cron
A turn is	one question, one spoken answer, then silence	possibly an hour of autonomous multi-step work
When unsure	degrades gracefully and stays quiet	stops and asks

What they share is the skeleton, deliberately: the same turn loop (route → tools → model → streamed answer), the same hosted models underneath, skills loaded from a config directory, a persistent memory store with a reflection pass, a gateway abstraction over front ends, and a persona defined in config rather than code. Hermit is what remains when that design is reduced to only what can run in milliseconds on a $50 board — and rewritten in Rust so it does.

Operator console

sudo-console ships with the device: a zero-dependency curses TUI with live RMS and wake-score meters, interim transcripts, the running conversation, and guarded lifecycle controls (m mic, s speaker, -/+ volume, r/b/p restart/reboot/poweroff behind a y-confirm and a narrowly scoped sudoers helper — the TUI itself never holds root).

Honest limitations

• Spotify Connect requires Premium; playback health further depends on upstream librespot ↔ Spotify key-service behaviour.
• Apple Music cannot be driven by an agent on Linux; AirPlay push works, agent control does not.
• This is a 16 kHz voice pipeline into a 3 W mono driver — engineered for speech, news, and casual listening, not hi-fi. The single-card topology is load-bearing (it is the AEC reference); do not "fix" it.
• Turbo web search covers English and Japanese; other Latin-script languages are not language-detected on-device (a language-ID model does not fit the RAM budget).
• Live radio has no "next track". The device says so instead of ignoring you.

Repository layout

hermit/          the agent: source, config, deploy, docs, tools
  src/           Rust daemon (edition 2024)
  config/        hermit.toml, prompts, skills, stations.toml
  deploy/        provision.sh, systemd units, Cross.toml, asound.conf
  tools/         sudo-console.py + tests
  docs/          BRINGUP.md hardware checklist
  scripts/       build-pi.sh, deploy.sh, bench.sh, phase0.sh
CLAUDE.md        agent-facing repo instructions

A deeper technical README — including hardware findings like the XVF3800's playback-slaved capture clock and measured AEC suppression — lives in hermit/README.md.