Memory — the bot's persistent, per-bot store

The durable memory subsystem: the per-bot store that outlives the perception snapshot ring, lets the bot remember "Guru prefers tea" across sessions, and backs the brain's recall/inspect/remember tools. This is the implementation + integration reference; the design rationale (why these axes, the keystone argument) is perception-to-brain.md Decision 4.

Status: built + tested. Lives in the perception module; written by the agent/operator (explicit remember) and by the perception summarizer's fact-extraction at trim time; consumed by the brain through a facade. (The background "long-term memory curator" job was removed 2026-07-10 — its consolidate/reconcile loop, memory/longterm/, the /api/perception/curator* REST + knobs, and the console panel are gone; fact extraction is now one output of the perception summarizer's per-clock-hour trim pass. See §0 / §5a.)

What memory is NOT: it is not a transcript archive, and not a bulk dump of perception. It holds discrete facts — recorded by the agent/operator, or extracted from the perception summarizer's trim pass (append + light semantic dedup, selective). Session chat, old sessions, and raw snapshot streams do not flow in wholesale — see §0.

Terms used below (defined elsewhere — this doc only references them): • Snapshot — one timestamped observation from one sensor stream (👁 vision / 🎙 speech / 👤 identity / 😮 emotion / 🤖 bodymotion), living in a short-lived in-memory ring; the raw perception feed memory outlives. → perception-pipeline.md. • Summary — a Gemini fusion of the snapshot streams over a window, cached per bot (LastSummary). • Grounding — the per-turn context string injected into every brain turn = the last summary (stamped with staleness) + the raw snapshots since it; it evaporates after the turn. → perception-to-brain.md Decision 3. The bridge to memory: perception → grounding → the agent reads it → if it chooses, remember() persists a durable fact (§0).

0. What actually populates memory today
- What does NOT flow into memory (the common misconception)
1. Where it lives — code map
2. The data model
3. The store API (MemoryStore)
4. Recall — structured ∧ semantic (recall())
5. Integration point A — perception (the home + the facade)
- 5a. How facts enter the store — the summarizer's fact-extraction
6. Integration point B — the brain (the tools)
6a. Integration point C — tasks (DIRECT, not the wire)
7. The face gallery — the first type:'person' memory
8. What's NOT wired yet (honest gaps)
See also

0. What actually populates memory today

Memory is written by a few deliberate paths — the agent/operator (explicit remember), and the perception summarizer's fact-extraction (automatic, at trim time): the same per-clock-hour trim pass that self-compresses aged perception into a span-summary also extracts durable facts into this store (append + light semantic dedup). Tasks may also touch memory directly as a side-effect. Raw snapshots are NOT bulk-distilled — extraction is selective (it promotes only the salient few).

Removed 2026-07-10 — the background curator. There used to be a separate always-on long-term memory curator loop (memory/longterm/: consolidate salient speech → derived beliefs, plus a reconcile pass revising/forgetting them, with its own watermark). It was deleted — the folder, curator.ts/sources.ts, the /api/perception/curator* REST and its knobs (PERCEPTION_CURATE, PERCEPTION_RECONCILE_MS, maxBatch/confMax/reconcileMs/ reconcileMin), and the console curator panel are all gone. Fact-writing is now folded into the perception summarizer (§5a). There is no separate consolidate/reconcile machinery.

Touches memory today	How	Path
The agent, mid-conversation	the brain decides a fact is worth keeping	`remember` / `update_memory` / `forget_memory` tools (`tools.ts`, conf 0.7)
A console operator	typing into the 4c memory inspector	`POST` / `PATCH` / forget on `/api/perception/memory*`
The perception SUMMARIZER (auto, at trim time)	the per-clock-hour trim pass that writes a span-summary also extracts durable facts into this store (append + light semantic dedup)	the summarizer's fact-extraction (§5a)
A task / behaviour (side effect)	a task that acts may record a side-effect belief — directly (shared code + db), not over the wire	`this.memory` on the Task base class (§6a)

Two memory tiers. The snapshot ring is short-term memory (real-time, volatile); this store is long-term (durable). The perception summarizer bridges the two — that producing/maintaining memory artifacts is the pipeline's job (not a behaviour's, whose job is to ACT) is the purpose-rule (behaviour-conductor.md §4a).

What does NOT flow into memory (the common misconception)

Session chat / transcripts. Conversation history lives in a separate store — the brain's per-session JSONL under .data/brain/ (brain/store.ts), not the memory sqlite (.data/orbit.db). The two stores never cross except when the agent manually calls remember.
Old sessions. On idle-close the brain runs #compactSummary (session.ts) — an LLM digest of the transcript — but it writes store.setSummary() (the brain session index, used to seed the next session's prompt), not memory.remember(). That summary is never promoted to a memory row.
Most perception snapshots / fusions. Vision/identity/emotion/bodymotion snapshots reach memory only indirectly via the agent (injected as the per-turn grounding block, perception-to-brain.md Decision 3; the brain calls remember if it chooses; grounding then evaporates). The automatic path is the summarizer's fact-extraction: at trim time the same pass that writes the span-summary pulls salient durable facts into this store (§5a) — selectively, not the whole stream.
The face gallery. Enrolled faces are still their own JSON file, not memory rows (§7).

The two stores, side by side:

Store	Backing	Holds	Lifetime
Brain session store	JSONL, `.data/brain/`	full turn transcripts + per-session close summary	bounded ring; summary seeds the next session
Memory store (this doc)	sqlite, `.data/orbit.db`	discrete agent-/operator-recorded facts	durable, supersede-not-delete

So a memory row comes from one of: an explicit agent/operator record, the summarizer's fact-extraction at trim time (selective, not bulk-distilled), or a task's side-effect write. Everything the robot merely saw or said otherwise lives in the short-term ring and ages out.

1. Where it lives — code map

All paths under brain/server/src/.

Piece	File	What
The store	`modules/perception/memory/store.ts`	`MemoryStore` — sqlite-backed CRUD + recall + lineage
The embedder	`modules/perception/memory/embedder.ts`	`geminiEmbedder()` — text → vector for semantic recall
Shared db	`core/db.ts`	`orbitDb()` — the one `.data/orbit.db` (also backs config/cost)
Facade (perception → brain seam)	`modules/perception/index.ts`	`MemoryApi` interface + `getMemoryApi()`
Fact-extraction (auto write)	the perception summarizer's trim pass (`retention.ts` `selfCompressAndTrim` + `summarizer.ts`)	extracts durable facts into `MemoryStore` at trim time, alongside the span-summary (§5a). (Replaces the removed `memory/longterm/` curator.)
Brain tools	`modules/brain/tools.ts`	`buildMemoryTools(bot, getMemory)` — the 7 model-facing tools
Task access (direct)	`tasks/_harness/memory.ts`	`this.memory` — `BotMemory` over the same store, bot-scoped (§6a)
Tool schemas/descriptions	`modules/brain/schemas.ts`	`RECALL_MEMORY_DESC`, `rememberSchema`, …
REST (the 4c inspector)	`modules/perception/index.ts`	`/api/perception/memory*` routes
Tests	`memory/store.test.ts`, `tasks/_harness/memory.test.ts`	store + direct task-access; tool tests in `brain/tools.test.ts`

2. The data model

A memory is described by three axes + confidence (don't conflate them), with first-class lineage and supersede-not-delete history.

memory(
  id, bot_id,      -- (renamed from dock_id; auto-migrated on boot)
  type,           -- person | summary | event | preference | fact | place
  subject,        -- normalized entity tag ('guru','kitchen'); '' if none
  claim,          -- the human-readable belief ("prefers tea")
  value_json,     -- structured extra (free-form), or null
  confidence,     -- 0..1, on EVERY memory (the agent hedges on it)
  derivation,     -- observed (from a snapshot) | derived (a summary/inference)
  status,         -- active | revised | forgotten
  created_at, valid_from, valid_to,   -- ms; valid_to NULL while active
  supersedes,     -- the prior memory id this revised, if any
  embedding BLOB  -- the claim's vector, for semantic recall
)
memory_lineage(memory_id, source_kind, source_id)   -- derived rows: what fed this
-- indexes: (bot_id,status) · (bot_id,subject) · (bot_id,type)

The axes (store.ts types MemoryType / Derivation / MemoryStatus):

type — the kind; retention/revision policy is keyed on this (a summary ages out; a person is revised, rarely purged). v1 set: person/summary/event/ preference/fact/place. Extend as needs surface.
subject — WHO/WHAT it's about; a normalized string tag (normSubject(): lowercased, trimmed, collapsed). An exact lookup, not a vector — vectors would fuzz it. (Promote to an entities table only when relationships between entities matter.)
derivation — observed (straight from a stream) vs derived (a summary/ inference). Derived rows carry lineage.
confidence — first-class, so the agent can say "I think you said…" vs "you said…".

Evolution is by supersede, never delete-in-place (MemoryStatus):

active = the current best belief; revised = a superseded prior version (kept for history/lineage, valid_to stamped, linked via supersedes); forgotten = purged (out of active recall, but the row stays). So "what did you believe last week?" is answerable.

3. The store API (`MemoryStore`)

Constructed new MemoryStore(db, embed) — both injected (tests pass :memory: + a deterministic fake embedder).

Method	What
`remember(NewMemory) → id`	insert a new active memory; embeds `subject: claim`. `derivation` default `observed`, `confidence` default 0.6
`revise(id, patch) → newId\|null`	insert a new active version (carrying axes unless overridden), mark the old `revised`, link `supersedes`. Re-embeds
`forget(id) → bool`	mark `forgotten` (not a row delete)
`recall(RecallFilter) → MemoryRow[]`	the workhorse — see §4
`get(id)` / `lineage(id)`	one row / its lineage edges (the "why" view)
`subjects(botId)` / `recent(botId,limit)`	orientation: who/what do I know about; newest memories
`count(botId)`	active count

Embeddings are stored as a BLOB (f32ToBlob/blobToF32, copied to avoid aliasing a reused sqlite buffer). If the embedder is down, the memory is still stored — just not semantically recallable until re-embedded.

4. Recall — structured ∧ semantic (`recall()`)

Two access modes, combinable (an LLM uses both):

Structured — indexed filters on (botId, type, subject, interval, status). Backs "what do I know about Guru", "what did I learn this morning". interval filters on valid_from (WHEN the belief took effect) — the intuitive "what happened in this window" grammar, not an open-ended validity overlap (an active memory's validity runs to ∞ and would match every window).
Semantic — pass a natural-language query; structured filters narrow the candidate set, then results are ranked by cosine (cosine()) of the query's embedding against each candidate's stored vector. No vector DB — an in-process brute-force scan over the bot's bounded memory set (hundreds–low-thousands of rows) is sub-ms. Embedder down → graceful fallback to recency.
Without a query, results are most-recent-first (valid_from DESC, rowid DESC).
active-only by default; includeInactive to see superseded/forgotten.

The embedder (embedder.ts): Gemini gemini-embedding-001 (env MEMORY_EMBED_MODEL). No GEMINI_API_KEY → a no-op embedder (always null) so the store still works, just recency-only. Embedder = (text) => Promise<Float32Array|null> is pluggable.

5. Integration point A — perception (the home + the facade)

Memory lives in the perception module (modules/perception/index.ts): the module constructs the single store —

const memory = new MemoryStore(orbitDb(), geminiEmbedder());

— and exposes it two ways:

MemoryApi facade (getMemoryApi()) — the in-process seam the brain reads (never importing MemoryStore directly, mirroring FaceToolsApi/PerceptionGroundingApi):

interface MemoryApi {
  recall(f: RecallFilter): Promise<MemoryRow[]>;
  inspect(id): { memory; lineage } | undefined;
  remember(m): Promise<string>;      // records as derivation:'observed'
  update(id, patch): Promise<string|null>;   // → store.revise
  forget(id): boolean;
  subjects(botId) / recent(botId,limit) / count(botId);
}

getMemoryStore() also exists for code that needs the raw store (the REST routes).

REST (/api/perception/memory*) — backs the console memory inspector (4c):
- GET /memory?bot=X[&query=&subject=&type=&inactive=1] → recall/list
- GET /memory/item/:id → one memory + its lineage (the "why do I believe this" view)
- POST /memory {bot,type,subject?,claim,confidence?} → add (console)
- PATCH /memory/item/:id {claim?,confidence?,subject?} → revise (supersede)
- DELETE / forget paths likewise.

5a. How facts enter the store — the summarizer's fact-extraction

This replaces the removed background curator (memory/longterm/, deleted 2026-07-10). There is no longer a separate always-on consolidate/reconcile loop with its own watermark. The design rationale for the unified model is perception-pipeline.md §7c. The old curator's design trace (decision-traces/long-term-memory-curator.md) is kept only as history.

Durable facts are written by the perception summarizer's trim pass — one pass, per closed clock-hour, with two outputs from the same call:

A span-summary — the aged raw perception, compressed into a durable hourly digest (span-summaries.jsonl). This is perception self-compressing its tail (perception-pipeline.md §7c).
Fact extraction — durable facts pulled into this MemoryStore (append + light semantic dedup against existing rows, so the same fact seen twice doesn't duplicate).

There is no separate checkpoint/watermark and no reconcile job. Retention is record-granular by closed clock-hour (selfCompressAndTrim in retention.ts), so once an hour closes its records are compressed → span-summary + facts, then trimmed. The MemoryStore remains a generic fact primitive (semantic recall + subject/type/confidence + supersede-history); the summarizer is simply one writer into it, the agent's remember tool the other.

6. Integration point B — the brain (the tools)

The brain reaches memory only through the facade, wired in session.ts deps (getMemory?: () => MemoryApi | undefined) and turned into tools per-turn:

...(this.#d.getMemory ? buildMemoryTools(this.bot, this.#d.getMemory) : [])

buildMemoryTools(bot, getMemory) (tools.ts) builds 7 model-facing tools (the agent's agency over memory — perceive / discover / inspect / mutate):

Tool	→ store	Purpose
`recall_memory({query?,subject?,type?})`	`recall`	the workhorse — semantic and/or structured
`list_subjects()`	`subjects`	orientation: who/what do I have memories about
`inspect_memory({id})`	`inspect`	the claim + lineage + confidence ("why do I believe this")
`remember({claim,subject?,type?})`	`remember` (conf 0.7)	record a fact learned in conversation
`update_memory({id,claim})`	`update`→`revise`	revise a corrected memory (keeps history)
`forget_memory({id})`	`forget`	purge
`remember_face` (the gallery)	face tools	the `type:'person'` specialization (see §7)

Descriptions/schemas live in schemas.ts (RECALL_MEMORY_DESC, rememberSchema, …). Memory now also feeds grounding directly (PASSIVE awareness). Beyond the pull tool, the per-turn grounding block (grounding.ts memoryGroundingSlice) appends a small, confidence-ranked slice of durable beliefs about who is currently present — so the agent passively knows what it knows without having to call recall_memory. It's deliberately curated to respect the noisy-derived- belief risk: filtered by confidence (GROUNDING_BELIEF_MIN_CONF, default 0.4), capped (GROUNDING_BELIEF_MAX, default 6), biased to present subjects (joined from the latest identity record), and tagged as "beliefs — weigh by confidence" so the agent hedges rather than treating them as hard fact. The live perception grounding (last summary + raw-since) is still the first part of the block — see perception-to-brain.md Decision 3.

6a. Integration point C — tasks (DIRECT, not the wire)

A task (a background process — see tasks.md) reaches memory directly, not over the WS. A task shares the station's code + .env, and memory is just a sqlite file + an env-keyed embedder — fully reconstructible in the task process. So the Task base class exposes this.memory (tasks/_harness/memory.ts): a BotMemory that opens the same MemoryStore on the same .data/orbit.db, lazily.

This is the "touch memory as a side effect" affordance the purpose-rule permits a task whose goal is acting (it may record that it acted). Two properties:

Bot-scoped — the bot is bound at construction from the task's verified identity, never an argument. recall only sees that bot; revise/forget/inspect refuse a belief belonging to another bot.
Direct, not a capability — memory failed the "needs live in-process state" test (it's sqlite + env), so it is NOT a wire capability (only the decoded frame / body link are). See tasks.md "Direct vs. the wire".

One-writer note. A task opening its own MemoryStore is a second sqlite connection to the same file (alongside the station's). better-sqlite3 serialises writes per the WAL; at this scale that's fine. A task's write is NOT observed live by the station's in-memory consumers (the curator's feed, the brain facade) — they see it on their next read. Nothing today needs live cross-process write-notification; if that changes, an event seam would be the fix, not the wire.

Tested in tasks/_harness/memory.test.ts (round-trips, supersede, the bot-scoping guards) over a real in-memory store.

7. The face gallery — the first `type:'person'` memory

The enrolled face gallery (face/gallery.ts, faces↔names) is conceptually the first evolving, provenance-carrying, per-bot memory — it improves (confirm_face appends a sample), corrects (forget_face), and carries provenance (each sample keeps its photo + addedAt). Today it's still its own JSON file with a bespoke shape, coupled to the module but not yet folded into this store. The plan: make it one type:'person' view over the unified store, keeping the FaceToolsApi facade stable so the brain is unaffected (design + migration: perception-to-brain.md 4.5/4.6). Until then, person-memory = the gallery; everything else = this store.

8. What's NOT wired yet (honest gaps)

Fact-extraction is new-shaped, not deeply verified. Facts now come from the summarizer's trim pass (§5a) rather than the removed curator. The extraction prompt + the light semantic-dedup threshold are first-guess and want tuning against a real busy hour; multi-source, event-time-aligned extraction (vision lag, late-emitting streams) is still the harder open problem.
No reconcile/revision loop any more. The removed curator had a reconcile pass that revised/forgot drifted beliefs on a slow interval. That is gone — facts are appended (with dedup) and revised only by the agent's update_memory/forget_memory tools or a future explicit path. Automatic contradiction-resolution/decay is not currently wired.
Retention/rollup tiers. Retention is now record-granular by closed clock-hour (selfCompressAndTrim); per-bot raw-vs-summary budgets are still TBD (perception-pipeline.md §7c).
Gallery → unified-store migration (§7).
Embedder choice is a v1 call (Gemini API); a local embedder is an open option.

Navigation Menu

Repository navigation

memory.md

memory.md

Memory — the bot's persistent, per-bot store

Contents

0. What actually populates memory today

What does NOT flow into memory (the common misconception)

1. Where it lives — code map

2. The data model

3. The store API (`MemoryStore`)

4. Recall — structured ∧ semantic (`recall()`)

5. Integration point A — perception (the home + the facade)

5a. How facts enter the store — the summarizer's fact-extraction

6. Integration point B — the brain (the tools)

6a. Integration point C — tasks (DIRECT, not the wire)

7. The face gallery — the first `type:'person'` memory

8. What's NOT wired yet (honest gaps)

See also

FilesExpand file tree

memory.md

Latest commit

History

memory.md

File metadata and controls

Memory — the bot's persistent, per-bot store

Contents

0. What actually populates memory today

What does NOT flow into memory (the common misconception)

1. Where it lives — code map

2. The data model

3. The store API (MemoryStore)

4. Recall — structured ∧ semantic (recall())

5. Integration point A — perception (the home + the facade)

5a. How facts enter the store — the summarizer's fact-extraction

6. Integration point B — the brain (the tools)

6a. Integration point C — tasks (DIRECT, not the wire)

7. The face gallery — the first type:'person' memory

8. What's NOT wired yet (honest gaps)

See also

3. The store API (`MemoryStore`)

4. Recall — structured ∧ semantic (`recall()`)

7. The face gallery — the first `type:'person'` memory