Merge dev into master

README.md

@@ -1 +1,115 @@
-# plugin-note-graph
+# Note Graph
+
+Note Graph is a [Joplin](https://joplinapp.org) plugin that visualizes your
+notes as an interactive graph. It connects notes by the links and tags you
+already use, and by semantic similarity using Joplin's own built-in AI. This
+way you can see how your notes actually relate to each other, not just how
+they're filed. An optional second stage, LLM enrichment (Pass B), adds topic
+labels and one-line explanations for those connections.
+
+## Features
+
+- **Explicit connections.** Notes linked with `[text](:/noteId)` or sharing
+  a tag are connected automatically, no setup required.
+- **Semantic connections.** With Joplin AI enabled, the plugin
+  embeds your notes and adds edges between notes that are related in
+  content even when nothing links them. Tunable threshold and edge count.
+- **Optional LLM enrichment (Pass B).** Labels each note with a short topic
+  category and each semantic connection with a one-line explanation of why
+  it exists, using Joplin AI chat. Off by default.
+- **Community detection.** Notes cluster into color-coded groups using the
+  Louvain method, with a keyword-based fallback for small or sparse vaults.
+- **Centrality-scaled nodes.** More connected notes render larger, so hubs
+  stand out at a glance.
+- **Live updates.** The graph updates as you edit, create, and delete
+  notes, without a full rebuild, and catches up automatically after a sync.
+- **Local and instant.** Embeddings, labels, and the last built graph are
+  cached in a local SQLite database, so reopening the panel doesn't mean
+  waiting again. Nothing is sent anywhere except to Joplin's own AI
+  subsystem, and only when semantic analysis or LLM enrichment is on.
+- **A panel built for exploring, not just looking.** Search, focus mode
+  (isolate a note's neighborhood), per-edge-type toggles, zoom, and
+  PNG/SVG/JSON export.
+
+## Requirements
+
+- Joplin desktop 3.5 or later.
+- Joplin 3.7 or later with AI enabled (Configuration screen's **AI** page),
+  if you want semantic connections or LLM enrichment. Everything else works
+  without it.
+
+## Installation
+
+### From the Joplin plugin marketplace
+
+1. In Joplin, open the Configuration screen and go to the **Plugins** page.
+2. Use the search box to look for **Note Graph**, or press the **Plugin
+   tools** (gear) button and choose **Browse all plugins**.
+3. Press **Install** next to Note Graph.
+4. Restart Joplin when prompted to complete installation.
+
+### From a `.jpl` file
+
+Build the plugin from source and install the resulting file:
+
+```sh
+npm install
+npm run dist
+```
+
+This produces a `.jpl` file under `publish/`. In Joplin, open the
+Configuration screen's **Plugins** page, press the **Plugin tools** (gear)
+button, choose **Install from file**, and select it. Restart Joplin after
+installing an update.
+
+## Usage
+
+Open it from the **Tools** menu: **Show Note Graph**. The graph builds from
+your current notes, tags and links; if AI analysis is enabled in the
+plugin's settings, semantic edges are added once your notes are embedded.
+If LLM enrichment is also enabled, category badges and relationship labels
+appear on hover once Pass B finishes labeling them.
+
+Click a node to open that note. Double-click to zoom in on it. Use the
+legend at the top of the panel to search, toggle edge types, enter focus
+mode on a selected note, or export the current view.
+
+## Configuration
+
+Available in the Configuration screen's **Note Graph** section:
+
+| Setting | Default | Effect |
+|---|---|---|
+| Enable AI-based semantic analysis | Off | Adds semantic similarity edges using Joplin AI |
+| Similarity threshold | 50% | Lower values surface more semantic edges |
+| Max semantic edges per note | 5 | Caps how many semantic connections each note keeps |
+| Enable LLM analysis | Off | Adds Pass B category labels and relationship explanations |
+| Retry AI embedding | Off | One-shot: re-runs AI-based semantic analysis, reusing cached embeddings |
+| Retry AI labels | Off | One-shot: retries Pass B for anything still unlabeled |
+
+Full details, including how the similarity score and Pass B labels are
+computed, are in [docs/settings.md](docs/settings.md),
+[docs/similarity-engine.md](docs/similarity-engine.md), and
+[docs/llm-enrichment.md](docs/llm-enrichment.md).
+
+## Documentation
+
+In-depth documentation lives in [`docs/`](docs/README.md): architecture,
+the data pipeline, the similarity engine, the graph model, LLM enrichment
+(Pass B), incremental updates, caching, development setup, and
+troubleshooting.
+
+## Development
+
+```sh
+npm install       # also builds the plugin (npm run prepare)
+npm test          # run the test suite
+npm run format    # apply the project's Prettier config
+```
+
+See [docs/development.md](docs/development.md) for the full build, test
+and project-layout reference.
+
+## License
+
+[MIT](LICENSE)

docs/README.md

@@ -0,0 +1,34 @@
+# Note Graph documentation
+
+This is the in-depth documentation for the Note Graph plugin: how it's put
+together, and why it's built the way it is. For install and quick-start
+instructions, see the [root README](../README.md).
+
+## Contents
+
+1. [Architecture](architecture.md) - how the plugin and the webview panel
+   fit together, and the overall request flow.
+2. [Data pipeline](data-pipeline.md) - fetching notes, tags, and links from
+   the Joplin API.
+3. [Similarity engine](similarity-engine.md) - how semantic edges are
+   scored from embedding vectors.
+4. [Graph model](graph-model.md) - nodes, edges, centrality, and community
+   detection.
+5. [LLM enrichment (Pass B)](llm-enrichment.md) - optional category labels
+   and relationship explanations via Joplin AI chat.
+6. [Incremental updates](incremental-updates.md) - how the graph stays live
+   as you edit, without a full rebuild.
+7. [Caching](caching.md) - the SQLite-backed vector and graph caches.
+8. [Settings reference](settings.md) - every setting, what it does, and
+   when changing it triggers a rebuild.
+9. [Development](development.md) - building, testing, and project layout.
+10. [Troubleshooting](troubleshooting.md) - common issues and what causes
+    them.
+
+## Reading order
+
+If you're new to the codebase, read them in order: architecture first for
+the map, then data pipeline through LLM enrichment for how a graph gets
+built from scratch, then incremental updates and caching for what happens
+after that. Settings and development are reference material you can jump
+to directly.

docs/architecture.md

@@ -0,0 +1,99 @@
+# Architecture
+
+Note Graph is a Joplin desktop plugin. It has no server component and sends
+no data anywhere except to Joplin's own AI subsystem (`joplin.ai`), and only
+when semantic analysis or LLM enrichment is turned on. Everything else runs
+inside the plugin sandbox that Joplin provides.
+
+Graph building happens in two stages. **Pass A** is required: it builds the
+graph itself, structural edges always and semantic edges when AI analysis is
+enabled (see [Similarity engine](similarity-engine.md) and [Graph
+model](graph-model.md)). **Pass B** is optional: it asks Joplin AI's chat
+model to label what Pass A already found, category tags on notes and
+relationship explanations on semantic edges, without discovering any new
+edges of its own (see [LLM enrichment](llm-enrichment.md)).
+
+The plugin is really two programs that talk to each other over Joplin's
+webview message bridge:
+
+- **The plugin script** (`src/index.ts` and everything under `src/data` and
+  `src/services`), which runs in Joplin's plugin host. It reads notes, tags
+  and links through the Joplin data API, builds the graph, and reacts to
+  workspace events.
+- **The webview panel** (`src/ui`), which renders the graph with
+  [Cytoscape.js](https://js.cytoscape.org/) inside an isolated webview. It
+  has no access to the Joplin API directly; it only receives messages from
+  the plugin script.
+
+## Module map
+
+| Path | Responsibility |
+|---|---|
+| `src/data` | Reads notes, tags and links from the Joplin API; extracts links from note bodies. |
+| `src/data/Database` | SQLite-backed caches: embedding vectors and the last built graph. |
+| `src/services/embeddings` | Resolves an embedding provider and turns notes into vectors, with caching. |
+| `src/services/similarity` | Turns embedding vectors and note metadata into scored note pairs, and those into graph edges. |
+| `src/services/graph` | Builds the renderable graph: nodes, edges, community detection, centrality, diffing. |
+| `src/services/llm` | Pass B: batches semantic edges to Joplin AI's chat model and parses category/relationship labels back onto the graph. |
+| `src/services/sync` | Listens to Joplin workspace events and turns them into incremental graph updates. |
+| `src/services/settings` | Registers and reads the plugin's settings. |
+| `src/services/AnalysisController.ts` | Orchestrates the above into a single graph-building pipeline; the one class `index.ts` talks to. |
+| `src/ui` | The webview panel: HTML shell, styling, and the Cytoscape-driven `graph-view.js` client. |
+
+## Request flow: opening the graph
+
+The **Show Note Graph** command shows the panel first, then calls
+`ensureGraphLoaded()`, which is a no-op if a graph is already built this
+session and otherwise does one of two things:
+
+- **A cached graph exists on disk:** post it immediately, then in the
+  background run a sync-complete sweep and, if labels are missing, an
+  enrichment backfill (see [Incremental updates](incremental-updates.md)
+  and [LLM enrichment](llm-enrichment.md)).
+- **Nothing cached:** load notes, post the structural graph
+  (`buildStructural`), then embed and post the semantic graph
+  (`embedAndBuildSemantic`). If LLM enrichment is enabled, a Pass B
+  follow-up then labels the semantic edges and posts a patch on top of the
+  already-posted graph.
+
+`ensureGraphLoaded()` is also the target of a callback the panel invokes
+when it polls with no data to show and is visible; a 30-second cooldown
+after a load failure keeps that from retrying in a tight loop. Concurrent
+callers share one in-flight load rather than triggering it twice.
+
+If semantic analysis is off or fails, the structural graph stays and a
+one-line status message explains why; Pass B then has nothing to enrich.
+
+## AnalysisController: the single orchestrator
+
+`AnalysisController` (`src/services/AnalysisController.ts`) is the only
+class `index.ts` calls into for building or rebuilding the graph. It owns:
+
+- the last set of notes and their embeddings, so settings changes (threshold,
+  top-K) can recompute the graph without re-fetching embeddings;
+- a monotonically increasing `runToken`, so a slow build that gets
+  superseded by a newer one (e.g. the user reopens the panel while an embed
+  is still running) discards its result instead of overwriting fresher data;
+- the last `GraphData`, diffed against each new build via `GraphDiffer` so
+  incremental updates can push a patch instead of a full graph;
+- the `LLMEnricher` instance for Pass B, whose in-memory cache it seeds from
+  the persisted graph cache on `loadFromCache()`, so labels already computed
+  in a previous session don't need to be re-requested from the model.
+
+See [Data pipeline](data-pipeline.md) for how notes are loaded and enriched,
+[Similarity engine](similarity-engine.md) for how semantic edges are scored,
+[Graph model](graph-model.md) for how nodes and edges are assembled, [LLM
+enrichment](llm-enrichment.md) for Pass B, and [Incremental
+updates](incremental-updates.md) for what happens after the initial load.
+
+## Persistence
+
+Two SQLite databases live in the plugin's data directory
+(`joplin.plugins.dataDir()`), opened lazily on first use:
+
+- `note-graph-vectors.sqlite`: one row per note's embedding vector, keyed by
+  note ID and model ID.
+- `note-graph-cache.sqlite`: the last successfully built graph (so reopening
+  the panel is instant) and the sync cursors used for incremental updates.
+
+Details in [Caching](caching.md).

docs/caching.md

@@ -0,0 +1,98 @@
+# Caching
+
+The plugin persists two things to disk so it doesn't have to re-fetch or
+re-embed everything on every panel open: embedding vectors, and the last
+built graph plus sync cursors. Both live in Joplin's per-plugin data
+directory (`joplin.plugins.dataDir()`), as separate SQLite files.
+
+## Why SQLite, and how it's accessed
+
+Native Node modules can't be bundled into a plugin the normal way, so the
+database access goes through Joplin's own bundled `sqlite3` module via
+`joplin.require('sqlite3')`. `VectorDatabase`
+(`src/data/Database/VectorDatabase.ts`) is a thin promisified wrapper around
+that callback-based driver: it owns the connection and runs the schema's
+`CREATE TABLE IF NOT EXISTS` statements on open, and exposes `run()` and
+`all()`. Query logic itself lives in the repository classes, not in this
+wrapper.
+
+`open()` is safe to call repeatedly and concurrently: if opening fails, both
+the in-progress promise and the (possibly half-created) connection are
+reset, so a later call retries cleanly instead of replaying a stale
+rejection or treating a half-open database as ready.
+
+## Vector cache
+
+**File:** `note-graph-vectors.sqlite`
+
+```sql
+CREATE TABLE IF NOT EXISTS note_vectors (
+    note_id TEXT PRIMARY KEY,
+    model_id TEXT NOT NULL,
+    updated_time INTEGER NOT NULL,
+    vector BLOB NOT NULL
+)
+```
+
+Managed by `VectorRepository` (`src/data/Database/VectorRepository.ts`),
+used by `EmbeddingOrchestrator` (see [Similarity
+engine](similarity-engine.md)). Vectors are stored as `Float32` BLOBs
+(`Buffer.from(Float32Array.buffer, ...)`), not JSON, to keep storage compact.
+Decoding copies the underlying bytes before viewing them as a
+`Float32Array`, because Node can place a small `Buffer` at an unaligned byte
+offset inside a shared pool, and viewing that directly would throw.
+
+A cached vector is only reused if both its `note_id` and `model_id` match;
+the caller (`EmbeddingOrchestrator`) also compares `updated_time` against
+the note's current value to decide freshness. Changing the AI model in
+Joplin settings naturally invalidates the whole cache, since every lookup
+will then miss on `model_id`.
+
+Reads are batched (up to 500 note IDs per `SELECT ... WHERE note_id IN (...)`,
+`QUERY_BATCH_SIZE`) to stay under SQLite's bound-parameter limit. Writes run
+inside a single transaction per `saveMany()` call and are serialized through
+an internal promise chain, since two interleaved transactions on the same
+connection would otherwise nest `BEGIN TRANSACTION` and error.
+
+## Graph cache and sync state
+
+**File:** `note-graph-cache.sqlite`
+
+```sql
+CREATE TABLE IF NOT EXISTS graph_cache (
+    id INTEGER PRIMARY KEY CHECK (id = 1),
+    notes_json TEXT NOT NULL,
+    graph_json TEXT NOT NULL,
+    updated_time INTEGER NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS sync_state (
+    id INTEGER PRIMARY KEY CHECK (id = 1),
+    events_cursor TEXT,
+    embeddings_cursor TEXT
+);
+```
+
+Managed by `GraphCacheRepository` (`src/data/Database/GraphCacheRepository.ts`).
+Both tables are single-row (`CHECK (id = 1)`), upserted with
+`ON CONFLICT(id) DO UPDATE`, since the plugin only ever needs "the current
+state," not history.
+
+- `graph_cache` holds the last successfully built `GraphData` plus the note
+  list it was built from, serialized as JSON. `AnalysisController` writes
+  this after every successful build (fire-and-forget; a write failure is
+  logged, not propagated) and `index.ts` reads it on panel open to render
+  instantly before a background sync sweep runs. Any Pass B `category` and
+  `relationshipLabel` fields on that graph ride along in the same JSON, and
+  `AnalysisController` reseeds `LLMEnricher`'s in-memory cache from them on
+  load; see [LLM enrichment](llm-enrichment.md).
+- `sync_state` holds the two pagination cursors used by
+  `IncrementalUpdater`'s sync-complete sweep: `events_cursor` for
+  `/events` (deletions and the AI-off change-detection fallback) and
+  `embeddings_cursor` for `joplin.ai.getEmbeddings()` (change detection
+  while AI analysis is on). See [Incremental
+  updates](incremental-updates.md).
+
+Graph-cache and sync-state writes go through the same kind of serialized
+write queue as the vector cache, for the same reason: SQLite transactions
+live on one shared connection.