Interactive Knowledge Base with LLM-driven queries

KnowledgeGraph centers on a KGX-compatible SQLite database that powers the app. The built-in skills can create and enrich that database from APIs, URLs, files, folders, and text. They can also emit linked markdown notes as an optional companion export for users who want a vault-style workflow.

Goal

To be able to query a database in plain English and have the answer light up or filter the graph with the option to run fully locally, no cloud.

---

Quick Start

1. Create the environment

conda env create -f environment.yml
conda activate knowledgegraph

The environment file installs:

• the editable APP package
• FastAPI/uvicorn and supporting packages for the app
• numpy and umap-learn for UMAP layout support
• test dependencies

2. Start local server

Pull the default model once, and start the ollama server:

ollama pull qwen3-coder:30b
ollama serve

Defaults to qwen3-coder:30b (30B MoE, 3B active). Pass --model <name> to use a different model.

3. Launch the app interface

Run the KGX from the APP/ directory:

cd APP
python -m kgx

The runtime_data/vault/vault.db within this repo is used by default. If it is missing, KGX bootstraps it from sample_data/3_db/vault.seed.db on first launch. If you want your original database unchanged, populate its copy to app's working directory runtime_data/.

• If you want to use a custom database in-place, provide its absolute path with --db argument:

python -m kgx --db /path/to_your_custom/vault.db

• If you want to use a config for the built-in module:

python -m kgx --config config/people.yaml

• If you want to avoid opening a browser:

python -m kgx --no-browser

4. Optional: set up a linked markdown vault

If you want a local markdown vault alongside the database, the repo can generate and style one for Obsidian-compatible browsing. This is optional; KGX itself reads vault.db, not the markdown files.

python skills/shared/setup_vault.py /path/to/vault
python skills/shared/setup_vault.py /path/to/vault --dry-run      # preview
python skills/shared/setup_vault.py /path/to/vault --colors-only   # graph + CSS only

If you use Obsidian, quit it before running this command because it overwrites graph.json on exit.

Build Custom Database

The app runs on a KGX-compatible SQLite database, vault.db, which is the key underlying structure used for storage, retrieval, and graph-driven exploration.

There are three primary ways to deliver or create one:

1. Bring your own vault.db. Use an existing KGX-compatible SQLite database and point the app at it.

2. Use the built-in skills to create or enrich a database. This is the main builder path. The skills pull from APIs, URLs, local files, folders, and pasted text, then write structured entities and relationships into vault.db.

3. Import an existing linked markdown vault. If you already have an Obsidian-style vault, the migration scripts can import it into vault.db. This is a compatibility path, not the primary product flow.

The bundled sample_data/ is provided for quick testing and to show the expected data structure.

folder	description	role	examples
1_source/	contains ready-made source data generated by built-in skill-driven content builders.	It serves as a direct input for automated source → db scenario.	people_nobel/api/laureates.json people_isu_biotech/biotech_staff.json genomics_scn/DATA.tsv + dataset.yaml + schema.yaml
2_vault/	contains ready-made Obsidian-style markdown knowledge base with module-specific artifacs.	It serves as a direct input for automated vault → db secnario.	nobel/ isu_biotech/
3_db/	contains ready-to-run SQLite databases.	It serves as a ready-made db example.	nobel50.db isu_biotech.db genomics_scn.db

Once you have a db, it can also be rendered back into a markdown vault.
supported transformations: vault -> db -> vault

Automated Skill-based Pipeline

When you don't have a SQLite db for your project...
The bundled skills can build vault.db for supported modules from a local source package or an existing markdown vault.

Follow the steps in the CLI:

1. Enter the root of this repo.

   cd <path_to>/KnowledgeGraph

2. Activate the LLM/backend you want to use, such as a local Ollama model or a hosted model through Codex, Claude, or similar tooling.

   codex
   # or
   ollama pull qwen3-coder:30b   # if not done already
   ollama serve

3. Formulate a task using one of the examples so the built-in skills compile vault.db. Alternatively, run the skill scripts directly (e.g. skills/genomics/run_genomics.py, or the vault importer in skills/shared/scripts/).

   FROM LOCAL SOURCE

   Use the built-in skills for the <genomics> module with the <functional_genomics> extension to infer standardized source metadata from sample_data/1_source/genomics_scn/DATA.tsv, review the mapping locally if needed, and compile the database at sample_data/3_db/genomics_scn.db.
   

   FROM MARKDOWN VAULT (Obsidian → SQLite)

   Import the Obsidian-style markdown vault at sample_data/2_vault/nobel/ into vault.db, preserving linked person and abstract records so I can continue exploring them in KGX.
   

   Import the Obsidian-style markdown vault at sample_data/2_vault/isu_biotech/ into vault.db, preserving linked person and abstract records so I can continue exploring them in KGX.
   

IMPORTANT! - If you are adapting KGX to a new module, use APP/prompts/kgx-data-setup.md with Claude, ChatGPT, or Codex. It walks through schema design, import scripting, tag cleanup, and config tuning for explore mode.

Modules

Currently supported modules include:

Domain	Description	supported artifacts	extensions
people	academic records with node: person	person, publication, tag ontology, related graph links, and person-linked markdown abstracts	isu_profile nobel_profile
genomics	gene-centered scientific records with linked sequence-derived entities	gene, transcript, protein, orthogroup, module-aware tags, and standardized local source packages	functional_genomics

PEOPLE

Use the people module to explore person-centric academic records — person, publication, and tag entities with their relationships and person-linked markdown abstracts. Two ready-made sample datasets are included and styled by APP/config/people.yaml:

• Nobel laureates — sample_data/3_db/nobel50.db (the default dataset the app opens).
• ISU Biotechnology staff — sample_data/3_db/isu_biotech.db.

Examples:

Nobel People — nobel50.db

A ready-made sample of the first 50 Nobel laureates, styled by the `nobel_profile` settings in `config/people.yaml`. This is the dataset the app opens by default.

APP USAGE:

The app uses runtime_data/ for its active working directory.

• If you copy .db to there, app will modify that local copy and leave the original file unchanged.
• If you start KGX with --db <custom-path/vault.db>, app will read from and write directly to that exact file.

# run from APP/ folder 
cd KnowledgeGraph/APP

# populate database copy to app's workdir
cp ../sample_data/3_db/nobel50.db ../runtime_data/vault/vault.db
python -m kgx --config config/people.yaml

# or point directly to the `.db` path 
python -m kgx --config config/people.yaml --db /custom_path/nobel50.db

ISU Biotech People — isu_biotech.db

A ready-made sample of ISU Office of Biotechnology staff, styled by the `isu_profile` settings in `config/people.yaml`.

The checked-in ISU biotech sample also tracks biotech-local tag ontology inputs under:

sample_data/1_source/people_isu_biotech/tags/

These files let the sample apply a sample-specific tag registry, aliases, and hierarchy instead of reusing the Nobel sample defaults.

APP USAGE:

The app uses runtime_data/ for its active working directory.

• If you copy .db to there, app will modify that local copy and leave the original file unchanged.
• If you start KGX with --db <custom-path/vault.db>, app will read from and write directly to that exact file.

# run from APP/ folder 
cd KnowledgeGraph/APP

# populate database copy to app's workdir
cp ../sample_data/3_db/isu_biotech.db ../runtime_data/vault/vault.db
python -m kgx --config config/people.yaml

# or point directly to the `.db` path 
python -m kgx --config config/people.yaml --db /custom_path/isu_biotech.db

GENOMICS

Use the genomics module when you build scientific graph databases from structured local source files such as TSV, CSV, or XLSX tables and specialize the flow with an extension such as:

• functional_genomics, includes: standardized source-package inference, deterministic local DB builds, and optional local-LLM review of ambiguous column mappings

The current genomics build treats local files as the authority and supports a staged workflow:

• raw local files such as DATA.tsv
• standardized metadata files such as dataset.yaml and schema.yaml
• deterministic DB build into sample_data/3_db/*.db

Related skill: genomics

Examples:

SCN Genomics - with functional_genomics extension

For the genomics sample, the automated builder starts from a local source table, standardizes its metadata, then compiles it into a graph database without online harvesting:

# operational flow
local table + optional notes ──> dataset.yaml + schema.yaml   # standardized source package
sample_data/1_source/genomics_scn/
    └── skills/genomics/run_genomics.py infer/review/build
            └──> sample_data/3_db/genomics_scn.db

CLI EXAMPLE:

cd KnowledgeGraph

python skills/genomics/run_genomics.py infer \
  --source-file sample_data/1_source/genomics_scn/DATA.tsv \
  --source-dir sample_data/1_source/genomics_scn \
  --apply

python skills/genomics/run_genomics.py build \
  --source-dir sample_data/1_source/genomics_scn \
  --db sample_data/3_db/genomics_scn.db \
  --fresh

If a dataset needs semantic review before build, the same workflow can optionally add:

• review to produce llm-review.yaml
• propose to turn that review into schema.patch.yaml
• apply-proposal to update schema.yaml deterministically before rebuild

APP USAGE:

# run from APP/ folder
cd KnowledgeGraph/APP

# populate database copy to app's workdir
cp ../sample_data/3_db/genomics_scn.db ../runtime_data/vault/vault.db
python -m kgx --config config/genomics.yaml

# or point directly to the `.db` path
python -m kgx --config config/genomics.yaml --db /custom_path/genomics_scn.db

---

Skills

genomics

Build local-first genomics graph databases from standardized source packages derived from TSV, CSV, or XLSX inputs.

cd skills/genomics
python run_genomics.py infer --source-file ../../sample_data/1_source/genomics_scn/DATA.tsv --source-dir ../../sample_data/1_source/genomics_scn --apply
python run_genomics.py build --source-dir ../../sample_data/1_source/genomics_scn --db ../../sample_data/3_db/genomics_scn.db --fresh

Validated use cases:

• infer a standardized dataset.yaml and schema.yaml from a local raw table
• review ambiguous mappings locally with Ollama before build
• generate and apply deterministic schema patch proposals
• compile a genomics KGX database without online harvesting

See skills/genomics/README.md for the full workflow.

---

Shared Utilities

All skills share common scripts in skills/shared/scripts/:

Script	Purpose
fetch_with_fallback.py	URL fetch with gzip decompression and bot-wall hard stop
extract_text.py	PDF/HTML/DOCX to plain text
extract_snippets.py	Topic and person snippet extraction with quote-priority
extract_names.py	Regex name extraction from text
inventory_folder.py	Recursive folder listing
tag_resolver.py	Fuzzy-match tags against vault registry (>80% threshold)
verify_extraction.py	Check extraction completeness
setup_vault.py	Optional linked-vault setup for Obsidian-compatible browsing
promote_person_tags.py	Promote supported publication tags onto profiled people using config policy
render_vault.py	Render SQLite content back into markdown people, abstracts, tags, and acknowledgement notes

---

Key Design Principles

1. Minimize LLM surface area — Python handles data fetching, transformation, and formatting. In the interactive skill path, the LLM provides only reasoning fields (~200 tokens: role, summary, tags). Deterministic sample builders can omit the LLM step entirely.
2. Module-level cache — Large datasets pass between tools via Python dicts, not LLM context.
3. Deterministic builders — All markdown output is assembled by Python, not generated by the LLM.
4. Tag resolution + sanitization — tag_resolver.py fuzzy-matches against the registry; _sanitize_tag() enforces kebab-case.
5. Raw data preservation — Original source content saved for future re-processing.

See docs/BEST_PRACTICES.md for the full set of 35 lessons learned.

---

File Structure

Top-Level Modules

Path	Description	Role	Required for app use?
APP/	KGX application code, UI, API, DB layer, prompts, and configs	Main runtime app	Yes
skills/	Genomics database-builder and shared data-building utilities	Builds and updates graph data	Optional for read-only app use, required to generate/update data
sample_data/	Checked-in example content and starter DB seed	Demo/reference data	No
runtime_data/	Live local DB, rendered output, logs, and temp files	Mutable working state	Yes in practice
docs/	Architecture notes, specs, examples, and historical material	Documentation and developer reference	No
environment.yml	Reproducible conda environment	Environment bootstrap	Yes in practice
README.md	Repo entry point and setup guide	User-facing overview	No

App Layout

APP/ is the product runtime:

• kgx/ contains the backend, config loader, DB layer, layouts, LLM helpers, and browser UI
• config/ contains the active default config plus domain presets
• prompts/ contains LLM-facing setup guidance such as kgx-data-setup.md

Skill System

The data-building side is split cleanly:

• skills/genomics/ Genomics database-builder skill
• skills/shared/ Shared scripts for extraction, tag handling, vault migration, and rendering

Docs Layout

docs/ holds non-runtime material:

• docs/app/ for KGX architecture and app-specific notes
• docs/specs/ for product and architecture specs

Data Layout

sample_data/
├── 1_source/
│   ├── people_nobel/           Nobel snapshot plus normalized match cache
│   └── people_isu_biotech/     ISU Office of Biotechnology staff snapshot with inline curation and local tag ontology inputs
├── 2_vault/
│   ├── nobel/
│   │   ├── people/             Generated Obsidian-style people vault
│   │   └── abstracts/          Linked publication abstract notes
│   └── isu_biotech/
│       ├── people/             Generated ISU biotech people vault
│       ├── abstracts/          Linked publication abstract notes
│       ├── acknowledgements/   Generated acknowledgement evidence notes when present
│       └── tags/               Rendered sample tag registry
└── 3_db/
    ├── nobel50.db              Built 50-laureate sample database
    ├── isu_biotech.db          Built ISU biotech sample database
    └── vault.seed.db           Tracked starter database used for bootstrap

runtime_data/
├── vault/
│   ├── vault.db                Live local database created/updated during use
│   ├── people/<slug>/
│   │   └── <slug>.md           Person profile
│   ├── abstracts/*.md          Per-paper notes with author wiki-links
│   ├── tags/
│   │   └── tag-registry.md     Approved tags
│   ├── acknowledgements/       Rendered acknowledgement evidence notes
│   └── .obsidian/
│       ├── graph.json          Graph node color groups
│       ├── snippets/entity-colors.css
│       └── appearance.json
├── rendered/                   Local rendered markdown output
├── logs/
└── tmp/

Graph node colors: abstracts (lavender), people (teal), tags (purple).

sample_data/ is checked-in example content. runtime_data/ is local mutable state and should not be committed.