[ahajdar]

Merge request type

• New feature
• Resolves bug
• Documentation
• Other

Relates to

• OPFLOW
• SOPFLOW
• SCOPFLOW
• TCOPFLOW
• CMake build system
• Spack configuration
• Manual
• Web docs
• Other (AgentiGrid)

This MR updates

• Header files
• Source code
• CMake build system
• Spack configuration
• Web docs
• Manual
• Other (AgentiGrid corpus, docs, tests, optional requirements file)

Summary

This PR adds optional retrieval grounding (RAG) to AgentiGrid. Before each LLM proposal, the agent can retrieve short reference material from a curated, provenance-tagged corpus and prepend it to the prompt. The corpus holds ExaGO tool help, case metadata, and certified examples of correct AgentiGrid responses. The goal is fewer proposals that are executable but structurally or methodologically wrong.

Depends on #139. This branch is stacked on the agent-loop PR, because both change agent_loop.py and the launcher. Until that merges, this PR also shows its commit. Please review only the last commit, "optional retrieval grounding (RAG) for LLM proposals". I'll rebase onto develop once #139 is merged.

Design points:

• Off by default. With RAG off, behavior and dependencies are unchanged. The new dependencies are optional and listed in requirements-rag.txt: chromadb and requests, plus sentence-transformers for the reranker. The core requirements.txt is untouched, and chromadb is imported lazily.
• Generation stage only. Retrieval changes what the LLM sees. The deterministic validator never sees it. Post-search analysis is not grounded.
• Fails safe and visibly. Any retrieval failure (embedding server down, missing index, missing chromadb) leaves the prompt unchanged, so the run behaves like the baseline. Calls, errors and empty results are counted in the journal (rag_config.stats), so a degraded run is visible.
• Three modes, set with AGENTIGRID_RAG_MODE or the launcher's Advanced panel:
  • off;
  • basic: the top 3 chunks with cosine similarity of at least 0.35;
  • corrective: retrieved chunks are graded (cosine, or an optional local cross-encoder), refined to the relevant sentences, and the context is withheld when retrieval is judged irrelevant.
• Air-gapped friendly. Embeddings come from a local Ollama model (nomic-embed-text) and the index is a local Chroma store. Nothing is sent to third parties beyond the configured LLM backend.

What's included:

Path	Content
agentigrid/agentigrid/rag/	Retrievers (basic, corrective), reranker grader, embeddings, Chroma store, ingest, corpus hash and chunking identity
agentigrid/agentigrid/engine/agent_loop.py, journal.py	Retrieval before each per-iteration LLM call; rag_enabled / rag_config in the journal
agentigrid/launcher/app.py, session_manager.py	RAG selector (default Off), grader choice for corrective mode, grounding caption on the live monitor and results page
agentigrid/rag/corpus/	Frozen corpus v1 (12 files) with corpus_manifest.json (SHA-256)
agentigrid/rag/tools/	rag_harvest.py (tool help, case metadata), rag_schema_exemplars.py (certified by AgentiGrid's own parser, validator and modifier), corpus_guard.py (leakage and personal-path audit, freeze)
agentigrid/docs/RAG_USAGE.md, README.md	Setup, modes, troubleshooting
agentigrid/tests/rag/	23 tests

Not included: the batch experiment harness (runner, evaluator, statistics) and research-specific specs. If useful upstream, these can follow as a separate PR.

Testing: 23 tests in tests/rag/. They use fake embeddings and fake models, so they need no Ollama, Chroma or model download. They cover:

• RAG is off by default;
• references reach the prompt and the journal;
• a failed retrieval leaves the prompt unchanged;
• retrieval errors are counted;
• the corrective grader contract;
• the chunking scheme;
• the shipped corpus matches its manifest and contains no personal paths;
• the schema exemplars certify on case118;
• the guard flags.

With chromadb installed and a fake embedder, ingesting rag/corpus reproduces the manifest hash (92 chunks). Basic mode returns 3 references, corrective returns refined strips, and off returns nothing.

python -m pytest -m "not slow" from agentigrid/, without the ExaGO binaries:

• with AgentiGrid: robust agent iterations, accurate UI timeline, LLM usage accounting #139 only: 1046 passed, 2 failed. The 2 failures are the OPFLOW fixture issues fixed by AgentiGrid: keep "};" on MATPOWER cell arrays so ExaGO reads genfuel #137.
• with AgentiGrid: keep "};" on MATPOWER cell arrays so ExaGO reads genfuel #137 also applied: 1051 passed, 0 failed.

The changelog entry is in agentigrid/claude_code_changes.md.

Issue(s) addressed

None filed. Discussed with the maintainers on 2026-10-06.