[RAG Projection] Add revision-aware derived indexing and rebuilds

.github/workflows/ci.yml

@@ -162,7 +162,8 @@ jobs:
             generic_rag \
             generic_rag.errors \
             generic_rag.contracts \
-            generic_rag.ports
+            generic_rag.ports \
+            generic_rag.projection
           do
             (
               cd "$rag_probe_dir"

PYTHON_MODULE_INDEX.md

@@ -4,9 +4,9 @@
 
 - Declared source root: `src`
 - Packaging source of truth: `pyproject.toml`
-- Importable production units: 4
-- Indexed production units: 4
-- Source/index parity: 4/4
+- Importable production units: 5
+- Indexed production units: 5
+- Source/index parity: 5/5
 - Package data: `src/generic_rag/py.typed`
 - Locked verification owner: `.github/workflows/ci.yml` (supporting workflow,
   not an importable unit)
@@ -41,9 +41,9 @@
 - Owned state or external resources: none.
 - Material side effects: none.
 - Verification: `tests/test_errors.py`, `tests/test_package_boundaries.py`,
-  `tests/support/clean_import_probe.py`, and the locked CI import and boundary
-  checks.
-- Documentation: `docs/api.md`.
+  `tests/test_projection.py`, `tests/support/clean_import_probe.py`, and the
+  locked CI import and boundary checks.
+- Documentation: `docs/api.md` and `docs/projection.md`.
 
 ## `generic_rag.contracts`
 
@@ -53,37 +53,68 @@
 - Supported public imports: `DocumentKey`, `DocumentIdentity`, `Document`,
   `FragmentIdentity`, `Fragment`, `EmbeddingIdentity`, `EmbeddingVector`,
   `VectorRecord`, `ProjectionIdentity`, `ProjectionCheckpoint`,
-  `ProjectionOutcome`, `ProjectionReceipt`, `RetrievalQuery`,
-  `RetrievalOutcome`, `RetrievalHit`, and `RetrievalResult` from
-  `generic_rag.contracts`.
+  `ProjectionOutcome`, `ProjectionReceipt`, `ChunkingPolicy`,
+  `ProjectionLimits`, `ProjectionRequest`, `ProjectionManifestEntry`,
+  `ProjectionManifest`, `ProjectionStateAvailability`,
+  `ProjectionStateSnapshot`, `ProjectionStateStatus`, `ProjectionResult`,
+  `RetrievalQuery`, `RetrievalOutcome`, `RetrievalHit`, and `RetrievalResult`
+  from `generic_rag.contracts`.
 - Re-exports: exactly the names in the module's `__all__`; none from the package
   root.
 - Direct internal dependencies: `generic_rag.errors`.
 - Owned state or external resources: none; instances own only immutable caller
   values.
 - Material side effects: none.
 - Verification: `tests/test_contract_values.py`,
-  `tests/test_projection_contracts.py`, `tests/test_retrieval_contracts.py`,
-  `tests/test_package_boundaries.py`, `tests/support/clean_import_probe.py`,
-  `tests/support/verify_artifacts.py`, and the locked CI import, boundary, and
-  artifact checks.
-- Documentation: `docs/api.md` and `docs/security-and-privacy.md`.
+  `tests/test_projection_contracts.py`, `tests/test_projection.py`,
+  `tests/test_retrieval_contracts.py`, `tests/test_package_boundaries.py`,
+  `tests/support/clean_import_probe.py`, `tests/support/verify_artifacts.py`, and
+  the locked CI import, boundary, and artifact checks.
+- Documentation: `docs/api.md`, `docs/projection.md`, and
+  `docs/security-and-privacy.md`.
 
 ## `generic_rag.ports`
 
 - Source: `src/generic_rag/ports.py`
 - Responsibility: define synchronous injected collaborator interfaces and
   explicit caller-owned borrowing semantics.
 - Supported public imports: `Borrowed`, `Embedder`, `VectorIndexWriter`,
-  `VectorIndexReader`, and `LexicalRetriever` from `generic_rag.ports`.
+  `VectorIndexResetter`, `VectorIndexReader`, and `LexicalRetriever` from
+  `generic_rag.ports`.
 - Re-exports: exactly the names in the module's `__all__`; none from the package
   root.
 - Direct internal dependencies: `generic_rag.contracts`.
 - Owned state or external resources: `Borrowed` retains a reference but never
   owns, acquires, releases, closes, or shuts down the resource.
 - Material side effects: none.
 - Verification: `tests/test_ports.py`, `tests/test_package_boundaries.py`,
-  `tests/support/clean_import_probe.py`, and the locked CI import and boundary
+  `tests/test_projection.py`, `tests/support/clean_import_probe.py`, and the
+  locked CI import and boundary checks.
+- Documentation: `docs/api.md`, `docs/lifecycle.md`, `docs/projection.md`, and
+  `docs/security-and-privacy.md`.
+
+## `generic_rag.projection`
+
+- Source: `src/generic_rag/projection.py`
+- Responsibility: deterministically plan and synchronously execute bounded,
+  revision-aware document projection against caller-supplied state.
+- Supported public imports: `ProjectionFailureStage`, `ProjectionStateError`,
+  `ProjectionOperationError`, `project_documents`, and `rebuild_projection`
+  from `generic_rag.projection`.
+- Re-exports: exactly the names in the module's `__all__`; none from the package
+  root.
+- Direct internal dependencies: `generic_rag.contracts`, `generic_rag.errors`,
+  and `generic_rag.ports`.
+- Owned state or external resources: none; planning state is immutable and
+  local to each call, while every embedder, writer, and resetter remains
+  caller-owned through `Borrowed`.
+- Material side effects: none at import time. At explicit workflow call time it
+  may invoke the borrowed embedder and vector writer, and full rebuild may
+  invoke the borrowed corpus resetter; it performs no persistence, network,
+  retry, acquisition, release, or lifecycle action itself.
+- Verification: `tests/test_projection.py`, `tests/test_package_boundaries.py`,
+  `tests/support/clean_import_probe.py`, `tests/support/verify_artifacts.py`, and
+  the locked CI test, lint, type, build, clean-install, import, and artifact
   checks.
-- Documentation: `docs/api.md`, `docs/lifecycle.md`, and
+- Documentation: `docs/projection.md`, `docs/api.md`, `docs/lifecycle.md`, and
   `docs/security-and-privacy.md`.

README.md

@@ -3,9 +3,10 @@
 `generic-rag` is a provider-neutral, runtime-dependency-free foundation for
 retrieval-augmented generation (RAG). Version 0.1.0 requires Python 3.11 or
 later and provides immutable contracts, typed error categories, synchronous
-collaborator protocols, and explicit caller-owned borrowing.
+collaborator protocols, deterministic bounded projection orchestration, and
+explicit caller-owned borrowing.
 
-No projection or retrieval algorithm is implemented in 0.1.0. The package has
+Retrieval and result composition are not implemented in 0.1.0. The package has
 no built-in adapter, provider, factory, persistence, network client,
 configuration system, authentication, citation mechanism, or CLI.
 
@@ -21,87 +22,125 @@ python -m pip install .
 The installed package has no runtime dependencies. Build and development tools
 are separate locked dependency groups.
 
-## Use the contracts in application code
+## Project approved documents
 
 The host application remains responsible for authorization and policy checks.
-After approving a source and query, application code can construct generic
-values and accept an application-owned provider through a protocol:
+After approving source content, construct a complete bounded target, wrap
+application-owned adapters in `Borrowed`, and call a projection workflow:
 
 ```python
 from generic_rag.contracts import (
+    ChunkingPolicy,
     Document,
     DocumentIdentity,
     DocumentKey,
+    EmbeddingIdentity,
     EmbeddingVector,
-    RetrievalQuery,
+    ProjectionIdentity,
+    ProjectionLimits,
+    ProjectionRequest,
+    ProjectionStateAvailability,
+    ProjectionStateSnapshot,
 )
-from generic_rag.ports import Borrowed, Embedder
+from generic_rag.ports import Borrowed
+from generic_rag.projection import rebuild_projection
 
-# Construct these values only after application-specific authorization.
-approved_document = Document(
-    identity=DocumentIdentity(
-        key=DocumentKey(corpus_id="corpus-a", document_id="document-1"),
-        revision_id="revision-3",
+class ExampleEmbedder:
+    identity = EmbeddingIdentity("example-model", 2)
+
+    def embed(self, texts, /):
+        return tuple(EmbeddingVector((float(len(text)), 0.0)) for text in texts)
+
+
+class ExampleWriter:
+    def replace_document(self, document, records, /):
+        # Replace the complete projection for this stable document key.
+        return None
+
+    def delete_document(self, document, /):
+        return None
+
+
+class ExampleResetter:
+    def reset_corpus(self, corpus_id, /):
+        # Remove every projected document for this corpus.
+        return None
+
+
+request = ProjectionRequest(
+    "corpus-a",
+    ProjectionIdentity("schema-v1", ExampleEmbedder.identity),
+    ChunkingPolicy(max_fragment_codepoints=800, overlap_codepoints=80),
+    ProjectionLimits(
+        max_documents=100,
+        max_document_codepoints=100_000,
+        max_embedding_batch_size=32,
+    ),
+    (
+        Document(
+            DocumentIdentity(
+                DocumentKey("corpus-a", "document-1"),
+                "revision-3",
+            ),
+            "Approved source text",
+            (("classification", "internal"),),
+        ),
     ),
-    text="Approved source text",
-    attributes=(("classification", "internal"),),
-)
-query = RetrievalQuery(
-    corpus_id=approved_document.identity.key.corpus_id,
-    text="What does the source say?",
-    hit_limit=5,
-    candidate_limit=20,
 )
 
+# Bootstrap and recovery are explicit and destructive: reset, then replace.
+result = rebuild_projection(
+    request,
+    ProjectionStateSnapshot(ProjectionStateAvailability.MISSING, None),
+    Borrowed(ExampleEmbedder()),
+    Borrowed(ExampleWriter()),
+    Borrowed(ExampleResetter()),
+)
 
-def application_embed_query(
-    provider: Embedder,
-    request: RetrievalQuery,
-) -> EmbeddingVector:
-    with Borrowed(provider) as embedder:
-        vectors = embedder.embed((request.text,))
-    if len(vectors) != 1:
-        raise ValueError("the provider violated the Embedder contract")
-    return vectors[0]
+# Persist result.manifest in application-owned state only after success.
 ```
 
-This is application orchestration, not a package retrieval workflow. Version
-0.1.0 defines the boundary that provider implementations and later generic
-workflows will use; it does not construct providers or call them on a user's
-behalf.
+For normal updates, load that manifest into a present
+`ProjectionStateSnapshot` and call `project_documents`; it changes only added,
+updated, removed, or rechunked documents. Use `rebuild_projection` only when an
+explicit corpus-wide reset is intended. See the [projection guide](docs/projection.md)
+for the complete lifecycle, state matrix, adapter obligations, and failure
+behavior.
 
 Public values must be imported from their owning modules:
 
 - `generic_rag.contracts`
 - `generic_rag.errors`
 - `generic_rag.ports`
+- `generic_rag.projection`
 
 The package root intentionally has no re-exports: `generic_rag.__all__ == ()`.
 See the [API reference](docs/api.md) for every supported name and invariant.
 
-## Planned RAG flow
+## Application RAG flow
 
 The package itself has no concept of a user or agent. A consuming application
 decides which sources a user may approve, which queries may be submitted, which
 provider implementations receive data, and whether retrieved fragments are
 shown to a user or supplied to a downstream tool or agent.
 
-- [Issue #3](https://github.com/Kims-DeveloperGroup/generic-rag/issues/3) is
-  planned to add generic projection orchestration. Its intended responsibility
-  is to accept caller-approved documents and explicitly injected collaborators,
-  derive fragments under a defined chunking policy, embed ordered fragment
-  text, replace or delete complete document projections, and report truthful
-  checkpoints and receipts. Its precise API and failure behavior are not part
-  of 0.1.0.
+- Projection accepts caller-approved documents and explicitly injected
+  collaborators. It derives deterministic fragments under a bounded chunking
+  policy, embeds ordered fragment text, replaces or deletes complete document
+  projections, and returns a manifest and truthful receipt for caller-owned
+  persistence.
 - [Issue #4](https://github.com/Kims-DeveloperGroup/generic-rag/issues/4) is
   planned to add retrieval and composition. Its intended responsibility is to
   use an injected `Embedder` and `VectorIndexReader` for semantic candidates
   and an injected `LexicalRetriever` for lexical candidates, then define
   deduplication, fusion, limiting, and outcome behavior. Provider rank will be
   the input; raw provider scores are not represented or assumed comparable.
 
-The caller/provider ownership model remains explicit throughout this plan. See
-[resource lifecycle](docs/lifecycle.md) and
+There is no end-user or agent query workflow yet. A consuming application can
+project data now, but must wait for or implement a separate reviewed retrieval
+layer before supplying retrieved context to users, tools, or agents. The
+caller/provider ownership model remains explicit throughout. See [resource
+lifecycle](docs/lifecycle.md) and
 [security and privacy](docs/security-and-privacy.md).
 
 ## Compatibility
@@ -111,9 +150,10 @@ assume compatibility across minor releases. For this release, direct imports
 from the documented owning modules are the supported public paths; root-level
 imports are not.
 
-The distribution includes `py.typed`. The wheel contains exactly the four
+The distribution includes `py.typed`. The wheel contains exactly the five
 importable modules `generic_rag`, `generic_rag.errors`,
-`generic_rag.contracts`, and `generic_rag.ports`, plus the typing marker.
+`generic_rag.contracts`, `generic_rag.ports`, and `generic_rag.projection`, plus
+the typing marker.
 
 ## Development verification
 
@@ -140,6 +180,5 @@ uv run --frozen python tests/support/verify_artifacts.py "$rag_dist_dir"
 
 CI is configured to run the tests on Python 3.11 and 3.14. On Python 3.11 it
 also runs lint, format, strict type, compilation, artifact, source-rebuild,
-clean-install, and isolated-import checks. A local Python 3.14.2 run currently
-contains 53 passing tests; the CI matrix is the authoritative cross-version
-result.
+clean-install, and isolated-import checks. The CI matrix is the authoritative
+cross-version result.