[RAG Retrieval] Add deterministic semantic and hybrid retrieval

PYTHON_MODULE_INDEX.md

@@ -4,9 +4,9 @@
 
 - Declared source root: `src`
 - Packaging source of truth: `pyproject.toml`
-- Importable production units: 5
-- Indexed production units: 5
-- Source/index parity: 5/5
+- Importable production units: 7
+- Indexed production units: 7
+- Source/index parity: 7/7
 - Package data: `src/generic_rag/py.typed`
 - Locked verification owner: `.github/workflows/ci.yml` (supporting workflow,
   not an importable unit)
@@ -57,8 +57,8 @@
   `ProjectionLimits`, `ProjectionRequest`, `ProjectionManifestEntry`,
   `ProjectionManifest`, `ProjectionStateAvailability`,
   `ProjectionStateSnapshot`, `ProjectionStateStatus`, `ProjectionResult`,
-  `RetrievalQuery`, `RetrievalOutcome`, `RetrievalHit`, and `RetrievalResult`
-  from `generic_rag.contracts`.
+  `RetrievalLimits`, `RetrievalQuery`, `RetrievalOutcome`, `RetrievalHit`, and
+  `RetrievalResult` from `generic_rag.contracts`.
 - Re-exports: exactly the names in the module's `__all__`; none from the package
   root.
 - Direct internal dependencies: `generic_rag.errors`.
@@ -70,7 +70,7 @@
   `tests/test_retrieval_contracts.py`, `tests/test_package_boundaries.py`,
   `tests/support/clean_import_probe.py`, `tests/support/verify_artifacts.py`, and
   the locked CI import, boundary, and artifact checks.
-- Documentation: `docs/api.md`, `docs/projection.md`, and
+- Documentation: `docs/api.md`, `docs/projection.md`, `docs/retrieval.md`, and
   `docs/security-and-privacy.md`.
 
 ## `generic_rag.ports`
@@ -90,8 +90,8 @@
 - Verification: `tests/test_ports.py`, `tests/test_package_boundaries.py`,
   `tests/test_projection.py`, `tests/support/clean_import_probe.py`, and the
   locked CI import and boundary checks.
-- Documentation: `docs/api.md`, `docs/lifecycle.md`, `docs/projection.md`, and
-  `docs/security-and-privacy.md`.
+- Documentation: `docs/api.md`, `docs/lifecycle.md`, `docs/projection.md`,
+  `docs/retrieval.md`, and `docs/security-and-privacy.md`.
 
 ## `generic_rag.projection`
 
@@ -104,7 +104,10 @@
 - Re-exports: exactly the names in the module's `__all__`; none from the package
   root.
 - Direct internal dependencies: `generic_rag.contracts`, `generic_rag.errors`,
-  and `generic_rag.ports`.
+  `generic_rag.ports`, and `generic_rag.projection_integrity`.
+- Integrity ownership: delegates deterministic source-digest, fragment-ID, and
+  checkpoint-token derivation and checkpoint validation to
+  `generic_rag.projection_integrity`.
 - Owned state or external resources: none; planning state is immutable and
   local to each call, while every embedder, writer, and resetter remains
   caller-owned through `Borrowed`.
@@ -118,3 +121,59 @@
   checks.
 - Documentation: `docs/projection.md`, `docs/api.md`, `docs/lifecycle.md`, and
   `docs/security-and-privacy.md`.
+
+## `generic_rag.projection_integrity`
+
+- Source: `src/generic_rag/projection_integrity.py`
+- Responsibility: own deterministic v1 derivation and validation algorithms
+  for projection integrity values.
+- Supported public imports: `derive_source_digest`, `derive_fragment_id`,
+  `derive_projection_checkpoint_token`, and
+  `has_valid_projection_checkpoint` from `generic_rag.projection_integrity`.
+- Re-exports: exactly the names in the module's `__all__`; none from the package
+  root.
+- Direct internal dependencies: `generic_rag.contracts` and
+  `generic_rag.errors`.
+- Owned state or external resources: none; each call derives or checks an
+  immutable value from exact caller-supplied contracts.
+- Material side effects: none at import or call time.
+- Verification: `tests/test_projection_integrity.py`,
+  `tests/test_projection.py`, `tests/test_retrieval.py`,
+  `tests/test_package_boundaries.py`, `tests/support/clean_import_probe.py`,
+  `tests/support/verify_artifacts.py`, and the locked CI test, lint, strict
+  type, compile, build, clean-install, import, and artifact checks.
+- Documentation: `docs/api.md`, `docs/projection.md`, `docs/retrieval.md`, and
+  `docs/security-and-privacy.md`.
+
+## `generic_rag.retrieval`
+
+- Source: `src/generic_rag/retrieval.py`
+- Responsibility: deterministically compose bounded semantic and hybrid
+  retrieval results from published caller-owned projection state.
+- Supported public imports: `retrieve_semantic` and `retrieve_hybrid` from
+  `generic_rag.retrieval`.
+- Re-exports: exactly the names in the module's `__all__`; none from the package
+  root.
+- Direct internal dependencies: `generic_rag.contracts`, `generic_rag.errors`,
+  `generic_rag.ports`, and `generic_rag.projection_integrity`.
+- Owned state or external resources: none; ranking state is local to each call,
+  while every embedder, vector reader, and lexical retriever remains
+  caller-owned through `Borrowed`.
+- Material side effects: none at import time. At explicit workflow call time it
+  may read one borrowed embedder identity, embed one query, search one borrowed
+  vector reader, and for hybrid retrieval search one borrowed lexical
+  retriever; it performs no persistence, network, retry, acquisition, release,
+  logging, authorization, citation, or lifecycle action itself.
+- Retrieval semantics: validates and deduplicates exact fragment identities,
+  filters candidates against the full current published document revision,
+  and preserves semantic provider order or fuses original semantic and lexical
+  ranks deterministically without comparing raw scores. Returned fragments
+  remain non-authoritative; this module does not authorize a query or source,
+  validate authoritative source text, or create citations.
+- Verification: `tests/test_retrieval.py`,
+  `tests/test_retrieval_contracts.py`, `tests/test_ports.py`,
+  `tests/test_package_boundaries.py`, `tests/support/clean_import_probe.py`,
+  `tests/support/verify_artifacts.py`, and the locked CI test, lint, strict
+  type, compile, build, clean-install, import, and artifact checks.
+- Documentation: `docs/retrieval.md`, `docs/api.md`, `docs/lifecycle.md`,
+  `docs/security-and-privacy.md`, and `README.md`.

README.md

@@ -3,12 +3,13 @@
 `generic-rag` is a provider-neutral, runtime-dependency-free foundation for
 retrieval-augmented generation (RAG). Version 0.1.0 requires Python 3.11 or
 later and provides immutable contracts, typed error categories, synchronous
-collaborator protocols, deterministic bounded projection orchestration, and
-explicit caller-owned borrowing.
+collaborator protocols, deterministic bounded projection orchestration,
+semantic retrieval, rank-based hybrid retrieval, and explicit caller-owned
+borrowing.
 
-Retrieval and result composition are not implemented in 0.1.0. The package has
-no built-in adapter, provider, factory, persistence, network client,
-configuration system, authentication, citation mechanism, or CLI.
+The package has no built-in adapter, provider, factory, persistence, network
+client, configuration system, authentication, authorization, citation
+mechanism, or CLI.
 
 ## Install from a checkout
 
@@ -107,12 +108,23 @@ explicit corpus-wide reset is intended. See the [projection guide](docs/projecti
 for the complete lifecycle, state matrix, adapter obligations, and failure
 behavior.
 
+Load the corresponding published manifest before retrieval, reauthorize each
+query and source in the host, and call `retrieve_semantic` or
+`retrieve_hybrid` with borrowed provider implementations. Returned fragment
+text is non-authoritative: resolve each identity against the still-authorized
+source revision and create host-owned citations before showing results to a
+user or injecting bounded context into an agent. See the [retrieval
+guide](docs/retrieval.md) for an executable independent-consumer example,
+deterministic fusion behavior, outcome handling, and the complete host flow.
+
 Public values must be imported from their owning modules:
 
 - `generic_rag.contracts`
 - `generic_rag.errors`
 - `generic_rag.ports`
 - `generic_rag.projection`
+- `generic_rag.projection_integrity`
+- `generic_rag.retrieval`
 
 The package root intentionally has no re-exports: `generic_rag.__all__ == ()`.
 See the [API reference](docs/api.md) for every supported name and invariant.
@@ -129,19 +141,20 @@ shown to a user or supplied to a downstream tool or agent.
   policy, embeds ordered fragment text, replaces or deletes complete document
   projections, and returns a manifest and truthful receipt for caller-owned
   persistence.
-- [Issue #4](https://github.com/Kims-DeveloperGroup/generic-rag/issues/4) is
-  planned to add retrieval and composition. Its intended responsibility is to
-  use an injected `Embedder` and `VectorIndexReader` for semantic candidates
-  and an injected `LexicalRetriever` for lexical candidates, then define
-  deduplication, fusion, limiting, and outcome behavior. Provider rank will be
-  the input; raw provider scores are not represented or assumed comparable.
-
-There is no end-user or agent query workflow yet. A consuming application can
-project data now, but must wait for or implement a separate reviewed retrieval
-layer before supplying retrieved context to users, tools, or agents. The
-caller/provider ownership model remains explicit throughout. See [resource
-lifecycle](docs/lifecycle.md) and
-[security and privacy](docs/security-and-privacy.md).
+- Retrieval accepts the matching caller-loaded published state and injected
+  semantic or lexical providers. It validates and filters current-revision
+  candidates, deduplicates exact identities, and returns bounded score-free
+  hits. Hybrid retrieval fuses provider ranks deterministically; it does not
+  compare raw provider scores.
+- The host reauthorizes every query, resolves each returned fragment identity
+  against authoritative source data, verifies the exact source slice, and
+  creates citations. It may then show cited results to a user or inject bounded
+  cited context into an agent; the agent must retain those citations.
+
+The package does not decide provider selection, authentication, authorization,
+prompt or tool policy, retry, display, logging, or resource lifecycle. See the
+[retrieval guide](docs/retrieval.md), [resource lifecycle](docs/lifecycle.md),
+and [security and privacy](docs/security-and-privacy.md).
 
 ## Compatibility
 
@@ -150,10 +163,11 @@ assume compatibility across minor releases. For this release, direct imports
 from the documented owning modules are the supported public paths; root-level
 imports are not.
 
-The distribution includes `py.typed`. The wheel contains exactly the five
+The distribution includes `py.typed`. The wheel contains exactly the seven
 importable modules `generic_rag`, `generic_rag.errors`,
-`generic_rag.contracts`, `generic_rag.ports`, and `generic_rag.projection`, plus
-the typing marker.
+`generic_rag.contracts`, `generic_rag.ports`, `generic_rag.projection`,
+`generic_rag.projection_integrity`, and `generic_rag.retrieval`, plus the typing
+marker.
 
 ## Development verification
 

docs/api.md

@@ -1,10 +1,11 @@
 # Public API
 
 Version 0.1.0 exposes immutable values, typed error categories, synchronous
-collaborator protocols, and deterministic bounded document projection. It does
-not expose retrieval or result-composition orchestration. See the [projection
-guide](projection.md), [resource lifecycle](lifecycle.md), and [security and
-privacy boundary](security-and-privacy.md) for the surrounding usage contract.
+collaborator protocols, deterministic bounded document projection, and
+score-free semantic and hybrid retrieval. See the [projection guide](projection.md),
+[retrieval guide](retrieval.md), [resource lifecycle](lifecycle.md), and
+[security and privacy boundary](security-and-privacy.md) for the surrounding
+usage contract.
 
 ## Import boundary
 
@@ -42,6 +43,7 @@ public symbols. Import names from their owning modules instead.
 - `ProjectionStateSnapshot`
 - `ProjectionStateStatus`
 - `ProjectionResult`
+- `RetrievalLimits`
 - `RetrievalQuery`
 - `RetrievalOutcome`
 - `RetrievalHit`
@@ -64,6 +66,18 @@ public symbols. Import names from their owning modules instead.
 - `project_documents`
 - `rebuild_projection`
 
+`generic_rag.projection_integrity` exports exactly:
+
+- `derive_source_digest`
+- `derive_fragment_id`
+- `derive_projection_checkpoint_token`
+- `has_valid_projection_checkpoint`
+
+`generic_rag.retrieval` exports exactly:
+
+- `retrieve_semantic`
+- `retrieve_hybrid`
+
 The package does not support importing any of these names from the package
 root.
 
@@ -127,6 +141,15 @@ An ordinary collaborator exception is chained as the operation error's cause.
 An invalid identity, malformed vector result, or non-`None` command result has
 no internal cause. `KeyboardInterrupt` and `SystemExit` pass through unchanged.
 
+Retrieval defines no workflow-specific public exception. Invalid top-level
+workflow inputs, including a query that exceeds `RetrievalLimits`, raise
+`ContractValidationError` before collaborator effects. After a collaborator
+boundary is entered, ordinary `Exception` failures and malformed collaborator
+returns contribute a content-free failure state. A failed branch contributes no
+fragment or exception text; a `PARTIAL` result can contain independently
+validated hits from another branch. No exception cause crosses the result
+boundary. `BaseException` subclasses pass through unchanged.
+
 ## Documents and fragments
 
 | Type | Fields | Construction rules |
@@ -174,6 +197,38 @@ The package produces manifests; the caller owns their persistence. Source
 digests, fragment IDs, and checkpoint tokens are deterministic under explicit
 v1 domains described in the [projection guide](projection.md#deterministic-projection-values).
 
+The public integrity functions are synchronous and positional-only:
+
+```python
+def derive_source_digest(document: Document, /) -> str: ...
+
+def derive_fragment_id(
+    document: DocumentIdentity,
+    start: int,
+    end: int,
+    /,
+) -> str: ...
+
+def derive_projection_checkpoint_token(
+    corpus_id: str,
+    projection: ProjectionIdentity,
+    chunking: ChunkingPolicy,
+    entries: tuple[ProjectionManifestEntry, ...],
+    /,
+) -> str: ...
+
+def has_valid_projection_checkpoint(
+    manifest: ProjectionManifest,
+    /,
+) -> bool: ...
+```
+
+They reproduce the same v1 integrity values used by projection and retrieval.
+They validate exact public contract shapes and canonical manifest-entry order;
+invalid inputs raise `ContractValidationError`. Checkpoint validation returns
+whether the supplied token equals the derived token. It does not inspect a
+provider index or establish authorization.
+
 ## Projection state and results
 
 `ProjectionStateAvailability` is a closed string enum:
@@ -264,6 +319,7 @@ failure matrices.
 
 | Type | Fields | Construction rules |
 | --- | --- | --- |
+| `RetrievalLimits` | `max_query_codepoints: int` | The query-text cap is a positive exact integer. |
 | `RetrievalQuery` | `corpus_id: str`, `text: str`, `hit_limit: int`, `candidate_limit: int` | Corpus and text are nonblank exact strings; limits are positive exact integers and `hit_limit <= candidate_limit`. Values are preserved exactly. |
 | `RetrievalHit` | `fragment: Fragment`, `rank: int` | Fragment requires its exact class and rank is a positive exact integer. There is no score field. |
 | `RetrievalResult` | `query: RetrievalQuery`, `outcome: RetrievalOutcome`, `hits: tuple[RetrievalHit, ...]`, `truncated: bool` | Nested values, the hit tuple, and the boolean require exact types. Hit count cannot exceed `query.hit_limit`. |
@@ -274,12 +330,12 @@ ranks are contiguous from one, fragment identities are unique, and every
 fragment belongs to the query corpus. `PARTIAL` requires at least one hit;
 `UNAVAILABLE`, `STALE`, and `FAILED` require no hits and `truncated=False`.
 
-`truncated=True` is the caller's explicit assertion that otherwise valid work
-or results were cut by the query budget. Hits and reader ports are score-free;
-raw provider scores are neither represented nor promised comparable.
-
-These are value contracts only. Version 0.1.0 has no package retrieval,
-composition, citation, user, tool, or agent workflow.
+The workflows set `truncated=True` exactly when validated, unique,
+current-revision candidates exceed `query.hit_limit`. Hits and reader ports are
+score-free; raw provider scores are neither represented nor promised
+comparable. When constructing a `RetrievalResult` directly, callers remain
+responsible for supplying a truthful `truncated` value because the value
+contract cannot reconstruct discarded candidates.
 
 ## Collaborator ports
 
@@ -302,12 +358,50 @@ Projection enforces the embedder result rules and requires each writer or
 resetter command to return exactly `None`. It cannot enforce external storage,
 atomicity, authorization, concurrency, or lifecycle behavior.
 
+Retrieval checks its embedder identity and output, provider tuple types and
+candidate bounds, fragment integrity, corpus and published revisions, and
+cross-provider identity consistency. The caller still owns provider selection,
+authorization, persistence, concurrency, retries, and authoritative source
+validation.
+
 `Borrowed[T]` is the companion ownership marker, not a provider port. Its exact
 behavior is documented in [resource lifecycle](lifecycle.md).
 
-## Planned retrieval workflow
+## Retrieval workflows
+
+Both public functions are synchronous and all parameters are positional-only:
+
+```python
+def retrieve_semantic(
+    query: RetrievalQuery,
+    state: ProjectionStateSnapshot,
+    limits: RetrievalLimits,
+    embedder: Borrowed[Embedder],
+    vector_reader: Borrowed[VectorIndexReader],
+    /,
+) -> RetrievalResult: ...
+
+def retrieve_hybrid(
+    query: RetrievalQuery,
+    state: ProjectionStateSnapshot,
+    limits: RetrievalLimits,
+    embedder: Borrowed[Embedder],
+    vector_reader: Borrowed[VectorIndexReader],
+    lexical_retriever: Borrowed[LexicalRetriever],
+    /,
+) -> RetrievalResult: ...
+```
 
-Retrieval and composition remain planned for Issue #4. The existing query,
-result, reader, and lexical contracts do not promise an implemented workflow,
-fusion algorithm, compatibility check, exception mapping, citation policy, or
-user/agent integration.
+Semantic retrieval embeds the query once, validates at most
+`candidate_limit` vector candidates, preserves provider order through
+current-revision filtering, and returns at most `hit_limit` hits. Hybrid
+retrieval also obtains at most `candidate_limit` lexical candidates, preserves
+each provider's original ranks, and fuses exact identities using deterministic
+reciprocal rank fusion with offset 60. It uses provider ranks rather than raw
+scores and applies opaque identity ordering to ties.
+
+The workflows return `complete`, `partial`, `unavailable`, `stale`, or `failed`
+according to published-state and collaborator results. They do not authorize,
+cite, persist, log, retry, or manage collaborator resources. See the
+[retrieval guide](retrieval.md) for candidate validation, exact outcome
+handling, deterministic fusion, and the required user and agent host flow.