Skip to content

Add C++ local-document ingest with immutable chunks - #126

Merged
usrname1git merged 1 commit into
mainfrom
feat/cpp-document-chunks
Sep 10, 2026
Merged

Add C++ local-document ingest with immutable chunks#126
usrname1git merged 1 commit into
mainfrom
feat/cpp-document-chunks

Conversation

@usrname1git

Copy link
Copy Markdown
Owner

Why

C++ memory-store still rejected document/chunks (Local-Document-Adapter). Go already stages immutable chunks and records adapter provenance on the run and source observation.

What

Same gates as Go: metadata and chunks together, content_sha256 of raw_transcript, contiguous UTF-8 byte ranges covering the transcript, forbidden-secret scan, unique chunks index, $setOnInsert so a retry is idempotent_noop.

Verified

  • memory-store.exe --self-test (document-ok, document-chunks-required)
  • throwaway godbrain_cpp_store_test: committed, chunks=1, second ingest idempotent_noop; db dropped

Not in this PR

  • Start/Heal still launch Go

Local-Document-Adapter payloads now parse document metadata and UTF-8
chunks, reject secrets, hash-bind content_sha256, and upsert the chunks
collection the same way as Go. Start/Heal still launch Go.
Copilot AI lite review requested due to automatic review settings September 10, 2026 18:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new JSON parsing path silently accepts malformed numeric fields (e.g., non-number ocr_confidence/confidence) due to ignored parse failures, which breaks strict input validation guarantees.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates the C++ cpp_memory_store ingestion protocol and MongoDB write path to support local-document ingestion (document + immutable chunks) in parity with the existing Go memory-store behavior, enabling idempotent chunk staging keyed by extractor identity and chunk index.

Changes:

  • Added document + chunks parsing/validation to the C++ stdin JSON protocol, including bounded chunk ranges and sensitive-content scanning.
  • Added MongoDB chunks unique index and chunk upsert + immutability conflict checks during staging.
  • Exposed sha256_hex() for document content_sha256 validation and documented the new “cut 11” capability.
File summaries
File Description
godbrain_core/cpp_memory_store/src/store.cpp Adds unique index and staging logic for immutable per-source chunks tied to extractor identity.
godbrain_core/cpp_memory_store/src/protocol.cpp Parses/validates document + chunks, including SHA-256 checks and secret scanning.
godbrain_core/cpp_memory_store/src/embedding.cpp Makes sha256_hex() available outside the anonymous namespace.
godbrain_core/cpp_memory_store/README.md Bumps to cut 11 and documents local-document ingest + immutable chunks.
godbrain_core/cpp_memory_store/include/godbrain/memory_store/protocol.hpp Adds DocumentMetadata/SourceChunk structs and payload fields.
godbrain_core/cpp_memory_store/include/godbrain/memory_store/embedding.hpp Exposes sha256_hex() in the public header.
Review details

Suppressed comments (1)

godbrain_core/cpp_memory_store/src/protocol.cpp:1091

  • json_number() return value is ignored for per-chunk confidence. If confidence is provided with a non-number type, the code marks has_confidence=true but leaves confidence at the default 0, silently accepting malformed payloads.
            if (json_has(item, "confidence")) {
                ch.has_confidence = true;
                json_number(item, "confidence", &ch.confidence);
            }
  • Files reviewed: 6/6 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +1049 to +1052
if (json_has(*doc, "ocr_confidence")) {
route->ingest.document.has_ocr_confidence = true;
json_number(*doc, "ocr_confidence", &route->ingest.document.ocr_confidence);
}
Comment on lines 1 to 5
#include "godbrain/memory_store/protocol.hpp"
#include "godbrain/memory_store/embedding.hpp"
#include "godbrain/memory_store/json.hpp"
#include "godbrain/memory_store/state_machine.hpp"
#include "godbrain/memory_store/embedding.hpp"
@usrname1git
usrname1git merged commit 6ec8ca7 into main Sep 10, 2026
1 check passed
@usrname1git
usrname1git deleted the feat/cpp-document-chunks branch September 10, 2026 18:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants