Repository navigation
Share search, graph and document semantics across clients - #405
Merged
Merged
Conversation
Each column's block layout is recorded in the segment footer and the reader decodes by it, rather than inferring the layout from the codec. A memtable cell, WAL row or string block whose bytes do not decode is a typed error, never a NULL or a skipped row. Read paths that skipped an unreadable flushed segment or timeseries partition now refuse the read and file a black-box report through an optional diagnostics feature. Memtable appends and batch inserts apply whole or not at all, and a flush encodes the segment before draining the memtable, so an encode error leaves every row readable.
…yped Add error variants for a numeric value out of range (22003), invalid text representation (22P02), datatype mismatch (42804), invalid datetime format (22007), datetime field overflow (22008), the node-label limit, and a full-text column fault. Each crosses the cluster wire as its own Data-Plane code and renders its SQLSTATE on every transport. RollbackFailed carries the typed cause of the reverse write that failed. Data-Plane handlers convert errors through ErrorCode::from instead of flattening them into Internal strings, and a bitmap sub-plan that fails fails its join instead of reading as an empty bitmap.
A u64 above i64::MAX reads as an exact Decimal and writes back as a MessagePack uint64 instead of wrapping negative. Numbers compare through one shared exact order: integers never round through f64, an integer against a float compares exactly, and NaN sorts above every number. The order backs filters, sorts, group keys and JSON comparisons, and a DECIMAL sort key orders by value. An integer literal past i64 stays an exact Decimal, or is refused with 22003 past the Decimal range. A bound float parameter stays a float. NaN and +/-Infinity render as their PostgreSQL text instead of JSON null. Timeseries ingest refuses an unsigned field an Int64 column cannot hold.
SUM and AVG accumulate integers exactly and floats with compensated addition, and fail with 22003 when an integer total leaves the Decimal range. MIN and MAX keep the original value and compare it exactly, so an integer column returns an integer. The rule applies to document, columnar and timeseries aggregates, window functions (now evaluated over typed values rather than JSON), HAVING, streaming materialized views, continuous aggregates and the Control-Plane post-aggregate. Spill runs, shuffle state and sketches encode as MessagePack so non-finite values survive.
ColumnType::Decimal carries an optional typmod. A plain DECIMAL keeps every digit it is given. DECIMAL(p,s) rounds a written value to s digits and refuses one with more than p-s integer digits with 22003. A typmod outside the supported range is refused at CREATE and ALTER with 22023.
Column definitions record the declared integer width (SMALLINT, INT) and float width (REAL). Strict, columnar, schemaless and KV writes re-type values under declared numeric columns and refuse one past the declared width, including computed assignments, counters, upserts, staged transaction writes and WAL replay. A narrowing ALTER COLUMN TYPE is refused. Declared VECTOR(n) columns of a schemaless collection are indexed like strict vector columns. The strict tuple encoder reports a value of the wrong kind and text that does not convert as distinct errors.
A TRANSFER on a field declared DECIMAL takes an exact INT or DECIMAL amount and moves it by decimal arithmetic. Every other field keeps float arithmetic. The amount is typed through the plan, WAL and replication records, and the computed balances meet the collection's declared column rules.
Struct-like variants such as Insert, Upsert, Merge, the search plans, the array plans and the vector-primary writes wrap named *Plan structs defined per family under types/plan/variants. Planner, visitor and converter matches follow. The catalog fold and aggregate wrap passes split into module directories.
A collection keyed by a declared primary key renders a row's identity under that column on every scan, filter, RLS image and RETURNING path, never as an extra id beside it. RLS injection resolves the identity column through the catalog. An assignment to the key column (UPDATE SET, ON CONFLICT DO UPDATE, MERGE, native document put) is refused unless it keeps the row's key. A WHERE on any declared key is a point lookup, a computed item over it is evaluated on the Control Plane, and to_jsonb(*) returns the whole row.
Point puts and deletes return undo entries for every R-tree, vector and sparse-vector mutation they make. A write abandoned after it applied (an enforcement refusal, a chain settle error, a commit error or a dropped batch) reverses them together with its cache entries and target row writes, and a failed reversal fail-stops the core. Undo failures carry a typed UndoError with their cause. An edge version the CSR refuses is taken back out of the edge store. Document collections no longer ingest geometry rows into a columnar memtable.
An import's write set comes from the operations it adds to the oplog, so a blob with pending changes still reports the rows its ready changes wrote. A delta that writes a non-map root container is refused. CRDT document writes capture the row image and restore it on abort. A dead-letter entry the queue or store refuses fails or holds the write instead of being logged, restart replay stops at a record its writer cannot produce, and both cases file a black-box report. Snapshot imports run without the peer byte and operation ceilings.
A stored strict row that does not decode no longer reaches the Event Plane with a missing or raw image. The write event carries an image fault, and delivery dead-letters it with an audit entry instead of running triggers, change streams, views or CRDT sync. Redo journalling of such a row fail-stops the core.
Columnar UPDATE and DELETE match current rows in flushed segments as well as the memtable, on the autocommit, resolve and transaction staging paths. An UPDATE assignment can be an expression evaluated against each row's pre-image. Assignments travel as UpdateValue through the plan, WAL and replication, and a filter that does not decode refuses the statement.
Every full-text table key gains a field component: a collection has a whole-document index plus one index per top-level string field. A document is indexed as (field, text) pairs through the WAL, sync and replication records, and a re-index retracts the fields the new body no longer fills. A segment that fails validation or is listed but missing is an error. Deletes are tracked per segment so merges drop dead postings. Bulk DELETE and TRUNCATE remove text in batched transactions, and a full TRUNCATE clears a collection in one purge. The writer and memtable split into module directories.
text_match and bm25_score take a column (or * for the whole document) and named options (mode, fuzzy). A column that cannot serve a search is a typed error (42703 or 42804). AND mode returns the true top-k of the documents holding every word, a word matching through its synonyms. Negative terms and residual WHERE and RLS filters apply before ranking, and equal scores order by row. A bm25_score in the SELECT list becomes a per-row score column: 0 for a row the index holds that does not match, NULL for a row it does not hold. Hybrid and three-source RRF searches take a field and mode and filter each leg before fusion. Staged transaction rows score through the same scorer, and a phrase is analyzed once with the collection's analyzer.
A vector search can rank among a set of primary keys, from a key IN list in SQL or allowed_ids on the native protocol, lowered to the candidate bitmap. A native metadata filter plans as the WHERE of the collection and fills the residual-filter slot, and the candidate window widens until top_k rows pass. A read policy joins the statement's WHERE predicates instead of replacing them. A search that names no field reads the collection's only vector index.
LABEL takes a list of labels on GRAPH TRAVERSE, GRAPH PATH and the CSR walks. A trailing EDGE WHERE clause crosses only edges whose properties match. Edge properties are stored as plain MessagePack. GRAPH TRAVERSE returns each node's depth and the crossed edges with their properties. Walks merge the session transaction's staged edges on every hop and drop a start node the graph does not hold. An unknown direction is refused.
Each Data-Plane core records its overlay journal lengths under a fresh savepoint id. The mark and the rewind go through every staged vShard, and a core rewinds to its own record, so a core hosting several staged vShards rewinds once and correctly. This replaces the per-vShard composite marker. A savepoint in an aborted block fails with 25P02.
One classifier recognises BEGIN and START TRANSACTION with their modes, COMMIT, END, ROLLBACK, ABORT and the savepoint spellings for pgwire and the native protocol. BEGIN refuses an unsupported isolation level or DEFERRABLE with 0A000 and stores an access mode. An error in any extended-query message aborts the block, and an aborted block admits only transaction control on both protocols.
The client's requested result formats resolve per column through one wire-type mapping, shared by Describe, Execute, DDL results and streaming. DECIMAL columns advertise numeric, UUID columns uuid, vectors float4 arrays and structured columns json. bytea renders as \x hex and non-finite floats as their PostgreSQL text. Output-schema derivation returns catalog errors instead of falling back to text.
The native and pgwire clients build the same search and graph SQL from shared modules: vector search honours allowed ids, text search takes mode and fuzzy options, graph traversal takes a direction and label set, and shortest path is available on both. Documents read back with the same fields whichever client wrote them. pgwire cells decode by column type and a cell that does not decode is an error naming the column, never a NULL. Search hits decode strictly. The protocol's text fields carry edge labels, a text mode and allowed ids. The NodeDb trait's provided methods split into modules.
farhan-syah
force-pushed
the
fix/lite-shared-api
branch
from
October 8, 2026 09:09
3d6f5d5 to
cc893da
Compare
farhan-syah
changed the title
Size the field vector through checked_decode_capacity so a corrupt count cannot request an oversized allocation.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Four panics in test modules print a hard-coded test value when an assertion fails. The output goes to test results, not a log, and no user data reaches it.
farhan-syah
force-pushed
the
fix/lite-shared-api
branch
from
October 8, 2026 10:24
fb195f4 to
b8175f7
Compare
farhan-syah
added
the
run-ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Features:
nodedb-client: native and pgwire clients build the same search and graph SQL from shared modules. Vector search takes allowed ids. Text search takes mode and fuzzy options.graph_traversetakes a direction and label set. Shortest path works on both clients. TheNodeDbtrait's provided methods split into modules.nodedb-fts,nodedb-query,nodedb-sql:text_matchandbm25_scoretake a column (or*for the whole document) and named options (mode,fuzzy). Abm25_scorein the SELECT list is a per-row score: 0 for an indexed row that does not match, NULL for a row the index does not hold. AND mode returns the true top-k of documents holding every word, synonyms included. Negative terms, residual WHERE and RLS filters apply before ranking. Hybrid and three-source RRF searches take a field and mode and filter each leg before fusion. Staged transaction rows score through the same scorer.nodedb-fts: each collection has a whole-document index plus one index per top-level string field. A re-index retracts fields the new body no longer fills. Deletes are tracked per segment so merges drop dead postings. Bulk DELETE and TRUNCATE remove text in batched transactions.INlist in SQL orallowed_idson the native protocol restricts candidates before the top-k cut. A native metadata filter plans as the collection's WHERE, and the candidate window widens untiltop_krows pass. A read policy joins the statement's WHERE predicates instead of replacing them.LABELtakes a list onGRAPH TRAVERSE,GRAPH PATHand the CSR walks. A trailingEDGE WHEREcrosses only edges whose properties match.GRAPH TRAVERSEreturns each node's depth and the crossed edges with properties. Walks merge the session transaction's staged edges on every hop.nodedb-crdt:remove_fieldsdeletes named scalar keys from a row.import_trackedreports which rows an import changed, so a derived index updates only those rows.nodedb-fts:FtsIndex::with_memtable_configsets the memtable spill thresholds.index_analyzed_documentindexes tokens the caller already analyzed. Sparse postings release spare capacity.nodedb-graph: traversal takes a direction and a label set (label_filter: &[&str]).TRANSFERon aDECIMALfield takes an exact INT or DECIMAL amount and uses decimal arithmetic.BEGIN,START TRANSACTION,COMMIT,END,ROLLBACK,ABORTand savepoint spellings for pgwire and the native protocol. After an error in any extended-query message, an aborted block admits only transaction control on both protocols. Each Data-Plane core records savepoints under its own savepoint id, replacing the per-vShard composite marker.DECIMALadvertises numeric,UUIDuuid, vectors float4 arrays, structured columns json.Fixes:
nodedb-crdt: an import's write set comes from the operations it adds to the oplog. A delta that writes a non-map root container is refused. CRDT document writes restore the row image on abort. A dead-letter entry the queue or store refuses fails or holds the write.i64::MAXreads as an exact Decimal and writes back as MessagePack uint64. Filters, sorts, group keys and JSON comparisons share one exact order, with NaN above every number. NaN and +/-Infinity render as PostgreSQL text instead of JSON null.DECIMAL(p,s)typmods are enforced. DeclaredSMALLINT,INTandREALwidths are enforced on strict, columnar, schemaless and KV writes, including computed assignments, counters, upserts, staged writes and WAL replay. DeclaredVECTOR(n)columns of a schemaless collection are indexed like strict ones.SqlPlanvariants wrap named*Planstructs undertypes/plan/variants.Breaking changes
The project is pre-1.0. None of these changes has a migration path.
(field, text)pairs through WAL, sync and replication records.UpdateValuethrough the plan, WAL and replication records. A filter that does not decode refuses the statement.TRANSFERamount is typed through the plan, WAL and replication records.ColumnType::Decimalcarries an optional typmod. Column definitions record declared integer and float widths.22003numeric value out of range, including integer totals past the Decimal range and aDECIMAL(p,s)overflow.22P02invalid text representation.42804datatype mismatch.22007invalid datetime format.22008datetime field overflow.22023DECIMALtypmod out of range at CREATE and ALTER.42703and42804for a column that cannot serve a text search.25P02for a savepoint in an aborted block.0A000for an unsupported isolation level orDEFERRABLEinBEGIN.RollbackFailedcarries the typed cause. Data-Plane handlers convert throughErrorCode::frominstead ofInternalstrings. A failed bitmap sub-plan fails its join.ALTER COLUMN TYPEis refused. An integer literal past the Decimal range is refused with22003. Timeseries ingest refuses an unsigned field anInt64column cannot hold.GRAPH TRAVERSEdrops a start node the graph does not hold.Why
Native and pgwire clients (including Lite) need identical search, graph and document semantics. Silent NULLs, wrapped integers and lossy numeric paths hid errors. These changes make each case a typed error or an exact value.
How to check
New tests:
nodedb/tests/wire/cases: FTS query semantics and options, hybrid and three-source RRF, FTS and graph overlays in transactions, graph label sets and edge predicates,DECIMALtypmods,TRANSFERonDECIMAL, integer and non-finite aggregates, u64 literals, columnar DML on flushed rows, declared-width computed writes, primary-key identity on UPDATE and ON CONFLICT, pgwire transaction spellings and extended-query abort.nodedb/tests/native: native transaction spellings and document identity update.nodedb/tests/inproc/cases: graph transaction overlays and edge predicates, field-scope re-index, FTS update re-index, streaming materialized views, sync compatibility.nodedb-clientfor SQL rendering and result decoding.Both test stages pass on the branch head with
--retries 0:cargo nextest run --workspace --exclude nodedb-cluster-tests --all-features --no-fail-fast --retries 0cargo nextest run -p nodedb-cluster-tests --all-features --no-fail-fast --retries 0cargo clippy --workspace --all-targets --all-features -- -D warningsis clean.