Repository navigation
fix: dispatch capacity refusals, KV sorted index ordering, array sync results, vector fixes - #416
Merged
Merged
Conversation
Every dispatcher capacity refusal is now built in one place that
increments nodedb_dispatch_capacity_busy_total{scope}. The counter is
shared with the node's metrics, so /metrics reads it without taking the
dispatcher lock.
A capacity refusal keeps its type through the system dispatch door and
every protocol surface, so clients see the retryable SERVER_OVERLOAD
class (SQLSTATE 57P03) instead of an internal error. Calvin catch-up
and deferred dispatch hold a refused transaction and re-send it once
capacity frees.
…ered search After a graph-only checkpoint restore, nodes hold no local vector and the vector lives in the attached backing. Neighbor pruning and the diversity heuristic read node storage directly, so every distance against a restored node errored and new nodes gained no incoming edges. Add HnswIndex::node_bytes, which resolves local storage first and then the backing, encoding to the index dtype when needed. Route insert, prune, and the f32 distance helper through it. dist_to_node_f32 now takes the metric, and the adaptive filter, ACORN, and NaviX traversals use it instead of get_vector, so restored and non-F32 indexes rank correctly under a caller-supplied metric.
nodedb_dispatch_capacity_busy_total measures dispatch pressure,
including refusals the server waits out itself. Add
nodedb_server_overload_responses_total{protocol} to measure client
impact: each protocol counts at the point it writes the overload
answer (pgwire 57P03, native 57P03 frame, HTTP 503 with Retry-After,
RESP BUSY). A gap between the two is pressure the server absorbs.
search_hit serves only the native and remote clients. Gate it on those features so a build with neither feature has no dead code. Add decoder tests for wrong cell types and a per-source score column.
Windowed rank, top-k, range and count now read each entry's timestamp from the timestamp column of its sort key instead of a separate lookup. Integer, float and ISO 8601 string columns decode to a timestamp. An entry with no decodable timestamp is hidden. WindowConfig::bounds returns a half-open [start, end) range, so a custom window admits nothing outside its range. Backfill extracts bool, null and composite values with the same bytes as the write path.
…ckpoint metric Distance evaluation reads node vectors through a probe, graph nodes may carry no vector, and checkpoint load checks the stored metric and structure before accepting an index.
…e bounds Encode sorted index values and numbers so byte order matches value order, parse WINDOW clauses in index DDL, and carry typed range bounds through the native sorted read path.
Add reject reasons and classes, per-op results on batch responses, advisory refusals, and a snapshot assembly lock with a diagnostic when its buffer is poisoned.
farhan-syah
added
the
run-ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #371
Closes #352
What changed
Dispatch capacity (#371, #352)
nodedb-bridge,nodedb/src/control: a full dispatch queue returnsError::DispatchCapacity. It maps to57P03/NDB-1403.503andRetry-After.XX000.nodedb_dispatch_capacity_busy_total{scope}counts refusals at the dispatcher.nodedb_server_overload_responses_total{protocol}counts overload answers sent to clients. It has a label per protocol:pgwire,native,http,resp.trace!, so a saturated queue no longer floods the log.main. This PR adds tests that pin it:dispatcher_refuses_over_cap_then_admits_once_a_slot_freesrebuild_against_a_saturated_tenant_defers_then_catches_upKV sorted index
kv/sorted_index/number.rs: integers and floats share one exact, order-preserving 10-byte encoding.11sorts above10.5.kv/sorted_index/value.rs: each value carries a type tag.WINDOW CUSTOMbounds accept ISO timestamps or milliseconds. A bad bound returns42601.KvSortedIndexRangetakes typed bounds (Value), not raw bytes.Array sync
ArrayRejectReasonlists every reason.class()tells a client whether to retry, resend the schema, re-authenticate, catch up, or give up.ArrayOpResults(0x98) returns one result per op for batches and snapshots. Before this, only the first reject reached the client.ArrayAdvisoryRefused(0x99) answers a refused ack or catch-up request. It is separate fromArrayReject.SnapshotBufferLostand records a capture.Vector
Client and CI
nodedb-client: the search-hit decoder compiles only when a client feature is enabled.nodedb-clientfeature subsets: none,native,remote.Docs (nodedb-docs, already on
main)SERVER_OVERLOADand the overload counters.WINDOW.Lite follow-ups
This PR changes shared-crate APIs. Lite must handle each change below:
ArrayRejectReasonlists every reason:SchemaTooNew=1, ArrayUnknown=2, ShapeInvalid=3, EngineUnavailable=4, RetentionFloor=5, EngineRefused=6, PermissionDenied=7, Unauthenticated=8, SnapshotCapacity=9, SnapshotBufferLost=10.EngineRejected,QuotaExceeded,WireVersionMismatch.ALLandclass().ArrayRejectClasswith valuesTransient,SchemaRequired,Reauthenticate,CatchUp,Permanent.ArrayRejectMsg::class()returns it.ArrayOpResults(0x98) withArrayOpResultsMsg,ArrayOpResultandArrayOpOutcome.ArrayDeltaBatchwith anArrayOpResults.ArrayReject{ShapeInvalid}.Unauthenticated.ArrayAdvisoryRefused(0x99) withArrayAdvisoryKind {Ack=1, CatchupRequest=2}andArrayAdvisoryRefusedMsg.ArrayReject.reason.class()and leaves the push window unchanged.TextFields::score_min/score_maxchange fromOption<Vec<u8>>toOption<Value>. Lite does not read these fields.dist_to_nodeerror text. Update them:restore/hnsw.rs:106engine/vector/durable/mod.rs:18search/lazy_load.rs:95MetaOp::CalvinFlusharm.hilbert_rangetoshard_bucket.How to check
cargo nextest run --workspace --exclude nodedb-cluster-tests --all-features --retries 0: 20901 of 20901 passed.cargo nextest run -p nodedb-cluster-tests --all-features --retries 0: 502 of 502 passed.nodedb-clientwith no features, withnative, and withremotescripts/ci/check_*gate passes, including--self-test.