Repository navigation
core: async-nats behind one module, and dead letters announce themselves - #14
Merged
Merged
Conversation
…ronous module nats 0.25 is deprecated and pins rustls 0.22, and its threads, connections and flusher floor had shaped the hot path. The decision: one module hosts async-nats on one tokio worker and everything else stays synchronous, with no async rewrite. The ADR records what the numbers decided (one connection, one worker), what the module has to rebuild because the new client does not give it (a real barrier, a reconnect that fails it, URL credentials), and what stays the same on the wire so that an existing JetStream store keeps working. Status: Proposed. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018wXgxT2UoFSZfqhAE1Tb5V
nats 0.25 is deprecated, 0.26 keeps rustls 0.22, and the client cost more than its size: two threads per connection and a connection per use, 1 011 threads and 401 connections at 200 flows, and a 5 ms flusher floor that three workarounds lived around. core/src/bus.rs hosts async-nats on one tokio worker and gives everything else a blocking API. No other module sees tokio or a NATS type, flows and connectors stay threads, and the interpreter still has no bus access. One connection carries the whole process, acks included: measured at 1, 10 and 50 flows, a second one buys nothing. publish_confirmed, fetch_iter_flushed and ack_nc go with the floor. Three things in the module are not obvious and matter. Client::flush only writes the socket, so it cannot stand for the sync client's PING. The barrier publishes to a subject only this connection subscribes to and waits to hear it back. And on a reconnect async-nats drops its unwritten buffer, so a round-trip alone could confirm a batch that lost its first half: a connection replaced while a batch was leaving fails the barrier, and the batch redelivers. Emits are held per batch. Handing each one over as it was made cost a cross-thread wake and a write per message, which the flusher floor had been coalescing without anyone asking: the flow hop fell from 20 300 to 12 600 evt/s, all of it in `process`. Held and sent with the barrier they cost one of each per round, and the rate came back. A quiet flow's emit still leaves at once. async-nats ignores credentials in the URL. bus.rs hands nats://user:pass@host and nats://token@host over itself; no test in the repo connects with credentials. Streams, buckets and durables are created with the JSON the sync client sent, so a fresh store comes out identical to v0.3.6's and an old one opens as it is. v0.3.6, then this build, then v0.3.6 again on one store: durable positions, a version overlay, a proposal, a dead letter, offsets, the lease and a detect snapshot all cross both ways. An emit subject with whitespace or a line break is now refused. v0.3.6 wrote it into the PUB line, so a flow building a subject from an event field let the event inject protocol: reproduced on that binary. The bus::live tests start their own nats-server and are ignored in CI like every test that needs one: cargo test bus::live -- --ignored Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018wXgxT2UoFSZfqhAE1Tb5V
Both standalone binaries held their own nats 0.25 client, and each carried a copy of publish_confirmed and fetch_round to get around the same flusher floor. They now compile core/src/bus.rs through #[path], which depends on nothing else in core, and those copies go. The source and sink loops keep their shape: publish before commit, put or confirm before ack. connectors/amqp moves from rustls 0.22 to 0.23 with the ring provider, with no source change to its TLS adapter. That was the last rustls 0.22 in the repo. Both crates pass their tests, the live lease and source tests included, and e2e/admission admits amqp_source and amqp_sink against a real RabbitMQ. "No tokio" no longer describes them, so SUBJECTS.md and COOKBOOK.md stop saying it. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018wXgxT2UoFSZfqhAE1Tb5V
RUSTSEC-2026-0049, -0098, -0099, -0104, -0346, -0349 and -0382 were ignored only because of rustls 0.22.4 and rustls-webpki 0.102.8, held by nats 0.25. No lockfile has them now: all five are on rustls 0.23.46 and rustls-webpki 0.103.16, and async-nats pulls that same 0.103 line, so the audit watches the real versions again. scripts/audit.sh is green with the entries gone. RUSTSEC-2024-0421 stays: idna 0.1.5 comes through amiquip, not through nats. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018wXgxT2UoFSZfqhAE1Tb5V
ARCHITECTURE gains the bus, the one thread that speaks NATS. The changelog records the move, the upgrade note on `_INBOX.>`, and what it fixed on the way: protocol injection through an emit subject, and the zombie left by a deleted consumer. Published numbers that changed, each against a run of the v0.3.6 binary in the same session: RSS under load 6-8 to 10 MB (v0.3.6 measures 9.2 today) and 49 to 38 MB at fifty flows, binary 6.2 to 11.2 MB (9.9 at v0.3.6), MQTT loopback 2 285 to 2 070 rt/s, and the multi-flow table. They change in README, bench/README, the book and site/index.html, which is deployed by hand. Left as published: cold start and the flow hop, which did not move; the paced p50 of 14 ms, which both binaries now beat, so this change did not earn it; the promote figure, not rerun on the old binary; the image size, not rebuilt; and bench/compare, a dated snapshot. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018wXgxT2UoFSZfqhAE1Tb5V
batch.publish refuses what can never leave, a payload over the server's limit above all. The consumer loop left the source message un-acked and went on, with no check of delivered, so that message redelivered for ever and never reached the DLQ, against the cap of ADR-0015. It now fails the run: same cap, same dead letter, the reason in last_error. Emits held before the refused one in that run still go out, so a redelivery can repeat them, within at-least-once. Detect units keep the old path: their subject comes from the program and their engine has already moved on, so a redelivery there is a different question. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018wXgxT2UoFSZfqhAE1Tb5V
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018wXgxT2UoFSZfqhAE1Tb5V
Nothing told anyone that a message had died: the self-healing loop waited for someone to open a session. to_dlq now publishes on <root>.dlq.events, an ordinary subject, so waking a person or an agent is a flow or a sink, as ADR-0024 wants of notifications. The event holds what the runtime wrote itself and never the payload or the error text: a dead letter is untrusted content and this is what wakes things up. A burst is one event per unit per VEJAS_DLQ_EVENT_SECS (default 60), with count for the letters it stands for, so that it wakes a reader once and loses nothing; a small thread sends what the window held back if no letter follows. A letter whose subject is the event subject is not announced, so a failing reader cannot feed itself. The documented route out is http-out. A child of an exec-* connector inherits the runtime's environment, VEJAS_APPROVAL_TOKEN included, so an agent must not be started that way. This is the runtime half of the trigger: the agent that answers it is still to be built. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018wXgxT2UoFSZfqhAE1Tb5V
The wait came from the sync client, which spawned threads on every connection and panicked in version_watch when the burst landed on its first one. The bus holds one connection from the start, so the test can be as strict as it was: ten bursts at boot gave no panic, and eight runs of this script pass. background() keeps its place as the net for any other panic, and its comment no longer names the client. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018wXgxT2UoFSZfqhAE1Tb5V
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The content of v0.5.0: the deprecated sync NATS client goes, behind one module, and dead letters announce themselves. The version bump and the release are not in this PR.
async-natsbehindcore/src/bus.rs(ADR-0035). One tokio worker hosts the client and the rest of the runtime keeps a blocking API, so flows, connectors and supervisors stay threads and no other module sees tokio or a NATS type.connectors/mqandconnectors/amqpcompile the same module.vx.dlq.events(ADR-0015 Amendment A). A dead letter is announced on an ordinary subject, so a flow or a sink can wake a person or an agent. Metadata only, never the payload or the error text; a burst is one event per unit perVEJAS_DLQ_EVENT_SECS, withcountfor the letters it stands for; a failing reader cannot feed itself.docs/SUBJECTS.md, part of the public contract, now lists the platform's events..cargo/audit.toml. One ignore remains, foramiquip.The cost: the binary grows from 9.9 to 11.2 MB, and the MQTT loopback falls by about 14 %. The flow hop holds its rate, within noise.
To upgrade
_INBOX.>on it. Without it nothing is acked and everything redelivers; the log names the permissions violation.Verification
Run locally on this branch: 55 unit tests, the 13 that need a NATS server, 22 golden cases, 50 scripts parse-checked,
scripts/audit.sh, andemit-subject,dlq-events,governed-gate,governed-agent,http-in-bound,api-write-gate,security-traversal,transport,detect,offset-resumeanddoc-examples.e2e/http-in-bound.shbursts during boot again: with the sync client that could panic the version watch, and ten runs here gave no panic.Not run locally on this branch:
e2e/admission,detect-demo, the image and mdBook, which CI covers, and the throughput benches, last measured before integration. The up-and-down upgrade, cold start and admission against a real RabbitMQ were checked onfeat/async-nats-bus, before it was integrated with master. Image size and multi-server failover have not been measured.Before the release
MANIFESTO.mdstill says "single-digit-megabyte"; RSS under load is about 10 MB.site/index.htmlchanges here and is deployed by hand.docs/adr/README.mddoes not list ADR-0035 yet.🤖 Generated with Claude Code
https://claude.ai/code/session_018wXgxT2UoFSZfqhAE1Tb5V