diff --git a/.cargo/audit.toml b/.cargo/audit.toml index 5132935..0d6806a 100644 --- a/.cargo/audit.toml +++ b/.cargo/audit.toml @@ -4,38 +4,6 @@ [advisories] ignore = [ - # rustls 0.22.4 and rustls-webpki 0.102.8 come with the sync `nats` 0.25 - # client (core, connectors/amqp, connectors/mq); connectors/amqp also uses - # rustls 0.22 directly for amqps://. nats 0.25 is the last release of the - # sync client, so this line gets no patched version; moving to async-nats - # removes all seven. Ignores go by advisory ID, so these would also hide - # 0346/0349 on rustls 0.23: keep core on >= 0.23.46. - # None is reachable the way the crates are used here: - - # Debug output of client session values leaks secrets only through a - # custom ClientSessionStore; neither nats nor vejas-amqp installs one. - "RUSTSEC-2026-0346", - # An unauthenticated ServerHello can drop later connections from hybrid - # post-quantum to classical key exchange. rustls 0.22 on ring never offers - # a post-quantum group, so there is nothing to downgrade. - "RUSTSEC-2026-0349", - # CRL distribution-point matching and CRL parsing: no CRL is ever loaded. - "RUSTSEC-2026-0049", - "RUSTSEC-2026-0104", - # URI and wildcard name constraints: reachable only after signature - # verification, with a certificate misissued by a CA the client already - # trusts, and only on a TLS connection (a nats:// URL to a server that does - # not require TLS stays plaintext). - "RUSTSEC-2026-0098", - "RUSTSEC-2026-0099", - # Certificate paths through the distrusted emSign intermediates: rustls-webpki - # 0.102 has no distrust list. Reachable only on a tls:// NATS_URL, from a - # network position that can impersonate the NATS server, with a chain the - # machine's OS trust store accepts (nats 0.25 loads the native roots). A NATS - # server on a plain or private-CA connection is outside it. This ID also - # covers rustls-webpki 0.103: keep core on >= 0.103.16, webpki-roots >= 1.0.10. - "RUSTSEC-2026-0382", - # idna 0.1.5 accepts Punycode labels that decode to plain ASCII. It comes # through url 1.7 <- amq-protocol 1.4 <- amiquip 0.4.2 (the last amiquip # release), in connectors/amqp only, where it parses the broker URL the diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 58a8671..84895f0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -96,6 +96,8 @@ jobs: run: e2e/emit-subject.sh - name: http-in survives a burst and a thread limit run: e2e/http-in-bound.sh + - name: Dead letters announce themselves, without feeding themselves + run: e2e/dlq-events.sh - name: Docs book builds (broken includes fail here, not on deploy) run: | curl -sL https://github.com/rust-lang/mdBook/releases/download/v0.5.4/mdbook-v0.5.4-x86_64-unknown-linux-gnu.tar.gz | tar xz diff --git a/CHANGELOG.md b/CHANGELOG.md index be117b3..3d32461 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,12 +7,53 @@ is `0`, minor versions may carry breaking changes — they are called out here. ## [Unreleased] +### Added +- **Dead letters announce themselves** on `vx.dlq.events`, an ordinary subject, + so that a flow or a sink can wake a person or an agent (ADR-0015 Amendment A). + The event carries metadata only, never the payload or the error text, and + `VEJAS_DLQ_EVENT_SECS` (default 60) groups a burst into one event per unit, + with `count` for the letters it stands for. `docs/examples/dlq_webhook.vjs.example` + posts it to a URL. + +### Security +- **Seven advisories leave `.cargo/audit.toml`**: RUSTSEC-2026-0049, -0098, -0099, + -0104, -0346, -0349 and -0382, all held by the sync `nats` client's rustls 0.22 + and rustls-webpki 0.102. Every lockfile is now on rustls 0.23.46 and + rustls-webpki 0.103.16, `connectors/amqp` included. + ### Fixed - **A burst of connections during boot could end the version watch for good.** Its first connection panicked when the OS had no thread to give: nothing showed it, and a promoted version no longer reached that instance until a restart. It needs a thread limit and a burst at that moment. The watch and the control loop now carry a name and start again after a panic. +- **A consumer deleted under a running flow no longer leaves a zombie.** The flow + stayed `running` for good, consuming nothing, and two empty dead letters from an + `_INBOX` subject landed in the queue. It now restarts and recreates the consumer. +- **An emit the server cannot take no longer redelivers forever.** A publish + refused before it leaves, for a payload over the server's limit, left its source + message un-acked with no cap on deliveries. It is now a failed run, dead-lettered + after `MAX_DELIVERIES` like any other. Detect units keep the old behaviour here: + their emits are small and their engine state makes a redelivery a different case. + +### Changed +- **The NATS client is now `async-nats`, behind one module (ADR-0035).** `nats` 0.25 + is deprecated. `core/src/bus.rs` runs `async-nats` on one tokio worker and gives + the rest of the runtime a blocking API: no other module sees tokio or a NATS type, + and flows, connectors and supervisors stay threads. A process now holds one + connection, where it held two per flow. At 200 flows that is 210 threads against + 1 011, 1 connection against 401, and 26 MB against 140. A flow is running 11 ms + after spawn, against 1.7 s. The flow hop holds its rate, ten flows run about 10 % + faster, and the 5 ms flusher floor is gone, with `publish_confirmed`, + `fetch_iter_flushed` and the second connection for acks. The costs: the binary + grows from 9.9 to 11.2 MB, RSS at one flow from 9.2 to 10.2 MB, and the MQTT + loopback falls from about 2 400 to 2 070 round-trips/s. Existing streams, + consumers, buckets and snapshots work as they are, and v0.3.6 reads back what this + version writes. **To upgrade:** a NATS user with a publish allow-list needs + `_INBOX.>` on it. Without it nothing is ever acked and everything redelivers; the + log names the permissions violation. +- **`connectors/mq` and `connectors/amqp` compile the same module**, and + `connectors/amqp` moves to rustls 0.23. ## [0.4.0] — 2026-11-06 diff --git a/README.md b/README.md index 9050dad..f5fff31 100644 --- a/README.md +++ b/README.md @@ -33,10 +33,10 @@ end-to-end test ([the script](e2e/bridge-film.mjs)) — the video cannot lie. ![The expert corrects the meaning; the pipes never move](docs/demo/bridge-teaser.gif) **Measured, not claimed** — reproducible on an 8-core dev machine -([methodology and comparisons](bench/)): cold start **11–13 ms**, **6–8 MB** -RSS under load (49 MB with fifty live flows), **6.2 MB** binary / **201 MB** +([methodology and comparisons](bench/)): cold start **11–13 ms**, **10 MB** +RSS under load (38 MB with fifty live flows), **11.2 MB** binary / **201 MB** image, end-to-end p50 **2 ms** uncongested / p50 14 ms at a paced 1 900/s -sustained with **every event hop persisted** in JetStream, **2 285/s** through a +sustained with **every event hop persisted** in JetStream, **2 070/s** through a real MQTT broker and back (QoS 1 both ways, PUBACK-gated). Same order of magnitude as engines that persist nothing, in ~25× less memory. diff --git a/bench/README.md b/bench/README.md index 02e03a9..b06776e 100644 --- a/bench/README.md +++ b/bench/README.md @@ -31,12 +31,12 @@ All five ceilings fell (#4+#3 `5cffb7b`, #2 `b34d9f3`, #1 `75fad83`, | Metric | v0 | all five fixes in | |---|---|---| | Cold start | 13–15 ms | **11–13 ms** | -| Runtime RSS under load | 6–8 MB | **6–8 MB** | -| Binary size | 3.9 MB | 6.2 MB (rustls) | +| Runtime RSS under load | 6–8 MB | **10 MB** | +| Binary size | 3.9 MB | 11.2 MB (rustls, async-nats) | | e2e sustained, paced ~1 900/s (`bench/paced.sh 2000 15`) | — | **p50 14 ms, p99 36 ms**, 20 000/20 000 | | e2e saturated (32 conns) | 65/s delivered | ~4 900/s ingest, **~2 650/s delivered** (sink-bound, queue absorbs the rest) | | e2e latency, uncongested (`bench/paced.sh 20 15`) | p50 859 ms | **p50 2 ms, p99 3 ms** | -| MQTT broker loopback, QoS 1 both ways (`bench/broker-mqtt.sh 5000`) | — | **2 285 rt/s**, 5 000/5 000 | +| MQTT broker loopback, QoS 1 both ways (`bench/broker-mqtt.sh 5000`) | — | **2 070 rt/s**, 5 000/5 000 | | Isolated flow hop | 171/s | **8 110/s** | The old "sustained (12 conns) 1 701/s, p50 18 ms" row died with ceiling #5: @@ -47,6 +47,31 @@ any concurrency, so steady-state latency is now measured **paced** Every hop persisted in JetStream throughout — the guarantee never moved. +### Since ADR-0035 + +The sync `nats` client left for `async-nats` behind `core/src/bus.rs`, so findings +#5 and #6 below describe a client that is gone, along with `publish_confirmed`, +`fetch_iter_flushed` and the second connection of #7. Against the v0.3.6 binary, +alternating in one session on this box: + +| | v0.3.6 | now | +|---|---:|---:| +| Threads at 1 / 10 / 50 / 200 idle flows | 16 / 61 / 261 / 1 011 | 11 / 20 / 60 / 210 | +| NATS connections, same | 3 / 21 / 101 / 401 | 1 | +| RSS, same, MB | 8.9 / 16.7 / 46.9 / 139.8 | 9.6 / 10.5 / 14.2 / 26.4 | +| Flow hop (`PUBS=16`, 64 000 events) | 13 100 – 13 700/s | 13 200 – 13 500/s | +| Flow hop, `bench/pub.py` as publisher | 20 100 – 20 500/s | 19 400 – 21 400/s | +| One detect unit | 30 800 – 35 400/s, 11 MB | 34 700 – 41 100/s, 12.7 MB | +| MQTT loopback | 2 370 – 2 430 rt/s | 2 050 – 2 090 rt/s | +| Spawn to a flow running | 1 714 ms | 10.9 ms | +| Paced 2 000/s, p50 / p99 | 3–4 / 14–17 ms | 3 / 14–15 ms | +| Binary | 9.9 MB | 11.2 MB | + +The paced row is why the p50 14 ms above stands as published: both binaries are far +below it today, so this change did not move it. The detect tables further down +predate this: add about 1.7 MB to their RSS. The method and the MQTT cost are in +ADR-0035. + The footprint numbers are the thesis, measured. The throughput and latency numbers are **known ceilings of the v0 I/O paths, not of the interpreter** — each one matches its cause exactly: @@ -134,13 +159,13 @@ they publish). Scaling the number of flows (`bench/multi-flow.sh`): | Flows | Aggregate rate | Runtime RSS | |---|---|---| -| 1 | 7–8 k/s | 6 MB | -| 10 | **13 746/s** | 16.6 MB | -| 50 | 8 410/s | 49.3 MB | +| 1 | 7–8 k/s | 10 MB | +| 10 | **13 700 – 14 700/s** | 14.5 MB | +| 50 | 11 100 – 12 100/s | 38 MB | Throughput *rises* with flow count (consumers parallelize; the bus, not the -interpreter, is the bound) and memory stays ~1 MB per running flow — fifty -live, persisted flows in under 50 MB. +interpreter, is the bound) and memory stays around half a megabyte per running +flow under load — fifty live, persisted flows in under 40 MB. ## Detect units (`bench/detect.sh`) @@ -252,7 +277,7 @@ one unit handles that at around 25 000 events/s in 18 MB. - **One instance saturates near 44 000 events/s** whatever the unit count. Beyond it, add instances: they share the durable consumers and the work (see *Clustering*). -- **Artefacts:** the binary is 8.2 MiB, the image 37.6 MiB, and a unit that is +- **Artefacts:** the binary is 10.7 MiB, the image 37.6 MiB, and a unit that is idle costs nothing measurable. Measured on the dev box in this file's header — 8 cores, WSL2 — two runs of diff --git a/connectors/amqp/Cargo.lock b/connectors/amqp/Cargo.lock index e3dd235..624e4c2 100644 --- a/connectors/amqp/Cargo.lock +++ b/connectors/amqp/Cargo.lock @@ -25,7 +25,7 @@ dependencies = [ "indexmap", "input_buffer", "log", - "mio", + "mio 0.6.23", "mio-extras", "percent-encoding 2.3.2", "snafu", @@ -70,29 +70,38 @@ dependencies = [ ] [[package]] -name = "async-channel" -version = "2.5.0" +name = "async-nats" +version = "0.52.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "924ed96dd52d1b75e9c1a3e6275715fd320f5f9439fb5a4a11fa51f4221158d2" +checksum = "015f0a7a3b0b29701b78c76a4d4a122624c97c71fa30363381664eb729d619c7" dependencies = [ - "concurrent-queue", - "event-listener-strategy", - "futures-core", - "pin-project-lite", + "base64", + "bytes", + "futures-util", + "memchr", + "pin-project", + "portable-atomic", + "rand", + "regex", + "ring", + "rustls-native-certs", + "rustls-pki-types", + "rustls-webpki", + "serde", + "serde_json", + "serde_nanos", + "serde_repr", + "thiserror", + "time", + "tokio", + "tokio-rustls", + "tokio-stream", + "tokio-util", + "tracing", + "tryhard", + "url 2.5.8", ] -[[package]] -name = "async-task" -version = "4.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b75356056920673b02621b35afd0f7dda9306d03c79a30f5c56c44cf256e3de" - -[[package]] -name = "atomic-waker" -version = "1.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" - [[package]] name = "autocfg" version = "1.5.1" @@ -101,24 +110,9 @@ checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" [[package]] name = "base64" -version = "0.13.1" +version = "0.22.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8" - -[[package]] -name = "base64-url" -version = "1.4.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67a99c239d0c7e77c85dddfa9cebce48704b3c49550fcd3b84dd637e4484899f" -dependencies = [ - "base64", -] - -[[package]] -name = "base64ct" -version = "1.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" [[package]] name = "bitflags" @@ -132,28 +126,6 @@ version = "2.13.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" -[[package]] -name = "block-buffer" -version = "0.10.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" -dependencies = [ - "generic-array", -] - -[[package]] -name = "blocking" -version = "1.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e83f8d02be6967315521be875afa792a316e28d57b5a2d401897e2a7921b7f21" -dependencies = [ - "async-channel", - "async-task", - "futures-io", - "futures-lite", - "piper", -] - [[package]] name = "built" version = "0.5.2" @@ -168,6 +140,9 @@ name = "bytes" version = "1.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" +dependencies = [ + "serde", +] [[package]] name = "cargo-lock" @@ -204,20 +179,16 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" [[package]] -name = "concurrent-queue" -version = "2.5.0" +name = "chacha20" +version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ca0197aee26d1ae37445ee532fefce43251d24cc7c166799f4d46817f1d3973" +checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06" dependencies = [ - "crossbeam-utils", + "cfg-if 1.0.4", + "cpufeatures", + "rand_core", ] -[[package]] -name = "const-oid" -version = "0.9.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" - [[package]] name = "cookie-factory" version = "0.2.4" @@ -226,9 +197,9 @@ checksum = "98a479f8099cc5ac64915a3dd76c87be27f929ba406ad705aacb13f19b791207" [[package]] name = "core-foundation" -version = "0.9.4" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f" +checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6" dependencies = [ "core-foundation-sys", "libc", @@ -242,9 +213,9 @@ checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" [[package]] name = "cpufeatures" -version = "0.2.17" +version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +checksum = "5ca28b0ae3115b884660db4118d803791fd6756b6e88f39c0f3f7859060d7566" dependencies = [ "libc", ] @@ -264,59 +235,6 @@ version = "0.8.22" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "61803da095bee82a81bb1a452ecc25d3b2f1416d1897eb86430c6159ef717c17" -[[package]] -name = "crypto-common" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" -dependencies = [ - "generic-array", - "typenum", -] - -[[package]] -name = "curve25519-dalek" -version = "4.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be" -dependencies = [ - "cfg-if 1.0.4", - "cpufeatures", - "curve25519-dalek-derive", - "digest", - "fiat-crypto", - "rustc_version", - "subtle", -] - -[[package]] -name = "curve25519-dalek-derive" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f46882e17999c6cc590af592290432be3bce0428cb0d5f8b6715e4dc7b383eb3" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "data-encoding" -version = "2.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4583a4551df46e2792f82ceeac45e850d2e2d5debba0b91f102385cda5b11f06" - -[[package]] -name = "der" -version = "0.7.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" -dependencies = [ - "const-oid", - "pem-rfc7468", - "zeroize", -] - [[package]] name = "deranged" version = "0.5.8" @@ -326,16 +244,6 @@ dependencies = [ "serde_core", ] -[[package]] -name = "digest" -version = "0.10.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" -dependencies = [ - "block-buffer", - "crypto-common", -] - [[package]] name = "displaydoc" version = "0.2.7" @@ -353,69 +261,6 @@ version = "0.3.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "780955b8b195a21ab8e4ac6b60dd1dbdcec1dc6c51c0617964b08c81785e12c9" -[[package]] -name = "ed25519" -version = "2.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "115531babc129696a58c64a4fef0a8bf9e9698629fb97e9e40767d235cfbcd53" -dependencies = [ - "signature", -] - -[[package]] -name = "ed25519-dalek" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70e796c081cee67dc755e1a36a0a172b897fab85fc3f6bc48307991f64e4eca9" -dependencies = [ - "curve25519-dalek", - "ed25519", - "sha2", - "signature", - "subtle", -] - -[[package]] -name = "event-listener" -version = "5.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a23add41df1562121a9393cb065eab5146a1242410f23a644851e90cfd669d2" -dependencies = [ - "parking", - "pin-project-lite", -] - -[[package]] -name = "event-listener-strategy" -version = "0.5.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8be9f3dfaaffdae2972880079a491a1a8bb7cbed0b8dd7a347f668b4150a3b93" -dependencies = [ - "event-listener", - "pin-project-lite", -] - -[[package]] -name = "fastrand" -version = "1.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e51093e27b0797c359783294ca4f0a911c270184cb10f85783b118614a1501be" -dependencies = [ - "instant", -] - -[[package]] -name = "fastrand" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" - -[[package]] -name = "fiat-crypto" -version = "0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d" - [[package]] name = "find-msvc-tools" version = "0.1.11" @@ -454,40 +299,51 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" [[package]] -name = "futures-io" +name = "futures-sink" version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53c0fa8157de1303bfffdaa1cc2a673bfffb60102f76b0ef4441659124373fed" +checksum = "1944426bf7d03f1d14f708785e4b33efd750b36d48a157b836b3efc15ede8e1d" [[package]] -name = "futures-lite" -version = "2.6.1" +name = "futures-task" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd" + +[[package]] +name = "futures-util" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f78e10609fe0e0b3f4157ffab1876319b5b0db102a2c60dc4626306dc46b44ad" +checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" dependencies = [ "futures-core", + "futures-sink", + "futures-task", "pin-project-lite", + "slab", ] [[package]] -name = "generic-array" -version = "0.14.7" +name = "getrandom" +version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" dependencies = [ - "typenum", - "version_check 0.9.5", + "cfg-if 1.0.4", + "libc", + "wasi", ] [[package]] name = "getrandom" -version = "0.2.17" +version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" dependencies = [ "cfg-if 1.0.4", "libc", - "wasi", + "r-efi", + "rand_core", ] [[package]] @@ -653,15 +509,6 @@ dependencies = [ "bytes", ] -[[package]] -name = "instant" -version = "0.1.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0242819d153cba4b4b05a5a8f2a7e9bbf97b6055b2a002b395c96b5ff3c0222" -dependencies = [ - "cfg-if 1.0.4", -] - [[package]] name = "iovec" version = "0.1.4" @@ -677,12 +524,6 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" -[[package]] -name = "json" -version = "0.12.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "078e285eafdfb6c4b434e0d31e8cfcb5115b651496faca5749b88fafd4f23bfd" - [[package]] name = "kernel32-sys" version = "0.2.2" @@ -717,15 +558,6 @@ version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "47d9d19d1d6efa0109d2f65ff4c85cddd50bd572e5a00127ab10987290bcefae" -[[package]] -name = "lock_api" -version = "0.4.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" -dependencies = [ - "scopeguard", -] - [[package]] name = "log" version = "0.4.34" @@ -763,6 +595,17 @@ dependencies = [ "winapi 0.2.8", ] +[[package]] +name = "mio" +version = "1.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1788edb87fdc09c7e26304471e2f5be8cdefb1b6930d6e3985fc02ff53bf86ee" +dependencies = [ + "libc", + "wasi", + "windows-sys 0.61.2", +] + [[package]] name = "mio-extras" version = "2.0.6" @@ -771,7 +614,7 @@ checksum = "52403fe290012ce777c4626790c8951324a2b9e3316b3143779c72b029742f19" dependencies = [ "lazycell", "log", - "mio", + "mio 0.6.23", "slab", ] @@ -787,43 +630,6 @@ dependencies = [ "ws2_32-sys", ] -[[package]] -name = "nats" -version = "0.25.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19090dd06e27eb59adfd16f332b85818a3e7dbbdbc13c448676c9bb69f964208" -dependencies = [ - "base64", - "base64-url", - "blocking", - "crossbeam-channel", - "fastrand 1.9.0", - "itoa", - "json", - "lazy_static", - "libc", - "log", - "memchr", - "nkeys", - "nuid", - "once_cell", - "parking_lot", - "portable-atomic", - "regex", - "ring", - "rustls", - "rustls-native-certs", - "rustls-pemfile", - "rustls-webpki", - "serde", - "serde_json", - "serde_nanos", - "serde_repr", - "time", - "url 2.5.8", - "winapi 0.3.9", -] - [[package]] name = "net2" version = "0.2.39" @@ -835,21 +641,6 @@ dependencies = [ "winapi 0.3.9", ] -[[package]] -name = "nkeys" -version = "0.4.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "879011babc47a1c7fdf5a935ae3cfe94f34645ca0cac1c7f6424b36fc743d1bf" -dependencies = [ - "data-encoding", - "ed25519", - "ed25519-dalek", - "getrandom", - "log", - "rand", - "signatory", -] - [[package]] name = "nom" version = "4.2.3" @@ -857,17 +648,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2ad2a91a8e869eeb30b9cb3119ae87773a8f4ae617f41b1eb9c154b2905f7bd6" dependencies = [ "memchr", - "version_check 0.1.5", -] - -[[package]] -name = "nuid" -version = "0.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20c1bb65186718d348306bf1afdeb20d9ab45b2ab80fb793c0fdcf59ffbb4f38" -dependencies = [ - "lazy_static", - "rand", + "version_check", ] [[package]] @@ -884,47 +665,9 @@ checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" [[package]] name = "openssl-probe" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d05e27ee213611ffe7d6348b942e8f942b37114c00cc03cec254295a4a17852e" - -[[package]] -name = "parking" -version = "2.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f38d5652c16fde515bb1ecef450ab0f6a219d619a7274976324d5e377f7dceba" - -[[package]] -name = "parking_lot" -version = "0.12.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" -dependencies = [ - "lock_api", - "parking_lot_core", -] - -[[package]] -name = "parking_lot_core" -version = "0.9.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" -dependencies = [ - "cfg-if 1.0.4", - "libc", - "redox_syscall", - "smallvec", - "windows-link", -] - -[[package]] -name = "pem-rfc7468" -version = "0.7.0" +version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412" -dependencies = [ - "base64ct", -] +checksum = "7e036b411754132fd6b74e94b74e9a13c5cf606fa93d1885c35860d3060820b1" [[package]] name = "percent-encoding" @@ -981,31 +724,30 @@ dependencies = [ ] [[package]] -name = "pin-project-lite" -version = "0.2.17" +name = "pin-project" +version = "1.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" +checksum = "2466b2336ed02bcdca6b294417127b90ec92038d1d5c4fbeac971a922e0e0924" +dependencies = [ + "pin-project-internal", +] [[package]] -name = "piper" -version = "0.2.5" +name = "pin-project-internal" +version = "1.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c835479a4443ded371d6c535cbfd8d31ad92c5d23ae9770a61bc155e4992a3c1" +checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b" dependencies = [ - "atomic-waker", - "fastrand 2.5.0", - "futures-io", + "proc-macro2", + "quote", + "syn 2.0.119", ] [[package]] -name = "pkcs8" -version = "0.10.2" +name = "pin-project-lite" +version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" -dependencies = [ - "der", - "spki", -] +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" [[package]] name = "portable-atomic" @@ -1028,15 +770,6 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" -[[package]] -name = "ppv-lite86" -version = "0.2.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" -dependencies = [ - "zerocopy", -] - [[package]] name = "proc-macro2" version = "1.0.107" @@ -1062,43 +795,27 @@ dependencies = [ ] [[package]] -name = "rand" -version = "0.8.7" +name = "r-efi" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22f6172bdec972074665ed81ed53b71da00bfc44b65a753cfde883ec4c702a1a" -dependencies = [ - "libc", - "rand_chacha", - "rand_core", -] +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" [[package]] -name = "rand_chacha" -version = "0.3.1" +name = "rand" +version = "0.10.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" +checksum = "65c9fb96cbc91e3478eaae79a69fcd3f1ae4ad052e471fe6732fff548984b4af" dependencies = [ - "ppv-lite86", + "chacha20", + "getrandom 0.4.3", "rand_core", ] [[package]] name = "rand_core" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" -dependencies = [ - "getrandom", -] - -[[package]] -name = "redox_syscall" -version = "0.5.18" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" -dependencies = [ - "bitflags 2.13.1", -] +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" [[package]] name = "regex" @@ -1137,28 +854,20 @@ checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" dependencies = [ "cc", "cfg-if 1.0.4", - "getrandom", + "getrandom 0.2.17", "libc", "untrusted", "windows-sys 0.52.0", ] -[[package]] -name = "rustc_version" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" -dependencies = [ - "semver", -] - [[package]] name = "rustls" -version = "0.22.4" +version = "0.23.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf4ef73721ac7bcd79b2b315da7779d8fc09718c6b3d2d1b2d94850eb8c18432" +checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf" dependencies = [ "log", + "once_cell", "ring", "rustls-pki-types", "rustls-webpki", @@ -1168,12 +877,11 @@ dependencies = [ [[package]] name = "rustls-native-certs" -version = "0.7.3" +version = "0.8.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5bfb394eeed242e909609f56089eecfe5fda225042e8b171791b9c95f5931e5" +checksum = "4a073f5dc7469f984c52ad2752b63b0807745133b6de880b7b64c1ac4c48aec4" dependencies = [ "openssl-probe", - "rustls-pemfile", "rustls-pki-types", "schannel", "security-framework", @@ -1199,9 +907,9 @@ dependencies = [ [[package]] name = "rustls-webpki" -version = "0.102.8" +version = "0.103.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "64ca1bc8749bd4cf37b5ce386cc146580777b4e8572c7b97baf22c83f444bee9" +checksum = "4443b1b6d39d7c663139942cc4558262562d2c84123bb6282ef177009ca8b281" dependencies = [ "ring", "rustls-pki-types", @@ -1226,17 +934,11 @@ dependencies = [ "windows-sys 0.61.2", ] -[[package]] -name = "scopeguard" -version = "1.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" - [[package]] name = "security-framework" -version = "2.11.1" +version = "3.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "897b2245f0b511c87893af39b033e5ca9cce68824c4d7e7630b5a1d339658d02" +checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d" dependencies = [ "bitflags 2.13.1", "core-foundation", @@ -1328,45 +1030,12 @@ dependencies = [ "syn 3.0.3", ] -[[package]] -name = "sha2" -version = "0.10.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" -dependencies = [ - "cfg-if 1.0.4", - "cpufeatures", - "digest", -] - [[package]] name = "shlex" version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" -[[package]] -name = "signatory" -version = "0.27.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1e303f8205714074f6068773f0e29527e0453937fe837c9717d066635b65f31" -dependencies = [ - "pkcs8", - "rand_core", - "signature", - "zeroize", -] - -[[package]] -name = "signature" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" -dependencies = [ - "digest", - "rand_core", -] - [[package]] name = "slab" version = "0.4.12" @@ -1402,13 +1071,13 @@ dependencies = [ ] [[package]] -name = "spki" -version = "0.7.3" +name = "socket2" +version = "0.6.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" +checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" dependencies = [ - "base64ct", - "der", + "libc", + "windows-sys 0.61.2", ] [[package]] @@ -1467,6 +1136,26 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "thiserror" +version = "2.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09e52cb86a36cede5cb101bf8908837b3e4c6e5e59fe7fd85c23fb56200d189e" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fe5197923287db20a58125f0bc85c062f7f2c892de97b18c356f9efb14b28524" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + [[package]] name = "time" version = "0.3.55" @@ -1522,6 +1211,66 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" +[[package]] +name = "tokio" +version = "1.54.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b52fd3255258d26fac7264ab5a76b53c13367136dc011d3d6ad4f8ed070396a3" +dependencies = [ + "bytes", + "libc", + "mio 1.2.4", + "pin-project-lite", + "socket2", + "tokio-macros", + "windows-sys 0.61.2", +] + +[[package]] +name = "tokio-macros" +version = "2.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "tokio-rustls" +version = "0.26.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c9cc2678c2cdd569ef8215e2afd7954ada2ae20b4fdd2c5fe6139a3b02d105db" +dependencies = [ + "rustls", + "tokio", +] + +[[package]] +name = "tokio-stream" +version = "0.1.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3d06f0b082ba57c26b79407372e57cf2a1e28124f78e9479fe80322cf53420b" +dependencies = [ + "futures-core", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "tokio-util" +version = "0.7.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e464cf451ba96ebfc6f9b6542f17ee8b8956e33f1e40d9690624e59d7a7f8a4b" +dependencies = [ + "bytes", + "futures-core", + "futures-sink", + "pin-project-lite", + "tokio", +] + [[package]] name = "toml" version = "0.5.11" @@ -1532,10 +1281,45 @@ dependencies = [ ] [[package]] -name = "typenum" -version = "1.20.1" +name = "tracing" +version = "0.1.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" +dependencies = [ + "pin-project-lite", + "tracing-attributes", + "tracing-core", +] + +[[package]] +name = "tracing-attributes" +version = "0.1.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "tracing-core" +version = "0.1.36" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" +dependencies = [ + "once_cell", +] + +[[package]] +name = "tryhard" +version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" +checksum = "9fe58ebd5edd976e0fe0f8a14d2a04b7c81ef153ea9a54eebc42e67c2c23b4e5" +dependencies = [ + "pin-project-lite", + "tokio", +] [[package]] name = "ucd-trie" @@ -1604,12 +1388,15 @@ name = "vejas-amqp" version = "0.1.0" dependencies = [ "amiquip", + "async-nats", + "bytes", "crossbeam-channel", - "mio", - "nats", + "futures-util", + "mio 0.6.23", "rustls", "rustls-pemfile", "serde_json", + "tokio", "webpki-roots 0.26.11", ] @@ -1619,12 +1406,6 @@ version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "914b1a6776c4c929a602fafd8bc742e06365d4bcbe48c30f9cca5824f70dc9dd" -[[package]] -name = "version_check" -version = "0.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" - [[package]] name = "walkdir" version = "2.5.0" @@ -1829,26 +1610,6 @@ dependencies = [ "synstructure", ] -[[package]] -name = "zerocopy" -version = "0.8.56" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "556764e583adb45a9f8d413c2a147fa7e8d821e48e12b14fd560b607998b75eb" -dependencies = [ - "zerocopy-derive", -] - -[[package]] -name = "zerocopy-derive" -version = "0.8.56" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2ab42fc20575779bd240faa45f94a74256f755c0fa9e89f0ede20d91d0cdfc1" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - [[package]] name = "zerofrom" version = "0.1.8" diff --git a/connectors/amqp/Cargo.toml b/connectors/amqp/Cargo.toml index 4c85196..31569fc 100644 --- a/connectors/amqp/Cargo.toml +++ b/connectors/amqp/Cargo.toml @@ -8,12 +8,17 @@ description = "Vejas AMQP 0-9-1 (RabbitMQ) connector spike: sync pure-Rust via a [dependencies] amiquip = { version = "0.4", default-features = false } crossbeam-channel = "0.5" -nats = "0.25" +# The bus: this crate compiles the runtime's core/src/bus.rs (ADR-0035). +async-nats = { version = "0.52", default-features = false, features = ["jetstream", "kv", "object-store", "ring", "server_2_10"] } +tokio = { version = "1", default-features = false, features = ["rt-multi-thread", "time"] } +bytes = "1" +futures-util = { version = "0.3", default-features = false, features = ["std"] } serde_json = "1" -# Pure-Rust TLS for amqps:// — rustls (already in the tree via nats, ring provider; -# no openssl-sys) wrapped over amiquip's mio-0.6 stream via a small Evented adapter -# (ADR-0026 Q1). This is the whole point of NOT using amiquip's native-tls default. -rustls = "0.22" +# Pure-Rust TLS for amqps:// — rustls (already in the tree via async-nats, ring +# provider; no openssl-sys) wrapped over amiquip's mio-0.6 stream via a small Evented +# adapter (ADR-0026 Q1). This is the whole point of NOT using amiquip's native-tls +# default. +rustls = { version = "0.23", default-features = false, features = ["ring", "std", "tls12", "logging"] } rustls-pemfile = "2" webpki-roots = "0.26" mio = "0.6" diff --git a/connectors/amqp/src/lease.rs b/connectors/amqp/src/lease.rs index 95a425a..691256e 100644 --- a/connectors/amqp/src/lease.rs +++ b/connectors/amqp/src/lease.rs @@ -8,10 +8,10 @@ //! //! Same JetStream KV mechanism as the core runtime (create = acquire, update-CAS = //! renew/fence, delete = release, bucket max_age = crash failover) — reimplemented -//! here because the connector is a standalone binary with its own NATS client, not -//! linked to the core. The bucket is shared (VEJAS_LEASES), keys namespaced, so it +//! here because the connector is a standalone binary, not linked to the core. The bucket is shared (VEJAS_LEASES), keys namespaced, so it //! coordinates with the fleet when they run on the same bus. +use crate::bus; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::Arc; use std::thread; @@ -56,23 +56,19 @@ pub fn source_key(queue: &str) -> String { /// Open (or create) the shared lease bucket. None if JetStream KV is unavailable — /// the caller then runs unleased (single-instance behaviour, no worse than today). -pub fn open_store(js: &nats::jetstream::JetStream) -> Option { - let cfg = nats::kv::Config { +pub fn open_store(js: &bus::JetStream) -> Option { + js.key_value(&bus::KvConfig { bucket: LEASE_BUCKET.to_string(), max_age: lease_ttl(), history: 1, - ..Default::default() - }; - js.create_key_value(&cfg) - .or_else(|_| js.key_value(LEASE_BUCKET)) - .ok() + }) } /// Held while this instance owns the lease. Dropping it stops renewal and releases /// the lease (instant handoff on shutdown). `lost()` goes true if the renewal /// thread finds the lease fenced or aged out — the caller stops getting. pub struct Lease { - store: nats::kv::Store, + store: bus::Kv, key: String, stop_renew: Arc, lost: Arc, @@ -100,7 +96,7 @@ impl Drop for Lease { /// thread. Returns None if `alive()` goes false before acquisition (asked to stop /// while standing by). pub fn acquire_blocking( - store: &nats::kv::Store, + store: &bus::Kv, key: &str, alive: impl Fn() -> bool, ) -> Option { @@ -167,8 +163,7 @@ mod tests { #[ignore] fn lease_is_mutually_exclusive() { let url = std::env::var("NATS_URL").unwrap_or_else(|_| "127.0.0.1:4222".into()); - let nc = nats::connect(&url).expect("connect"); - let js = nats::jetstream::new(nc); + let js = bus::connect(&url).expect("connect").jetstream(); let store = open_store(&js).expect("lease store"); let key = format!("amqp_test_{}", std::process::id()); let _ = store.delete(&key); @@ -192,8 +187,7 @@ mod tests { #[ignore] fn stale_revision_is_fenced() { let url = std::env::var("NATS_URL").unwrap_or_else(|_| "127.0.0.1:4222".into()); - let nc = nats::connect(&url).expect("connect"); - let js = nats::jetstream::new(nc); + let js = bus::connect(&url).expect("connect").jetstream(); let store = open_store(&js).expect("lease store"); let key = format!("amqp_fence_{}", std::process::id()); let _ = store.delete(&key); diff --git a/connectors/amqp/src/main.rs b/connectors/amqp/src/main.rs index e11f17e..cc6d5c2 100644 --- a/connectors/amqp/src/main.rs +++ b/connectors/amqp/src/main.rs @@ -15,6 +15,11 @@ //! path. The transactional ordering is exercised against a fake below with fault //! injection; real-RabbitMQ certification is the declared CI exception (ADR-0017). +// The runtime's own bus module, compiled in as it is: one sync API over `async-nats` +// on one small tokio runtime, and no NATS type anywhere else (ADR-0035). +#[allow(dead_code)] +#[path = "../../../core/src/bus.rs"] +mod bus; mod lease; mod tls; @@ -77,48 +82,9 @@ pub trait AmqpSink { } // ───────────────────────── loops ───────────────────────── -/// Publish to a JetStream subject and wait for the pub-ack, flushing DIRECTLY past -/// nats 0.25's 5ms flusher floor (client.rs MIN_FLUSH_BETWEEN) — that floor caps -/// every sequential pub-ack at ~200/s, and the AMQP source is a sequential -/// consume→publish→ack loop (the spike measured 186/s, dead on the floor). Faithful -/// re-impl of the crate's request path incl. the no-responders guard + a direct -/// flush; Ok still means a real pub-ack (persisted), so the ack-after-bus-publish -/// ordering is unchanged. (Named bus_* to avoid the AmqpSink::publish_confirmed -/// method, which is the RabbitMQ *broker* confirm.) -fn bus_publish_confirmed( - nc: &nats::Connection, - subject: &str, - payload: &[u8], - wait: Duration, -) -> Result<(), String> { - let reply = nc.new_inbox(); - let sub = nc.subscribe(&reply).map_err(|e| e.to_string())?; - nc.publish_request(subject, &reply, payload) - .map_err(|e| e.to_string())?; - nc.flush().map_err(|e| e.to_string())?; // direct flush — bypass the 5ms floor - let msg = sub - .next_timeout(wait) - .map_err(|_| "timed out waiting for JetStream pub-ack".to_string())?; - if msg.is_no_responders() { - return Err("no JetStream responder for the subject (stream missing?)".into()); - } - let ack: serde_json::Value = - serde_json::from_slice(&msg.data).map_err(|e| format!("bad pub-ack: {e}"))?; - if let Some(err) = ack.get("error") { - return Err(format!("JetStream rejected the publish: {err}")); - } - if ack.get("stream").is_none() { - return Err(format!( - "unexpected pub-ack (no stream): {}", - String::from_utf8_lossy(&msg.data) - )); - } - Ok(()) -} - fn run_source( src: &mut dyn AmqpSource, - nc: &nats::Connection, + js: &bus::JetStream, subject: &str, wait: Duration, alive: impl Fn() -> bool, @@ -126,7 +92,7 @@ fn run_source( eprintln!("[vejas-amqp] source: AMQP → {subject} (consume→publish→ack)"); while alive() { let Some(msg) = src.recv(wait)? else { continue }; - match bus_publish_confirmed(nc, subject, &msg.body, Duration::from_secs(5)) { + match js.publish(subject, &msg.body) { Ok(_) => src.ack(msg.tag)?, // bus durable → ack the broker Err(e) => { eprintln!("[vejas-amqp] publish failed ({e}); nack+requeue"); @@ -139,13 +105,13 @@ fn run_source( fn run_sink( sink: &mut dyn AmqpSink, - sub: &nats::jetstream::PullSubscription, + sub: &bus::Puller, routing_key: &str, alive: impl Fn() -> bool, ) -> Result<(), String> { eprintln!("[vejas-amqp] sink: bus → AMQP (publish→confirm→ack)"); while alive() { - for m in fetch_round(sub)? { + for m in sub.round()? { if !alive() { return Ok(()); } @@ -161,16 +127,6 @@ fn run_sink( Ok(()) } -fn fetch_round(sub: &nats::jetstream::PullSubscription) -> Result, String> { - let iter = sub - .timeout_fetch( - nats::jetstream::BatchOptions { batch: 64, expires: Some(500_000_000), no_wait: true }, - Duration::from_millis(750), - ) - .map_err(|e| e.to_string())?; - Ok(iter.map_while(|m| m.ok()).collect()) -} - // ───────────────────────── amiquip-backed impls ───────────────────────── /// Real source: an amiquip Consumer plus a tag→Delivery map (amiquip acks a /// Delivery by value, so the loop-facing tag API holds the deliveries until acked). @@ -249,14 +205,13 @@ fn main() { let wait = Duration::from_secs(env("VEJAS_AMQP_WAIT_SECS").and_then(|s| s.parse().ok()).unwrap_or(3)); let competing = matches!(env("VEJAS_AMQP_COMPETING").as_deref(), Some("1") | Some("true")); - let nc = match nats::connect(&nats_url) { - Ok(nc) => nc, + let js = match bus::connect(&nats_url) { + Ok(bus) => bus.jetstream(), Err(e) => { eprintln!("[vejas-amqp] cannot connect to NATS at {nats_url}: {e}"); std::process::exit(1); } }; - let js = nats::jetstream::new(nc.clone()); // amqps:// → pure-Rust rustls over amiquip's mio stream (ADR-0026 Q1); amqp:// // → plaintext. Both avoid amiquip's native-tls (openssl-sys) default. @@ -301,7 +256,7 @@ fn main() { None }; let source_alive = || alive() && held.as_ref().map_or(true, |l| !l.lost()); - run_source_amqp(&channel, &nc, &queue, &subject, wait, source_alive) + run_source_amqp(&channel, &js, &queue, &subject, wait, source_alive) } else { run_sink_amqp(&channel, &js, &stream, &subject, &durable, &routing_key) }; @@ -317,7 +272,7 @@ fn main() { fn run_source_amqp( channel: &Channel, - nc: &nats::Connection, + js: &bus::JetStream, queue: &str, subject: &str, wait: Duration, @@ -331,12 +286,12 @@ fn run_source_amqp( .basic_consume(queue, ConsumerOptions::default()) .map_err(|e| e.to_string())?; let mut src = AmqpConsumer { consumer, pending: HashMap::new() }; - run_source(&mut src, nc, subject, wait, alive) + run_source(&mut src, js, subject, wait, alive) } fn run_sink_amqp( channel: &Channel, - js: &nats::jetstream::JetStream, + js: &bus::JetStream, stream: &str, subject: &str, durable: &str, @@ -346,22 +301,11 @@ fn run_sink_amqp( let confirms = channel.listen_for_publisher_confirms().map_err(|e| e.to_string())?; let exchange = Exchange::direct(channel); // Bind (creating if absent) the durable pull consumer on the stream — the - // runtime owns the stream, but pull_subscribe_with_options only BINDS an - // existing durable, so a standalone connector must declare it first (idempotent). - let _ = js.add_consumer( - stream, - nats::jetstream::ConsumerConfig { - durable_name: Some(durable.to_string()), - filter_subject: subject.to_string(), - ..Default::default() - }, - ); - let sub = js - .pull_subscribe_with_options( - subject, - &nats::jetstream::PullSubscribeOptions::new().durable_name(durable.to_string()), - ) - .map_err(|e| e.to_string())?; + // runtime owns the stream, but pull_durable only BINDS an existing durable, + // so a standalone connector must declare it first (idempotent). + let filter = [subject.to_string()]; + let _ = js.create_pull_consumer(stream, durable, &filter, Duration::ZERO, None); + let sub = js.pull_durable(stream, durable, Some(subject))?; let mut sink = AmqpPublisher { exchange, confirms }; run_sink(&mut sink, &sub, routing_key, alive) } diff --git a/connectors/mq/Cargo.lock b/connectors/mq/Cargo.lock index 3ea092c..37001fb 100644 --- a/connectors/mq/Cargo.lock +++ b/connectors/mq/Cargo.lock @@ -12,49 +12,43 @@ dependencies = [ ] [[package]] -name = "async-channel" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "924ed96dd52d1b75e9c1a3e6275715fd320f5f9439fb5a4a11fa51f4221158d2" -dependencies = [ - "concurrent-queue", - "event-listener-strategy", - "futures-core", - "pin-project-lite", -] - -[[package]] -name = "async-task" -version = "4.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b75356056920673b02621b35afd0f7dda9306d03c79a30f5c56c44cf256e3de" - -[[package]] -name = "atomic-waker" -version = "1.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" - -[[package]] -name = "base64" -version = "0.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8" - -[[package]] -name = "base64-url" -version = "1.4.13" +name = "async-nats" +version = "0.52.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67a99c239d0c7e77c85dddfa9cebce48704b3c49550fcd3b84dd637e4484899f" +checksum = "015f0a7a3b0b29701b78c76a4d4a122624c97c71fa30363381664eb729d619c7" dependencies = [ "base64", + "bytes", + "futures-util", + "memchr", + "pin-project", + "portable-atomic", + "rand", + "regex", + "ring", + "rustls-native-certs", + "rustls-pki-types", + "rustls-webpki", + "serde", + "serde_json", + "serde_nanos", + "serde_repr", + "thiserror", + "time", + "tokio", + "tokio-rustls", + "tokio-stream", + "tokio-util", + "tracing", + "tryhard", + "url", ] [[package]] -name = "base64ct" -version = "1.8.3" +name = "base64" +version = "0.22.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" [[package]] name = "bitflags" @@ -63,25 +57,12 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" [[package]] -name = "block-buffer" -version = "0.10.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" -dependencies = [ - "generic-array", -] - -[[package]] -name = "blocking" -version = "1.6.2" +name = "bytes" +version = "1.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e83f8d02be6967315521be875afa792a316e28d57b5a2d401897e2a7921b7f21" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" dependencies = [ - "async-channel", - "async-task", - "futures-io", - "futures-lite", - "piper", + "serde", ] [[package]] @@ -101,25 +82,21 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" [[package]] -name = "concurrent-queue" -version = "2.5.0" +name = "chacha20" +version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ca0197aee26d1ae37445ee532fefce43251d24cc7c166799f4d46817f1d3973" +checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06" dependencies = [ - "crossbeam-utils", + "cfg-if", + "cpufeatures", + "rand_core", ] -[[package]] -name = "const-oid" -version = "0.9.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" - [[package]] name = "core-foundation" -version = "0.9.4" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f" +checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6" dependencies = [ "core-foundation-sys", "libc", @@ -133,81 +110,13 @@ checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" [[package]] name = "cpufeatures" -version = "0.2.17" +version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +checksum = "5ca28b0ae3115b884660db4118d803791fd6756b6e88f39c0f3f7859060d7566" dependencies = [ "libc", ] -[[package]] -name = "crossbeam-channel" -version = "0.5.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d85363c37faeca707aef026efa9f3b34d077bce547e48f770770625c6013679e" -dependencies = [ - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-utils" -version = "0.8.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61803da095bee82a81bb1a452ecc25d3b2f1416d1897eb86430c6159ef717c17" - -[[package]] -name = "crypto-common" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" -dependencies = [ - "generic-array", - "typenum", -] - -[[package]] -name = "curve25519-dalek" -version = "4.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be" -dependencies = [ - "cfg-if", - "cpufeatures", - "curve25519-dalek-derive", - "digest", - "fiat-crypto", - "rustc_version", - "subtle", -] - -[[package]] -name = "curve25519-dalek-derive" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f46882e17999c6cc590af592290432be3bce0428cb0d5f8b6715e4dc7b383eb3" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "data-encoding" -version = "2.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4583a4551df46e2792f82ceeac45e850d2e2d5debba0b91f102385cda5b11f06" - -[[package]] -name = "der" -version = "0.7.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" -dependencies = [ - "const-oid", - "pem-rfc7468", - "zeroize", -] - [[package]] name = "deranged" version = "0.5.8" @@ -217,16 +126,6 @@ dependencies = [ "serde_core", ] -[[package]] -name = "digest" -version = "0.10.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" -dependencies = [ - "block-buffer", - "crypto-common", -] - [[package]] name = "displaydoc" version = "0.2.7" @@ -238,69 +137,6 @@ dependencies = [ "syn 3.0.3", ] -[[package]] -name = "ed25519" -version = "2.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "115531babc129696a58c64a4fef0a8bf9e9698629fb97e9e40767d235cfbcd53" -dependencies = [ - "signature", -] - -[[package]] -name = "ed25519-dalek" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70e796c081cee67dc755e1a36a0a172b897fab85fc3f6bc48307991f64e4eca9" -dependencies = [ - "curve25519-dalek", - "ed25519", - "sha2", - "signature", - "subtle", -] - -[[package]] -name = "event-listener" -version = "5.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a23add41df1562121a9393cb065eab5146a1242410f23a644851e90cfd669d2" -dependencies = [ - "parking", - "pin-project-lite", -] - -[[package]] -name = "event-listener-strategy" -version = "0.5.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8be9f3dfaaffdae2972880079a491a1a8bb7cbed0b8dd7a347f668b4150a3b93" -dependencies = [ - "event-listener", - "pin-project-lite", -] - -[[package]] -name = "fastrand" -version = "1.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e51093e27b0797c359783294ca4f0a911c270184cb10f85783b118614a1501be" -dependencies = [ - "instant", -] - -[[package]] -name = "fastrand" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" - -[[package]] -name = "fiat-crypto" -version = "0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d" - [[package]] name = "find-msvc-tools" version = "0.1.11" @@ -323,40 +159,51 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" [[package]] -name = "futures-io" +name = "futures-sink" version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53c0fa8157de1303bfffdaa1cc2a673bfffb60102f76b0ef4441659124373fed" +checksum = "1944426bf7d03f1d14f708785e4b33efd750b36d48a157b836b3efc15ede8e1d" [[package]] -name = "futures-lite" -version = "2.6.1" +name = "futures-task" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd" + +[[package]] +name = "futures-util" +version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f78e10609fe0e0b3f4157ffab1876319b5b0db102a2c60dc4626306dc46b44ad" +checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" dependencies = [ "futures-core", + "futures-sink", + "futures-task", "pin-project-lite", + "slab", ] [[package]] -name = "generic-array" -version = "0.14.7" +name = "getrandom" +version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" dependencies = [ - "typenum", - "version_check", + "cfg-if", + "libc", + "wasi", ] [[package]] name = "getrandom" -version = "0.2.17" +version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" dependencies = [ "cfg-if", "libc", - "wasi", + "r-efi", + "rand_core", ] [[package]] @@ -463,33 +310,12 @@ dependencies = [ "icu_properties", ] -[[package]] -name = "instant" -version = "0.1.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0242819d153cba4b4b05a5a8f2a7e9bbf97b6055b2a002b395c96b5ff3c0222" -dependencies = [ - "cfg-if", -] - [[package]] name = "itoa" version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" -[[package]] -name = "json" -version = "0.12.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "078e285eafdfb6c4b434e0d31e8cfcb5115b651496faca5749b88fafd4f23bfd" - -[[package]] -name = "lazy_static" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" - [[package]] name = "libc" version = "0.2.189" @@ -502,21 +328,6 @@ version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "47d9d19d1d6efa0109d2f65ff4c85cddd50bd572e5a00127ab10987290bcefae" -[[package]] -name = "lock_api" -version = "0.4.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" -dependencies = [ - "scopeguard", -] - -[[package]] -name = "log" -version = "0.4.34" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6" - [[package]] name = "memchr" version = "2.8.3" @@ -524,65 +335,14 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" [[package]] -name = "nats" -version = "0.25.0" +name = "mio" +version = "1.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19090dd06e27eb59adfd16f332b85818a3e7dbbdbc13c448676c9bb69f964208" +checksum = "1788edb87fdc09c7e26304471e2f5be8cdefb1b6930d6e3985fc02ff53bf86ee" dependencies = [ - "base64", - "base64-url", - "blocking", - "crossbeam-channel", - "fastrand 1.9.0", - "itoa", - "json", - "lazy_static", "libc", - "log", - "memchr", - "nkeys", - "nuid", - "once_cell", - "parking_lot", - "portable-atomic", - "regex", - "ring", - "rustls", - "rustls-native-certs", - "rustls-pemfile", - "rustls-webpki", - "serde", - "serde_json", - "serde_nanos", - "serde_repr", - "time", - "url", - "winapi", -] - -[[package]] -name = "nkeys" -version = "0.4.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "879011babc47a1c7fdf5a935ae3cfe94f34645ca0cac1c7f6424b36fc743d1bf" -dependencies = [ - "data-encoding", - "ed25519", - "ed25519-dalek", - "getrandom", - "log", - "rand", - "signatory", -] - -[[package]] -name = "nuid" -version = "0.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20c1bb65186718d348306bf1afdeb20d9ab45b2ab80fb793c0fdcf59ffbb4f38" -dependencies = [ - "lazy_static", - "rand", + "wasi", + "windows-sys 0.61.2", ] [[package]] @@ -599,81 +359,42 @@ checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" [[package]] name = "openssl-probe" -version = "0.1.6" +version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d05e27ee213611ffe7d6348b942e8f942b37114c00cc03cec254295a4a17852e" +checksum = "7e036b411754132fd6b74e94b74e9a13c5cf606fa93d1885c35860d3060820b1" [[package]] -name = "parking" -version = "2.2.1" +name = "percent-encoding" +version = "2.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f38d5652c16fde515bb1ecef450ab0f6a219d619a7274976324d5e377f7dceba" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" [[package]] -name = "parking_lot" -version = "0.12.5" +name = "pin-project" +version = "1.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" +checksum = "2466b2336ed02bcdca6b294417127b90ec92038d1d5c4fbeac971a922e0e0924" dependencies = [ - "lock_api", - "parking_lot_core", + "pin-project-internal", ] [[package]] -name = "parking_lot_core" -version = "0.9.12" +name = "pin-project-internal" +version = "1.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" +checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b" dependencies = [ - "cfg-if", - "libc", - "redox_syscall", - "smallvec", - "windows-link", -] - -[[package]] -name = "pem-rfc7468" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412" -dependencies = [ - "base64ct", + "proc-macro2", + "quote", + "syn 2.0.119", ] -[[package]] -name = "percent-encoding" -version = "2.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" - [[package]] name = "pin-project-lite" version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" -[[package]] -name = "piper" -version = "0.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c835479a4443ded371d6c535cbfd8d31ad92c5d23ae9770a61bc155e4992a3c1" -dependencies = [ - "atomic-waker", - "fastrand 2.5.0", - "futures-io", -] - -[[package]] -name = "pkcs8" -version = "0.10.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" -dependencies = [ - "der", - "spki", -] - [[package]] name = "portable-atomic" version = "1.15.0" @@ -695,15 +416,6 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" -[[package]] -name = "ppv-lite86" -version = "0.2.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" -dependencies = [ - "zerocopy", -] - [[package]] name = "proc-macro2" version = "1.0.107" @@ -723,43 +435,27 @@ dependencies = [ ] [[package]] -name = "rand" -version = "0.8.7" +name = "r-efi" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22f6172bdec972074665ed81ed53b71da00bfc44b65a753cfde883ec4c702a1a" -dependencies = [ - "libc", - "rand_chacha", - "rand_core", -] +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" [[package]] -name = "rand_chacha" -version = "0.3.1" +name = "rand" +version = "0.10.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" +checksum = "65c9fb96cbc91e3478eaae79a69fcd3f1ae4ad052e471fe6732fff548984b4af" dependencies = [ - "ppv-lite86", + "chacha20", + "getrandom 0.4.3", "rand_core", ] [[package]] name = "rand_core" -version = "0.6.4" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" -dependencies = [ - "getrandom", -] - -[[package]] -name = "redox_syscall" -version = "0.5.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" -dependencies = [ - "bitflags", -] +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" [[package]] name = "regex" @@ -798,28 +494,19 @@ checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" dependencies = [ "cc", "cfg-if", - "getrandom", + "getrandom 0.2.17", "libc", "untrusted", "windows-sys 0.52.0", ] -[[package]] -name = "rustc_version" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" -dependencies = [ - "semver", -] - [[package]] name = "rustls" -version = "0.22.4" +version = "0.23.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf4ef73721ac7bcd79b2b315da7779d8fc09718c6b3d2d1b2d94850eb8c18432" +checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf" dependencies = [ - "log", + "once_cell", "ring", "rustls-pki-types", "rustls-webpki", @@ -829,26 +516,16 @@ dependencies = [ [[package]] name = "rustls-native-certs" -version = "0.7.3" +version = "0.8.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5bfb394eeed242e909609f56089eecfe5fda225042e8b171791b9c95f5931e5" +checksum = "4a073f5dc7469f984c52ad2752b63b0807745133b6de880b7b64c1ac4c48aec4" dependencies = [ "openssl-probe", - "rustls-pemfile", "rustls-pki-types", "schannel", "security-framework", ] -[[package]] -name = "rustls-pemfile" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dce314e5fee3f39953d46bb63bb8a46d40c2f8fb7cc5a3b6cab2bde9721d6e50" -dependencies = [ - "rustls-pki-types", -] - [[package]] name = "rustls-pki-types" version = "1.15.1" @@ -860,9 +537,9 @@ dependencies = [ [[package]] name = "rustls-webpki" -version = "0.102.8" +version = "0.103.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "64ca1bc8749bd4cf37b5ce386cc146580777b4e8572c7b97baf22c83f444bee9" +checksum = "4443b1b6d39d7c663139942cc4558262562d2c84123bb6282ef177009ca8b281" dependencies = [ "ring", "rustls-pki-types", @@ -878,17 +555,11 @@ dependencies = [ "windows-sys 0.61.2", ] -[[package]] -name = "scopeguard" -version = "1.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" - [[package]] name = "security-framework" -version = "2.11.1" +version = "3.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "897b2245f0b511c87893af39b033e5ca9cce68824c4d7e7630b5a1d339658d02" +checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d" dependencies = [ "bitflags", "core-foundation", @@ -907,12 +578,6 @@ dependencies = [ "libc", ] -[[package]] -name = "semver" -version = "1.0.28" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" - [[package]] name = "serde" version = "1.0.229" @@ -976,17 +641,6 @@ dependencies = [ "syn 3.0.3", ] -[[package]] -name = "sha2" -version = "0.10.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" -dependencies = [ - "cfg-if", - "cpufeatures", - "digest", -] - [[package]] name = "shlex" version = "2.0.1" @@ -994,26 +648,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" [[package]] -name = "signatory" -version = "0.27.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1e303f8205714074f6068773f0e29527e0453937fe837c9717d066635b65f31" -dependencies = [ - "pkcs8", - "rand_core", - "signature", - "zeroize", -] - -[[package]] -name = "signature" -version = "2.2.0" +name = "slab" +version = "0.4.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" -dependencies = [ - "digest", - "rand_core", -] +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" [[package]] name = "smallvec" @@ -1022,13 +660,13 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" [[package]] -name = "spki" -version = "0.7.3" +name = "socket2" +version = "0.6.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" +checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" dependencies = [ - "base64ct", - "der", + "libc", + "windows-sys 0.61.2", ] [[package]] @@ -1076,6 +714,26 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "thiserror" +version = "2.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09e52cb86a36cede5cb101bf8908837b3e4c6e5e59fe7fd85c23fb56200d189e" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fe5197923287db20a58125f0bc85c062f7f2c892de97b18c356f9efb14b28524" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + [[package]] name = "time" version = "0.3.55" @@ -1117,10 +775,105 @@ dependencies = [ ] [[package]] -name = "typenum" -version = "1.20.1" +name = "tokio" +version = "1.54.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b52fd3255258d26fac7264ab5a76b53c13367136dc011d3d6ad4f8ed070396a3" +dependencies = [ + "bytes", + "libc", + "mio", + "pin-project-lite", + "socket2", + "tokio-macros", + "windows-sys 0.61.2", +] + +[[package]] +name = "tokio-macros" +version = "2.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "tokio-rustls" +version = "0.26.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c9cc2678c2cdd569ef8215e2afd7954ada2ae20b4fdd2c5fe6139a3b02d105db" +dependencies = [ + "rustls", + "tokio", +] + +[[package]] +name = "tokio-stream" +version = "0.1.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3d06f0b082ba57c26b79407372e57cf2a1e28124f78e9479fe80322cf53420b" +dependencies = [ + "futures-core", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "tokio-util" +version = "0.7.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e464cf451ba96ebfc6f9b6542f17ee8b8956e33f1e40d9690624e59d7a7f8a4b" +dependencies = [ + "bytes", + "futures-core", + "futures-sink", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "tracing" +version = "0.1.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" +dependencies = [ + "pin-project-lite", + "tracing-attributes", + "tracing-core", +] + +[[package]] +name = "tracing-attributes" +version = "0.1.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "tracing-core" +version = "0.1.36" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" +checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" +dependencies = [ + "once_cell", +] + +[[package]] +name = "tryhard" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9fe58ebd5edd976e0fe0f8a14d2a04b7c81ef153ea9a54eebc42e67c2c23b4e5" +dependencies = [ + "pin-project-lite", + "tokio", +] [[package]] name = "unicode-ident" @@ -1156,44 +909,19 @@ checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" name = "vejas-mq" version = "0.1.0" dependencies = [ - "nats", + "async-nats", + "bytes", + "futures-util", "serde_json", + "tokio", ] -[[package]] -name = "version_check" -version = "0.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" - [[package]] name = "wasi" version = "0.11.1+wasi-snapshot-preview1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" -[[package]] -name = "winapi" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" -dependencies = [ - "winapi-i686-pc-windows-gnu", - "winapi-x86_64-pc-windows-gnu", -] - -[[package]] -name = "winapi-i686-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" - -[[package]] -name = "winapi-x86_64-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" - [[package]] name = "windows-link" version = "0.2.1" @@ -1311,26 +1039,6 @@ dependencies = [ "synstructure", ] -[[package]] -name = "zerocopy" -version = "0.8.56" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "556764e583adb45a9f8d413c2a147fa7e8d821e48e12b14fd560b607998b75eb" -dependencies = [ - "zerocopy-derive", -] - -[[package]] -name = "zerocopy-derive" -version = "0.8.56" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2ab42fc20575779bd240faa45f94a74256f755c0fa9e89f0ede20d91d0cdfc1" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - [[package]] name = "zerofrom" version = "0.1.8" diff --git a/connectors/mq/Cargo.toml b/connectors/mq/Cargo.toml index 6067b08..c5eed03 100644 --- a/connectors/mq/Cargo.toml +++ b/connectors/mq/Cargo.toml @@ -8,7 +8,8 @@ description = "Vejas IBM MQ connector: a first-class bus citizen (ADR-0023). Own # NOT a member of the runtime workspace, and — unlike the SAP/Salesforce exec # children — this one holds its OWN NATS client (ADR-0023): a destructive MQGET # under syncpoint may only MQCMIT *after* the bus publish is confirmed, and a -# stdout-only child cannot know the bus confirmed. So it needs `nats` directly. +# stdout-only child cannot know the bus confirmed. So it needs the bus directly: it +# compiles the runtime's own core/src/bus.rs (ADR-0035), hence the NATS client below. # # The IBM MQ C client (libmqic_r.so) is dlopen'd at RUNTIME (like libsapnwrfc, # ADR-0014) — there is NO build-time MQ dependency, so this crate builds anywhere. @@ -17,7 +18,10 @@ description = "Vejas IBM MQ connector: a first-class bus citizen (ADR-0023). Own # via the standard MQSERVER / CCDT environment (no MQCD struct needed). [dependencies] -nats = "0.25" +async-nats = { version = "0.52", default-features = false, features = ["jetstream", "kv", "object-store", "ring", "server_2_10"] } +tokio = { version = "1", default-features = false, features = ["rt-multi-thread", "time"] } +bytes = "1" +futures-util = { version = "0.3", default-features = false, features = ["std"] } serde_json = "1" [[bin]] diff --git a/connectors/mq/src/lease.rs b/connectors/mq/src/lease.rs index dde76d3..d6bcba4 100644 --- a/connectors/mq/src/lease.rs +++ b/connectors/mq/src/lease.rs @@ -8,10 +8,10 @@ //! //! Same JetStream KV mechanism as the core runtime (create = acquire, update-CAS = //! renew/fence, delete = release, bucket max_age = crash failover) — reimplemented -//! here because the connector is a standalone binary with its own NATS client, not -//! linked to the core. The bucket is shared (VEJAS_LEASES), keys namespaced, so it +//! here because the connector is a standalone binary, not linked to the core. The bucket is shared (VEJAS_LEASES), keys namespaced, so it //! coordinates with the fleet when they run on the same bus. +use crate::bus; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::Arc; use std::thread; @@ -56,23 +56,19 @@ pub fn source_key(queue: &str) -> String { /// Open (or create) the shared lease bucket. None if JetStream KV is unavailable — /// the caller then runs unleased (single-instance behaviour, no worse than today). -pub fn open_store(js: &nats::jetstream::JetStream) -> Option { - let cfg = nats::kv::Config { +pub fn open_store(js: &bus::JetStream) -> Option { + js.key_value(&bus::KvConfig { bucket: LEASE_BUCKET.to_string(), max_age: lease_ttl(), history: 1, - ..Default::default() - }; - js.create_key_value(&cfg) - .or_else(|_| js.key_value(LEASE_BUCKET)) - .ok() + }) } /// Held while this instance owns the lease. Dropping it stops renewal and releases /// the lease (instant handoff on shutdown). `lost()` goes true if the renewal /// thread finds the lease fenced or aged out — the caller stops getting. pub struct Lease { - store: nats::kv::Store, + store: bus::Kv, key: String, stop_renew: Arc, lost: Arc, @@ -100,7 +96,7 @@ impl Drop for Lease { /// thread. Returns None if `alive()` goes false before acquisition (asked to stop /// while standing by). pub fn acquire_blocking( - store: &nats::kv::Store, + store: &bus::Kv, key: &str, alive: impl Fn() -> bool, ) -> Option { @@ -167,8 +163,7 @@ mod tests { #[ignore] fn lease_is_mutually_exclusive() { let url = std::env::var("NATS_URL").unwrap_or_else(|_| "127.0.0.1:4222".into()); - let nc = nats::connect(&url).expect("connect"); - let js = nats::jetstream::new(nc); + let js = bus::connect(&url).expect("connect").jetstream(); let store = open_store(&js).expect("lease store"); let key = format!("mq_test_{}", std::process::id()); let _ = store.delete(&key); @@ -192,8 +187,7 @@ mod tests { #[ignore] fn stale_revision_is_fenced() { let url = std::env::var("NATS_URL").unwrap_or_else(|_| "127.0.0.1:4222".into()); - let nc = nats::connect(&url).expect("connect"); - let js = nats::jetstream::new(nc); + let js = bus::connect(&url).expect("connect").jetstream(); let store = open_store(&js).expect("lease store"); let key = format!("mq_fence_{}", std::process::id()); let _ = store.delete(&key); diff --git a/connectors/mq/src/main.rs b/connectors/mq/src/main.rs index 5942b51..74b534d 100644 --- a/connectors/mq/src/main.rs +++ b/connectors/mq/src/main.rs @@ -17,6 +17,11 @@ //! broker with fault injection; real-queue-manager certification is the declared //! CI exception (ADR-0023, like SAP/ADR-0017). +// The runtime's own bus module, compiled in as it is: one sync API over `async-nats` +// on one small tokio runtime, and no NATS type anywhere else (ADR-0035). +#[allow(dead_code)] +#[path = "../../../core/src/bus.rs"] +mod bus; mod lease; mod mqi; @@ -86,50 +91,13 @@ fn load_config() -> Result { } // ───────────────────────── source: MQ → bus ───────────────────────── -/// Publish and wait for the JetStream pub-ack, flushing DIRECTLY past nats 0.25's -/// 5ms flusher floor (client.rs MIN_FLUSH_BETWEEN) — that floor caps every -/// sequential pub-ack at ~200/s, and this get→publish→commit loop is exactly that. -/// Reimplements the crate's request path (inbox, buffered publish-with-reply, -/// no-responders guard) + a direct flush; Ok still means a real pub-ack (persisted), -/// so the at-least-once ordering (publish-before-MQCMIT) is unchanged. -fn publish_confirmed( - nc: &nats::Connection, - subject: &str, - payload: &[u8], - wait: Duration, -) -> Result<(), String> { - let reply = nc.new_inbox(); - let sub = nc.subscribe(&reply).map_err(|e| e.to_string())?; - nc.publish_request(subject, &reply, payload) - .map_err(|e| e.to_string())?; - nc.flush().map_err(|e| e.to_string())?; // direct flush — bypass the 5ms floor - let msg = sub - .next_timeout(wait) - .map_err(|_| "timed out waiting for JetStream pub-ack".to_string())?; - if msg.is_no_responders() { - return Err("no JetStream responder for the subject (stream missing?)".into()); - } - let ack: serde_json::Value = - serde_json::from_slice(&msg.data).map_err(|e| format!("bad pub-ack: {e}"))?; - if let Some(err) = ack.get("error") { - return Err(format!("JetStream rejected the publish: {err}")); - } - if ack.get("stream").is_none() { - return Err(format!( - "unexpected pub-ack (no stream): {}", - String::from_utf8_lossy(&msg.data) - )); - } - Ok(()) -} - /// Get under syncpoint, publish to the bus awaiting the JetStream pub-ack, then — /// and only then — MQCMIT. Any failure before the commit is an MQBACK, so the /// message stays on the queue and is re-got: at-least-once, no loss. The broker is /// the durable cursor (no offset KV — contrast Kafka/ADR-0022). fn run_source( broker: &mut dyn Broker, - nc: &nats::Connection, + js: &bus::JetStream, subject: &str, wait: Duration, alive: impl Fn() -> bool, @@ -141,7 +109,7 @@ fn run_source( .get_syncpoint(wait) .map_err(|e| format!("get: {e}"))?; let Some(msg) = got else { continue }; // idle: no message within the wait - match publish_confirmed(nc, subject, &msg.body, Duration::from_secs(5)) { + match js.publish(subject, &msg.body) { Ok(_) => { // JetStream pub-ack received (message is durable on the bus) → // safe to remove it from MQ. @@ -176,13 +144,16 @@ fn run_source( /// CorrelId (VEJAS_MQ_DEDUP_FIELD) lets a downstream consumer dedup. fn run_sink( broker: &mut dyn Broker, - sub: &nats::jetstream::PullSubscription, + sub: &bus::Puller, dedup_field: Option<&str>, alive: impl Fn() -> bool, ) -> Result<(), String> { eprintln!("[vejas-mq] sink: bus → MQ (put→commit→ack)"); while alive() { - for m in fetch_round(sub)? { + // One bounded round as a continuous long-poll: messages the instant they + // arrive, else the pull is held server-side for the window and comes back + // empty (no client sleep, no coverage gap, the shape the runtime uses). + for m in sub.round()? { if !alive() { return Ok(()); // stopped mid-batch: leave un-acked, it redelivers } @@ -190,7 +161,7 @@ fn run_sink( .and_then(|f| dedup_key(&m.data, f)) .map(|k| mqi::correlid_from_key(&k)) .unwrap_or_default(); - let out = MqMessage { body: m.data.clone(), correlid }; + let out = MqMessage { body: m.data.to_vec(), correlid }; match broker.put_syncpoint(&out) { Ok(()) => match broker.commit() { Ok(()) => { @@ -211,21 +182,6 @@ fn run_sink( Ok(()) } -/// Pull one bounded batch as a continuous long-poll: messages the instant they -/// arrive, else the pull is held server-side for the window and returns empty (no -/// client sleep, no coverage gap — the same shape the core runtime uses). -fn fetch_round( - sub: &nats::jetstream::PullSubscription, -) -> Result, String> { - let iter = sub - .timeout_fetch( - nats::jetstream::BatchOptions { batch: 64, expires: Some(500_000_000), no_wait: true }, - Duration::from_millis(750), - ) - .map_err(|e| e.to_string())?; - Ok(iter.map_while(|m| m.ok()).collect()) -} - /// Pull a stringified idempotency key out of the JSON body at a top-level field. fn dedup_key(body: &[u8], field: &str) -> Option { let v: serde_json::Value = serde_json::from_slice(body).ok()?; @@ -251,14 +207,13 @@ fn main() { ); } - let nc = match nats::connect(&cfg.nats_url) { - Ok(nc) => nc, + let js = match bus::connect(&cfg.nats_url) { + Ok(bus) => bus.jetstream(), Err(e) => { eprintln!("[vejas-mq] cannot connect to NATS at {}: {e}", cfg.nats_url); std::process::exit(1); } }; - let js = nats::jetstream::new(nc.clone()); let result = if cfg.mode == "source" { // Singleton by default (ADR-0023, "because order"): take the lease so one @@ -282,27 +237,16 @@ fn main() { // stop getting if the lease is lost (fenced / aged out) as well as on signal let source_alive = || alive() && held.as_ref().map_or(true, |l| !l.lost()); MqiQueue::open(&cfg.qmgr, &cfg.queue, false) - .and_then(|mut q| run_source(&mut q, &nc, &cfg.subject, cfg.wait, source_alive)) + .and_then(|mut q| run_source(&mut q, &js, &cfg.subject, cfg.wait, source_alive)) } else { // durable pull consumer, competing-safe by its own durable (SUBJECTS.md). - // pull_subscribe_with_options only BINDS an existing durable — the runtime - // owns the stream, but a standalone connector must declare the consumer - // first (idempotent), or binding 404s. VEJAS_STREAM names the stream. + // pull_durable only BINDS an existing durable — the runtime owns the + // stream, but a standalone connector must declare the consumer first + // (idempotent), or binding 404s. VEJAS_STREAM names the stream. let stream = env_or("VEJAS_STREAM", "VEJAS"); - let _ = js.add_consumer( - &stream, - nats::jetstream::ConsumerConfig { - durable_name: Some(cfg.durable.clone()), - filter_subject: cfg.subject.clone(), - ..Default::default() - }, - ); - let sub = js - .pull_subscribe_with_options( - &cfg.subject, - &nats::jetstream::PullSubscribeOptions::new().durable_name(cfg.durable.clone()), - ) - .map_err(|e| e.to_string()); + let filter = [cfg.subject.clone()]; + let _ = js.create_pull_consumer(&stream, &cfg.durable, &filter, Duration::ZERO, None); + let sub = js.pull_durable(&stream, &cfg.durable, Some(&cfg.subject)); match (MqiQueue::open(&cfg.qmgr, &cfg.queue, true), sub) { (Ok(mut q), Ok(sub)) => run_sink(&mut q, &sub, cfg.dedup_field.as_deref(), alive), (Err(e), _) | (_, Err(e)) => Err(e), @@ -436,20 +380,20 @@ mod tests { #[ignore] fn source_publishes_to_bus_e2e() { let url = std::env::var("NATS_URL").unwrap_or_else(|_| "127.0.0.1:4222".into()); - let nc = nats::connect(&url).expect("connect"); - let js = nats::jetstream::new(nc.clone()); + let nc = bus::connect(&url).expect("connect"); + let js = nc.jetstream(); let pid = std::process::id(); let stream = format!("MQPROBE_{pid}"); let subject = format!("mqprobe.{pid}.in"); let _ = js.delete_stream(&stream); - js.add_stream(nats::jetstream::StreamConfig { + js.add_stream(&bus::StreamConfig { name: stream.clone(), subjects: vec![format!("mqprobe.{pid}.>")], ..Default::default() }) .expect("add_stream"); // a plain subscriber to observe what lands on the bus - let sub = nc.subscribe(&subject).expect("sub"); + let mut sub = nc.subscribe(&subject).expect("sub"); let mut broker = FakeBroker::with(&["a1", "b2", "c3"]); // stop once the queue has drained (get returns None) — bounded, no globals. @@ -468,7 +412,7 @@ mod tests { fn backout(&mut self) -> Result<(), mqi::MqError> { self.inner.backout() } } let mut draining = Draining { inner: std::mem::take(&mut broker), flag: &seen_empty }; - run_source(&mut draining, &nc, &subject, Duration::from_millis(50), alive).expect("source"); + run_source(&mut draining, &js, &subject, Duration::from_millis(50), alive).expect("source"); let mut got = Vec::new(); while let Ok(m) = sub.next_timeout(Duration::from_millis(300)) { diff --git a/core/Cargo.lock b/core/Cargo.lock index e20a5fd..127da93 100644 --- a/core/Cargo.lock +++ b/core/Cargo.lock @@ -33,77 +33,56 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d92bec98840b8f03a5ff5413de5293bfcd8bf96467cf5452609f939ec6f5de16" [[package]] -name = "async-channel" -version = "2.5.0" +name = "async-nats" +version = "0.52.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "924ed96dd52d1b75e9c1a3e6275715fd320f5f9439fb5a4a11fa51f4221158d2" +checksum = "015f0a7a3b0b29701b78c76a4d4a122624c97c71fa30363381664eb729d619c7" dependencies = [ - "concurrent-queue", - "event-listener-strategy", - "futures-core", - "pin-project-lite", + "base64", + "bytes", + "futures-util", + "memchr", + "pin-project", + "portable-atomic", + "rand", + "regex", + "ring", + "rustls-native-certs", + "rustls-pki-types", + "rustls-webpki", + "serde", + "serde_json", + "serde_nanos", + "serde_repr", + "thiserror", + "time", + "tokio", + "tokio-rustls", + "tokio-stream", + "tokio-util", + "tracing", + "tryhard", + "url", ] -[[package]] -name = "async-task" -version = "4.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b75356056920673b02621b35afd0f7dda9306d03c79a30f5c56c44cf256e3de" - -[[package]] -name = "atomic-waker" -version = "1.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" - [[package]] name = "autocfg" version = "1.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" -[[package]] -name = "base64" -version = "0.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8" - [[package]] name = "base64" version = "0.22.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" -[[package]] -name = "base64-url" -version = "1.4.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67a99c239d0c7e77c85dddfa9cebce48704b3c49550fcd3b84dd637e4484899f" -dependencies = [ - "base64 0.13.1", -] - -[[package]] -name = "base64ct" -version = "1.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" - [[package]] name = "bitflags" version = "2.13.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" -[[package]] -name = "block-buffer" -version = "0.10.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" -dependencies = [ - "generic-array", -] - [[package]] name = "block-buffer" version = "0.12.1" @@ -122,25 +101,21 @@ dependencies = [ "objc2", ] -[[package]] -name = "blocking" -version = "1.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e83f8d02be6967315521be875afa792a316e28d57b5a2d401897e2a7921b7f21" -dependencies = [ - "async-channel", - "async-task", - "futures-io", - "futures-lite", - "piper", -] - [[package]] name = "bumpalo" version = "3.20.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" +dependencies = [ + "serde", +] + [[package]] name = "cc" version = "1.4.3" @@ -170,8 +145,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06" dependencies = [ "cfg-if", - "cpufeatures 0.3.1", - "rand_core 0.10.1", + "cpufeatures", + "rand_core", ] [[package]] @@ -194,21 +169,6 @@ version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6e4de3bc4ea267985becf712dc6d9eed8b04c953b3fcfb339ebc87acd9804901" -[[package]] -name = "concurrent-queue" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ca0197aee26d1ae37445ee532fefce43251d24cc7c166799f4d46817f1d3973" -dependencies = [ - "crossbeam-utils", -] - -[[package]] -name = "const-oid" -version = "0.9.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" - [[package]] name = "const-oid" version = "0.10.2" @@ -217,9 +177,9 @@ checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" [[package]] name = "core-foundation" -version = "0.9.4" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f" +checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6" dependencies = [ "core-foundation-sys", "libc", @@ -231,15 +191,6 @@ version = "0.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" -[[package]] -name = "cpufeatures" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" -dependencies = [ - "libc", -] - [[package]] name = "cpufeatures" version = "0.3.1" @@ -258,31 +209,6 @@ dependencies = [ "cfg-if", ] -[[package]] -name = "crossbeam-channel" -version = "0.5.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d85363c37faeca707aef026efa9f3b34d077bce547e48f770770625c6013679e" -dependencies = [ - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-utils" -version = "0.8.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61803da095bee82a81bb1a452ecc25d3b2f1416d1897eb86430c6159ef717c17" - -[[package]] -name = "crypto-common" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" -dependencies = [ - "generic-array", - "typenum", -] - [[package]] name = "crypto-common" version = "0.2.2" @@ -303,49 +229,6 @@ dependencies = [ "windows-sys 0.61.2", ] -[[package]] -name = "curve25519-dalek" -version = "4.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be" -dependencies = [ - "cfg-if", - "cpufeatures 0.2.17", - "curve25519-dalek-derive", - "digest 0.10.7", - "fiat-crypto", - "rustc_version", - "subtle", -] - -[[package]] -name = "curve25519-dalek-derive" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f46882e17999c6cc590af592290432be3bce0428cb0d5f8b6715e4dc7b383eb3" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "data-encoding" -version = "2.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4583a4551df46e2792f82ceeac45e850d2e2d5debba0b91f102385cda5b11f06" - -[[package]] -name = "der" -version = "0.7.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" -dependencies = [ - "const-oid 0.9.6", - "pem-rfc7468", - "zeroize", -] - [[package]] name = "deranged" version = "0.5.8" @@ -355,25 +238,15 @@ dependencies = [ "serde_core", ] -[[package]] -name = "digest" -version = "0.10.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" -dependencies = [ - "block-buffer 0.10.4", - "crypto-common 0.1.7", -] - [[package]] name = "digest" version = "0.11.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" dependencies = [ - "block-buffer 0.12.1", - "const-oid 0.10.2", - "crypto-common 0.2.2", + "block-buffer", + "const-oid", + "crypto-common", ] [[package]] @@ -405,75 +278,12 @@ version = "1.0.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" -[[package]] -name = "ed25519" -version = "2.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "115531babc129696a58c64a4fef0a8bf9e9698629fb97e9e40767d235cfbcd53" -dependencies = [ - "signature", -] - -[[package]] -name = "ed25519-dalek" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70e796c081cee67dc755e1a36a0a172b897fab85fc3f6bc48307991f64e4eca9" -dependencies = [ - "curve25519-dalek", - "ed25519", - "sha2 0.10.9", - "signature", - "subtle", -] - [[package]] name = "equivalent" version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" -[[package]] -name = "event-listener" -version = "5.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a23add41df1562121a9393cb065eab5146a1242410f23a644851e90cfd669d2" -dependencies = [ - "parking", - "pin-project-lite", -] - -[[package]] -name = "event-listener-strategy" -version = "0.5.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8be9f3dfaaffdae2972880079a491a1a8bb7cbed0b8dd7a347f668b4150a3b93" -dependencies = [ - "event-listener", - "pin-project-lite", -] - -[[package]] -name = "fastrand" -version = "1.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e51093e27b0797c359783294ca4f0a911c270184cb10f85783b118614a1501be" -dependencies = [ - "instant", -] - -[[package]] -name = "fastrand" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" - -[[package]] -name = "fiat-crypto" -version = "0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d" - [[package]] name = "find-msvc-tools" version = "0.1.11" @@ -518,20 +328,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" [[package]] -name = "futures-io" +name = "futures-sink" version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53c0fa8157de1303bfffdaa1cc2a673bfffb60102f76b0ef4441659124373fed" - -[[package]] -name = "futures-lite" -version = "2.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f78e10609fe0e0b3f4157ffab1876319b5b0db102a2c60dc4626306dc46b44ad" -dependencies = [ - "futures-core", - "pin-project-lite", -] +checksum = "1944426bf7d03f1d14f708785e4b33efd750b36d48a157b836b3efc15ede8e1d" [[package]] name = "futures-task" @@ -546,21 +346,12 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" dependencies = [ "futures-core", + "futures-sink", "futures-task", "pin-project-lite", "slab", ] -[[package]] -name = "generic-array" -version = "0.14.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" -dependencies = [ - "typenum", - "version_check", -] - [[package]] name = "getrandom" version = "0.2.17" @@ -581,7 +372,7 @@ dependencies = [ "cfg-if", "libc", "r-efi", - "rand_core 0.10.1", + "rand_core", ] [[package]] @@ -769,15 +560,6 @@ dependencies = [ "serde_core", ] -[[package]] -name = "instant" -version = "0.1.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0242819d153cba4b4b05a5a8f2a7e9bbf97b6055b2a002b395c96b5ff3c0222" -dependencies = [ - "cfg-if", -] - [[package]] name = "itoa" version = "1.0.18" @@ -795,18 +577,6 @@ dependencies = [ "wasm-bindgen", ] -[[package]] -name = "json" -version = "0.12.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "078e285eafdfb6c4b434e0d31e8cfcb5115b651496faca5749b88fafd4f23bfd" - -[[package]] -name = "lazy_static" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" - [[package]] name = "libc" version = "0.2.189" @@ -819,15 +589,6 @@ version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "47d9d19d1d6efa0109d2f65ff4c85cddd50bd572e5a00127ab10987290bcefae" -[[package]] -name = "lock_api" -version = "0.4.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" -dependencies = [ - "scopeguard", -] - [[package]] name = "log" version = "0.4.33" @@ -905,40 +666,14 @@ dependencies = [ ] [[package]] -name = "nats" -version = "0.25.0" +name = "mio" +version = "1.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19090dd06e27eb59adfd16f332b85818a3e7dbbdbc13c448676c9bb69f964208" +checksum = "1788edb87fdc09c7e26304471e2f5be8cdefb1b6930d6e3985fc02ff53bf86ee" dependencies = [ - "base64 0.13.1", - "base64-url", - "blocking", - "crossbeam-channel", - "fastrand 1.9.0", - "itoa", - "json", - "lazy_static", "libc", - "log", - "memchr", - "nkeys", - "nuid", - "once_cell", - "parking_lot", - "portable-atomic", - "regex", - "ring", - "rustls 0.22.4", - "rustls-native-certs", - "rustls-pemfile", - "rustls-webpki 0.102.8", - "serde", - "serde_json", - "serde_nanos", - "serde_repr", - "time", - "url", - "winapi", + "wasi", + "windows-sys 0.61.2", ] [[package]] @@ -953,31 +688,6 @@ dependencies = [ "libc", ] -[[package]] -name = "nkeys" -version = "0.4.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "879011babc47a1c7fdf5a935ae3cfe94f34645ca0cac1c7f6424b36fc743d1bf" -dependencies = [ - "data-encoding", - "ed25519", - "ed25519-dalek", - "getrandom 0.2.17", - "log", - "rand 0.8.7", - "signatory", -] - -[[package]] -name = "nuid" -version = "0.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20c1bb65186718d348306bf1afdeb20d9ab45b2ab80fb793c0fdcf59ffbb4f38" -dependencies = [ - "lazy_static", - "rand 0.8.7", -] - [[package]] name = "num-conv" version = "0.2.2" @@ -1016,47 +726,9 @@ checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" [[package]] name = "openssl-probe" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d05e27ee213611ffe7d6348b942e8f942b37114c00cc03cec254295a4a17852e" - -[[package]] -name = "parking" -version = "2.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f38d5652c16fde515bb1ecef450ab0f6a219d619a7274976324d5e377f7dceba" - -[[package]] -name = "parking_lot" -version = "0.12.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" -dependencies = [ - "lock_api", - "parking_lot_core", -] - -[[package]] -name = "parking_lot_core" -version = "0.9.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" -dependencies = [ - "cfg-if", - "libc", - "redox_syscall", - "smallvec", - "windows-link", -] - -[[package]] -name = "pem-rfc7468" -version = "0.7.0" +version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412" -dependencies = [ - "base64ct", -] +checksum = "7e036b411754132fd6b74e94b74e9a13c5cf606fa93d1885c35860d3060820b1" [[package]] name = "percent-encoding" @@ -1107,31 +779,30 @@ dependencies = [ ] [[package]] -name = "pin-project-lite" -version = "0.2.17" +name = "pin-project" +version = "1.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" +checksum = "2466b2336ed02bcdca6b294417127b90ec92038d1d5c4fbeac971a922e0e0924" +dependencies = [ + "pin-project-internal", +] [[package]] -name = "piper" -version = "0.2.5" +name = "pin-project-internal" +version = "1.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c835479a4443ded371d6c535cbfd8d31ad92c5d23ae9770a61bc155e4992a3c1" +checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b" dependencies = [ - "atomic-waker", - "fastrand 2.5.0", - "futures-io", + "proc-macro2", + "quote", + "syn 2.0.119", ] [[package]] -name = "pkcs8" -version = "0.10.2" +name = "pin-project-lite" +version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" -dependencies = [ - "der", - "spki", -] +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" [[package]] name = "portable-atomic" @@ -1154,15 +825,6 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" -[[package]] -name = "ppv-lite86" -version = "0.2.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" -dependencies = [ - "zerocopy", -] - [[package]] name = "proc-macro2" version = "1.0.107" @@ -1187,17 +849,6 @@ version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" -[[package]] -name = "rand" -version = "0.8.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22f6172bdec972074665ed81ed53b71da00bfc44b65a753cfde883ec4c702a1a" -dependencies = [ - "libc", - "rand_chacha", - "rand_core 0.6.4", -] - [[package]] name = "rand" version = "0.10.3" @@ -1206,26 +857,7 @@ checksum = "65c9fb96cbc91e3478eaae79a69fcd3f1ae4ad052e471fe6732fff548984b4af" dependencies = [ "chacha20", "getrandom 0.4.3", - "rand_core 0.10.1", -] - -[[package]] -name = "rand_chacha" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" -dependencies = [ - "ppv-lite86", - "rand_core 0.6.4", -] - -[[package]] -name = "rand_core" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" -dependencies = [ - "getrandom 0.2.17", + "rand_core", ] [[package]] @@ -1234,15 +866,6 @@ version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" -[[package]] -name = "redox_syscall" -version = "0.5.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" -dependencies = [ - "bitflags", -] - [[package]] name = "ref-cast" version = "1.0.27" @@ -1312,29 +935,6 @@ version = "2.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d" -[[package]] -name = "rustc_version" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" -dependencies = [ - "semver", -] - -[[package]] -name = "rustls" -version = "0.22.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf4ef73721ac7bcd79b2b315da7779d8fc09718c6b3d2d1b2d94850eb8c18432" -dependencies = [ - "log", - "ring", - "rustls-pki-types", - "rustls-webpki 0.102.8", - "subtle", - "zeroize", -] - [[package]] name = "rustls" version = "0.23.46" @@ -1345,33 +945,23 @@ dependencies = [ "once_cell", "ring", "rustls-pki-types", - "rustls-webpki 0.103.16", + "rustls-webpki", "subtle", "zeroize", ] [[package]] name = "rustls-native-certs" -version = "0.7.3" +version = "0.8.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5bfb394eeed242e909609f56089eecfe5fda225042e8b171791b9c95f5931e5" +checksum = "4a073f5dc7469f984c52ad2752b63b0807745133b6de880b7b64c1ac4c48aec4" dependencies = [ "openssl-probe", - "rustls-pemfile", "rustls-pki-types", "schannel", "security-framework", ] -[[package]] -name = "rustls-pemfile" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dce314e5fee3f39953d46bb63bb8a46d40c2f8fb7cc5a3b6cab2bde9721d6e50" -dependencies = [ - "rustls-pki-types", -] - [[package]] name = "rustls-pki-types" version = "1.15.1" @@ -1381,17 +971,6 @@ dependencies = [ "zeroize", ] -[[package]] -name = "rustls-webpki" -version = "0.102.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "64ca1bc8749bd4cf37b5ce386cc146580777b4e8572c7b97baf22c83f444bee9" -dependencies = [ - "ring", - "rustls-pki-types", - "untrusted", -] - [[package]] name = "rustls-webpki" version = "0.103.16" @@ -1444,17 +1023,11 @@ dependencies = [ "syn 3.0.3", ] -[[package]] -name = "scopeguard" -version = "1.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" - [[package]] name = "security-framework" -version = "2.11.1" +version = "3.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "897b2245f0b511c87893af39b033e5ca9cce68824c4d7e7630b5a1d339658d02" +checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d" dependencies = [ "bitflags", "core-foundation", @@ -1473,12 +1046,6 @@ dependencies = [ "libc", ] -[[package]] -name = "semver" -version = "1.0.28" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" - [[package]] name = "serde" version = "1.0.229" @@ -1553,17 +1120,6 @@ dependencies = [ "syn 3.0.3", ] -[[package]] -name = "sha2" -version = "0.10.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" -dependencies = [ - "cfg-if", - "cpufeatures 0.2.17", - "digest 0.10.7", -] - [[package]] name = "sha2" version = "0.11.0" @@ -1571,8 +1127,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" dependencies = [ "cfg-if", - "cpufeatures 0.3.1", - "digest 0.11.3", + "cpufeatures", + "digest", ] [[package]] @@ -1581,28 +1137,6 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" -[[package]] -name = "signatory" -version = "0.27.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1e303f8205714074f6068773f0e29527e0453937fe837c9717d066635b65f31" -dependencies = [ - "pkcs8", - "rand_core 0.6.4", - "signature", - "zeroize", -] - -[[package]] -name = "signature" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" -dependencies = [ - "digest 0.10.7", - "rand_core 0.6.4", -] - [[package]] name = "simd-adler32" version = "0.3.10" @@ -1622,13 +1156,13 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" [[package]] -name = "spki" -version = "0.7.3" +name = "socket2" +version = "0.6.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" +checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" dependencies = [ - "base64ct", - "der", + "libc", + "windows-sys 0.61.2", ] [[package]] @@ -1748,6 +1282,66 @@ dependencies = [ "zerovec", ] +[[package]] +name = "tokio" +version = "1.54.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b52fd3255258d26fac7264ab5a76b53c13367136dc011d3d6ad4f8ed070396a3" +dependencies = [ + "bytes", + "libc", + "mio", + "pin-project-lite", + "socket2", + "tokio-macros", + "windows-sys 0.61.2", +] + +[[package]] +name = "tokio-macros" +version = "2.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "tokio-rustls" +version = "0.26.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c9cc2678c2cdd569ef8215e2afd7954ada2ae20b4fdd2c5fe6139a3b02d105db" +dependencies = [ + "rustls", + "tokio", +] + +[[package]] +name = "tokio-stream" +version = "0.1.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3d06f0b082ba57c26b79407372e57cf2a1e28124f78e9479fe80322cf53420b" +dependencies = [ + "futures-core", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "tokio-util" +version = "0.7.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e464cf451ba96ebfc6f9b6542f17ee8b8956e33f1e40d9690624e59d7a7f8a4b" +dependencies = [ + "bytes", + "futures-core", + "futures-sink", + "pin-project-lite", + "tokio", +] + [[package]] name = "tracing" version = "0.1.44" @@ -1779,6 +1373,16 @@ dependencies = [ "once_cell", ] +[[package]] +name = "tryhard" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9fe58ebd5edd976e0fe0f8a14d2a04b7c81ef153ea9a54eebc42e67c2c23b4e5" +dependencies = [ + "pin-project-lite", + "tokio", +] + [[package]] name = "typenum" version = "1.20.1" @@ -1815,11 +1419,11 @@ version = "2.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "02d1a66277ed75f640d608235660df48c8e3c19f3b4edb6a263315626cc3c01d" dependencies = [ - "base64 0.22.1", + "base64", "flate2", "log", "once_cell", - "rustls 0.23.46", + "rustls", "rustls-pki-types", "url", "webpki-roots 0.26.11", @@ -1927,12 +1531,12 @@ dependencies = [ "hashlink", "hex", "indexmap", - "rand 0.10.3", + "rand", "regex", "rustc-hash", "serde", "serde_json", - "sha2 0.11.0", + "sha2", "smallvec", "thiserror", "tracing", @@ -1978,22 +1582,19 @@ dependencies = [ name = "vejas-runtime" version = "0.4.0" dependencies = [ + "async-nats", + "bytes", "ctrlc", - "nats", - "rustls 0.23.46", + "futures-util", + "rustls", "serde_json", "tiny_http", + "tokio", "ureq", "varpulis-engine", "webpki-roots 0.26.11", ] -[[package]] -name = "version_check" -version = "0.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" - [[package]] name = "wasi" version = "0.11.1+wasi-snapshot-preview1" @@ -2063,28 +1664,6 @@ dependencies = [ "rustls-pki-types", ] -[[package]] -name = "winapi" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" -dependencies = [ - "winapi-i686-pc-windows-gnu", - "winapi-x86_64-pc-windows-gnu", -] - -[[package]] -name = "winapi-i686-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" - -[[package]] -name = "winapi-x86_64-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" - [[package]] name = "windows-core" version = "0.62.2" @@ -2255,26 +1834,6 @@ dependencies = [ "synstructure", ] -[[package]] -name = "zerocopy" -version = "0.8.56" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "556764e583adb45a9f8d413c2a147fa7e8d821e48e12b14fd560b607998b75eb" -dependencies = [ - "zerocopy-derive", -] - -[[package]] -name = "zerocopy-derive" -version = "0.8.56" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2ab42fc20575779bd240faa45f94a74256f755c0fa9e89f0ede20d91d0cdfc1" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - [[package]] name = "zerofrom" version = "0.1.8" diff --git a/core/Cargo.toml b/core/Cargo.toml index ff681cc..37c6768 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -11,7 +11,12 @@ serde_json = "1" # `termination` so SIGTERM/SIGHUP (a k8s rolling restart) hit the graceful drain # path, not default immediate kill — needed for lease handoff (ADR-0020). ctrlc = { version = "3.4", features = ["termination"] } -nats = "0.25" +# The NATS client and what it runs on. All four stay inside src/bus.rs (ADR-0035): +# the rest of the runtime is synchronous and sees neither tokio nor a NATS type. +async-nats = { version = "0.52", default-features = false, features = ["jetstream", "kv", "object-store", "ring", "server_2_10"] } +tokio = { version = "1", default-features = false, features = ["rt-multi-thread", "time"] } +bytes = "1" +futures-util = { version = "0.3", default-features = false, features = ["std"] } # Pooled, blocking HTTP(S) client for the http-out/http-poll/oauth-poll drivers # (replaces one curl process per request). Pure-Rust TLS (rustls), self-contained. ureq = "2" diff --git a/core/src/bus.rs b/core/src/bus.rs new file mode 100644 index 0000000..6e75c7f --- /dev/null +++ b/core/src/bus.rs @@ -0,0 +1,1551 @@ +// The bus island (ADR-0035): every NATS byte of the runtime goes through here. +// +// The runtime is synchronous and thread-based; the NATS client is not. So +// `async-nats` lives on one small tokio runtime owned by this module, and the +// rest of the code sees a blocking API and no NATS type. A flow, a connector or +// a supervisor is still a plain OS thread that calls a plain function. +// +// Three rules keep the island honest: +// +// * Every call that leaves a thread for the island goes through `wait`, which +// parks that OS thread. `wait` is never called from inside the runtime: the +// event callback and the echo task below are the only code that runs there, +// and neither calls back into this API. tokio panics rather than deadlocks +// if that ever changes (a test pins it). +// * The connection is shared. `connect` returns the one `Bus` of a URL, whoever +// asks, so a thousand flows are a thousand threads on one socket instead of +// two thousand sockets. +// * What the sync client gave for free is rebuilt here on purpose, because +// `async-nats` does not give it: see `Batch::confirm`. + +use std::cell::RefCell; +use std::collections::HashMap; +use std::future::Future; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, Mutex, OnceLock}; +use std::time::{Duration, Instant}; + +use async_nats::client::{Request, Statistics}; +use async_nats::connection::State; +use async_nats::jetstream::{self, kv, object_store}; +use async_nats::{Client, ConnectOptions, Event, ServerAddr, StatusCode, Subject, Subscriber}; +use bytes::Bytes; +use futures_util::{FutureExt, StreamExt}; +use serde_json::{json, Value}; +use tokio::runtime::Runtime; +use tokio::sync::oneshot; +use tokio::time::timeout; + +// ───────────────────────── the runtime ───────────────────────── + +/// Worker threads of the island. One connection is one task, and the worker runs +/// nothing that blocks: a second one bought nothing at 1, 10 or 50 flows (ADR-0035). +const WORKERS: usize = 1; + +/// How long a publish may wait for room in a connection's queue. Past it the +/// connection is gone for good enough reasons to say so to the caller. +const SEND_WAIT: Duration = Duration::from_secs(5); + +/// How long a barrier may wait for its echo (the sync client's flush timeout). +const BARRIER_WAIT: Duration = Duration::from_secs(10); + +/// After a failed first connect, later callers get the same answer for this +/// long instead of queueing on the registry lock behind a dead address. +const CONNECT_MEMO: Duration = Duration::from_secs(1); + +fn runtime() -> &'static Runtime { + static RUNTIME: OnceLock = OnceLock::new(); + RUNTIME.get_or_init(|| { + tokio::runtime::Builder::new_multi_thread() + .worker_threads(WORKERS) + .thread_name("vejas-bus") + .enable_all() + .build() + .expect("the bus runtime") + }) +} + +/// Park this OS thread on a future of the island. Never call it from a task of +/// the runtime itself. +fn wait(future: F) -> F::Output { + runtime().block_on(future) +} + +/// `wait`, for at most `within`; `None` if the time ran out. The timer has to be +/// built inside the runtime, which is why this is not `wait(timeout(..))`. +fn wait_for(within: Duration, future: F) -> Option { + wait(async { timeout(within, future).await.ok() }) +} + +/// `future`, bounded by `SEND_WAIT`, with its error as text. +async fn sent( + future: impl Future>, +) -> Result { + match timeout(SEND_WAIT, future).await { + Ok(result) => result.map_err(|e| e.to_string()), + Err(_) => Err("timed out: the bus connection is not taking writes".into()), + } +} + +// ───────────────────────── connections ───────────────────────── + +/// One NATS connection, and the way to prove the server has read it. +struct Link { + client: Client, + stats: Arc, + /// A subject only this connection listens to, for the barrier. + echo: Subject, + next_id: AtomicU64, + waiting: Arc>>>, +} + +impl Link { + async fn open(servers: Vec) -> Result, String> { + let client = with_url_credentials(ConnectOptions::new(), &servers) + .name("vejas-runtime") + .event_callback(|event| async move { + match event { + Event::Connected | Event::Closed => {} + Event::Disconnected => eprintln!("[bus] disconnected, reconnecting"), + other => eprintln!("[bus] {other}"), + } + }) + .connect(servers) + .await + .map_err(|e| e.to_string())?; + let echo: Subject = client.new_inbox().into(); + let mut echoes = client + .subscribe(echo.clone()) + .await + .map_err(|e| e.to_string())?; + let waiting: Arc>>> = Arc::default(); + let waiters = waiting.clone(); + tokio::spawn(async move { + while let Some(echoed) = echoes.next().await { + let id = <[u8; 8]>::try_from(&echoed.payload[..]).map(u64::from_be_bytes); + if let Ok(id) = id { + if let Some(waiter) = waiters.lock().unwrap().remove(&id) { + let _ = waiter.send(()); + } + } + } + }); + Ok(Arc::new(Link { + stats: client.statistics(), + client, + echo, + next_id: AtomicU64::new(0), + waiting, + })) + } + + fn up(&self) -> bool { + self.client.connection_state() == State::Connected + } + + /// Which connection this link is on: it moves on every reconnect. + fn epoch(&self) -> u64 { + self.stats.connects.load(Ordering::Relaxed) + } + + /// Prove the server has read everything sent on this link so far: publish + /// to ourselves and wait to hear it. A connection's messages are handled in + /// the order they were sent, so the echo cannot overtake them. + async fn barrier(&self) -> Result<(), String> { + let id = self.next_id.fetch_add(1, Ordering::Relaxed); + let (tx, rx) = oneshot::channel(); + self.waiting.lock().unwrap().insert(id, tx); + let ask = self + .client + .publish(self.echo.clone(), Bytes::copy_from_slice(&id.to_be_bytes())); + let heard = match sent(ask).await { + Err(error) => Err(error), + Ok(()) => match timeout(BARRIER_WAIT, rx).await { + Ok(Ok(())) => Ok(()), + Ok(Err(_)) => Err("the barrier was dropped".to_string()), + Err(_) => Err("timed out waiting for the server".to_string()), + }, + }; + if heard.is_err() { + self.waiting.lock().unwrap().remove(&id); + } + heard + } + + /// The barrier, on the connection `epoch` names: one that was replaced in + /// the meantime fails it, whether or not anything was lost with it. + async fn barrier_since(&self, epoch: u64) -> Result<(), String> { + self.barrier().await?; + if self.epoch() != epoch { + return Err("the connection was replaced during the batch".into()); + } + Ok(()) + } +} + +/// The process's connection to one NATS deployment: a cheap handle, shared by +/// every thread that talks to it. +#[derive(Clone)] +pub struct Bus(Arc); + +struct Shared { + /// One connection for everything: publishes, requests, subscriptions, pull + /// rounds and acks. A connection of their own for the acks bought nothing + /// here at any flow count (ADR-0035). + link: Arc, + js: jetstream::Context, +} + +#[derive(Default)] +struct Registry { + open: HashMap, + failed: HashMap, +} + +/// The `Bus` of `url`, connecting on first use. `url` is what `NATS_URL` holds: +/// one server or a comma-separated list, with or without a scheme. Once it has +/// connected, a `Bus` reconnects on its own for as long as the process lives, +/// and threads already holding it pick up where they were. While it is down +/// this answers an error: a unit that starts now backs off and comes back, +/// as it did when every start dialled its own connection. +pub fn connect(url: &str) -> Result { + static BUSES: OnceLock> = OnceLock::new(); + let mut registry = BUSES + .get_or_init(Default::default) + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + if let Some(bus) = registry.open.get(url) { + if !bus.0.link.up() { + return Err("the connection to NATS is down".into()); + } + return Ok(bus.clone()); + } + if let Some((at, error)) = registry.failed.get(url) { + if at.elapsed() < CONNECT_MEMO { + return Err(error.clone()); + } + } + match open(url) { + Ok(bus) => { + registry.open.insert(url.to_string(), bus.clone()); + Ok(bus) + } + Err(error) => { + registry + .failed + .insert(url.to_string(), (Instant::now(), error.clone())); + Err(error) + } + } +} + +/// `nats://user:pass@host` and `nats://token@host`: `async-nats` parses these and +/// then connects without them, so they are handed over here. One set for the +/// connection: the first server that names any. +fn with_url_credentials(options: ConnectOptions, servers: &[ServerAddr]) -> ConnectOptions { + let named = servers + .iter() + .find_map(|server| Some((server.username()?, server.password()))); + match named { + Some((user, Some(pass))) => options.user_and_password(user.to_string(), pass.to_string()), + Some((token, None)) => options.token(token.to_string()), + None => options, + } +} + +fn servers_of(url: &str) -> Result, String> { + url.split(',') + .map(|server| { + server + .trim() + .parse::() + .map_err(|e| e.to_string()) + }) + .collect() +} + +fn open(url: &str) -> Result { + let servers = servers_of(url)?; + wait(async move { + let link = Link::open(servers).await?; + let js = jetstream::new(link.client.clone()); + Ok(Bus(Arc::new(Shared { link, js }))) + }) +} + +impl Bus { + pub fn jetstream(&self) -> JetStream { + JetStream(self.clone()) + } + + /// Publish one message now, with no promise: for what stands for nothing to + /// ack. An error means it was never handed to the connection. + pub fn publish(&self, subject: &str, payload: impl Into) -> Result<(), String> { + let (client, payload) = (&self.0.link.client, payload.into()); + wait(sent(client.publish(subject.to_string(), payload))) + } + + /// A batch of publishes that leave together, see [`Batch`]. + pub fn batch(&self) -> Batch { + Batch { + bus: self.clone(), + held: Vec::new(), + } + } + + /// Ack every message, in one go. An ack that does not make it only means a + /// redelivery. + pub fn ack_all<'a>( + &self, + messages: impl IntoIterator, + ) -> Result<(), String> { + let client = &self.0.link.client; + wait(async { + for message in messages { + if let Some(reply) = &message.reply { + sent(client.publish(reply.clone(), Bytes::new())).await?; + } + } + Ok(()) + }) + } + + /// A core request: the answer, or an error when nobody answers within `within`. + pub fn request( + &self, + subject: &str, + payload: impl Into, + within: Duration, + ) -> Result { + let request = Request::new().payload(payload.into()).timeout(Some(within)); + let answer = wait( + self.0 + .link + .client + .send_request(subject.to_string(), request), + ) + .map_err(|e| e.to_string())?; + Ok(Message::new(answer, self.0.link.clone())) + } + + pub fn subscribe(&self, subject: &str) -> Result { + let client = &self.0.link.client; + let sub = wait(sent(client.subscribe(subject.to_string())))?; + Ok(Subscription { + sub, + link: self.0.link.clone(), + }) + } + + pub fn queue_subscribe(&self, subject: &str, group: &str) -> Result { + let client = &self.0.link.client; + let sub = wait(sent( + client.queue_subscribe(subject.to_string(), group.to_string()), + ))?; + Ok(Subscription { + sub, + link: self.0.link.clone(), + }) + } +} + +/// The publishes of one batch, held back until [`Batch::confirm`]. A loop that +/// publishes before it acks keeps its emits here, one batch at a time. +/// +/// Handing each emit to the connection as it is made costs a cross-thread wake +/// and a write per message, the most expensive thing on the hot path (~33 µs +/// each on the flow hop, measured: 20 000 evt/s fell to 12 500). Held until the +/// end of the batch they cost one of each per batch, and the batch is still +/// whatever has arrived, so a quiet flow's emit leaves as soon as it is made. +pub struct Batch { + bus: Bus, + held: Vec<(Subject, Bytes)>, +} + +impl Batch { + /// Hold a publish. An error means this one can never leave (a subject the + /// protocol cannot carry, a payload over the server's limit), and only this + /// one: the caller treats its source message as a failed run, and the rest of + /// the batch goes on. + pub fn publish(&mut self, subject: &str, payload: impl Into) -> Result<(), String> { + let payload = payload.into(); + if subject.is_empty() || subject.contains([' ', '\t', '\r', '\n']) { + return Err(format!("invalid subject {subject:?}")); + } + let limit = self.bus.0.link.client.max_payload(); + if payload.len() > limit { + return Err(format!( + "{} bytes is over the server's limit of {limit}", + payload.len() + )); + } + self.held.push((subject.to_string().into(), payload)); + Ok(()) + } + + /// Send what is held and prove it arrived: the publish-before-ack barrier. + /// Ok means the server has read every publish of the batch, so their source + /// messages may be acked; a batch with nothing held has nothing to prove. + /// The batch is empty afterwards, either way. Two things stand in here for + /// what the sync client's `flush` was. + /// + /// `async_nats::Client::flush` only writes the socket; it proves nothing + /// about the server. The barrier is a round-trip through it instead, see + /// [`Link::barrier`]. + /// + /// And on a reconnect `async-nats` drops what it had not yet written: the + /// old connection's buffer goes with it, in silence, and whatever comes next + /// leaves on the new one. A round-trip alone would then confirm a batch that + /// lost its first half. So a connection replaced while the batch was leaving + /// fails the barrier, and the batch redelivers. + pub fn confirm(&mut self) -> Result<(), String> { + if self.held.is_empty() { + return Ok(()); + } + let link = &self.bus.0.link; + if !link.up() { + self.held.clear(); + return Err("the connection is down".into()); + } + let epoch = link.epoch(); + let held = &mut self.held; + wait(async { + for (subject, payload) in held.drain(..) { + sent(link.client.publish(subject, payload)).await?; + } + link.barrier_since(epoch).await + }) + } +} + +/// A plain subscription: the messages arrive whether or not anyone is waiting, +/// and go back with `respond`. +pub struct Subscription { + sub: Subscriber, + link: Arc, +} + +impl Subscription { + /// The next message, or an error if none arrives within `within`. + pub fn next_timeout(&mut self, within: Duration) -> Result { + let next = match self.sub.next().now_or_never() { + Some(next) => next, + None => wait_for(within, self.sub.next()).ok_or("timed out")?, + }; + match next { + Some(message) => Ok(Message::new(message, self.link.clone())), + None => Err("subscription closed".into()), + } + } +} + +/// A message off the bus. +pub struct Message { + pub subject: String, + pub data: Bytes, + reply: Option, + link: Arc, +} + +/// What JetStream says about a delivery, read from its reply subject. +#[derive(Debug, PartialEq, Eq)] +pub struct Delivery { + /// How many times this message has been delivered, this time included. + pub delivered: i64, + pub stream_seq: u64, +} + +impl Message { + fn new(message: async_nats::Message, link: Arc) -> Message { + Message { + subject: message.subject.to_string(), + data: message.payload, + reply: message.reply, + link, + } + } + + /// The JetStream delivery behind this message, if it is one. + pub fn delivery(&self) -> Option { + parse_ack_subject(self.reply.as_deref()?) + } + + pub fn ack(&self) -> Result<(), String> { + self.answer(Bytes::new()) + } + + pub fn nak(&self) -> Result<(), String> { + self.answer(Bytes::from_static(b"-NAK")) + } + + /// Answer a request. A message that expects no answer is left alone. + pub fn respond(&self, payload: impl AsRef<[u8]>) -> Result<(), String> { + self.answer(Bytes::copy_from_slice(payload.as_ref())) + } + + fn answer(&self, payload: Bytes) -> Result<(), String> { + let Some(reply) = &self.reply else { + return Ok(()); + }; + wait(sent(self.link.client.publish(reply.clone(), payload))) + } +} + +/// `$JS.ACK.......`, +/// or the same with `.` in front and a token behind. +fn parse_ack_subject(reply: &str) -> Option { + let tokens: Vec<&str> = reply.strip_prefix("$JS.ACK.")?.split('.').collect(); + let at = match tokens.len() { + 7 => 2, + n if n >= 9 => 4, + _ => return None, + }; + Some(Delivery { + delivered: tokens[at].parse().ok()?, + stream_seq: tokens[at + 1].parse().ok()?, + }) +} + +// ───────────────────────── pull rounds ───────────────────────── + +/// Messages asked for in one pull request. +const PULL_BATCH: usize = 64; + +/// The idle poll window (`expires`). One outstanding pull covers this whole span, +/// so a message arriving any time inside it is delivered at once: there is NO gap +/// between rounds where a message would sit unpolled. It is therefore two things +/// at once: the idle pull cadence (one pull per window when quiet) AND the +/// worst-case stop-response latency (the loop re-checks its `alive` flag each +/// time a pull returns). 500 ms keeps idle chatter to ~2 pulls/s per consumer +/// while leaving stop far inside the transport suite's 3 s bound (T5). +const PULL_EXPIRES_MS: u64 = 500; + +/// A continuous long-poll. `no_wait` returns the instant messages are available +/// (a single message is delivered at once, no batch-fill latency, and a high-rate +/// drain still fills 64-batches in one round). When the subject is idle the pull +/// is HELD server-side for `expires` and comes back empty, so a loop paces itself +/// on the server with no client sleep and no coverage gap. Nothing parks in +/// `max_waiting`: every request resolves within `expires`, so a stopped loop is +/// never wedged on one. (An unbounded fetch parks forever on an idle subject: a +/// stopped flow's thread stayed in `recv()`, reported "running", and processed +/// one more message before dying, the zombie-consumer trap caught while +/// rehearsing the demo.) +const PULL_REQUEST: &[u8] = b"{\"batch\":64,\"expires\":500000000,\"no_wait\":true}"; + +/// Client-side backstop, comfortably past the server's `expires`, so that the +/// server's own status ends a round and not this timeout. +const PULL_WAIT: Duration = Duration::from_millis(PULL_EXPIRES_MS + 250); + +/// A pull consumer, bound to one inbox of its own on the shared connection. The +/// messages of every outstanding request land in that inbox, in order, so a +/// caller can keep the next request in flight while it works through a round. +pub struct Puller { + next: Subject, + inbox: Subject, + sub: RefCell, + /// Set when a round learned that the consumer is gone: the next round + /// fails, and the supervisor's attempt starts over from the top. + fault: RefCell>, + link: Arc, + /// The stream and name of an ephemeral consumer, deleted with this handle. + owned: Option<(String, String)>, + js: JetStream, +} + +impl Puller { + /// Send one pull request and return its round. The request leaves now, and + /// the messages come back one by one as they arrive. + /// + /// With the connection down no request is sent: the round times out empty, + /// and the caller's loop keeps its pace and its stop flag until the + /// connection is back, when it picks up where it was. + pub fn round(&self) -> Result, String> { + if let Some(fault) = self.fault.borrow_mut().take() { + return Err(fault); + } + if self.link.up() { + let request = self.link.client.publish_with_reply( + self.next.clone(), + self.inbox.clone(), + Bytes::from_static(PULL_REQUEST), + ); + wait(sent(request))?; + } + Ok(Round { + puller: self, + taken: 0, + }) + } + + fn receive(&self, deadline: Instant) -> Option { + let mut sub = self.sub.borrow_mut(); + match sub.next().now_or_never() { + Some(next) => next, + None => wait_for(deadline - Instant::now(), sub.next()).flatten(), + } + } +} + +impl Drop for Puller { + fn drop(&mut self) { + if let Some((stream, consumer)) = self.owned.take() { + if self.link.up() { + let _ = self.js.delete_consumer(&stream, &consumer); + } + } + } +} + +/// The messages of one pull request: it ends with the batch, on the server's own +/// status, or at the backstop. With a request in flight behind it, a status may +/// be that request's rather than this one's: both mean "nothing more for now". +pub struct Round<'a> { + puller: &'a Puller, + taken: usize, +} + +impl Iterator for Round<'_> { + type Item = Message; + + fn next(&mut self) -> Option { + let deadline = Instant::now() + PULL_WAIT; + while self.taken < PULL_BATCH { + let message = self.puller.receive(deadline)?; + match classify(&message) { + Pulled::Message => { + self.taken += 1; + return Some(Message::new(message, self.puller.link.clone())); + } + Pulled::Nothing => {} + Pulled::Over => return None, + Pulled::Gone(why) => { + *self.puller.fault.borrow_mut() = Some(why); + return None; + } + } + } + None + } +} + +enum Pulled { + Message, + /// 404: a request came back empty; keep waiting for the rest of the round. + Nothing, + /// 408, or a 409 that leaves the consumer standing: this round is over. + Over, + /// The consumer or JetStream is not there any more. + Gone(String), +} + +fn classify(message: &async_nats::Message) -> Pulled { + let description = message.description.as_deref().unwrap_or(""); + match message.status { + None | Some(StatusCode::OK) => Pulled::Message, + Some(StatusCode::NOT_FOUND) => Pulled::Nothing, + Some(StatusCode::TIMEOUT) => Pulled::Over, + Some(StatusCode::REQUEST_TERMINATED) if !description.contains("Deleted") => Pulled::Over, + Some(status) => Pulled::Gone(format!("consumer gone: {status} {description}")), + } +} + +// ───────────────────────── JetStream ───────────────────────── + +/// The persisted side of a `Bus`. +#[derive(Clone)] +pub struct JetStream(Bus); + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum Discard { + #[default] + Old, + New, +} + +/// What a stream is created with. The fields not named here keep the server's +/// defaults, and `add_stream` sends these and only these, so a stream comes out +/// the same as one made by any earlier release of the runtime. +#[derive(Clone, Default)] +pub struct StreamConfig { + pub name: String, + pub subjects: Vec, + pub max_msgs: i64, + pub max_msgs_per_subject: i64, + pub max_age: Duration, + pub discard: Discard, + pub allow_rollup: bool, + pub deny_delete: bool, +} + +/// A key-value bucket. The stream behind it is the one the sync client made, +/// `allow_direct` off included, so buckets of earlier releases and new ones are +/// the same thing. +#[derive(Clone, Default)] +pub struct KvConfig { + pub bucket: String, + pub history: i64, + pub max_age: Duration, +} + +/// A consumer that is not kept: named by the server, deleted with its `Puller`, +/// reaped by the server if the process dies first. +pub struct EphemeralConfig<'a> { + pub filter: &'a str, + /// Start at this stream sequence; `None` follows the tail. + pub start_seq: Option, + pub inactive_threshold: Duration, +} + +#[derive(Clone, Copy, Debug)] +pub struct StreamState { + pub first_seq: u64, + pub last_seq: u64, + pub messages: u64, +} + +pub struct StoredMessage { + pub sequence: u64, + pub data: Bytes, +} + +impl JetStream { + fn shared(&self) -> &Shared { + &(self.0).0 + } + + /// The JetStream API, as the server speaks it: a request and a JSON reply. + /// Administration goes this way rather than through the client's typed + /// configs because each of those is free to carry defaults of its own, and + /// an in-place upgrade must find the same streams and consumers it left. + fn api(&self, verb: String, body: Bytes) -> Result { + let client = &self.shared().link.client; + let reply = + wait(client.request(format!("$JS.API.{verb}"), body)).map_err(|e| e.to_string())?; + let answer: Value = + serde_json::from_slice(&reply.payload).map_err(|e| format!("unreadable reply: {e}"))?; + match answer.get("error") { + None => Ok(answer), + Some(error) => { + let what = error + .get("description") + .and_then(Value::as_str) + .unwrap_or("error"); + Err(what.to_string()) + } + } + } + + /// Publish and wait for the stream to say it stored the message. Ok means + /// persisted; the stream sequence comes back. + pub fn publish(&self, subject: &str, payload: impl AsRef<[u8]>) -> Result { + let bytes = Bytes::copy_from_slice(payload.as_ref()); + wait(async { + let ack = self.shared().js.publish(subject.to_string(), bytes).await; + let ack = ack.map_err(|e| e.to_string())?.await; + ack.map(|ack| ack.sequence).map_err(|e| e.to_string()) + }) + } + + /// Create a stream, or leave it as it is: one that exists is never changed, + /// and a different config on it is an error the caller may ignore. + pub fn add_stream(&self, config: &StreamConfig) -> Result<(), String> { + let discard = match config.discard { + Discard::Old => "old", + Discard::New => "new", + }; + let mut body = json!({ + "name": config.name, + "subjects": config.subjects, + "max_msgs": config.max_msgs, + "max_msgs_per_subject": config.max_msgs_per_subject, + "max_age": config.max_age.as_nanos() as u64, + "discard": discard, + "retention": "limits", + "storage": "file", + }); + if config.allow_rollup { + body["allow_rollup_hdrs"] = json!(true); + } + if config.deny_delete { + body["deny_delete"] = json!(true); + } + self.api( + format!("STREAM.CREATE.{}", config.name), + body.to_string().into(), + ) + .map(|_| ()) + } + + #[cfg(test)] + pub fn delete_stream(&self, stream: &str) -> Result<(), String> { + self.api(format!("STREAM.DELETE.{stream}"), Bytes::new()) + .map(|_| ()) + } + + pub fn stream_state(&self, stream: &str) -> Result { + let info = self.api(format!("STREAM.INFO.{stream}"), Bytes::new())?; + let seq = |field: &str| info["state"][field].as_u64().unwrap_or(0); + Ok(StreamState { + first_seq: seq("first_seq"), + last_seq: seq("last_seq"), + messages: seq("messages"), + }) + } + + /// The message at a stream sequence. + pub fn get_message(&self, stream: &str, sequence: u64) -> Result { + let js = &self.shared().js; + let message = wait(async { + let stream = js + .get_stream_no_info(stream) + .await + .map_err(|e| e.to_string())?; + stream + .get_raw_message(sequence) + .await + .map_err(|e| e.to_string()) + })?; + Ok(StoredMessage { + sequence: message.sequence, + data: message.payload, + }) + } + + pub fn delete_message(&self, stream: &str, sequence: u64) -> Result<(), String> { + let body = json!({"seq": sequence}).to_string(); + self.api(format!("STREAM.MSG.DELETE.{stream}"), body.into()) + .map(|_| ()) + } + + /// Empty a stream; how many messages went comes back. + pub fn purge_stream(&self, stream: &str) -> Result { + let reply = self.api(format!("STREAM.PURGE.{stream}"), Bytes::new())?; + Ok(reply["purged"].as_u64().unwrap_or(0)) + } + + /// Create (or verify) a durable pull consumer that delivers every one of + /// `filters` in stream order, from the beginning or, with `start_seq`, from + /// that stream sequence on. Idempotent: the same request on an existing + /// consumer is accepted, and 2.10 updates the filters of an existing one in + /// place (its ack floor stays). The reply's filters must be exactly the ones + /// asked for: a server that ignored `filter_subjects` would deliver the + /// whole stream. + pub fn create_pull_consumer( + &self, + stream: &str, + durable: &str, + filters: &[String], + ack_wait: Duration, + start_seq: Option, + ) -> Result<(), String> { + let mut config = json!({ + "durable_name": durable, + "ack_policy": "explicit", + "ack_wait": ack_wait.as_nanos() as u64, + "deliver_policy": "all", + "replay_policy": "instant", + }); + // Resuming after a snapshot: start at the sequence after it. The server + // cannot change an existing consumer's deliver policy, so the caller + // deletes the durable first. + if let Some(seq) = start_seq { + config["deliver_policy"] = Value::from("by_start_sequence"); + config["opt_start_seq"] = Value::from(seq); + } + match filters { + [one] => config["filter_subject"] = Value::String(one.clone()), + many => config["filter_subjects"] = Value::from(many.to_vec()), + } + let body = json!({ "stream_name": stream, "config": config }).to_string(); + let reply = self + .api( + format!("CONSUMER.DURABLE.CREATE.{stream}.{durable}"), + body.into(), + ) + .map_err(|e| format!("consumer {durable}: {e}"))?; + let mut kept = filters_of(&reply["config"]); + let mut asked = filters.to_vec(); + kept.sort(); + asked.sort(); + if kept != asked { + return Err(format!( + "consumer {durable}: the server kept the filters {kept:?}, not {asked:?} \ + (several `.from()` subjects need NATS 2.10 or newer)" + )); + } + Ok(()) + } + + pub fn delete_consumer(&self, stream: &str, consumer: &str) -> Result<(), String> { + self.api(format!("CONSUMER.DELETE.{stream}.{consumer}"), Bytes::new()) + .map(|_| ()) + } + + /// The stream sequence below which a consumer has acked everything. + pub fn ack_floor(&self, stream: &str, consumer: &str) -> Result { + let info = self.api(format!("CONSUMER.INFO.{stream}.{consumer}"), Bytes::new())?; + Ok(info["ack_floor"]["stream_seq"].as_u64().unwrap_or(0)) + } + + /// Bind to a durable consumer that must exist. With `filter`, it must also + /// be the one consuming that subject. + pub fn pull_durable( + &self, + stream: &str, + durable: &str, + filter: Option<&str>, + ) -> Result { + let info = self + .api(format!("CONSUMER.INFO.{stream}.{durable}"), Bytes::new()) + .map_err(|e| format!("provided durable consumer doesn't exist: {e}"))?; + if let Some(filter) = filter { + let filters = filters_of(&info["config"]); + if !filters.is_empty() && filters != [filter] { + return Err("subjects do not match".into()); + } + } + self.puller(stream, durable, None) + } + + /// A read-only cursor over a stream: a consumer of its own, never acked, so + /// nothing a live consumer does or sees changes. + pub fn pull_ephemeral(&self, stream: &str, config: &EphemeralConfig) -> Result { + let mut consumer = json!({ + "deliver_policy": "new", + "ack_policy": "explicit", + "filter_subject": config.filter, + "replay_policy": "instant", + "inactive_threshold": config.inactive_threshold.as_nanos() as u64, + }); + if let Some(seq) = config.start_seq { + consumer["deliver_policy"] = Value::from("by_start_sequence"); + consumer["opt_start_seq"] = Value::from(seq); + } + let body = json!({ "stream_name": stream, "config": consumer }).to_string(); + let info = self.api(format!("CONSUMER.CREATE.{stream}"), body.into())?; + let name = info["name"] + .as_str() + .ok_or("the server named no consumer")? + .to_string(); + self.puller(stream, &name, Some((stream.to_string(), name.clone()))) + } + + fn puller( + &self, + stream: &str, + consumer: &str, + owned: Option<(String, String)>, + ) -> Result { + let shared = self.shared(); + let inbox: Subject = shared.link.client.new_inbox().into(); + let sub = wait(sent(shared.link.client.subscribe(inbox.clone())))?; + Ok(Puller { + next: format!("$JS.API.CONSUMER.MSG.NEXT.{stream}.{consumer}").into(), + inbox, + sub: RefCell::new(sub), + fault: RefCell::new(None), + link: shared.link.clone(), + owned, + js: self.clone(), + }) + } + + /// Open a bucket, creating it if it is not there. None if JetStream KV is + /// unavailable. + pub fn key_value(&self, config: &KvConfig) -> Option { + let _ = self.add_stream(&StreamConfig { + name: format!("KV_{}", config.bucket), + subjects: vec![format!("$KV.{}.>", config.bucket)], + max_msgs_per_subject: config.history.max(1), + max_age: config.max_age, + discard: Discard::New, + allow_rollup: true, + deny_delete: true, + ..Default::default() + }); + let store = wait(self.shared().js.get_key_value(config.bucket.clone())).ok()?; + Some(Kv { store }) + } + + /// Open an object bucket, creating it if it is not there. None if JetStream + /// is unavailable. + pub fn object_store(&self, bucket: &str) -> Option { + let _ = self.add_stream(&StreamConfig { + name: format!("OBJ_{bucket}"), + subjects: vec![format!("$O.{bucket}.C.>"), format!("$O.{bucket}.M.>")], + discard: Discard::New, + allow_rollup: true, + ..Default::default() + }); + let store = wait(self.shared().js.get_object_store(bucket)).ok()?; + Some(Objects { store }) + } +} + +/// The subject filters of a consumer's config, however it holds them. +fn filters_of(config: &Value) -> Vec { + match config.get("filter_subjects").and_then(Value::as_array) { + Some(all) => all + .iter() + .filter_map(|x| x.as_str().map(str::to_string)) + .collect(), + None => config + .get("filter_subject") + .and_then(Value::as_str) + .map(|one| vec![one.to_string()]) + .unwrap_or_default(), + } +} + +// ───────────────────────── key-value ───────────────────────── + +/// A JetStream KV bucket. Revisions are stream sequences; `create` and `update` +/// are compare-and-set on them, which is what the singleton lease stands on +/// (ADR-0020). +#[derive(Clone)] +pub struct Kv { + store: kv::Store, +} + +impl Kv { + pub fn get(&self, key: &str) -> Result, String> { + wait(self.store.get(key)).map_err(|e| e.to_string()) + } + + pub fn put(&self, key: &str, value: impl AsRef<[u8]>) -> Result { + let value = Bytes::copy_from_slice(value.as_ref()); + wait(self.store.put(key, value)).map_err(|e| e.to_string()) + } + + /// Atomic create-if-absent: a deleted key counts as absent. The revision + /// comes back; a key that exists is an error. + pub fn create(&self, key: &str, value: impl AsRef<[u8]>) -> Result { + let value = Bytes::copy_from_slice(value.as_ref()); + wait(self.store.create(key, value)).map_err(|e| e.to_string()) + } + + /// Compare-and-set: fails unless `revision` is still the key's latest. + pub fn update(&self, key: &str, value: impl AsRef<[u8]>, revision: u64) -> Result { + let value = Bytes::copy_from_slice(value.as_ref()); + wait(self.store.update(key, value, revision)).map_err(|e| e.to_string()) + } + + pub fn delete(&self, key: &str) -> Result<(), String> { + wait(self.store.delete(key)).map_err(|e| e.to_string()) + } + + /// The keys that hold a value now. + pub fn keys(&self) -> Result, String> { + let collect = async { + let mut keys = self.store.keys().await.map_err(|e| e.to_string())?; + let mut all = Vec::new(); + while let Some(key) = keys.next().await { + all.push(key.map_err(|e| e.to_string())?); + } + Ok(all) + }; + wait_for(BARRIER_WAIT, collect).unwrap_or_else(|| Err("timed out listing keys".into())) + } + + /// Every change to the bucket, starting with its current keys. + pub fn watch_all(&self) -> Result { + let watch = wait(self.store.watch_with_history(">")).map_err(|e| e.to_string())?; + Ok(KvWatch { watch }) + } +} + +pub struct KvWatch { + watch: kv::Watch, +} + +impl KvWatch { + /// The next key that changed, `None` if nothing did within `within`, an + /// error when the watch has ended and must be opened again. + pub fn next_timeout(&mut self, within: Duration) -> Result, String> { + match wait_for(within, self.watch.next()) { + None => Ok(None), + Some(Some(Ok(entry))) => Ok(Some(entry.key)), + Some(Some(Err(e))) => Err(e.to_string()), + Some(None) => Err("the watch ended".into()), + } + } +} + +// ───────────────────────── objects ───────────────────────── + +/// A JetStream object bucket. +#[derive(Clone)] +pub struct Objects { + store: object_store::ObjectStore, +} + +impl Objects { + pub fn put(&self, name: &str, mut bytes: &[u8]) -> Result<(), String> { + wait(self.store.put(name, &mut bytes)) + .map(|_| ()) + .map_err(|e| e.to_string()) + } + + /// The object's bytes, or `None` if there is no such object. + pub fn get(&self, name: &str) -> Result>, String> { + use object_store::GetErrorKind; + use tokio::io::AsyncReadExt; + wait(async { + let mut object = match self.store.get(name).await { + Ok(object) => object, + Err(e) if e.kind() == GetErrorKind::NotFound => return Ok(None), + Err(e) => return Err(e.to_string()), + }; + let mut bytes = Vec::new(); + object + .read_to_end(&mut bytes) + .await + .map_err(|e| e.to_string())?; + Ok(Some(bytes)) + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn an_ack_subject_gives_its_delivery() { + let short = "$JS.ACK.VEJAS.flow_x.3.17.9.1793984542000000000.0"; + assert_eq!( + parse_ack_subject(short), + Some(Delivery { + delivered: 3, + stream_seq: 17 + }) + ); + let long = "$JS.ACK.dom.ACC.VEJAS.flow_x.2.41.9.1793984542000000000.5.token"; + assert_eq!( + parse_ack_subject(long), + Some(Delivery { + delivered: 2, + stream_seq: 41 + }) + ); + assert_eq!(parse_ack_subject("_INBOX.abc"), None); + assert_eq!(parse_ack_subject("$JS.ACK.too.few"), None); + } + + #[test] + fn the_pull_request_says_what_the_constants_say() { + let request: Value = serde_json::from_slice(PULL_REQUEST).unwrap(); + assert_eq!(request["batch"], PULL_BATCH); + assert_eq!(request["expires"], PULL_EXPIRES_MS * 1_000_000); + assert_eq!(request["no_wait"], true); + } + + #[test] + fn a_url_is_one_server_or_a_list() { + assert_eq!(servers_of("nats://127.0.0.1:4222").unwrap().len(), 1); + assert_eq!(servers_of("127.0.0.1:4222").unwrap().len(), 1); + assert_eq!(servers_of("nats://a:4222, nats://b:4222").unwrap().len(), 2); + assert!(servers_of("http://a:4222").is_err()); + } + + #[test] + fn statuses_end_a_round_or_a_consumer() { + let status = |code: StatusCode, description: &str| async_nats::Message { + subject: "_INBOX.x".into(), + reply: None, + payload: Bytes::new(), + headers: None, + status: Some(code), + description: Some(description.to_string()), + length: 0, + }; + let verdict = |m: async_nats::Message| match classify(&m) { + Pulled::Message => "message", + Pulled::Nothing => "nothing", + Pulled::Over => "over", + Pulled::Gone(_) => "gone", + }; + assert_eq!( + verdict(status(StatusCode::NOT_FOUND, "No Messages")), + "nothing" + ); + assert_eq!( + verdict(status(StatusCode::TIMEOUT, "Request Timeout")), + "over" + ); + assert_eq!( + verdict(status( + StatusCode::REQUEST_TERMINATED, + "Exceeded MaxWaiting" + )), + "over" + ); + assert_eq!( + verdict(status(StatusCode::REQUEST_TERMINATED, "Consumer Deleted")), + "gone" + ); + assert_eq!(verdict(status(StatusCode::NO_RESPONDERS, "")), "gone"); + } + + /// `wait` inside a task of the island would deadlock a worker; tokio refuses. + #[test] + fn blocking_inside_the_runtime_panics_instead_of_deadlocking() { + let task = runtime().spawn(async { wait(async {}) }); + assert!(wait(task).unwrap_err().is_panic()); + } +} + +/// Against a real server: each test starts a `nats-server` of its own, so that +/// it can also kill it. Excluded from CI like every test that needs one. Run: +/// `cargo test bus::live -- --ignored` +#[cfg(test)] +mod live { + use super::*; + use std::path::PathBuf; + use std::process::{Child, Command, Stdio}; + + struct Server { + child: Child, + port: u16, + store: PathBuf, + } + + impl Server { + fn start() -> Server { + Server::start_with(&[]) + } + + fn start_with(args: &[&str]) -> Server { + let port = std::net::TcpListener::bind("127.0.0.1:0") + .unwrap() + .local_addr() + .unwrap() + .port(); + let store = + std::env::temp_dir().join(format!("vejas-bus-{}-{port}", std::process::id())); + let child = Server::spawn(port, &store, args); + Server { child, port, store } + } + + fn spawn(port: u16, store: &PathBuf, args: &[&str]) -> Child { + let mut child = Command::new("nats-server") + .args(["-js", "-a", "127.0.0.1", "-p", &port.to_string()]) + .args(args) + .arg("-sd") + .arg(store) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .spawn() + .expect("nats-server on PATH"); + for _ in 0..100 { + if std::net::TcpStream::connect(("127.0.0.1", port)).is_ok() { + return child; + } + std::thread::sleep(Duration::from_millis(20)); + } + let _ = child.kill(); + let _ = child.wait(); + panic!("nats-server did not come up"); + } + + fn url(&self) -> String { + format!("nats://127.0.0.1:{}", self.port) + } + + fn restart(&mut self) { + self.child.kill().unwrap(); + self.child.wait().unwrap(); + self.child = Server::spawn(self.port, &self.store, &[]); + } + } + + impl Drop for Server { + fn drop(&mut self) { + let _ = self.child.kill(); + let _ = self.child.wait(); + let _ = std::fs::remove_dir_all(&self.store); + } + } + + fn until(what: &str, done: impl Fn() -> bool) { + let start = Instant::now(); + while !done() { + assert!( + start.elapsed() < Duration::from_secs(15), + "never happened: {what}" + ); + std::thread::sleep(Duration::from_millis(10)); + } + } + + #[test] + #[ignore] + fn the_barrier_comes_back_after_everything_published_before_it() { + let server = Server::start(); + let bus = connect(&server.url()).unwrap(); + let mut sub = bus.subscribe("t.barrier").unwrap(); + let mut batch = bus.batch(); + for i in 0..5000 { + batch.publish("t.barrier", format!("{i}")).unwrap(); + } + batch.confirm().unwrap(); + // The echo left the server after these: they are all here, unwaited. + for i in 0..5000 { + let message = sub.next_timeout(Duration::ZERO).unwrap(); + assert_eq!(&message.data[..], format!("{i}").as_bytes()); + } + } + + #[test] + #[ignore] + fn a_batch_that_spans_a_reconnect_fails_its_barrier() { + let mut server = Server::start(); + let bus = connect(&server.url()).unwrap(); + let link = &bus.0.link; + let epoch = link.epoch(); + bus.publish("t.lost", "published before the crash").unwrap(); + server.restart(); + until("the bus is back", || link.up() && link.epoch() != epoch); + // The echo comes back on the new connection, and proves nothing about + // the old one. + assert!(wait(link.barrier()).is_ok()); + assert!( + wait(link.barrier_since(epoch)).is_err(), + "that publish may be gone" + ); + // The next batch stands on the new connection. + let mut batch = bus.batch(); + batch.publish("t.lost", "published after").unwrap(); + batch.confirm().unwrap(); + } + + #[test] + #[ignore] + fn one_unsendable_emit_fails_alone_and_nothing_held_needs_no_barrier() { + let server = Server::start(); + let bus = connect(&server.url()).unwrap(); + let mut batch = bus.batch(); + batch.confirm().unwrap(); + assert!(batch.publish("two words", "x").is_err()); + assert!( + batch.publish("a\r\nPUB b 0", "x").is_err(), + "no protocol injection" + ); + assert!(batch.publish("", "x").is_err()); + assert!(batch.publish("t.big", vec![0u8; 2 << 20]).is_err()); + batch.publish("t.fine", "x").unwrap(); + batch.confirm().unwrap(); + } + + #[test] + #[ignore] + fn connect_refuses_while_the_connection_is_down_and_recovers() { + let mut server = Server::start(); + let url = server.url(); + let bus = connect(&url).unwrap(); + server.child.kill().unwrap(); + server.child.wait().unwrap(); + until("the bus to notice", || !bus.0.link.up()); + assert!(connect(&url).is_err()); + let mut batch = bus.batch(); + batch.publish("t.x", "x").unwrap(); + assert!(batch.confirm().is_err()); + server.child = Server::spawn(server.port, &server.store, &[]); + until("the bus to reconnect", || bus.0.link.up()); + assert!(connect(&url).is_ok()); + } + + #[test] + #[ignore] + fn a_durable_survives_a_restart_and_keeps_its_place() { + let mut server = Server::start(); + let js = connect(&server.url()).unwrap().jetstream(); + js.add_stream(&StreamConfig { + name: "T".into(), + subjects: vec!["t.>".into()], + ..Default::default() + }) + .unwrap(); + let filter = ["t.in".to_string()]; + js.create_pull_consumer("T", "d", &filter, Duration::ZERO, None) + .unwrap(); + let puller = js.pull_durable("T", "d", Some("t.in")).unwrap(); + assert!( + js.pull_durable("T", "d", Some("t.other")).is_err(), + "bound to its subject" + ); + js.publish("t.in", "one").unwrap(); + let first: Vec = puller.round().unwrap().collect(); + assert_eq!(first.len(), 1); + assert_eq!( + first[0].delivery().unwrap(), + Delivery { + delivered: 1, + stream_seq: 1 + } + ); + first[0].ack().unwrap(); + // Let the server take the ack before it goes down. + wait(connect(&server.url()).unwrap().0.link.barrier()).unwrap(); + server.restart(); + until("the bus is back", || connect(&server.url()).is_ok()); + js.publish("t.in", "two").unwrap(); + until("the next message", || { + puller + .round() + .unwrap() + .next() + .is_some_and(|m| &m.data[..] == b"two") + }); + } + + #[test] + #[ignore] + fn a_round_ends_at_the_pull_expiry_and_a_deleted_consumer_fails_the_next() { + let server = Server::start(); + let js = connect(&server.url()).unwrap().jetstream(); + js.add_stream(&StreamConfig { + name: "T".into(), + subjects: vec!["t.>".into()], + ..Default::default() + }) + .unwrap(); + let filter = ["t.in".to_string()]; + js.create_pull_consumer("T", "d", &filter, Duration::ZERO, None) + .unwrap(); + let puller = js.pull_durable("T", "d", None).unwrap(); + let t0 = Instant::now(); + assert_eq!(puller.round().unwrap().count(), 0); + let idle = t0.elapsed(); + assert!( + idle >= Duration::from_millis(400) && idle < Duration::from_millis(700), + "{idle:?}" + ); + // A deleted consumer is told to the request waiting on it, as a 409. + let waiting = puller.round().unwrap(); + std::thread::sleep(Duration::from_millis(50)); + js.delete_consumer("T", "d").unwrap(); + assert_eq!(waiting.count(), 0); + let error = puller.round().err().expect("the consumer is gone"); + assert!(error.contains("consumer gone"), "{error}"); + } + + #[test] + #[ignore] + fn a_lease_is_compare_and_set() { + let server = Server::start(); + let js = connect(&server.url()).unwrap().jetstream(); + let kv = js + .key_value(&KvConfig { + bucket: "T".into(), + history: 1, + max_age: Duration::from_secs(60), + }) + .unwrap(); + let rev = kv.create("lease", "a").unwrap(); + assert!(kv.create("lease", "b").is_err(), "exactly one wins"); + let renewed = kv.update("lease", "a", rev).unwrap(); + assert!( + kv.update("lease", "a", rev).is_err(), + "a stale revision is fenced" + ); + assert_eq!(kv.get("lease").unwrap().as_deref(), Some(&b"a"[..])); + kv.delete("lease").unwrap(); + assert_eq!(kv.get("lease").unwrap(), None); + assert!( + kv.update("lease", "a", renewed).is_err(), + "a released lease is not renewed" + ); + kv.create("lease", "b").unwrap(); + assert_eq!(kv.keys().unwrap(), vec!["lease".to_string()]); + } + + #[test] + #[ignore] + fn a_watch_opens_with_what_is_there_and_follows() { + let server = Server::start(); + let js = connect(&server.url()).unwrap().jetstream(); + let kv = js + .key_value(&KvConfig { + bucket: "T".into(), + history: 4, + ..Default::default() + }) + .unwrap(); + assert!(kv.keys().unwrap().is_empty()); + kv.put("before", "1").unwrap(); + let mut watch = kv.watch_all().unwrap(); + let wait = Duration::from_secs(2); + assert_eq!(watch.next_timeout(wait).unwrap().as_deref(), Some("before")); + kv.put("after", "2").unwrap(); + assert_eq!(watch.next_timeout(wait).unwrap().as_deref(), Some("after")); + assert_eq!(watch.next_timeout(Duration::from_millis(50)).unwrap(), None); + } + + #[test] + #[ignore] + fn a_watch_goes_on_after_a_restart_or_says_it_ended() { + let mut server = Server::start(); + let url = server.url(); + let js = connect(&url).unwrap().jetstream(); + let kv = js + .key_value(&KvConfig { + bucket: "T".into(), + history: 4, + ..Default::default() + }) + .unwrap(); + let mut watch = kv.watch_all().unwrap(); + server.restart(); + until("the bus is back", || connect(&url).is_ok()); + kv.put("after", "1").unwrap(); + let start = Instant::now(); + loop { + assert!( + start.elapsed() < Duration::from_secs(30), + "the watch went silent" + ); + match watch.next_timeout(Duration::from_millis(500)) { + Ok(Some(key)) => { + assert_eq!(key, "after"); + break; + } + Ok(None) => continue, + // `version_watch` opens it again from here, and reads what is there + Err(_) => watch = kv.watch_all().unwrap(), + } + } + } + + #[test] + #[ignore] + fn credentials_in_the_url_are_used() { + let server = Server::start_with(&["--user", "alice", "--pass", "s3cret"]); + let at = format!("127.0.0.1:{}", server.port); + assert!( + connect(&format!("nats://{at}")).is_err(), + "the server wants credentials" + ); + assert!(connect(&format!("nats://alice:wrong@{at}")).is_err()); + connect(&format!("nats://alice:s3cret@{at}")).unwrap(); + let server = Server::start_with(&["--auth", "t0ken"]); + connect(&format!("nats://t0ken@127.0.0.1:{}", server.port)).unwrap(); + } + + #[test] + #[ignore] + fn an_object_comes_back_whole_and_a_missing_one_is_none() { + let server = Server::start(); + let js = connect(&server.url()).unwrap().jetstream(); + let objects = js.object_store("T").unwrap(); + assert_eq!(objects.get("detect:x").unwrap(), None); + let big: Vec = (0..300_000u32).map(|i| i as u8).collect(); + objects.put("detect:x", &big).unwrap(); + assert_eq!(objects.get("detect:x").unwrap().as_deref(), Some(&big[..])); + objects.put("detect:x", b"smaller").unwrap(); + assert_eq!( + objects.get("detect:x").unwrap().as_deref(), + Some(&b"smaller"[..]) + ); + } +} diff --git a/core/src/cluster.rs b/core/src/cluster.rs index 96ee8dc..16adf7a 100644 --- a/core/src/cluster.rs +++ b/core/src/cluster.rs @@ -6,11 +6,11 @@ //! instance, or N instances produce N× the events. Such a driver acquires a //! **lease** in a JetStream KV bucket before it runs: //! -//! - acquire = `Store::create` (atomic create-if-absent — exactly one wins) -//! - renew = `Store::update(key, id, revision)` (compare-and-set; a paused +//! - acquire = `Kv::create` (atomic create-if-absent — exactly one wins) +//! - renew = `Kv::update(key, id, revision)` (compare-and-set; a paused //! leader that wakes with a stale revision fails and stands down — //! fencing, so two instances never *keep* running the same unit) -//! - release = `Store::delete` on graceful shutdown (instant handoff) +//! - release = `Kv::delete` on graceful shutdown (instant handoff) //! - failover = the bucket's `max_age` ages the value out after a crash, so a //! stand-by's `create` then succeeds (bounded by the TTL) //! @@ -21,6 +21,8 @@ use std::sync::{Arc, OnceLock}; use std::thread; use std::time::Duration; +use crate::bus; + const LEASE_BUCKET: &str = "VEJAS_LEASES"; /// This process's identity, stable for its lifetime. `VEJAS_INSTANCE` overrides @@ -75,22 +77,18 @@ pub fn lease_ttl() -> Duration { /// Open (or create) the shared lease bucket. Its own sibling root, like the DLQ /// and audit streams. Returns None if JetStream KV is unavailable — the caller /// then runs the driver unleased (single-instance behaviour, no worse than today). -pub fn open_lease_store(js: &nats::jetstream::JetStream) -> Option { - let cfg = nats::kv::Config { +pub fn open_lease_store(js: &bus::JetStream) -> Option { + js.key_value(&bus::KvConfig { bucket: LEASE_BUCKET.to_string(), max_age: lease_ttl(), history: 1, - ..Default::default() - }; - js.create_key_value(&cfg) - .or_else(|_| js.key_value(LEASE_BUCKET)) - .ok() + }) } /// Held while this instance owns a singleton's lease. Dropping it stops the /// renewal thread and releases the lease (instant handoff on shutdown). pub struct LeaseGuard { - store: nats::kv::Store, + store: bus::Kv, key: String, stop_renew: Arc, } @@ -112,7 +110,7 @@ impl Drop for LeaseGuard { /// and the supervisor re-enters standby. Returns None if `alive()` goes false /// before the lease is acquired (asked to stop while standing by). pub fn acquire_blocking( - store: &nats::kv::Store, + store: &bus::Kv, key: &str, running: Arc, alive: impl Fn() -> bool, diff --git a/core/src/connectors.rs b/core/src/connectors.rs index 2d2fde7..42c6551 100644 --- a/core/src/connectors.rs +++ b/core/src/connectors.rs @@ -16,13 +16,16 @@ use std::io::{Read, Write}; use std::net::{TcpListener, TcpStream}; +use std::collections::HashMap; use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; -use std::sync::Arc; +use std::sync::{Arc, Mutex, OnceLock}; use std::thread; use std::time::{Duration, Instant}; use serde_json::{Map, Value}; +use crate::bus; + // ───────────────────────── config & context ───────────────────────── /// A connector instance's config: the UPPERCASE literals of its manifest. @@ -89,23 +92,14 @@ impl Ctx { fn alive(&self) -> bool { self.running.load(Ordering::SeqCst) } - fn jetstream(&self) -> Result { - Ok(self.jetstream_and_conn()?.0) - } - /// Like `jetstream()`, but also returns the raw `Connection` (a cheap clone — - /// `Connection` is `Arc` inside) so a driver can use `publish_confirmed`, which - /// needs the connection to flush directly past the 5ms flusher floor. - fn jetstream_and_conn( - &self, - ) -> Result<(nats::jetstream::JetStream, nats::Connection), String> { - let nc = nats::connect(&self.nats_url).map_err(|e| e.to_string())?; - let js = nats::jetstream::new(nc.clone()); - let _ = js.add_stream(&nats::jetstream::StreamConfig { + fn jetstream(&self) -> Result { + let js = bus::connect(&self.nats_url)?.jetstream(); + let _ = js.add_stream(&bus::StreamConfig { name: self.stream.clone(), subjects: vec![format!("{}.>", self.subj_root)], ..Default::default() }); - Ok((js, nc)) + Ok(js) } /// Full subject from a config value that may be a bare suffix or a vx.* path. fn subject(&self, raw: &str) -> String { @@ -197,10 +191,7 @@ impl Driver for HttpIn { let max_connections = ctx.config.u64_or("MAX_CONNECTIONS", 512).max(1) as usize; let open = Arc::new(AtomicUsize::new(0)); let mut refused = 0u64; - // nc for publish_confirmed (direct flush past the 5ms flusher floor — - // measured: /ingest p50 5.2ms→0.77ms, no throughput regression at conc=32); - // _js keeps the stream ensured. - let (_js, nc) = ctx.jetstream_and_conn()?; + let js = ctx.jetstream()?; // Optional ALLOW: the subjects this unauthenticated surface may publish to. // Absent = any suffix (compat); set = only these (defence in depth, ADR-0029 // R7 F2 — recommended when the port is reachable by anyone but the operator). @@ -237,7 +228,7 @@ impl Driver for HttpIn { refuse(sock); continue; }; - let nc = nc.clone(); + let js = js.clone(); let root = ctx.subj_root.clone(); let name = ctx.name.clone(); let allow = allow.clone(); @@ -248,7 +239,7 @@ impl Driver for HttpIn { let spawned = thread::Builder::new().name(format!("{name}:conn")).spawn( move || { let _slot = slot; - handle_http_in(&name, &mut sock, &nc, &root, &allow) + handle_http_in(&name, &mut sock, &js, &root, &allow) }, ); if let Err(e) = spawned { @@ -314,7 +305,7 @@ const REQUEST_READ_LIMIT: Duration = Duration::from_secs(30); fn handle_http_in( name: &str, sock: &mut std::net::TcpStream, - nc: &nats::Connection, + js: &bus::JetStream, subj_root: &str, allow: &[String], ) { @@ -394,7 +385,7 @@ fn handle_http_in( ("403 Forbidden", format!("{{\"error\":\"subject {suffix:?} not permitted by ALLOW\"}}")) } else { let subject = format!("{subj_root}.{suffix}"); - match publish_confirmed(nc, &subject, body, Duration::from_secs(5)) { + match js.publish(&subject, body) { Ok(_) => { trace_pub(name, &subject, body); ("202 Accepted", format!("{{\"published\":\"{subject}\"}}")) @@ -544,12 +535,6 @@ impl Driver for HttpPoll { // ───────────────────────── pull rounds ───────────────────────── -/// One bounded pull round: up to 10 messages, and the server resolves the -/// request within PULL_EXPIRES_NS (messages or a 408 timeout status), so the -/// calling loop gets control back and can re-check its stop flag. A plain -/// `fetch()` parks forever on an idle subject: a stopped flow's thread stayed -/// wedged in `recv()`, reported "running", and processed one more message -/// before dying (the zombie-consumer trap caught while rehearsing the demo). /// Consumer ack-wait (redelivery latency): a message not acked within this window /// redelivers (at-least-once, ADR-0002). `VEJAS_ACK_WAIT_SECS` tunes it; unset (or /// ≤0) keeps `Duration::ZERO`, which JetStream reads as its 30s default — the @@ -566,141 +551,11 @@ pub fn ack_wait() -> Duration { .unwrap_or_default() } -/// The idle poll window (`expires`). One outstanding pull covers this whole span, -/// so a message arriving any time inside it is delivered at once — there is NO gap -/// between rounds where a message would sit unpolled. It is therefore two things at -/// once: the idle pull cadence (one pull per window when quiet) AND the worst-case -/// stop-response latency (the loop re-checks its `alive` flag each time a pull -/// returns). 500ms keeps idle chatter to ~2 pulls/s per consumer while leaving stop -/// far inside the transport suite's 3s bound (T5). -const PULL_EXPIRES_MS: u64 = 500; - -pub fn fetch_round(sub: &nats::jetstream::PullSubscription) -> Result, String> { - Ok(fetch_iter_with(sub, None)?.collect()) -} - -/// [`fetch_round`] for a hot loop: the pull request leaves NOW, and the -/// messages come back one by one as the reader thread hands them over. -/// -/// Two things a hot loop needs that a one-off scan does not. -/// -/// **The request leaves now.** A pull request is a publish, and the `nats` -/// 0.25 client only *buffers* a publish: its flusher thread writes the buffer -/// out, and it waits at least 5 ms between two writes (`MIN_FLUSH_BETWEEN` in -/// its `client.rs`, a never-done `TODO(dlc)`). So every round of the consumer -/// loop paid up to 5 ms before the server even saw the request. Measured on -/// the flow loop under a backlog: 64 messages per round, ~7 ms per round, -/// ~9 000 evt/s — with the interpreter itself at 10 µs per event. Flushing -/// the request in the calling thread (`Connection::flush` writes a PING and -/// flushes the writer directly, the same bypass `publish_confirmed` uses for -/// pub-acks) turns that wait into one round-trip: 13 500 evt/s. -/// -/// **Streaming, not collecting.** Collecting the whole batch into a `Vec` -/// before processing meant the reader thread decoded 64 messages while this -/// thread waited, then this thread processed 64 while the reader idled — -/// two costs added instead of overlapped. Iterating as messages arrive lets -/// both run at once. The batch boundary is unchanged: the caller still does -/// its publish-before-ack barrier once the iterator ends. -/// -/// One-off scans (replay hydration, the audit list, the shadow watch) keep -/// [`fetch_round`]: they pull a handful of rounds and a few milliseconds each -/// do not matter there. -pub fn fetch_iter_flushed<'a>( - sub: &'a nats::jetstream::PullSubscription, - nc: &nats::Connection, -) -> Result, String> { - fetch_iter_with(sub, Some(nc)) -} - -/// One pull round: the messages of one request, ending when the request is -/// exhausted (batch filled, or the server's own timeout status). -pub struct FetchRound<'a>(Box> + 'a>); - -impl Iterator for FetchRound<'_> { - type Item = nats::Message; - fn next(&mut self) -> Option { - self.0.next()?.ok() - } -} - -fn fetch_iter_with<'a>( - sub: &'a nats::jetstream::PullSubscription, - flush_on: Option<&nats::Connection>, -) -> Result, String> { - // A continuous long-poll. `no_wait` returns the instant messages are available - // (a single message is delivered at once — no batch-fill latency, and the - // benched high-rate drain still fills 64-batches in one round). When the subject - // is idle the pull is HELD server-side for `expires` and returns empty — so the - // loop paces itself on the server with no client sleep, and crucially no - // coverage gap: previously a 150ms idle sleep sat between rounds with no pull - // outstanding, and a message landing in that window waited it out (measured tail - // ~150ms — the connector-latency bug the MQTT bench surfaced). No pull parks in - // max_waiting: one is in flight at a time and each resolves within `expires`, so - // the anti-zombie invariant holds and stop is re-checked every round. - let expires_ns = (PULL_EXPIRES_MS * 1_000_000) as usize; - let iter = sub - .timeout_fetch( - nats::jetstream::BatchOptions { - batch: 64, - expires: Some(expires_ns), - no_wait: true, - }, - // client-side backstop: comfortably past the server's `expires` so the - // server's own empty/timeout status ends the round, not this timeout. - Duration::from_millis(PULL_EXPIRES_MS + 250), - ) - .map_err(|e| e.to_string())?; - // `timeout_fetch` has buffered the pull request; send it before waiting on it. - if let Some(nc) = flush_on { - nc.flush().map_err(|e| e.to_string())?; - } - Ok(FetchRound(Box::new(iter))) -} - -/// Publish to a JetStream subject and wait for the pub-ack, flushing the write -/// DIRECTLY in this thread rather than via the shared flusher. -/// -/// `nats` 0.25's flusher thread imposes a hard-coded 5ms floor between flushes -/// (`client.rs` `MIN_FLUSH_BETWEEN`, a never-done `TODO(dlc)`). Since a JetStream -/// pub-ack is a request/reply, EVERY sequential `js.publish` is capped at ~200/s by -/// that floor — the ceiling the MQTT loopback bench surfaced (js.publish p50 -/// ≈5.1ms, dead on the tick). `Connection::flush()` does a direct writer flush + -/// PING/PONG in the calling thread, bypassing the floor, which lifts sequential -/// publish to ~1-2k/s. This reimplements `request_with_headers_or_timeout` (nats -/// `lib.rs`) verbatim — inbox, buffered publish-with-reply, the **no-responders -/// guard** — and only adds the direct flush, so the at-least-once contract is -/// unchanged: a returned Ok still means the message was persisted (a real pub-ack -/// with a `stream`/`seq`), never a mis-read of a 503 or an error ack. -pub fn publish_confirmed( - nc: &nats::Connection, - subject: &str, - payload: &[u8], - wait: Duration, -) -> Result<(), String> { - let reply = nc.new_inbox(); - let sub = nc.subscribe(&reply).map_err(|e| e.to_string())?; - nc.publish_request(subject, &reply, payload) - .map_err(|e| e.to_string())?; - nc.flush().map_err(|e| e.to_string())?; // direct flush — bypass the 5ms floor - let msg = sub - .next_timeout(wait) - .map_err(|_| "timed out waiting for JetStream pub-ack".to_string())?; - if msg.is_no_responders() { - return Err("no JetStream responder for the subject (stream missing?)".into()); - } - // ApiResponse: an "error" key = rejected; a "stream" key = persisted. - let ack: serde_json::Value = - serde_json::from_slice(&msg.data).map_err(|e| format!("bad pub-ack: {e}"))?; - if let Some(err) = ack.get("error") { - return Err(format!("JetStream rejected the publish: {err}")); - } - if ack.get("stream").is_none() { - return Err(format!( - "unexpected pub-ack (no stream): {}", - String::from_utf8_lossy(&msg.data) - )); - } - Ok(()) +/// One whole pull round, collected: for one-off scans (replay hydration, the +/// audit list, the shadow watch). A hot loop iterates `Puller::round` instead, +/// so that it works on messages as they arrive. +pub fn fetch_round(sub: &bus::Puller) -> Result, String> { + Ok(sub.round()?.collect()) } /// Hydrate up to `n` recent REAL events for `subject` from the persisted stream, @@ -714,16 +569,13 @@ pub fn publish_confirmed( /// sequence rather than from the beginning: at most `window` messages are /// scanned, of which the last `n` matching `subject` are kept. pub fn hydrate_recent( - js: &nats::jetstream::JetStream, + js: &bus::JetStream, stream: &str, subject: &str, n: usize, ) -> Result, String> { let n = n.clamp(1, 5000); - let last = js - .stream_info(stream) - .map(|si| si.state.last_seq) - .unwrap_or(0); + let last = js.stream_state(stream).map(|state| state.last_seq).unwrap_or(0); if last == 0 { return Ok(Vec::new()); // empty stream: nothing persisted yet } @@ -731,24 +583,16 @@ pub fn hydrate_recent( // but keep it bounded (an operator action, not the hot path). let window = (n as u64).saturating_mul(8).max(256).min(200_000); let start = last.saturating_sub(window).max(1); - let cfg = nats::jetstream::ConsumerConfig { - deliver_policy: nats::jetstream::DeliverPolicy::ByStartSeq, - opt_start_seq: Some(start), - // Explicit but never acked: the consumer is ephemeral and discarded, so - // un-acked messages simply expire with it — nothing redelivers to the - // flow, nothing is consumed off the stream. - ack_policy: nats::jetstream::AckPolicy::Explicit, - filter_subject: subject.to_string(), + // Never acked: the consumer is ephemeral and discarded, so un-acked messages + // simply expire with it — nothing redelivers to the flow, nothing is consumed + // off the stream. + let cfg = bus::EphemeralConfig { + filter: subject, + start_seq: Some(start), // auto-reap the ephemeral consumer shortly after we stop pulling inactive_threshold: Duration::from_secs(30), - ..Default::default() }; - let sub = js - .pull_subscribe_with_options( - subject, - &nats::jetstream::PullSubscribeOptions::new().consumer_config(cfg), - ) - .map_err(|e| e.to_string())?; + let sub = js.pull_ephemeral(stream, &cfg)?; let mut ring: std::collections::VecDeque<(u64, String, serde_json::Value)> = std::collections::VecDeque::with_capacity(n); let mut scanned = 0u64; @@ -762,7 +606,7 @@ pub fn hydrate_recent( if let Ok(v) = serde_json::from_slice::(&m.data) { // the stream sequence is the intrinsic join key for time-travel / // canary diffs (ADR-0021) — no per-flow key to invent. - let seq = m.jetstream_message_info().map(|i| i.stream_seq).unwrap_or(0); + let seq = m.delivery().map(|i| i.stream_seq).unwrap_or(0); ring.push_back((seq, m.subject.clone(), v)); while ring.len() > n { ring.pop_front(); @@ -1010,7 +854,7 @@ impl Driver for OAuthPoll { e })?; let publish_page = - |parsed: &Value, logical: &str, js: &nats::jetstream::JetStream| -> Result<(), String> { + |parsed: &Value, logical: &str, js: &bus::JetStream| -> Result<(), String> { let envelope = serde_json::json!({ "endpoint": logical, "fetched_at": iso8601_utc(crate::now_secs()), @@ -1181,70 +1025,6 @@ pub fn subject_matches(pattern: &str, subject: &str) -> bool { } } -/// Create (or verify) a durable pull consumer that delivers every one of -/// `filters` in stream order, from the beginning or, with `start_seq`, from -/// that stream sequence on. The sync `nats` client knows one filter subject -/// per consumer; the JetStream API knows several since 2.10, so the request is -/// sent raw. Idempotent: the same request on an existing consumer is accepted, -/// and 2.10 updates the filters of an existing one in place (its ack floor -/// stays). The reply's filters must be exactly the ones asked for — a server -/// that ignored `filter_subjects` would deliver the whole stream. -pub fn create_pull_consumer( - nc: &nats::Connection, - stream: &str, - durable: &str, - filters: &[String], - ack_wait: Duration, - start_seq: Option, -) -> Result<(), String> { - let mut config = serde_json::json!({ - "durable_name": durable, - "ack_policy": "explicit", - "ack_wait": ack_wait.as_nanos() as u64, - "deliver_policy": "all", - "replay_policy": "instant", - }); - // Resuming after a snapshot: start at the sequence after it. The server - // cannot change an existing consumer's deliver policy, so the caller - // deletes the durable first. - if let Some(seq) = start_seq { - config["deliver_policy"] = Value::from("by_start_sequence"); - config["opt_start_seq"] = Value::from(seq); - } - match filters { - [one] => config["filter_subject"] = Value::String(one.clone()), - many => config["filter_subjects"] = Value::from(many.to_vec()), - } - let body = serde_json::json!({ "stream_name": stream, "config": config }).to_string(); - let reply = nc - .request(&format!("$JS.API.CONSUMER.DURABLE.CREATE.{stream}.{durable}"), body) - .map_err(|e| format!("consumer {durable}: {e}"))?; - let v: Value = serde_json::from_slice(&reply.data) - .map_err(|e| format!("consumer {durable}: unreadable reply: {e}"))?; - if let Some(err) = v.get("error") { - let what = err.get("description").and_then(Value::as_str).unwrap_or("error"); - return Err(format!("consumer {durable}: {what}")); - } - let mut kept: Vec = match v["config"].get("filter_subjects").and_then(Value::as_array) { - Some(a) => a.iter().filter_map(|x| x.as_str().map(str::to_string)).collect(), - None => v["config"] - .get("filter_subject") - .and_then(Value::as_str) - .map(|x| vec![x.to_string()]) - .unwrap_or_default(), - }; - let mut asked = filters.to_vec(); - kept.sort(); - asked.sort(); - if kept != asked { - return Err(format!( - "consumer {durable}: the server kept the filters {kept:?}, not {asked:?} \ - (several `.from()` subjects need NATS 2.10 or newer)" - )); - } - Ok(()) -} - // ───────────────────────── detect snapshots ───────────────────────── pub const DETECT_STATE_BUCKET: &str = "VEJAS_DETECT_STATE"; @@ -1262,21 +1042,13 @@ pub struct Snapshot { /// The object store the detect units keep their snapshots in: one object per /// unit, the latest only. Create-or-open, like every bucket in this repo. -pub fn open_state_store( - js: &nats::jetstream::JetStream, -) -> Option { - let cfg = nats::object_store::Config { - bucket: DETECT_STATE_BUCKET.to_string(), - ..Default::default() - }; - js.create_object_store(&cfg) - .or_else(|_| js.object_store(DETECT_STATE_BUCKET)) - .ok() +pub fn open_state_store(js: &bus::JetStream) -> Option { + js.object_store(DETECT_STATE_BUCKET) } /// Store a snapshot: one JSON header line, then the engine's bytes as they are. pub fn snapshot_put( - store: &nats::object_store::ObjectStore, + store: &bus::Objects, unit: &str, snap: &Snapshot, ) -> Result<(), String> { @@ -1290,34 +1062,14 @@ pub fn snapshot_put( .into_bytes(); bytes.push(b'\n'); bytes.extend_from_slice(&snap.state); - store - .put(unit, &mut bytes.as_slice()) - .map(|_| ()) - .map_err(|e| format!("snapshot put: {e}")) + store.put(unit, &bytes).map_err(|e| format!("snapshot put: {e}")) } /// The unit's snapshot, if it has one. -pub fn snapshot_get( - store: &nats::object_store::ObjectStore, - unit: &str, -) -> Result, String> { - let mut object = match store.get(unit) { - Ok(o) => o, - Err(e) => { - let text = e.to_string(); - if e.kind() == std::io::ErrorKind::NotFound - || text.contains("not found") - || text.contains("no message") - { - return Ok(None); - } - return Err(format!("snapshot get: {e}")); - } +pub fn snapshot_get(store: &bus::Objects, unit: &str) -> Result, String> { + let Some(bytes) = store.get(unit).map_err(|e| format!("snapshot get: {e}"))? else { + return Ok(None); }; - let mut bytes = Vec::new(); - object - .read_to_end(&mut bytes) - .map_err(|e| format!("snapshot read: {e}"))?; let nl = bytes .iter() .position(|b| *b == b'\n') @@ -1374,12 +1126,12 @@ pub const DLQ_MAX_MSGS: i64 = 100_000; /// Ensure the dead-letter stream exists (idempotent). Bounded + discard-oldest: /// a full DLQ is itself an operator signal, never a silent unbounded growth. -pub fn ensure_dlq_stream(js: &nats::jetstream::JetStream) { - let _ = js.add_stream(&nats::jetstream::StreamConfig { +pub fn ensure_dlq_stream(js: &bus::JetStream) { + let _ = js.add_stream(&bus::StreamConfig { name: DLQ_STREAM.to_string(), subjects: vec![format!("{DLQ_ROOT}.>")], max_msgs: DLQ_MAX_MSGS, - discard: nats::jetstream::DiscardPolicy::Old, + discard: bus::Discard::Old, ..Default::default() }); } @@ -1402,7 +1154,7 @@ fn epoch_secs() -> u64 { /// JetStream has stored it — the caller must ack the original only on Ok. #[allow(clippy::too_many_arguments)] pub fn to_dlq( - js: &nats::jetstream::JetStream, + js: &bus::JetStream, unit: &str, subject: &str, attempts: i64, @@ -1429,7 +1181,125 @@ pub fn to_dlq( })) .map_err(|e| e.to_string())?; let dlq_subject = format!("{DLQ_ROOT}.{}", dlq_unit_token(unit)); - js.publish(&dlq_subject, body).map(|_| ()).map_err(|e| e.to_string()) + js.publish(&dlq_subject, body).map_err(|e| e.to_string())?; + announce_dead_letter(js, unit, subject, attempts, version); + Ok(()) +} + +// ───────────────────────── dead-letter events (ADR-0015, Amendment A) ───────── +// +// Nothing used to tell anyone that a message died: the self-healing loop waited +// for someone to open a session. Each dead letter is now announced on +// `.dlq.events`, an ordinary subject, so waking a person or an agent is a +// flow or a sink, like every other notification here (ADR-0024). The event holds +// only what the runtime wrote itself, never the payload or the error text: a dead +// letter is untrusted content, and this is what wakes things up. A reader that +// needs the letter fetches it through `vejas_dlq`. + +/// Dead letters of one unit not announced yet, and the last one's coordinates. +#[derive(Default)] +struct DlqAnnouncement { + last_sent: u64, + count: u64, + subject: String, + attempts: i64, + version: String, +} + +static DLQ_ANNOUNCEMENTS: OnceLock>> = OnceLock::new(); +static DLQ_FLUSHER: AtomicBool = AtomicBool::new(false); + +fn dlq_announcements() -> &'static Mutex> { + DLQ_ANNOUNCEMENTS.get_or_init(|| Mutex::new(HashMap::new())) +} + +fn dlq_events_subject() -> String { + let root = std::env::var("VEJAS_SUBJECT_ROOT").unwrap_or_else(|_| "vx".into()); + format!("{root}.dlq.events") +} + +/// At most one event per unit per this many seconds, so that a burst of poison +/// wakes a reader once and not ten thousand times. 0 announces every letter. +fn dlq_event_gap() -> u64 { + std::env::var("VEJAS_DLQ_EVENT_SECS").ok().and_then(|v| v.parse().ok()).unwrap_or(60) +} + +/// The events due now. Each carries `count`, the dead letters it stands for, so +/// the window delays news without losing any. +fn take_due_dlq_events( + pending: &mut HashMap, + now: u64, + gap: u64, +) -> Vec { + pending + .iter_mut() + .filter(|(_, a)| a.count > 0 && now.saturating_sub(a.last_sent) >= gap) + .map(|(unit, a)| { + let event = serde_json::json!({ + "transition": "dead-lettered", "ts": now, "unit": unit, + "subject": a.subject, "attempts": a.attempts, "version": a.version, + "count": a.count, + }); + a.last_sent = now; + a.count = 0; + event + }) + .collect() +} + +/// Best-effort, like the proposal events: the letter is stored and the caller +/// goes on to ack, whatever happens to its announcement. +fn publish_dlq_event(js: &bus::JetStream, event: &Value) { + let body = serde_json::to_vec(event).unwrap_or_default(); + if let Err(e) = js.publish(&dlq_events_subject(), body) { + eprintln!("[vejas] dead-letter event not published: {e}"); + } +} + +fn announce_dead_letter( + js: &bus::JetStream, + unit: &str, + subject: &str, + attempts: i64, + version: &str, +) { + // A reader of these events that keeps failing dead-letters them in turn. If + // that were announced too, the reader would feed itself forever. + if subject == dlq_events_subject() { + return; + } + let gap = dlq_event_gap(); + let due = { + let mut pending = dlq_announcements().lock().unwrap(); + let a = pending.entry(unit.to_string()).or_default(); + a.count += 1; + a.subject = subject.to_string(); + a.attempts = attempts; + a.version = version.to_string(); + take_due_dlq_events(&mut pending, epoch_secs(), gap) + }; + for event in &due { + publish_dlq_event(js, event); + } + // What the window held back must still go out if no letter follows it. + if gap > 0 && !DLQ_FLUSHER.swap(true, Ordering::SeqCst) { + let js = js.clone(); + let flusher = thread::Builder::new().name("dlq-events".into()).spawn(move || { + while crate::RUNNING.load(Ordering::SeqCst) { + thread::sleep(Duration::from_secs(1)); + let due = { + let mut pending = dlq_announcements().lock().unwrap(); + take_due_dlq_events(&mut pending, epoch_secs(), gap) + }; + for event in &due { + publish_dlq_event(&js, event); + } + } + }); + if flusher.is_err() { + DLQ_FLUSHER.store(false, Ordering::SeqCst); + } + } } // ───────────────────────── exec-source offset resume (ADR-0011/0021) ───────────────────────── @@ -1443,18 +1313,15 @@ pub fn to_dlq( pub const OFFSET_BUCKET: &str = "VEJAS_OFFSETS"; -pub fn open_offset_store(js: &nats::jetstream::JetStream) -> Option { - let cfg = nats::kv::Config { +pub fn open_offset_store(js: &bus::JetStream) -> Option { + js.key_value(&bus::KvConfig { bucket: OFFSET_BUCKET.to_string(), history: 1, ..Default::default() - }; - js.create_key_value(&cfg) - .or_else(|_| js.key_value(OFFSET_BUCKET)) - .ok() + }) } -pub fn offset_get(store: &nats::kv::Store, key: &str) -> Option { +pub fn offset_get(store: &bus::Kv, key: &str) -> Option { store .get(key) .ok() @@ -1463,7 +1330,7 @@ pub fn offset_get(store: &nats::kv::Store, key: &str) -> Option { .filter(|s| !s.is_empty()) } -pub fn offset_put(store: &nats::kv::Store, key: &str, value: &str) { +pub fn offset_put(store: &bus::Kv, key: &str, value: &str) { let _ = store.put(key, value.as_bytes()); } @@ -1477,12 +1344,12 @@ pub const AUDIT_ROOT: &str = "vxaudit"; // `max_age` on the hot stream can touch it. pub const AUDIT_MAX_MSGS: i64 = 1_000_000; -pub fn ensure_audit_stream(js: &nats::jetstream::JetStream) { - let _ = js.add_stream(&nats::jetstream::StreamConfig { +pub fn ensure_audit_stream(js: &bus::JetStream) { + let _ = js.add_stream(&bus::StreamConfig { name: AUDIT_STREAM.to_string(), subjects: vec![format!("{AUDIT_ROOT}.>")], max_msgs: AUDIT_MAX_MSGS, - discard: nats::jetstream::DiscardPolicy::Old, + discard: bus::Discard::Old, ..Default::default() }); } @@ -1492,7 +1359,7 @@ pub fn ensure_audit_stream(js: &nats::jetstream::JetStream) { /// caller can trust the trail; the live-promote path logs but does not fail the /// promote if this errors (git remains the trail for committed changes). pub fn to_audit( - js: &nats::jetstream::JetStream, + js: &bus::JetStream, unit: &str, record: &serde_json::Value, ) -> Result<(), String> { @@ -1504,7 +1371,7 @@ pub fn to_audit( /// Recent audit records for a unit, oldest last (read-only, ephemeral consumer). pub fn audit_recent( - js: &nats::jetstream::JetStream, + js: &bus::JetStream, unit: &str, n: usize, ) -> Result, String> { @@ -1525,29 +1392,17 @@ fn run_sink( subject: &str, handler: impl Fn(&[u8]) -> Result, String>, ) -> Result<(), String> { - let (js, nc) = ctx.jetstream_and_conn()?; + let js = ctx.jetstream()?; let durable = ctx.name.replace([':', '.', '-'], "_"); - let _ = js.add_consumer( - &ctx.stream, - nats::jetstream::ConsumerConfig { - durable_name: Some(durable.clone()), - filter_subject: subject.to_string(), - ack_wait: ack_wait(), - ..Default::default() - }, - ); - let sub = js - .pull_subscribe_with_options( - subject, - &nats::jetstream::PullSubscribeOptions::new().durable_name(durable), - ) - .map_err(|e| e.to_string())?; + let filter = [subject.to_string()]; + let _ = js.create_pull_consumer(&ctx.stream, &durable, &filter, ack_wait(), None); + let sub = js.pull_durable(&ctx.stream, &durable, Some(subject))?; eprintln!("[{}] consuming {subject}", ctx.name); loop { if !ctx.alive() { return Ok(()); } - for msg in fetch_iter_flushed(&sub, &nc)? { + for msg in sub.round()? { if !ctx.alive() { // stopped mid-batch: leave the message un-acked, it redelivers return Ok(()); @@ -1575,7 +1430,7 @@ fn run_sink( }); // poison guard: give up after MAX_DELIVERIES attempts let delivered = - msg.jetstream_message_info().map(|i| i.delivered).unwrap_or(1); + msg.delivery().map(|i| i.delivered).unwrap_or(1); if delivered >= crate::MAX_DELIVERIES { // Poison: park it in the DLQ, then ack — publish before ack // so it is never lost (ADR-0015). On DLQ failure, nak. @@ -1597,7 +1452,7 @@ fn run_sink( Some(format!("{e} — DLQ publish failed: {de}")), vec![], preview, None, None, ); - let _ = msg.ack_kind(nats::jetstream::AckKind::Nak); + let _ = msg.nak(); } } } else { @@ -1605,7 +1460,7 @@ fn run_sink( crate::record_trace_full( &ctx.name, subject, false, Some(e), vec![], preview, None, None, ); - let _ = msg.ack_kind(nats::jetstream::AckKind::Nak); + let _ = msg.nak(); } } } @@ -1806,9 +1661,7 @@ impl Driver for ExecStreamSource { let subject = ctx.subject(&ctx.config.str("SUBJECT").ok_or("SUBJECT required")?); let env = ctx.config.env_vars(); let restart = ctx.config.u64_or("RESTART_SECS", 2).max(1); - // js for the offset KV store; nc for publish_confirmed (this streams stdout - // line-by-line sequentially — the exact ~200/s flusher-floor case). - let (js, nc) = ctx.jetstream_and_conn()?; + let js = ctx.jetstream()?; // Optional offset resume: when OFFSET_KV is set, hand the child the last // committed offset as $OFFSET at (re)start (e.g. `kcat -o $OFFSET`) and // commit each record's OFFSET_FIELD after publishing it. @@ -1817,8 +1670,8 @@ impl Driver for ExecStreamSource { let offset_start = ctx.config.str("OFFSET_START").unwrap_or_else(|| "end".into()); let offset_store = offset_kv.as_ref().and_then(|_| open_offset_store(&js)); // Offset commit CADENCE (keeps the KV put out of the per-record hot path). - // A KV put is itself a throttled request/reply, so committing per record - // re-caps the stream at ~200/s even with the publish path lifted. Commit + // A KV put is itself a request/reply, and committing per record would + // add a round-trip to every record. Commit // every OFFSET_COMMIT_MS (default 100ms) and once more when the child // drains/exits: publish-before-commit still holds (an offset is committed // only for a record already on the bus), the crash duplicate window just @@ -1890,7 +1743,7 @@ impl Driver for ExecStreamSource { continue; } if let Ok(rec) = serde_json::from_str::(line) { - match publish_confirmed(&nc, &subject, line.as_bytes(), Duration::from_secs(5)) { + match js.publish(&subject, line) { Ok(_) => { trace_pub(&ctx.name, &subject, line.as_bytes()); // publish-before-commit: advance the offset in @@ -2049,9 +1902,7 @@ impl Driver for MqttIn { let client_id = mqtt_client_id(ctx); let user = ctx.config.str("USERNAME"); let pass = ctx.config.str("PASSWORD"); - // nc for publish_confirmed (direct flush past the 5ms flusher floor — the - // MQTT-loopback ceiling); _js keeps the stream ensured. - let (_js, nc) = ctx.jetstream_and_conn()?; + let js = ctx.jetstream()?; while ctx.alive() { let attempt = (|| -> Result<(), String> { let mut client = mqtt_connect( @@ -2066,7 +1917,7 @@ impl Driver for MqttIn { // publish-before-ack: bus first (await JetStream pub-ack), // THEN PUBACK — a crash before the bus write re-delivers // from the broker (ADR-0025). - publish_confirmed(&nc, &subject, &payload, Duration::from_secs(5)) + js.publish(&subject, &payload) .map_err(|e| format!("bus publish: {e}"))?; trace_pub(&ctx.name, &subject, &payload); if pq > 0 { @@ -2175,7 +2026,7 @@ impl Driver for ExecRpc { )); } let env = ctx.config.env_vars(); - let nc = nats::connect(&ctx.nats_url).map_err(|e| e.to_string())?; + let nc = bus::connect(&ctx.nats_url)?; let mut child = Command::new("sh") .arg("-c") @@ -2197,9 +2048,7 @@ impl Driver for ExecRpc { // Queue group (not a plain subscribe) so N clustered instances share the // requests — exactly one answers each, load-balanced (ADR-0020). Each // instance keeps its own child process (e.g. its own SAP logon). - let sub = nc - .queue_subscribe(&subject, "vejas") - .map_err(|e| e.to_string())?; + let mut sub = nc.queue_subscribe(&subject, "vejas")?; loop { if !ctx.alive() { let _ = child.kill(); @@ -2208,7 +2057,7 @@ impl Driver for ExecRpc { match sub.next_timeout(Duration::from_millis(300)) { Ok(msg) => { // One request line in, one reply line out (serialized). - let mut req = msg.data.clone(); + let mut req = msg.data.to_vec(); req.push(b'\n'); if stdin.write_all(&req).is_err() { return Err("child stdin closed".into()); @@ -2391,6 +2240,41 @@ fn next_url(base: &str, body: &serde_json::Value, field: &str) -> Option #[cfg(test)] mod tests { + fn letter(pending: &mut HashMap, unit: &str) { + let a = pending.entry(unit.to_string()).or_default(); + a.count += 1; + a.subject = "vx.in".into(); + } + + #[test] + fn a_burst_of_dead_letters_is_one_event_that_counts_them() { + let mut pending = HashMap::new(); + letter(&mut pending, "flow:a"); + let first = take_due_dlq_events(&mut pending, 1000, 60); + assert_eq!(first.len(), 1, "the first letter is news at once"); + assert_eq!(first[0]["count"], 1); + for _ in 0..3 { + letter(&mut pending, "flow:a"); + } + assert!(take_due_dlq_events(&mut pending, 1030, 60).is_empty(), "inside the window"); + let held = take_due_dlq_events(&mut pending, 1060, 60); + assert_eq!(held[0]["count"], 3, "the window delays news, it loses none"); + assert!(take_due_dlq_events(&mut pending, 2000, 60).is_empty(), "nothing pending"); + } + + #[test] + fn units_are_windowed_apart_and_zero_means_every_letter() { + let mut pending = HashMap::new(); + letter(&mut pending, "flow:a"); + take_due_dlq_events(&mut pending, 1000, 60); + letter(&mut pending, "flow:a"); + letter(&mut pending, "flow:b"); + let due = take_due_dlq_events(&mut pending, 1001, 60); + assert_eq!(due.len(), 1); + assert_eq!(due[0]["unit"], "flow:b"); + assert_eq!(take_due_dlq_events(&mut pending, 1001, 0).len(), 1, "gap 0 lets flow:a through"); + } + use super::*; use serde_json::json; @@ -2486,51 +2370,40 @@ mod tests { assert!(Config::default().headers().is_empty()); } - /// Verifies `publish_confirmed` clears the nats flusher throttle AND keeps the - /// at-least-once contract: messages actually persist, and a subject with no - /// stream errors (never a false Ok). Needs a live NATS on 127.0.0.1:4222; - /// excluded from CI. Run: - /// `cargo test publish_confirmed_beats_throttle -- --ignored --nocapture` + /// Verifies a confirmed publish keeps the at-least-once contract: messages + /// actually persist, and a subject with no stream errors (never a false Ok). + /// Needs a live NATS on 127.0.0.1:4222; excluded from CI. Run: + /// `cargo test confirmed_publish -- --ignored --nocapture` #[test] #[ignore] - fn publish_confirmed_beats_throttle_and_persists() { + fn confirmed_publish_persists_or_errors() { let url = std::env::var("NATS_URL").unwrap_or_else(|_| "127.0.0.1:4222".into()); - let nc = nats::connect(&url).expect("connect"); - let js = nats::jetstream::new(nc.clone()); + let js = bus::connect(&url).expect("connect").jetstream(); let pid = std::process::id(); let stream = format!("PBFAST_{pid}"); let subject = format!("pbfast.{pid}.in"); let _ = js.delete_stream(&stream); - js.add_stream(nats::jetstream::StreamConfig { + js.add_stream(&bus::StreamConfig { name: stream.clone(), subjects: vec![format!("pbfast.{pid}.>")], ..Default::default() }) .expect("add_stream"); - const N: usize = 200; - // (A) baseline: throttled js.publish + const N: usize = 400; let t = Instant::now(); for _ in 0..N { js.publish(&subject, b"x").expect("js.publish"); } - let base = N as f64 / t.elapsed().as_secs_f64(); - // (B) direct-flush publish_confirmed - let t = Instant::now(); - for _ in 0..N { - publish_confirmed(&nc, &subject, b"x", Duration::from_secs(5)).expect("confirmed"); - } - let fast = N as f64 / t.elapsed().as_secs_f64(); - eprintln!("[pbfast] js.publish={base:.0}/s publish_confirmed={fast:.0}/s ({:.1}x)", fast / base); - assert!(fast > base * 2.0, "direct flush should clear the 5ms floor (got {fast:.0} vs {base:.0}/s)"); + eprintln!("[pbfast] js.publish={:.0}/s", N as f64 / t.elapsed().as_secs_f64()); - // (C) durability: all 2N messages actually landed on the stream - let msgs = js.stream_info(&stream).expect("info").state.messages; - assert!(msgs >= (2 * N) as u64, "every confirmed publish persisted (have {msgs})"); + // durability: all N messages actually landed on the stream + let msgs = js.stream_state(&stream).expect("info").messages; + assert!(msgs >= N as u64, "every confirmed publish persisted (have {msgs})"); - // (D) no-responders guard: a subject with no stream must ERROR, not false-Ok + // no-responders guard: a subject with no stream must ERROR, not false-Ok let orphan = format!("orphan.{pid}.no.stream"); - let r = publish_confirmed(&nc, &orphan, b"x", Duration::from_secs(1)); + let r = js.publish(&orphan, b"x"); assert!(r.is_err(), "publish to a stream-less subject must error, got {r:?}"); let _ = js.delete_stream(&stream); @@ -2544,33 +2417,22 @@ mod tests { #[ignore] fn fetch_round_latency_probe() { let url = std::env::var("NATS_URL").unwrap_or_else(|_| "127.0.0.1:4222".into()); - let nc = nats::connect(&url).expect("connect nats"); - let js = nats::jetstream::new(nc.clone()); + let nc = bus::connect(&url).expect("connect nats"); + let js = nc.jetstream(); let pid = std::process::id(); let stream = format!("PROBE_{pid}"); let subject = format!("probe.{pid}.in"); let _ = js.delete_stream(&stream); - js.add_stream(nats::jetstream::StreamConfig { + js.add_stream(&bus::StreamConfig { name: stream.clone(), subjects: vec![format!("probe.{pid}.>")], ..Default::default() }) .expect("add_stream"); let durable = format!("probe_dur_{pid}"); - let _ = js.add_consumer( - &stream, - nats::jetstream::ConsumerConfig { - durable_name: Some(durable.clone()), - filter_subject: subject.clone(), - ..Default::default() - }, - ); - let sub = js - .pull_subscribe_with_options( - &subject, - &nats::jetstream::PullSubscribeOptions::new().durable_name(durable), - ) - .expect("pull_subscribe"); + let filter = [subject.clone()]; + js.create_pull_consumer(&stream, &durable, &filter, Duration::ZERO, None).expect("consumer"); + let sub = js.pull_durable(&stream, &durable, Some(&subject)).expect("pull_subscribe"); // (A) idle: how long does an empty round block? let mut empties = Vec::new(); @@ -2583,9 +2445,8 @@ mod tests { eprintln!("[probe] empty-round ms: {empties:?}"); // (B) publish->receipt: a message lands at a SWEPT offset into an idle - // stretch (20..~300ms) so samples hit every phase of the poll cycle — most - // importantly the coverage GAP where no pull is outstanding (the current - // 150ms idle sleep). Detection latency = receipt - the publish offset. + // stretch (20..~300ms) so samples hit every phase of the poll cycle. + // Detection latency = receipt - the publish offset. let mut lat = Vec::new(); for i in 0..30u32 { let delay_ms = 20 + (i as u64 * 23) % 280; // sweep across the ~252ms cycle @@ -2593,8 +2454,7 @@ mod tests { let t0 = Instant::now(); let pubber = std::thread::spawn(move || { std::thread::sleep(Duration::from_millis(delay_ms)); - ncp.publish(&subj, payload.as_bytes()).expect("publish"); - ncp.flush().ok(); + ncp.publish(&subj, payload).expect("publish"); }); let detected = loop { let msgs = fetch_round(&sub).expect("fetch"); diff --git a/core/src/control.rs b/core/src/control.rs index 65f724e..321e7d9 100644 --- a/core/src/control.rs +++ b/core/src/control.rs @@ -53,8 +53,8 @@ fn supervise(registry: Registry, root: PathBuf, tenant: String) { return; } let attempt = (|| -> Result<(), String> { - let nc = nats::connect(&url).map_err(|e| e.to_string())?; - let sub = nc.subscribe(&cmd_subj).map_err(|e| e.to_string())?; + let nc = crate::bus::connect(&url)?; + let mut sub = nc.subscribe(&cmd_subj)?; eprintln!("[control] tenant {tenant}: serving {cmd_subj}, status every {}s", status_every.as_secs()); // first status leaves immediately: the console sees us on connect let mut last_status = Instant::now() - status_every; @@ -90,9 +90,7 @@ fn supervise(registry: Registry, root: PathBuf, tenant: String) { })) .unwrap_or_default(), ); - if msg.reply.is_some() { - let _ = msg.respond(serde_json::to_vec(&reply).unwrap_or_default()); - } + let _ = msg.respond(serde_json::to_vec(&reply).unwrap_or_default()); } Err(_) => {} // timeout: loop to re-check RUNNING / status schedule } diff --git a/core/src/main.rs b/core/src/main.rs index b056160..96770bd 100644 --- a/core/src/main.rs +++ b/core/src/main.rs @@ -24,6 +24,7 @@ // POST /flows/new agent CLI writes a VejasScript flow // POST /reload rescan; restart changed files (mtime) +mod bus; mod cluster; mod connectors; mod control; @@ -234,8 +235,7 @@ fn replay_literal( let hydrated = before.source.as_deref().and_then(|subject| { let url = env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".into()); let stream = env::var("VEJAS_STREAM").unwrap_or_else(|_| "VEJAS".into()); - let nc = nats::connect(&url).ok()?; - let js = nats::jetstream::new(nc); + let js = bus::connect(&url).ok()?.jetstream(); connectors::hydrate_recent(&js, &stream, subject, n).ok() }); match hydrated { @@ -313,8 +313,7 @@ fn time_travel(root: &Path, file: &str, candidate_src: &str, n: usize) -> Result let pkg = pkg_of_path(&path); let url = env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".into()); let stream = env::var("VEJAS_STREAM").unwrap_or_else(|_| "VEJAS".into()); - let nc = nats::connect(&url).map_err(|e| format!("NATS unreachable: {e}"))?; - let js = nats::jetstream::new(nc); + let js = bus::connect(&url).map_err(|e| format!("NATS unreachable: {e}"))?.jetstream(); // diff against the CURRENT EFFECTIVE version (a live overlay if one is valid) let store = versions::open_store(&js); let (current_src, _) = versions::resolve_source(store.as_ref(), &unit, &baseline, now_secs()); @@ -401,8 +400,7 @@ fn canary_start(root: &Path, file: &str, candidate_src: &str) -> Result Result s, Err(_) => return, }; @@ -476,7 +471,7 @@ fn canary_start(root: &Path, file: &str, candidate_src: &str) -> Result(&m.data) else { continue; }; - let seq = m.jetstream_message_info().map(|i| i.stream_seq).unwrap_or(0); + let seq = m.delivery().map(|i| i.stream_seq).unwrap_or(0); let b = shadow_emits(¤t, &event, &root, &pkg); let a = shadow_emits(&candidate, &event, &root, &pkg); let ch = b != a; @@ -551,8 +546,7 @@ fn promote_version( let baseline = fs::read_to_string(&path).map_err(|e| e.to_string())?; let unit = flow_proc_name(&path); let url = env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".into()); - let nc = nats::connect(&url).map_err(|e| format!("NATS unreachable: {e}"))?; - let js = nats::jetstream::new(nc); + let js = bus::connect(&url).map_err(|e| format!("NATS unreachable: {e}"))?.jetstream(); let store = versions::open_store(&js).ok_or("version store unavailable")?; // current effective source: a prior overlay if valid, else the baseline let (current, _) = versions::resolve_source(Some(&store), &unit, &baseline, now_secs()); @@ -571,7 +565,7 @@ fn promote_version( /// The current effective-source hash of a flow file — the baseline a proposal is /// pinned to (the overlay-or-baseline the evidence saw, ADR-0021/0024). fn effective_source_hash( - js: &nats::jetstream::JetStream, + js: &bus::JetStream, root: &Path, file: &str, ) -> Result { @@ -585,7 +579,7 @@ fn effective_source_hash( /// Emit a proposal transition on the bus so a notification flow can route it /// (Slack/PagerDuty/email) — the platform dogfoods its own connectors (ADR-0024). -fn emit_proposal_event(js: &nats::jetstream::JetStream, transition: &str, proposal: &Value) { +fn emit_proposal_event(js: &bus::JetStream, transition: &str, proposal: &Value) { let subj_root = env::var("VEJAS_SUBJECT_ROOT").unwrap_or_else(|_| "vx".into()); let subject = format!("{subj_root}.proposals.events"); let ev = json!({"transition": transition, "ts": now_secs(), "proposal": proposal}); @@ -690,8 +684,7 @@ fn submit_proposal( } let file = payload["file"].as_str().ok_or("payload.file required")?.to_string(); let url = env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".into()); - let nc = nats::connect(&url).map_err(|e| format!("NATS unreachable: {e}"))?; - let js = nats::jetstream::new(nc); + let js = bus::connect(&url).map_err(|e| format!("NATS unreachable: {e}"))?.jetstream(); let baseline = effective_source_hash(&js, root, &file)?; let store = proposals::open_store(&js).ok_or("proposal store unavailable")?; let id = proposals::new_id(now_secs()); @@ -705,8 +698,7 @@ fn submit_proposal( /// never shows an approvable change the evidence never saw (ADR-0024). fn reconcile_and_list(root: &Path) -> Result, String> { let url = env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".into()); - let nc = nats::connect(&url).map_err(|e| format!("NATS unreachable: {e}"))?; - let js = nats::jetstream::new(nc); + let js = bus::connect(&url).map_err(|e| format!("NATS unreachable: {e}"))?.jetstream(); let store = proposals::open_store(&js).ok_or("proposal store unavailable")?; let mut out = Vec::new(); for p in proposals::list(&store) { @@ -769,8 +761,7 @@ fn apply_approved( let baseline = fs::read_to_string(&path).map_err(|e| e.to_string())?; let unit = flow_proc_name(&path); let url = env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".into()); - let nc = nats::connect(&url).map_err(|e| e.to_string())?; - let js = nats::jetstream::new(nc); + let js = bus::connect(&url)?.jetstream(); if cluster::clustered() { let store = versions::open_store(&js).ok_or("version store unavailable")?; let parent = versions::hash_content(&baseline); @@ -799,8 +790,7 @@ fn decide_proposal( decided_by: &str, ) -> Result { let url = env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".into()); - let nc = nats::connect(&url).map_err(|e| format!("NATS unreachable: {e}"))?; - let js = nats::jetstream::new(nc); + let js = bus::connect(&url).map_err(|e| format!("NATS unreachable: {e}"))?.jetstream(); let store = proposals::open_store(&js).ok_or("proposal store unavailable")?; let proposal = proposals::get(&store, id).ok_or("proposal not found")?; if proposal["status"] != "pending" { @@ -911,14 +901,14 @@ fn record_promote( let Some(path) = guard_path(root, file) else { return }; let unit = flow_proc_name(&path); let url = env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".into()); - let Ok(nc) = nats::connect(&url) else { + let Ok(bus) = bus::connect(&url) else { eprintln!( "[vejas] audit: NATS unreachable — live promote of {name} not recorded \ (git remains the trail for committed changes)" ); return; }; - let js = nats::jetstream::new(nc); + let js = bus.jetstream(); let record = json!({ "ts": now_secs(), "actor": actor, "unit": unit, "file": file, "name": name, "key": key, @@ -944,8 +934,7 @@ fn rollback_literal( let path = guard_path(root, file).ok_or("path not allowed")?; let unit = flow_proc_name(&path); let url = env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".into()); - let nc = nats::connect(&url).map_err(|e| format!("NATS unreachable: {e}"))?; - let js = nats::jetstream::new(nc); + let js = bus::connect(&url).map_err(|e| format!("NATS unreachable: {e}"))?.jetstream(); let records = connectors::audit_recent(&js, &unit, 500)?; // newest first: the most recent promote of exactly this literal let target = records @@ -1116,7 +1105,7 @@ fn supervise_vjs(handle: Arc, root: PathBuf) { // file iff it forked from exactly this baseline; a stale overlay // (baseline advanced / flow git-deleted) is evicted here — git wins. let src = { - let vconn = nats::connect(&url).ok().map(nats::jetstream::new); + let vconn = bus::connect(&url).ok().map(|bus| bus.jetstream()); let vstore = vconn.as_ref().and_then(versions::open_store); let (s, outcome) = versions::resolve_source( vstore.as_ref(), @@ -1168,41 +1157,23 @@ fn supervise_vjs(handle: Arc, root: PathBuf) { } return Err("no `source` declaration".into()); }; - let nc = nats::connect(&url).map_err(|e| e.to_string())?; - // Acks travel on their own connection. An ack is a publish to the - // message's reply subject, and any connection may send it. On the - // consuming connection the 64 acks of a round sat in front of the - // next round's PING, and every PONG waited for the server to work - // through them (~30 µs each, ~2 ms a round, measured). On a - // connection of their own they cost the server the same and this - // loop nothing. If that connection drops, un-acked messages simply - // redeliver (at-least-once, ADR-0002). - let ack_nc = nats::connect(&url).map_err(|e| e.to_string())?; - let js = nats::jetstream::new(nc.clone()); - let _ = js.add_stream(&nats::jetstream::StreamConfig { + let nc = bus::connect(&url)?; + let js = nc.jetstream(); + let _ = js.add_stream(&bus::StreamConfig { name: stream.clone(), subjects: vec![format!("{subj_root}.>")], ..Default::default() }); let durable = handle.spec.name.replace([':', '.'], "_"); // create the durable pull consumer (idempotent-ish), then bind - let _ = js.add_consumer( + let _ = js.create_pull_consumer( &stream, - nats::jetstream::ConsumerConfig { - durable_name: Some(durable.clone()), - deliver_subject: None, - filter_subject: source.clone(), - ack_wait: connectors::ack_wait(), - ..Default::default() - }, + &durable, + std::slice::from_ref(&source), + connectors::ack_wait(), + None, ); - let sub = js - .pull_subscribe_with_options( - &source, - &nats::jetstream::PullSubscribeOptions::new() - .durable_name(durable), - ) - .map_err(|e| e.to_string())?; + let sub = js.pull_durable(&stream, &durable, Some(&source))?; set_state(&handle, |st| { st.status = "running".into(); st.started_at = Some(now_secs()); @@ -1225,24 +1196,25 @@ fn supervise_vjs(handle: Arc, root: PathBuf) { // (`PULL_EXPIRES_MS`) and its round comes back empty. A stop // leaves at most one request outstanding, whose messages redeliver // un-acked after `ack_wait` — the same as stopping mid-batch. - let mut round = connectors::fetch_iter_flushed(&sub, &nc)?; + let mut round = sub.round()?; + let mut batch = nc.batch(); loop { if !RUNNING.load(Ordering::SeqCst) || handle.stop.load(Ordering::SeqCst) { return Ok(()); } engine.invalidate(); // pick up live service edits let round_t0 = Instant::now(); - let next = connectors::fetch_iter_flushed(&sub, &nc)?; - // Messages are processed as the reader thread hands them over, not - // once the whole batch has landed, so the two threads overlap - // instead of taking turns (see `fetch_iter_flushed`). + let next = sub.round()?; + // Messages are processed as they arrive, not once the whole batch + // has landed, so reading and working overlap instead of taking + // turns. let fetch = &mut round; let mut fetched = 0usize; let mut first_at: Option = None; - // Emits are buffered (core publish) and confirmed by ONE flush per + // Emits are core publishes, held and confirmed by ONE barrier per // batch before their source messages are acked — one round-trip a // batch instead of one per emit, at-least-once preserved (ADR-0002). - let mut to_ack: Vec = Vec::new(); + let mut to_ack: Vec = Vec::new(); for msg in fetch { fetched += 1; first_at.get_or_insert_with(Instant::now); @@ -1259,7 +1231,7 @@ fn supervise_vjs(handle: Arc, root: PathBuf) { let err = format!("bad json: {e}"); eprintln!("[vejas] {}: {err}", handle.spec.name); let delivered = - msg.jetstream_message_info().map(|i| i.delivered).unwrap_or(1); + msg.delivery().map(|i| i.delivered).unwrap_or(1); let prev: String = String::from_utf8_lossy(&msg.data).chars().take(160).collect(); metrics::observe(&handle.spec.name, false, 0, 0.0); @@ -1273,7 +1245,7 @@ fn supervise_vjs(handle: Arc, root: PathBuf) { Err(de) => { record_trace(&handle.spec.name, &msg.subject, false, Some(format!("{err} — DLQ publish failed: {de}")), vec![], prev, None); - let _ = msg.ack_kind(nats::jetstream::AckKind::Nak); + let _ = msg.nak(); } } continue; @@ -1281,26 +1253,25 @@ fn supervise_vjs(handle: Arc, root: PathBuf) { }; let t0 = Instant::now(); let start_nanos = metrics::now_nanos(); - match vjs::run(&prog, &event, &mut engine) { + // Holding an emit is local, so a refusal is final for this message: + // it can never leave, however often it redelivers. That makes it + // a failed run, capped by MAX_DELIVERIES and dead-lettered. + let outcome = vjs::run(&prog, &event, &mut engine).and_then(|ctx| { + for (subj, payload) in &ctx.emits { + let bytes = serde_json::to_vec(payload).unwrap_or_default(); + // held; sent and confirmed once per batch below + batch.publish(subj, bytes).map_err(|e| format!("publish {subj}: {e}"))?; + } + Ok(ctx) + }); + match outcome { Ok(ctx) => { - let mut ok = true; - for (subj, payload) in &ctx.emits { - let bytes = serde_json::to_vec(payload).unwrap_or_default(); - // buffered core publish; flushed once per batch below - if let Err(e) = nc.publish(subj, bytes) { - eprintln!( - "[vejas] {}: publish {subj}: {e}", - handle.spec.name - ); - ok = false; - } - } let emits = ctx.emits.len() as u64; - metrics::observe(&handle.spec.name, ok, emits, t0.elapsed().as_secs_f64()); + metrics::observe(&handle.spec.name, true, emits, t0.elapsed().as_secs_f64()); metrics::span(metrics::Span { unit: handle.spec.name.clone(), subject: msg.subject.clone(), - ok, + ok: true, error: None, start_nanos, end_nanos: metrics::now_nanos(), @@ -1309,8 +1280,8 @@ fn supervise_vjs(handle: Arc, root: PathBuf) { record_trace( &handle.spec.name, &msg.subject, - ok, - (!ok).then(|| "publish failed (will redeliver)".to_string()), + true, + None, ctx.emits.iter().map(|(s, _)| s.clone()).collect(), // The preview is derived from the stored event when // /events is read; serialising it here cost every @@ -1318,9 +1289,7 @@ fn supervise_vjs(handle: Arc, root: PathBuf) { String::new(), Some(event), ); - if ok { - to_ack.push(msg); // acked after the batch flush - } + to_ack.push(msg); // acked after the batch flush } Err(e) => { eprintln!("[vejas] {}: {e}", handle.spec.name); @@ -1338,7 +1307,7 @@ fn supervise_vjs(handle: Arc, root: PathBuf) { // dead-lettered (ADR-0015) rather than dropped; publish // before ack, nak if the DLQ write fails. let delivered = - msg.jetstream_message_info().map(|i| i.delivered).unwrap_or(1); + msg.delivery().map(|i| i.delivered).unwrap_or(1); if delivered >= MAX_DELIVERIES { match connectors::to_dlq(&js, &handle.spec.name, &msg.subject, delivered, &e, &msg.data, &flow_version) { Ok(()) => { @@ -1352,7 +1321,7 @@ fn supervise_vjs(handle: Arc, root: PathBuf) { record_trace(&handle.spec.name, &msg.subject, false, Some(format!("{e} — DLQ publish failed: {de}")), vec![], String::new(), Some(event.clone())); - let _ = msg.ack_kind(nats::jetstream::AckKind::Nak); + let _ = msg.nak(); } } } else { @@ -1364,36 +1333,22 @@ fn supervise_vjs(handle: Arc, root: PathBuf) { } } } - // Publish-before-ack barrier for the batch: one flush confirms the - // server received every buffered emit, THEN ack the fully-published - // source messages. On flush failure, ack none — the batch - // redelivers (nothing is lost; at-least-once, ADR-0002). + // Publish-before-ack barrier for the batch: one round-trip confirms + // the server received every emit, THEN ack the fully-published + // source messages. On failure, ack none — the batch redelivers + // (nothing is lost; at-least-once, ADR-0002). metrics::observe_fetch(&handle.spec.name, fetched); let processed_at = Instant::now(); - if !to_ack.is_empty() { - match nc.flush() { - Ok(_) => { - for m in &to_ack { - match m.reply.as_deref() { - Some(reply) => { - if let Err(e) = ack_nc.publish(reply, b"") { - eprintln!( - "[vejas] {}: ack: {e} (redelivers)", - handle.spec.name - ); - } - } - None => { - let _ = m.ack(); - } - } - } + match batch.confirm() { + Ok(()) => { + if let Err(e) = nc.ack_all(&to_ack) { + eprintln!("[vejas] {}: ack: {e} (redelivers)", handle.spec.name); } - Err(e) => eprintln!( - "[vejas] {}: flush before ack failed: {e} (batch redelivers)", - handle.spec.name - ), } + Err(e) => eprintln!( + "[vejas] {}: flush before ack failed: {e} (batch redelivers)", + handle.spec.name + ), } round = next; let first = first_at.unwrap_or(processed_at); @@ -1503,10 +1458,9 @@ fn supervise_detect(handle: Arc, _root: PathBuf) { .next() .unwrap_or(&handle.spec.name) .to_string(); - let nc = nats::connect(&url).map_err(|e| e.to_string())?; - let ack_nc = nats::connect(&url).map_err(|e| e.to_string())?; - let js = nats::jetstream::new(nc.clone()); - let _ = js.add_stream(&nats::jetstream::StreamConfig { + let nc = bus::connect(&url)?; + let js = nc.jetstream(); + let _ = js.add_stream(&bus::StreamConfig { name: stream.clone(), subjects: vec![format!("{subj_root}.>")], ..Default::default() @@ -1521,7 +1475,7 @@ fn supervise_detect(handle: Arc, _root: PathBuf) { // this attempt, so a restart never leaves a stale reader behind. let attempt_stop = Arc::new(AtomicBool::new(false)); let _stop_reader = StopOnDrop(attempt_stop.clone()); - let (tx, rx) = std::sync::mpsc::channel::(); + let (tx, rx) = std::sync::mpsc::channel::(); let durable = handle.spec.name.replace([':', '.'], "_"); let filters: Vec = sources.iter().map(|(s, _)| s.clone()).collect(); // Snapshot and resume (ADR-0031 points 4-5), for a program with state @@ -1570,33 +1524,24 @@ fn supervise_detect(handle: Arc, _root: PathBuf) { Some(seq) => { // The server cannot move an existing consumer: re-create it after the snapshot. let _ = js.delete_consumer(&stream, &durable); - connectors::create_pull_consumer(&nc, &stream, &durable, &filters, connectors::ack_wait(), Some(seq))?; + js.create_pull_consumer(&stream, &durable, &filters, connectors::ack_wait(), Some(seq))?; metrics::observe_restore(&handle.spec.name, seq); } - None => connectors::create_pull_consumer(&nc, &stream, &durable, &filters, connectors::ack_wait(), None)?, + None => js.create_pull_consumer(&stream, &durable, &filters, connectors::ack_wait(), None)?, } // A first snapshot at the consumer's position, so a crash before the // first cadence still replays from here rather than resuming past // events the state never saw. if let (Some(store), None) = (&store, resume_seq) { - let floor = js - .consumer_info(&stream, &durable) - .map(|i| i.ack_floor.stream_seq) - .unwrap_or(0); + let floor = js.ack_floor(&stream, &durable).unwrap_or(0); if let Err(e) = snapshot_unit(store, &handle.spec.name, &version, &program, floor) { eprintln!("[vejas] {}: first snapshot: {e}", handle.spec.name); } } let sub = js - .pull_subscribe_with_options( - &filters[0], - &nats::jetstream::PullSubscribeOptions::new() - .durable_name(durable.clone()) - .bind_stream(stream.clone()), - ) + .pull_durable(&stream, &durable, None) .map_err(|e| format!("{durable}: {e}"))?; { - let reader_nc = nc.clone(); let stop = attempt_stop.clone(); let unit = handle.clone(); let name = handle.spec.name.clone(); @@ -1605,7 +1550,7 @@ fn supervise_detect(handle: Arc, _root: PathBuf) { && !unit.stop.load(Ordering::SeqCst) && !stop.load(Ordering::SeqCst) { - match connectors::fetch_iter_flushed(&sub, &reader_nc) { + match sub.round() { Ok(round) => { let mut n = 0usize; for msg in round { @@ -1642,6 +1587,7 @@ fn supervise_detect(handle: Arc, _root: PathBuf) { let mut retrying: std::collections::BTreeSet = std::collections::BTreeSet::new(); let mut acks_since_snapshot: u64 = 0; let mut last_snapshot = Instant::now(); + let mut outbox = nc.batch(); let result = (|| -> Result<(), String> { loop { if !RUNNING.load(Ordering::SeqCst) || handle.stop.load(Ordering::SeqCst) { @@ -1694,12 +1640,12 @@ fn supervise_detect(handle: Arc, _root: PathBuf) { Err(_) => break, } } - let mut to_ack: Vec<(nats::Message, u64)> = Vec::new(); + let mut to_ack: Vec<(bus::Message, u64)> = Vec::new(); for msg in batch { if !RUNNING.load(Ordering::SeqCst) || handle.stop.load(Ordering::SeqCst) { return Ok(()); // un-acked: redelivers to whoever holds the durable next } - let seq = msg.jetstream_message_info().map(|i| i.stream_seq).unwrap_or(0); + let seq = msg.delivery().map(|i| i.stream_seq).unwrap_or(0); // The consumer's filters are the `.from()` subjects, so one // matches; the first declared wins when two overlap. let Some(event_type) = sources @@ -1725,7 +1671,7 @@ fn supervise_detect(handle: Arc, _root: PathBuf) { let err = format!("bad payload: {e}"); eprintln!("[vejas] {}: {err}", handle.spec.name); let delivered = - msg.jetstream_message_info().map(|i| i.delivered).unwrap_or(1); + msg.delivery().map(|i| i.delivered).unwrap_or(1); metrics::observe(&handle.spec.name, false, 0, 0.0); match connectors::to_dlq(&js, &handle.spec.name, &msg.subject, delivered, &err, &msg.data, &version) { Ok(()) => { @@ -1734,7 +1680,7 @@ fn supervise_detect(handle: Arc, _root: PathBuf) { retrying.remove(&seq); } Err(_) => { - let _ = msg.ack_kind(nats::jetstream::AckKind::Nak); + let _ = msg.nak(); retrying.insert(seq); } } @@ -1744,7 +1690,7 @@ fn supervise_detect(handle: Arc, _root: PathBuf) { let mut subjects = Vec::with_capacity(emits.len()); for emit in &emits { let subject = emit_subject(emit, &subj_root, &stem); - if let Err(e) = nc.publish(&subject, emit.to_payload()) { + if let Err(e) = outbox.publish(&subject, emit.to_payload()) { eprintln!("[vejas] {}: publish {subject}: {e}", handle.spec.name); ok = false; } @@ -1769,7 +1715,7 @@ fn supervise_detect(handle: Arc, _root: PathBuf) { eprintln!("[vejas] {}: {e}", handle.spec.name); metrics::observe(&handle.spec.name, false, 0, t0.elapsed().as_secs_f64()); let delivered = - msg.jetstream_message_info().map(|i| i.delivered).unwrap_or(1); + msg.delivery().map(|i| i.delivered).unwrap_or(1); if delivered >= MAX_DELIVERIES { match connectors::to_dlq(&js, &handle.spec.name, &msg.subject, delivered, &e, &msg.data, &version) { Ok(()) => { @@ -1778,7 +1724,7 @@ fn supervise_detect(handle: Arc, _root: PathBuf) { retrying.remove(&seq); } Err(_) => { - let _ = msg.ack_kind(nats::jetstream::AckKind::Nak); + let _ = msg.nak(); retrying.insert(seq); } } @@ -1792,29 +1738,20 @@ fn supervise_detect(handle: Arc, _root: PathBuf) { } } let processed_at = Instant::now(); - // Publish-before-ack barrier, one flush per batch (ADR-0002). - if !to_ack.is_empty() { - match nc.flush() { - Ok(_) => { - for (m, seq) in &to_ack { - match m.reply.as_deref() { - Some(reply) => { - let _ = ack_nc.publish(reply, b""); - } - None => { - let _ = m.ack(); - } - } - retrying.remove(seq); - acked_max = acked_max.max(*seq); - } - acks_since_snapshot += to_ack.len() as u64; + // Publish-before-ack barrier, one round-trip per batch (ADR-0002). + match outbox.confirm() { + Ok(()) => { + let _ = nc.ack_all(to_ack.iter().map(|(m, _)| m)); + for (_, seq) in &to_ack { + retrying.remove(seq); + acked_max = acked_max.max(*seq); } - Err(e) => eprintln!( - "[vejas] {}: flush before ack failed: {e} (batch redelivers)", - handle.spec.name - ), + acks_since_snapshot += to_ack.len() as u64; } + Err(e) => eprintln!( + "[vejas] {}: flush before ack failed: {e} (batch redelivers)", + handle.spec.name + ), } metrics::observe_round( &handle.spec.name, @@ -1883,7 +1820,7 @@ fn program_is_stateful(program: &varpulis_engine::Program) -> bool { /// Take the unit's snapshot standing for stream sequence `seq` and store it. fn snapshot_unit( - store: &nats::object_store::ObjectStore, + store: &bus::Objects, unit: &str, version: &str, program: &varpulis_engine::Program, @@ -1930,10 +1867,10 @@ fn contained(attempt: impl FnOnce() -> Result<(), String>) -> Result<(), String> } /// Start a background loop of the runtime itself, which no supervisor watches. -/// `nats::connect` spawns threads and panics when the OS has none, and that used -/// to end the loop for good with nothing to show for it: a burst of connections -/// during boot took `version_watch`, and a promoted version no longer reached this -/// instance. The thread carries a name, and a panic starts the loop again. +/// A panic used to end such a loop for good with nothing to show for it: in v0.4.0 +/// a burst of connections during boot took `version_watch`, and a promoted version +/// no longer reached that instance. The thread carries a name, and a panic starts +/// the loop again. fn background(name: &'static str, body: impl Fn() + Send + 'static) { let spawned = thread::Builder::new().name(name.into()).spawn(move || { while RUNNING.load(Ordering::SeqCst) { @@ -2062,8 +1999,7 @@ fn supervise_connector(handle: Arc, root: PathBuf) { // graceful stop. If the lease is lost mid-run, the renewal thread clears // `running`, the driver returns, and the supervisor re-enters standby. let _lease = if cluster::is_singleton(driver.kind()) { - let nc = nats::connect(&url).map_err(|e| e.to_string())?; - let js = nats::jetstream::new(nc); + let js = bus::connect(&url)?.jetstream(); match cluster::open_lease_store(&js) { Some(store) => { let key = connectors::dlq_unit_token(&handle.spec.name); @@ -2129,31 +2065,32 @@ fn supervise_connector(handle: Arc, root: PathBuf) { fn version_watch(registry: Registry, root: PathBuf) { let url = env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".into()); while RUNNING.load(Ordering::SeqCst) { - let store = nats::connect(&url) + let store = bus::connect(&url) .ok() - .map(nats::jetstream::new) - .and_then(|js| versions::open_store(&js)); + .and_then(|bus| versions::open_store(&bus.jetstream())); let Some(store) = store else { thread::sleep(Duration::from_secs(2)); continue; }; - let watch = match store.watch_all() { + let mut watch = match store.watch_all() { Ok(w) => w, Err(_) => { thread::sleep(Duration::from_secs(2)); continue; } }; - for entry in watch { - if !RUNNING.load(Ordering::SeqCst) { - return; - } + while RUNNING.load(Ordering::SeqCst) { + let key = match watch.next_timeout(Duration::from_millis(500)) { + Ok(Some(key)) => key, + Ok(None) => continue, + Err(_) => break, + }; // map the KV key token back to a supervised flow unit and restart it // (it then re-resolves the overlay: apply on a promote, baseline on // an eviction/delete). let target = { let reg = registry.lock().unwrap(); - reg.keys().find(|n| versions::key_token(n) == entry.key).cloned() + reg.keys().find(|n| versions::key_token(n) == key).cloned() }; if let Some(name) = target { eprintln!("[vejas] version change for {name} → hot-reloading"); @@ -3056,19 +2993,18 @@ fn sap_rpc_request(root: &Path, op: &Value) -> Result { "no exec-rpc SAP connector found — add a manifest with driver \"exec-rpc\" and REQUEST_SUBJECT (see docs/examples/sap_rpc.vjs.example)", )?; let url = env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".into()); - let nc = nats::connect(&url).map_err(|e| e.to_string())?; + let nc = bus::connect(&url)?; let payload = serde_json::to_vec(op).map_err(|e| e.to_string())?; let msg = nc - .request_timeout(&subject, payload, Duration::from_secs(30)) + .request(&subject, payload, Duration::from_secs(30)) .map_err(|e| format!("SAP connector did not answer on {subject}: {e}"))?; serde_json::from_slice::(&msg.data).map_err(|e| e.to_string()) } // ───────────────────────── dead-letter queue (ADR-0015) ───────────────────────── -fn dlq_js() -> Result { +fn dlq_js() -> Result { let url = env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".into()); - let nc = nats::connect(&url).map_err(|e| e.to_string())?; - let js = nats::jetstream::new(nc); + let js = bus::connect(&url)?.jetstream(); connectors::ensure_dlq_stream(&js); Ok(js) } @@ -3076,10 +3012,10 @@ fn dlq_js() -> Result { /// List dead letters, newest first (each carries its stream `seq` for replay/purge). fn dlq_list(limit: usize) -> Result, String> { let js = dlq_js()?; - let info = js.stream_info(connectors::DLQ_STREAM).map_err(|e| e.to_string())?; - let (first, last) = (info.state.first_seq, info.state.last_seq); + let state = js.stream_state(connectors::DLQ_STREAM)?; + let (first, last) = (state.first_seq, state.last_seq); let mut out = Vec::new(); - if last == 0 || info.state.messages == 0 { + if last == 0 || state.messages == 0 { return Ok(out); } let mut seq = last; @@ -3103,7 +3039,7 @@ fn dlq_list(limit: usize) -> Result, String> { /// Record a replay or a purge: one summary per call, best-effort like the /// live-promote path. A purge in particular leaves no other trace. fn dlq_audit( - js: &nats::jetstream::JetStream, + js: &bus::JetStream, kind: &str, actor: &str, seq: Option, @@ -3134,7 +3070,7 @@ fn dlq_replay(seq: Option, unit: Option<&str>, actor: &str) -> Result, unit: Option<&str>, actor: &str) -> Result, unit: Option<&str>, actor: &str) -> Result { let js = dlq_js()?; if seq.is_none() && unit.is_none() { - let r = js.purge_stream(connectors::DLQ_STREAM).map_err(|e| e.to_string())?; - dlq_audit(&js, "dlq-purge", actor, None, None, r.purged); - return Ok(json!({"purged": r.purged})); + let purged = js.purge_stream(connectors::DLQ_STREAM)?; + dlq_audit(&js, "dlq-purge", actor, None, None, purged); + return Ok(json!({"purged": purged})); } let mut n = 0u64; if let Some(s) = seq { - js.delete_message(connectors::DLQ_STREAM, s).map_err(|e| e.to_string())?; + js.delete_message(connectors::DLQ_STREAM, s)?; n = 1; } else if let Some(u) = unit { for env in dlq_list(100_000)? { @@ -3416,11 +3352,12 @@ fn publish_emits(emits: &[(String, Value)]) { return; } let url = env::var("NATS_URL").unwrap_or_else(|_| "nats://127.0.0.1:4222".into()); - if let Ok(nc) = nats::connect(&url) { + if let Ok(nc) = bus::connect(&url) { + let mut batch = nc.batch(); for (subj, payload) in emits { - let _ = nc.publish(subj, serde_json::to_vec(payload).unwrap_or_default()); + let _ = batch.publish(subj, serde_json::to_vec(payload).unwrap_or_default()); } - let _ = nc.flush(); + let _ = batch.confirm(); } } diff --git a/core/src/proposals.rs b/core/src/proposals.rs index a516cc9..758a5aa 100644 --- a/core/src/proposals.rs +++ b/core/src/proposals.rs @@ -13,6 +13,8 @@ use std::sync::atomic::{AtomicU64, Ordering}; +use crate::bus; + const PROPOSALS_BUCKET: &str = "VEJAS_PROPOSALS"; const PROPOSALS_HISTORY: i64 = 64; // bounded live queue; audit is the durable record @@ -21,15 +23,12 @@ static SEQ: AtomicU64 = AtomicU64::new(1); /// Open (or create) the shared proposal bucket. None if JetStream KV is /// unavailable (proposals need the shared bus; a single dev instance without it /// simply has no queue). -pub fn open_store(js: &nats::jetstream::JetStream) -> Option { - let cfg = nats::kv::Config { +pub fn open_store(js: &bus::JetStream) -> Option { + js.key_value(&bus::KvConfig { bucket: PROPOSALS_BUCKET.to_string(), history: PROPOSALS_HISTORY, ..Default::default() - }; - js.create_key_value(&cfg) - .or_else(|_| js.key_value(PROPOSALS_BUCKET)) - .ok() + }) } /// A fresh proposal id — `ts` is the wall-clock second (stamped by the caller; @@ -41,7 +40,7 @@ pub fn new_id(ts: u64) -> String { /// Submit a proposal (status "pending"). Returns the stored record. #[allow(clippy::too_many_arguments)] pub fn submit( - store: &nats::kv::Store, + store: &bus::Kv, id: &str, kind: &str, payload: serde_json::Value, @@ -66,7 +65,7 @@ pub fn submit( Ok(record) } -pub fn get(store: &nats::kv::Store, id: &str) -> Option { +pub fn get(store: &bus::Kv, id: &str) -> Option { store .get(id) .ok() @@ -75,10 +74,10 @@ pub fn get(store: &nats::kv::Store, id: &str) -> Option { } /// Every proposal, newest first (by created_at). -pub fn list(store: &nats::kv::Store) -> Vec { +pub fn list(store: &bus::Kv) -> Vec { let mut all: Vec = store .keys() - .map(|keys| keys.filter_map(|k| get(store, &k)).collect()) + .map(|keys| keys.iter().filter_map(|k| get(store, k)).collect()) .unwrap_or_default(); all.sort_by_key(|p| std::cmp::Reverse(p["created_at"].as_u64().unwrap_or(0))); all @@ -87,7 +86,7 @@ pub fn list(store: &nats::kv::Store) -> Vec { /// Transition a proposal's status (and stamp who/when for a human decision). /// Returns the updated record, or an error if it is gone. pub fn set_status( - store: &nats::kv::Store, + store: &bus::Kv, id: &str, status: &str, decided_by: Option<&str>, diff --git a/core/src/versions.rs b/core/src/versions.rs index 6e4bc33..53da923 100644 --- a/core/src/versions.rs +++ b/core/src/versions.rs @@ -15,6 +15,8 @@ use std::sync::{Mutex, OnceLock}; +use crate::bus; + const VERSIONS_BUCKET: &str = "VEJAS_VERSIONS"; const VERSIONS_HISTORY: i64 = 64; // bounded rollback/audit history per flow (ADR-0021) @@ -44,19 +46,16 @@ fn key_of(flow: &str) -> String { /// Open (or create) the shared version bucket. None if JetStream KV is /// unavailable — the caller then uses the baseline file (single-instance, no /// overlay), no worse than before versioning existed. -pub fn open_store(js: &nats::jetstream::JetStream) -> Option { - let cfg = nats::kv::Config { +pub fn open_store(js: &bus::JetStream) -> Option { + js.key_value(&bus::KvConfig { bucket: VERSIONS_BUCKET.to_string(), history: VERSIONS_HISTORY, ..Default::default() - }; - js.create_key_value(&cfg) - .or_else(|_| js.key_value(VERSIONS_BUCKET)) - .ok() + }) } /// The current overlay for a flow, if any (the parsed snapshot value). -pub fn get_version(store: &nats::kv::Store, flow: &str) -> Option { +pub fn get_version(store: &bus::Kv, flow: &str) -> Option { store .get(&key_of(flow)) .ok() @@ -68,7 +67,7 @@ pub fn get_version(store: &nats::kv::Store, flow: &str) -> Option Vec { /// is evicted here — git wins, loudly (logged + noted for the panel; the caller /// also writes an ADR-0018 audit record on `Evicted`). Returns (source, outcome). pub fn resolve_source( - store: Option<&nats::kv::Store>, + store: Option<&bus::Kv>, flow: &str, baseline: &str, ts: u64, diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 65c65e5..c556b9f 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -31,11 +31,11 @@ docker-compose.yml two containers: nats + vejas The runtime is a single OS process. Inside it: -- **one supervisor thread per flow** — connects to NATS, creates a durable pull - consumer on the flow's `source`, and loops: fetch a bounded batch (the pull - carries a server-side expiry so a stopped flow actually stops), run each - event through the interpreter, publish the emits **before** acking - (at-least-once; a crash means redelivery, never a lost emit), and record the +- **one supervisor thread per flow** — creates a durable pull consumer on the + flow's `source`, and loops: fetch a bounded batch (the pull carries a + server-side expiry so a stopped flow actually stops), run each event through + the interpreter, publish the emits **before** acking (at-least-once; a crash + means redelivery, never a lost emit), and record the event in the in-memory trace ring (last 50 per flow, `/events`; the full event is kept internally as shadow-replay fuel). A message that still fails after 5 deliveries is dropped (acked) and the drop is traced — redelivery is @@ -48,6 +48,11 @@ The runtime is a single OS process. Inside it: endpoints, the editing endpoints, and the MCP endpoint. One request = one thread (so a slow `/flows/new` never blocks monitoring), at most `VEJAS_HTTP_MAX_INFLIGHT` at once; past it the request gets a 503. +- **the bus** (`core/src/bus.rs`, ADR-0035) — the only thread that speaks NATS. + One tokio worker runs the `async-nats` client on one connection, shared by the + whole process. Everything above stays synchronous: a thread that needs the bus + calls a blocking function and parks until it returns. No other module sees + tokio or a NATS type, and the interpreter has no access to the bus. A panic in a unit's supervisor is caught around each attempt and takes the path of any other failure: `restarting`, the error in `last_error`, backoff, another diff --git a/docs/COOKBOOK.md b/docs/COOKBOOK.md index 99786b9..6f6c08c 100644 --- a/docs/COOKBOOK.md +++ b/docs/COOKBOOK.md @@ -175,7 +175,7 @@ destructive, so competing instances never duplicate. > vault) and put every message on the bus as `vx.amqp.orders.in`. Like IBM MQ, a **first-class standalone binary** (`connectors/amqp`) — pure -Rust, synchronous, no tokio, and TLS through rustls (no OpenSSL anywhere). +Rust, a synchronous broker client, and TLS through rustls (no OpenSSL anywhere). The at-least-once contract is the platform's, spoken in AMQP: the source acks RabbitMQ only after the bus pub-ack; the sink acks the bus only after the publisher confirm. The agent fills the env recipe diff --git a/docs/SUBJECTS.md b/docs/SUBJECTS.md index 4bec851..68f90ae 100644 --- a/docs/SUBJECTS.md +++ b/docs/SUBJECTS.md @@ -7,7 +7,7 @@ Bundled connectors are **native Rust drivers** run from a declarative manifest (`connectors/.vjs`: `driver "..."` + literal config, editable in the panel, hot-addable). No subprocess, no Python. Drivers today: -- **http-in** (source:webhook) — `POST /ingest/` → `vx.`. Config: PORT. +- **http-in** (source:webhook) — `POST /ingest/` → `vx.`. Config: PORT, MAX_CONNECTIONS. - **timer** (source:interval) — emits PAYLOAD on SUBJECT every INTERVAL_SECS (an object payload gains a `ts` field, ISO 8601 UTC, when absent). - **http-poll** (source:poll) — GETs URL every INTERVAL_SECS → SUBJECT. Optional HEADERS; optional `ENVELOPE = true` publishes `{endpoint, fetched_at, body}` (like oauth-poll) so a stateless flow gets a `collected_at`. @@ -37,7 +37,7 @@ hot-addable). No subprocess, no Python. Drivers today: MQI FFI, `dlopen` of the MQ client at runtime — builds with no MQ present. Configured by env (`VEJAS_MQ_*`, see the recipe), not by a manifest. - **amqp source/sink** (standalone binary, `connectors/amqp`) — RabbitMQ / - AMQP 0-9-1 as a first-class citizen (ADR-0026): pure Rust, sync, no tokio, + AMQP 0-9-1 as a first-class citizen (ADR-0026): pure Rust, a sync broker client, TLS via rustls over amiquip's mio loop (no OpenSSL). Source acks AMQP only after the bus pub-ack; sink acks the bus only after the publisher confirm. Configured by env (`VEJAS_AMQP_*`, see the recipe). Certified against a @@ -65,3 +65,15 @@ for content changes). It lives OUTSIDE `vx.>` on purpose: control traffic is transient and must never be captured by a stream (a stream on the command subject would hijack request/reply with its pub-ack). Specification: [CONTROL.md](CONTROL.md), decision: ADR-0013. + +## Events the platform publishes + +Ordinary subjects under the same root, so a flow or a sink reacts to them like to +anything else. Both carry metadata the runtime wrote itself. + +- `vx.dlq.events`: a message reached the DLQ (ADR-0015 Amendment A). `{transition, + ts, unit, subject, attempts, version, count}`, at most one per unit per + `VEJAS_DLQ_EVENT_SECS`, with `count` for the letters it stands for. Never the + payload or the error text. +- `vx.proposals.events`: a proposal changed state (ADR-0024). `{transition, ts, + proposal}`. diff --git a/docs/adr/0015-dead-letter-queue.md b/docs/adr/0015-dead-letter-queue.md index 0db18f2..197fdf7 100644 --- a/docs/adr/0015-dead-letter-queue.md +++ b/docs/adr/0015-dead-letter-queue.md @@ -83,6 +83,19 @@ The operator loop is: a dead letter shows the failure → correct the literal permanently unpublishable loops without a delivery cap. Out of scope here (it is a bus/config fault, not a poison payload), noted for a later pass. +**Amendment A (2026-11-06, normative):** a dead letter announces itself. Nothing +told anyone a message had died, so the self-healing loop of ADR-0024 waited for +someone to open a session. `to_dlq` now publishes on `.dlq.events`, an +ordinary subject: waking a person or an agent is a flow or a sink, following +ADR-0024's "notifications are flows". The event carries metadata the runtime +wrote itself, never the payload or the error text, because a dead letter is +untrusted content and this event is what wakes things up. One event per unit per +`VEJAS_DLQ_EVENT_SECS` (default 60), with `count` for the letters it stands for, +so that a burst wakes a reader once and loses nothing. A letter whose subject is +the event subject is not announced, so a failing reader cannot feed itself. An +agent started from an `exec-*` connector would inherit the runtime's environment, +so the documented route out is `http-out`. + ## Alternatives considered - **Keep dropping after the cap** — rejected: the failed work is lost and diff --git a/docs/adr/0035-async-nats-bus-island.md b/docs/adr/0035-async-nats-bus-island.md new file mode 100644 index 0000000..48bad51 --- /dev/null +++ b/docs/adr/0035-async-nats-bus-island.md @@ -0,0 +1,218 @@ +# 0035 — The NATS client moves to async-nats, behind one synchronous module + +- Status: Accepted +- Date: 2026-11-06 + +## Context + +The runtime is synchronous and thread-based, and it stays that way: a flow, a +connector and a supervisor are OS threads, and no async runtime was in the tree. +That was also true of the NATS client, `nats` 0.25, which is the problem. The +crate is deprecated. `nats` 0.26 does not help: it keeps rustls 0.22.4 and the +same advisories, and seven entries of `.cargo/audit.toml` existed only for it. + +It also cost more than its size. The client runs two threads per connection, and +the runtime opened a connection per use, 25 sites in all. Measured on v0.3.6: 11 +threads idle, then 5 threads and 2 connections per flow. Two hundred flows were +1 011 threads, 401 connections and 140 MB. + +And a hard-coded 5 ms floor in its flusher thread shaped the hot path. Findings +#5 and #6 of `bench/README.md` come from it, and three workarounds followed: +`publish_confirmed`, `fetch_iter_flushed`, and a second connection per flow for +the acks (#7). + +## Decision + +One new module, `core/src/bus.rs`, hosts `async-nats` on one tokio runtime and +exposes a synchronous API. Nothing else in the runtime imports tokio or a NATS +type. Flows, the interpreter, connectors and supervisors do not change shape. +`Engine::live` and `Engine::shadow` are untouched and the interpreter has no bus +access, so `Program::writes_bus()` means what it meant. + +There is no async rewrite. Threads are what make a flow simple to read, and the +cost above was the client's. + +The eight points that make the module. + +### 1. One connection per process + +`bus::connect` returns the one `Bus` of a URL, whoever asks. Pull rounds, +publishes, requests, subscriptions and acks all share it. + +Whether data and acks need a connection each was measured, because finding #7 +said they did. They do not, at any flow count, and that decides it: + +| Events/s, drained with `bench/pub.py` | v0.3.6 | 2 conns, 2 workers | 1 conn, 2 workers | 2 conns, 1 worker | 1 conn, 1 worker | +|---|---:|---:|---:|---:|---:| +| 1 flow | 20 100 – 20 500 | 19 500 – 19 800 | 20 700 | 20 000 – 20 400 | 19 400 – 21 400 | +| 10 flows | 35 000 – 35 200 | 36 500 – 37 600 | 37 900 – 38 400 | 36 600 | 35 200 – 36 000 | +| 50 flows | 22 100 | 26 900 | 27 000 | 26 900 | 27 400 | +| 1 detect unit | 33 200 – 35 400 | 37 700 – 40 500 | 35 500 – 37 300 | not run | 40 700 – 41 100 | + +### 2. One worker + +One connection is one task, and the runtime runs nothing that blocks. In the table +above a second worker is about 5 % ahead at ten flows, about 10 % behind on a detect +unit and level elsewhere, for one more thread. The simpler one stays. + +`wait` parks the calling OS thread on a future. It is never called from inside +the runtime: tokio panics rather than deadlocks if it ever is, and a test pins +that. The event callback and the echo task are the only code of ours that runs +there. + +### 3. The barrier is rebuilt, because `flush` does not mean what it meant + +The at-least-once rule of a batch is: emits leave, the server confirms it has read +them, then the source messages are acked. The sync client's `flush` was a PING and +a PONG. `async_nats::Client::flush` only writes the socket. Using it would have +compiled, passed every test and acked messages the server never saw. + +There is no public PING, so the barrier publishes to a subject only this +connection subscribes to and waits to hear it back. A connection's messages are +handled in order, so the echo cannot overtake them. Two alternatives were +rejected. A request to a subject nobody listens to, relying on the 503, stalls +every flow as soon as someone runs `nats sub '>'`. Draining a subscription does +write a PING, but the client does not wait for the PONG. + +### 4. A reconnect fails the barrier + +On a reconnect `async-nats` replaces the connection and its write buffer with it: +what was not yet written is dropped, in silence, and what comes next leaves on the +new connection. A round-trip alone would then confirm a batch that lost its first +half. `Statistics::connects` moves on every connection, before any later command +is handled, so the barrier reads it before the batch leaves and again after the +echo. If it moved, the batch fails and redelivers, whether or not anything was +lost. The sync client could complete a pending PONG on reconnect, so this is +stricter than what it replaces. + +### 5. Emits are held per batch + +A first version handed each emit to the connection as it was made. It ran the flow +hop at 12 600 evt/s against 20 300 for v0.3.6, and the round phases on `/metrics` +put all of it in `process`: 33 µs more per message, CPU tripled. Each emit cost a +cross-thread wake and a write of its own. The 5 ms floor had been coalescing those +all along, as a side effect. So `Batch` holds a round's emits and hands them over +with the barrier: one wake and one write per round, and throughput came back. + +A batch is whatever has arrived, so a quiet flow's emit still leaves at once: the +server ends a round as soon as the consumer has caught up. Subject and size are +checked when an emit is held, so one bad emit fails its own message and not the +batch. + +### 6. Pull rounds keep their wire format + +`async-nats`'s pull streams manage their own requests and thresholds, which is not +what "one request in flight ahead of the round being processed" asks for. So +`Puller` owns an inbox on the shared connection and sends the same request as +before: +`{"batch":64,"expires":500000000,"no_wait":true}`. `Round` is the same iterator, +with one change: a 503, or a 409 saying the consumer was deleted, now fails the +next round so the supervisor recreates it. The old iterator handed those statuses +to the flow as empty messages. Deleting a consumer under a running flow on v0.3.6 +put two dead letters from an `_INBOX` subject in the queue and left the flow +`running`, consuming nothing, for good. Here it restarts once and carries on. + +While the connection is down no request is sent. The round times out empty and the +loop keeps its pace and its stop flag, then picks up where it was. + +### 7. Administration keeps its wire format too + +Streams, buckets and consumers are created through raw JetStream API requests, +with the bodies the sync client sent. The typed configs of `async-nats` carry +defaults of their own: a bucket would have come out with `allow_direct` on. KV and +object stores then open through `async-nats` on top of those streams. + +The result was checked three ways. `nats stream info` and `nats consumer info` on +a fresh store are identical to v0.3.6 for all eight streams and every durable; only +the ordered consumers behind a KV watch differ, and they hold no state. An upgrade +test ran v0.3.6 against a real store, then this build on it, then v0.3.6 again: +durable positions, the version overlay, a pending proposal, a dead letter, offsets +and the lease survive both ways, and a sequence opened under one binary closed +under the other, from the other client's detect snapshot. + +### 8. Two things the client does not do + +`async-nats` parses `nats://user:pass@host` and `nats://token@host` and connects +without them, where the sync client used them. `bus.rs` hands them over. No test +in the repo connects with credentials, so this would have failed in production. + +And `connect` refuses while the connection is down, as `nats::connect` did: a unit +that starts during an outage backs off and comes back, and an HTTP route says so. +Threads already running stay in place and resume on reconnect. + +## Measured + +v0.3.6 and this build alternating, one session on the 8-core WSL2 box of +`bench/README.md`, each run started under a load of 6. + +| | v0.3.6 | this build | +|---|---:|---:| +| Threads at 0 / 1 / 10 / 50 / 200 flows | 11 / 16 / 61 / 261 / 1 011 | 10 / 11 / 20 / 60 / 210 | +| NATS connections, same | 1 / 3 / 21 / 101 / 401 | 1 / 1 / 1 / 1 / 1 | +| Idle RSS, same, MB | 7.1 / 8.9 / 16.7 / 46.9 / 139.8 | 9.1 / 9.6 / 10.5 / 14.2 / 26.4 | +| Flow hop, `bench/flow-only.sh`, 16 publishers | 13 100 – 13 700/s | 13 200 – 13 500/s | +| Flow hop, drained with `bench/pub.py` | 20 100 – 20 500/s | 19 400 – 21 400/s | +| RSS under load, one flow | 9.2 MB | 10.1 – 10.3 MB | +| `bench/multi-flow.sh`, 10 flows | 12 300 – 13 000/s, 18.6 MB | 13 700 – 14 700/s, 14.5 MB | +| `bench/multi-flow.sh`, 50 flows | 11 300/s, 58 MB | 11 100 – 12 100/s, 38 MB | +| One detect unit, `bench/detect.sh` | 30 800 – 35 400/s, 11 MB | 34 700 – 41 100/s, 12.7 MB | +| Paced 2 000/s, p50 / p99 | 3–4 / 14–17 ms | 3 / 14–15 ms | +| Paced 20/s, p50 / p99 | 2 / 3–4 ms | 2 / 3 ms | +| `bench/run.sh 15 32`, ingest / delivered per s | 4 930 – 4 960 / 2 850 – 2 890 | 4 640 – 5 110 / 2 570 – 2 910 | +| MQTT loopback, `bench/broker-mqtt.sh 5000` | 2 370 – 2 430 rt/s | 2 050 – 2 090 rt/s | +| Spawn to `/healthz` 200 | 4.0 ms | 3.9 ms | +| Spawn to a flow running | 1 714 ms | 10.9 ms | +| Binary | 9.90 MB | 11.24 MB | + +`bench/flow-only.sh` is bound by its `nats` publishers on both sides, which is why +the second line is there. `bench/cluster.sh` delivers 20 000/20 000 through a +`kill -9` with the timer staying single, `bench/cluster-promote.sh` converges in +10 ms with 40 000/40 000 and no interleave, and all the e2e suites pass, the +connector admission job included. + +The MQTT loopback is the one number that fell, by 14 %. Both its legs work one +message at a time, a pub-ack on `mqtt-in` and an ack on `mqtt-out`, and each now +crosses to the worker and back where the sync client wrote from the calling thread. +That is the likely cost, not an isolated one. A batch has no such hop to lose, so +the flow hop did not feel it. + +## Connectors + +`connectors/mq` and `connectors/amqp` hold their own NATS client (ADR-0023, +ADR-0026) and move too, by compiling the same file: `#[path]` on +`core/src/bus.rs`, which depends on nothing else in core. Their `publish_confirmed` +and `fetch_round` go away. `connectors/amqp` also moves from rustls 0.22 to 0.23, +with no source change to its TLS adapter. Both are sync programs; "no tokio" is no +longer true of them, in the same contained sense as the runtime. + +## Consequences + +- `.cargo/audit.toml` loses seven entries, and every lockfile is on rustls 0.23.46 + and rustls-webpki 0.103.16. `core/Cargo.lock` goes from 251 packages to 201. +- An emit subject with whitespace or a line break is refused. The sync client + wrote it into the `PUB` line, so event data could inject protocol: reproduced on + the v0.3.6 binary, see the changelog. The refused message stays un-acked and + redelivers. It does not reach the dead-letter queue, because a failed emit was + always treated as transient; a payload over the server's limit behaves the same. +- A runtime whose NATS user has a publish allow-list needs `_INBOX.>` on it, for + the barrier. It already needed to subscribe there, and to answer `exec-rpc` and + control requests. Without it the emits leave, no batch is ever confirmed, nothing + is acked, and everything redelivers; the log names the permissions violation. No + deployment in this repo sets such a list, so this is an upgrade note. +- tokio is in the tree. It stays inside one file, and the runtime has one more + thing it depends on behaving. What `bus.rs` rebuilt (points 3, 4 and 8) is what a + client upgrade could break. The `bus::live` tests hold it against a real server; + like every test that needs one they are ignored in CI, so run them on an upgrade: + `cargo test bus::live -- --ignored`. +- One connection is one fate: when it drops, every flow waits at once, and a server + that closes it as a slow consumer stops the whole process. A pull bounds what a + flow can have in flight to 128 messages, which is what keeps that far away. +- The binary grows by 1.3 MB and RSS at one flow by about 1 MB. Past a few flows + the runtime is smaller than it was. + +## Not measured + +TLS performance (`tls://` was checked for behaviour: a private CA in the trust +store, an upgrade from `nats://`, an unknown CA refused). Failover between the +servers of a NATS cluster, or more than one server in `NATS_URL`: the runs here +restart one server in place. diff --git a/docs/book/src/guides/dlq-replay.md b/docs/book/src/guides/dlq-replay.md index f78068d..2c88f7b 100644 --- a/docs/book/src/guides/dlq-replay.md +++ b/docs/book/src/guides/dlq-replay.md @@ -23,9 +23,37 @@ post-mortem knows *which* logic failed, even after a promote. and `vejas_dlq_purge` are refused, and the HTTP routes want the `X-Approval-Token` header, which the panel sends. +## Being told + +Each dead letter is announced on `vx.dlq.events`, an ordinary subject, so what +happens next is a flow or a sink like any other: + +```json +{"transition": "dead-lettered", "ts": 1793980000, "unit": "flow:orders", + "subject": "vx.orders.in", "attempts": 5, "version": "d5768a2de6e8cbb0", + "count": 1} +``` + +- **Metadata only.** The payload and the error text stay in the DLQ: a dead + letter is untrusted content, and this event is what wakes a person or an + agent. Read the letter with `vejas_dlq`. +- **One event per unit per window** (`VEJAS_DLQ_EVENT_SECS`, 60 s). The first + letter is news at once; `count` says how many letters the next event stands + for, so a burst wakes a reader once and loses nothing. Each instance of a + cluster keeps its own window. +- **It cannot feed itself.** A reader of these events that keeps failing + dead-letters them, and those letters are not announced. + +[`dlq_webhook.vjs.example`](https://github.com/cpoder/vejas/blob/master/docs/examples/dlq_webhook.vjs.example) +posts the event to a URL: a chat channel, or a runner that starts your agent +with the agent's token alone. Do not start an agent from an `exec-*` connector: +a child inherits the runtime's environment, `VEJAS_APPROVAL_TOKEN` and +`VEJAS_SECRET_*` included. + ## The loop that makes it self-healing -1. A message dies; the envelope carries version `v3`. +1. A message dies; the envelope carries version `v3`, and `vx.dlq.events` says + so. 2. An agent reads `vejas_dlq`, drafts a candidate flow, proves it: replay yesterday's real traffic through it ([time-travel](change-safely.md)), watch it shadow live traffic diff --git a/docs/book/src/introduction.md b/docs/book/src/introduction.md index 6df5a40..f7b1123 100644 --- a/docs/book/src/introduction.md +++ b/docs/book/src/introduction.md @@ -18,7 +18,7 @@ Three design commitments shape everything: approves (ADR-0006, ADR-0024). 3. **Measured, not claimed.** Every number in these docs comes from a reproducible benchmark in [`bench/`](reference/benchmarks.md): cold start - 11 ms, 6–8 MB RSS under load, end-to-end p50 2 ms uncongested, every hop + 11 ms, 10 MB RSS under load, end-to-end p50 2 ms uncongested, every hop persisted, cluster promote in 60 ms — lossless. ## Where to go diff --git a/docs/book/src/reference/benchmarks.md b/docs/book/src/reference/benchmarks.md index 6541d37..c00c1f4 100644 --- a/docs/book/src/reference/benchmarks.md +++ b/docs/book/src/reference/benchmarks.md @@ -10,13 +10,13 @@ yours. | Metric | Value | Reproduce with | |---|---|---| | Cold start (spawn → healthz) | 11–13 ms | `bench/run.sh` | -| Runtime RSS under load | 6–8 MB (49 MB with 50 live flows) | `bench/run.sh`, `bench/multi-flow.sh` | -| Binary / image | 6.2 MB / 201 MB | — | +| Runtime RSS under load | 10 MB (38 MB with 50 live flows) | `bench/flow-only.sh`, `bench/multi-flow.sh` | +| Binary / image | 11.2 MB / 201 MB | — | | e2e latency, uncongested (webhook→flow→sink) | **p50 2 ms, p99 3 ms** | `bench/paced.sh 20 15` | | e2e paced sustained ~1 900/s | p50 14 ms, p99 36 ms, 20 000/20 000 | `bench/paced.sh 2000 15` | | e2e saturated (32 conns) | ~4 900/s ingest, ~2 650/s delivered (sink-bound) | `bench/run.sh 15 32` | | Isolated flow hop | 13 652/s (13 746/s aggregate over 10 flows) | `bench/flow-only.sh`, `bench/multi-flow.sh` | -| MQTT loopback, QoS 1 both ways, real mosquitto | 2 285 rt/s, 5 000/5 000 | `bench/broker-mqtt.sh 5000` | +| MQTT loopback, QoS 1 both ways, real mosquitto | 2 070 rt/s, 5 000/5 000 | `bench/broker-mqtt.sh 5000` | | Cluster: instance kill -9 under load | 20 000/20 000, zero loss | `bench/cluster.sh` | | Cluster: singleton failover | ~2.6 s graceful / ~5.9 s crash (TTL-bound) | `bench/cluster-gaps.sh` | | Cluster-wide version promote | 60 ms convergence, lossless mid-burst | `bench/cluster-promote.sh` | diff --git a/docs/book/src/reference/env-vars.md b/docs/book/src/reference/env-vars.md index 1deb4a0..25ad3cb 100644 --- a/docs/book/src/reference/env-vars.md +++ b/docs/book/src/reference/env-vars.md @@ -19,6 +19,7 @@ directory is a complete system. | `VEJAS_TENANT` | *(unset)* | Tenant label for provisioned packages. | | `VEJAS_BUNDLE` | *(unset)* | Bundle path for provisioning (`vejas_provision`). | | `VEJAS_AGENT_CMD` | *(unset)* | Command the panel uses for its agent-assist box. | +| `VEJAS_DLQ_EVENT_SECS` | `60` | At most one dead-letter event per unit per this many seconds on `vx.dlq.events`. `0` announces every letter. | | `VEJAS_HTTP_MAX_INFLIGHT` | `256` | Requests handled at once on the panel/MCP port, one thread each. Past it: 503. | ## Secrets (ADR-0008) diff --git a/docs/examples/dlq_webhook.vjs.example b/docs/examples/dlq_webhook.vjs.example new file mode 100644 index 0000000..0dde399 --- /dev/null +++ b/docs/examples/dlq_webhook.vjs.example @@ -0,0 +1,11 @@ +# connector: dlq_webhook — tell someone when a message dies (ADR-0015). +# Each dead letter is announced on vx.dlq.events: unit, subject, attempts, version, +# and count, the letters the event stands for. Metadata only: the payload stays in +# the DLQ, so what this wakes reads it with vejas_dlq, as untrusted content. +# +# The URL is a chat webhook, or a runner of your own that starts your agent with +# the agent's VEJAS_TOKEN alone. Keep the agent out of the runtime: a child of an +# exec-* connector inherits its environment, VEJAS_APPROVAL_TOKEN included. +driver "http-out" +SUBJECT = "vx.dlq.events" +URL = secret("ops/dlq_webhook_url") diff --git a/e2e/dlq-events.sh b/e2e/dlq-events.sh new file mode 100755 index 0000000..17f19a6 --- /dev/null +++ b/e2e/dlq-events.sh @@ -0,0 +1,84 @@ +#!/bin/bash +# A dead letter announces itself on vx.dlq.events (ADR-0015 Amendment A): metadata +# only, one event per unit per window with `count` for the letters it stands for, +# through an ordinary sink, and a failing reader of the events cannot feed itself. +# e2e/dlq-events.sh (needs the nats CLI) +set -uo pipefail +cd "$(dirname "$0")/.." +BIN="$PWD/core/target/release/vejas-runtime" +[ -x "$BIN" ] || { echo "build first" >&2; exit 1; } +S=$(mktemp -d); R=$(mktemp -d) +RT=""; NP=""; SUB=""; HOOK="" +trap 'kill $RT $NP $SUB $HOOK 2>/dev/null; wait $RT $NP $SUB $HOOK 2>/dev/null; rm -rf "$S" "$R"' EXIT +fail=0 +ok(){ echo " ✓ $1"; } +ko(){ echo " ✗ $1"; fail=1; } + +URL=nats://127.0.0.1:4280; H="http://127.0.0.1:8752"; HOOK_PORT=18791 +mkdir -p "$R/flows" "$R/connectors" +printf '# flow: demo\nsource "vx.demo.in"\nemit "vx.demo.out", {id: id}\n' > "$R/flows/demo.vjs" +# a reader of the events that fails on every one of them +printf '# flow: watcher\nsource "vx.dlq.events"\nx = secret("missing/x")\nemit "vx.demo.out", {x: x}\n' > "$R/flows/watcher.vjs" +printf '# connector: hook\ndriver "http-out"\nSUBJECT = "vx.dlq.events"\nURL = "http://127.0.0.1:%s/hook"\n' $HOOK_PORT \ + > "$R/connectors/hook.vjs" + +python3 -c ' +import http.server, sys +class H(http.server.BaseHTTPRequestHandler): + def do_POST(self): + body = self.rfile.read(int(self.headers.get("content-length", 0))) + open(sys.argv[2], "ab").write(body + b"\n") + self.send_response(200); self.send_header("Content-Length", "0"); self.end_headers() + def log_message(self, *a): pass +http.server.HTTPServer(("127.0.0.1", int(sys.argv[1])), H).serve_forever() +' $HOOK_PORT "$S/hook.out" & HOOK=$! + +nats-server -js -sd "$S/n" -a 127.0.0.1 -p 4280 > /dev/null 2>&1 & NP=$! +sleep 0.5 +NATS_URL=$URL VEJAS_ROOT="$R" VEJAS_HTTP_ADDR=127.0.0.1:8752 VEJAS_ACK_WAIT_SECS=1 VEJAS_DLQ_EVENT_SECS=2 \ + "$BIN" > "$S/rt.log" 2>&1 & RT=$! +until curl -sf -o /dev/null "$H/healthz"; do sleep 0.1; done +for _ in $(seq 1 100); do + curl -s -m 5 -H 'Content-Type: application/json' \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"vejas_topology","arguments":{}}}' "$H/mcp" \ + | grep -q 'flow:demo.*running\|running.*flow:demo' && break + sleep 0.1 +done +sleep 1 +nats -s $URL sub vx.dlq.events --raw > "$S/events" 2>/dev/null & SUB=$! +sleep 0.5 +counted(){ python3 -c ' +import json, sys +print(sum(json.loads(l).get("count", 0) for l in open(sys.argv[1]) if l.startswith("{") and json.loads(l).get("unit") == sys.argv[2])) +' "$S/events" "$1"; } + +echo "── one poison message: announced at once, metadata only" +nats -s $URL pub vx.demo.in 'CANARY-not-json' > /dev/null 2>&1 +for _ in $(seq 1 50); do grep -q 'dead-lettered' "$S/events" && break; sleep 0.1; done +grep -q '"unit":"flow:demo"' "$S/events" && ok "the event names the unit" || ko "no event: $(cut -c1-120 "$S/events")" +grep -q '"count":1' "$S/events" && ok "it counts one letter" || ko "wrong count" +grep -q 'CANARY' "$S/events" && ko "the payload reached the event" || ok "the payload stays in the DLQ" +grep -q 'bad json' "$S/events" && ko "the error text reached the event" || ok "the error text stays in the DLQ" + +echo "── a burst: one event, nothing lost" +for i in $(seq 1 20); do nats -s $URL pub vx.demo.in "poison-$i" > /dev/null 2>&1; done +LINES_DURING=$(grep -c 'flow:demo' "$S/events") +[ "$LINES_DURING" -le 3 ] && ok "$LINES_DURING event(s) during 20 letters" || ko "$LINES_DURING events for 20 letters" +for _ in $(seq 1 60); do [ "$(counted flow:demo)" = "21" ] && break; sleep 0.1; done +[ "$(counted flow:demo)" = "21" ] && ok "the counts add up to all 21 letters" || ko "the counts add up to $(counted flow:demo)" + +echo "── an ordinary sink carries it" +for _ in $(seq 1 50); do grep -q 'dead-lettered' "$S/hook.out" 2>/dev/null && break; sleep 0.1; done +grep -q 'dead-lettered' "$S/hook.out" 2>/dev/null && ok "http-out posted the event" || ko "the sink got nothing" + +echo "── a failing reader does not feed itself" +watcher_letters(){ curl -s -m 5 "$H/dlq" | python3 -c ' +import json, sys +print(len([d for d in json.load(sys.stdin)["dead_letters"] if d.get("unit") == "flow:watcher"]))'; } +for _ in $(seq 1 200); do [ "$(watcher_letters)" -ge 1 ] && break; sleep 0.1; done +[ "$(watcher_letters)" -ge 1 ] && ok "the watcher dead-lettered the events it failed on" || ko "the watcher never dead-lettered" +sleep 4 +grep -q '"unit":"flow:watcher"' "$S/events" && ko "a failing reader announced its own letters" || ok "its letters are not announced" + +[ $fail -eq 0 ] && echo "DLQ EVENTS HOLD" || echo "DLQ EVENTS REGRESSION — do not ship" +exit $fail diff --git a/e2e/emit-subject.sh b/e2e/emit-subject.sh index 25f9032..9a7497b 100755 --- a/e2e/emit-subject.sh +++ b/e2e/emit-subject.sh @@ -19,6 +19,9 @@ URL=nats://127.0.0.1:4281; H="http://127.0.0.1:8753"; IN=18794 mkdir -p "$R/flows" "$R/connectors" printf '# flow: dyn\nsource "vx.dyn.in"\nemit f"vx.out.{tenant}", {id: id}\n' > "$R/flows/dyn.vjs" printf '# connector: http_in\ndriver "http-in"\nPORT = %s\n' $IN > "$R/connectors/http_in.vjs" +# an emit over the server's 1 MB limit: 16 bytes doubled seventeen times +printf '# flow: big\nsource "vx.big.in"\ns = "xxxxxxxxxxxxxxxx"\nfor i in items:\n s = s + s\nend\nemit "vx.big.out", {s: s}\n' \ + > "$R/flows/big.vjs" nats-server -js -sd "$S/n" -a 127.0.0.1 -p 4281 > /dev/null 2>&1 & NP=$! sleep 0.5 @@ -57,6 +60,15 @@ print(len([d for d in json.load(sys.stdin)["dead_letters"] if "not a valid subje for _ in $(seq 1 150); do [ "$(letters)" = "2" ] && break; sleep 0.1; done [ "$(letters)" = "2" ] && ok "both messages reached the DLQ with the reason" || ko "$(letters) of 2 reached the DLQ" +echo "── an emit the server would refuse takes it too" +curl -s -m 5 -o /dev/null -X POST -H 'Content-Type: application/json' \ + -d '{"items":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17]}' "http://127.0.0.1:$IN/ingest/big.in" +big_letters(){ curl -s -m 5 "$H/dlq" | python3 -c ' +import json, sys +print(len([d for d in json.load(sys.stdin)["dead_letters"] if d.get("unit") == "flow:big"]))'; } +for _ in $(seq 1 150); do [ "$(big_letters)" = "1" ] && break; sleep 0.1; done +[ "$(big_letters)" = "1" ] && ok "an oversized emit reached the DLQ" || ko "the oversized emit never reached the DLQ" + echo "── a shadow run shows the same error" curl -s -m 10 -H 'Content-Type: application/json' -d \ '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"vejas_run_flow","arguments":{"file":"flows/dyn.vjs","input":{"id":1,"tenant":"a b"}}}}' \ diff --git a/e2e/http-in-bound.sh b/e2e/http-in-bound.sh index 501712f..e683967 100755 --- a/e2e/http-in-bound.sh +++ b/e2e/http-in-bound.sh @@ -98,9 +98,6 @@ else debian:trixie-slim vejas-runtime > /dev/null for _ in $(seq 1 100); do curl -sf -o /dev/null "$H2/healthz" && break; sleep 0.1; done wait_ingest $IN2 || ko "http-in never came up in the container" - # Past the boot: a burst that lands on the first NATS connections is another - # case, covered by background(), and it made this check flaky. - sleep 5 hold $IN2 300 3; HOLD=$! sleep 1 wait $HOLD 2>/dev/null diff --git a/site/index.html b/site/index.html index b7fab6d..55e07f4 100644 --- a/site/index.html +++ b/site/index.html @@ -287,7 +287,7 @@

Business surface

1 binary  ·  1 dependency (NATS)  ·  0 builder UI  ·  nothing proprietary to export

-

measured: cold start 11–13 ms  ·  6–8 MB RSS  ·  p50 2 ms, every event hop persisted  ·  cluster promote 60 ms, lossless  ·  the benchmarks

+

measured: cold start 11–13 ms  ·  10 MB RSS  ·  p50 2 ms, every event hop persisted  ·  cluster promote 60 ms, lossless  ·  the benchmarks