Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ is `0`, minor versions may carry breaking changes — they are called out here.

## [Unreleased]

### Changed
- Published numbers corrected against v0.5.1: cluster promote 20 ms, graceful handoff
at most 2 s, crash failover 4–5 s, image 40 MB to pull (100 MB unpacked), and against
Redpanda Connect ~20× less memory and a ~30× smaller binary. The README also said
port 8686 binds to localhost: that is the bundled compose, the raw binary listens on
`0.0.0.0:8686`.

## [0.5.1] — 2026-11-07

If an `exec` recipe read a runtime credential by inheritance, it now has to declare
Expand Down
61 changes: 39 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,11 @@ end-to-end test ([the script](e2e/bridge-film.mjs)) — the video cannot lie.

**Measured, not claimed** — reproducible on an 8-core dev machine
([methodology and comparisons](bench/)): cold start **11–13 ms**, **10 MB**
RSS under load (38 MB with fifty live flows), **11.2 MB** binary / **118 MB**
image, end-to-end p50 **2 ms** uncongested / p50 14 ms at a paced 1 900/s
RSS under load (38 MB with fifty live flows), **11.2 MB** binary / **40 MB**
image to pull, end-to-end p50 **2 ms** uncongested / p50 14 ms at a paced 1 900/s
sustained with **every event hop persisted** in JetStream, **2 070/s** through a
real MQTT broker and back (QoS 1 both ways, PUBACK-gated). Same order of
magnitude as engines that persist nothing, in ~25× less memory.
magnitude as engines that persist nothing, in ~20× less memory.

![Prompt-to-flow and the pipeline, derived from the code itself](docs/screenshots/panel-top.png)

Expand All @@ -60,9 +60,9 @@ it without touching code:
- **Clustering** — run N runtimes on the same NATS: flows load-balance and
fail over natively (measured: an instance kill -9'd under load, every
message still delivered exactly-all), singleton sources take a KV lease
(graceful handoff 2.6 s, crash failover bounded by the TTL), and a
(graceful handoff at most 2 s, crash failover bounded by the TTL), and a
clustered instance refuses local-file mutations — a live promote instead
publishes a **version** every instance converges on (measured: 60 ms
publishes a **version** every instance converges on (measured: 20 ms
convergence, lossless mid-burst). (ADR-0020/0021, measured in
[bench/](bench/))
- **Language** — VejasScript: `source` in, `emit` out, `invoke` to compose
Expand All @@ -72,19 +72,24 @@ it without touching code:
through `EXPORTS` (private by default) or the bus. (ADR-0003, ADR-0004)
- **Connectors** — a typed Rust **driver SDK** (`Driver` trait, Source/Sink).
Bundled drivers: `http-in`, `timer`, `http-poll`, `oauth-poll` (a generic
OAuth2 REST poller that stands in for much of a SaaS catalog), `slack-out`,
`http-out`, plus the exec bridges (`exec-source`/`exec-sink`/
`exec-stream-source`/`exec-rpc`), which wrap a program in **any language**
over stdio, isolated by process. That is how the **SAP connector** ships:
OAuth2 REST poller that stands in for much of a SaaS catalog), `mqtt-in`,
`mqtt-out`, `slack-out`, `http-out`, plus the exec bridges
(`exec-source`/`exec-sink`/`exec-stream-source`/`exec-rpc`), which wrap a
program in **any language** over stdio, isolated by process. That is how the
**SAP connector** ships:
native Rust over the official `libsapnwrfc` C library — BAPI/RFC calls,
IDocs in and out over tRFC, no JVM (ADR-0014) — and the **Salesforce**
connector (OAuth2 + Bulk API 2.0, streaming). An instance is a declarative
`.vjs` manifest, hot-addable, and an agent can write one from a prompt. The
subject convention stays the whole interface, so an out-of-process connector
is first-class. (ADR-0007, ADR-0011, ADR-0014)
- **Secrets** — `secret("path/key")` resolves from a Vault (HashiCorp KV v2; an
env backend for dev) at run time. A secret is never a literal, so it never
lands in git, the panel, or the business surface. (ADR-0008)
connector (OAuth2 + Bulk API 2.0, streaming). Brokers that need their own
transactional loop are standalone binaries (`connectors/amqp`,
`connectors/mq`). An instance is a declarative `.vjs` manifest, hot-addable,
and an agent can write one from a prompt. The subject convention stays the
whole interface, so an out-of-process connector is first-class. (ADR-0007,
ADR-0011, ADR-0014)
- **Secrets** — `secret("path/key")` resolves at run time from a Vault
(HashiCorp KV v2), a local file, or an env backend for dev. A secret is never
a literal, so it never lands in git, the panel, or the business surface, and
every path that shows a run (preview, replay, time-travel, canary) gets a
placeholder instead of the value. (ADR-0008)
- **UI** — no builder. Monitoring (pipeline graph, statuses, a live feed of the
last processed events) + a business panel where experts review and correct
the business surface (rendered from literals in the code). A correction is
Expand All @@ -93,8 +98,15 @@ it without touching code:
- **Versioning** — a candidate version of a flow replays **yesterday's
real traffic** (time-travel) or shadow-follows **live** traffic (canary),
diffed side by side per event; unproven code never emits for real. Promote
publishes the version cluster-wide (measured: 60 ms, lossless), audited,
publishes the version cluster-wide (measured: 20 ms, lossless), audited,
rollback included. (ADR-0021)
- **Dead letters** — a message that keeps failing is parked with its error and
the flow version that killed it, never dropped. Replay is explicit, and each
dead letter is announced on `vx.dlq.events`, so a flow can wake someone.
(ADR-0015)
- **Governed mode** — with `VEJAS_REQUIRE_APPROVAL=1`, agents propose and a
human approves. Every mutating path, MCP and HTTP, refuses a direct write, and
approval takes a token the agent never holds. (ADR-0024)
- **MCP & API** — the runtime is its own MCP server; a flow that declares
`tool "…"` becomes a first-class MCP tool, and one that declares
`api "VERB /path"` becomes a synchronous HTTP endpoint with an auto-generated
Expand All @@ -111,18 +123,23 @@ claude mcp add --transport http vejas http://localhost:8686/mcp # or any MCP c
# The panel: http://localhost:8686 — a webhook entry: POST :8787/ingest/<subject>
```

`docker compose up` builds from source. Prebuilt: `ghcr.io/cpoder/vejas-runtime`
and a Linux x86_64 binary on [Releases](https://github.com/cpoder/vejas/releases/latest).

Develop and test:

```bash
cargo test --manifest-path core/Cargo.toml # language unit tests
core/target/release/vejas-runtime vjs-test tests/vjs # golden end-to-end cases
```

Security defaults: the panel/MCP port (8686) binds to localhost only — that
surface can write flows and run commands (exec connectors), so expose it
deliberately. Set `VEJAS_TOKEN` and every write (POST, `/mcp` included)
requires `Authorization: Bearer <token>`. The webhook port (8787) only
publishes events onto the bus.
Security defaults: the bundled compose publishes the panel/MCP port (8686) on
localhost only, because that surface can write flows and run commands (exec
connectors). The raw binary listens on `0.0.0.0:8686`, with no token: set
`VEJAS_HTTP_ADDR` and `VEJAS_TOKEN` before running it anywhere reachable. With
`VEJAS_TOKEN` set, every write (POST, `/mcp` included) requires
`Authorization: Bearer <token>`. The webhook port (8787) only publishes events
onto the bus.

## License

Expand Down
12 changes: 8 additions & 4 deletions bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,10 @@ fifth under load; if your numbers disagree, check `uptime` first.

## Clustering (ADR-0020, measured)

The table below is v0.5.1 on `async-nats`, re-measured on 2026-11-07. Leave a few
seconds between runs: each script frees its port on exit, and a run chained straight
after another printed no result.

Two instances, one NATS, `bench/cluster.sh` + `bench/cluster-gaps.sh` — and
the same probe at **three** (`bench/cluster.sh 3 20000`): 20 000/20 000
delivered, the singleton timer ticked 8 times in 8 seconds rather than 24,
Expand All @@ -294,12 +298,12 @@ them (8 243 and 7 611).

| Invariant | Result |
|---|---|
| Flows under kill -9 (1.5s into load) | **20 000/20 000 exactly-all**, ~8 k/s aggregate |
| Flows under kill -9 (1.5s into load) | **20 000/20 000 exactly-all**; what the dead instance held comes back after the ack wait, 31.5 s in all with the default 30 s, on v0.3.6 and v0.5.1 alike |
| Singleton duplication (timer, 2 instances) | **eliminated** — 8 ticks/8 s (was 16 pre-lease) |
| Graceful handoff (SIGTERM leader) | **2.6 s** ≈ tick interval + 1 s standby retry |
| Crash failover (kill -9 leader, TTL 3 s) | **5.9 s** ≈ TTL + retry + tick (worst case) |
| Graceful handoff (SIGTERM leader) | **at most 2 s** ≈ tick interval + standby retry (four runs: three under 1.5 s, one at 2.0 s) |
| Crash failover (kill -9 leader, TTL 3 s) | **4–5 s** ≈ TTL + retry + tick (four runs: 5.0, 5.0, 4.0, 4.0 s) |
| Split-brain guard | clustered instance answers 409 on local-file mutation, file untouched |
| Live promote across the cluster (ADR-0021) | **60 ms convergence**, first new-version emit at 680 ms, zero interleave, 40 000/40 000 delivered mid-burst |
| Live promote across the cluster (ADR-0021) | **20 ms convergence**, first new-version emit at 20–30 ms, zero interleave, 40 000/40 000 delivered mid-burst |

## Not measured yet

Expand Down
4 changes: 2 additions & 2 deletions bench/compare/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ bench/compare/run-n8n.sh 20 32 # n8n (official image, tune
| Flow-hop rate (no HTTP) | 171/s | **8 110/s** (9 948/s over 10 flows) | — | — |
| Cold start | 15 ms | **11–13 ms** | 391 ms | 7–17 s (container) |
| RSS under load | **6–8 MB** | **10 MB** (38 MB at 50 flows) | 202 MB | 1.2–1.3 GB |
| Distribution size | 3.9 MB binary | **11.2 MB binary / 118 MB image** | 338 MB binary | 372 MB image |
| Distribution size | 3.9 MB binary | **11.2 MB binary / 40 MB image to pull** (100 MB unpacked) | 338 MB binary | 372 MB image |
| Persistence | every hop (JetStream) | **every hop (JetStream)** | in-flight only | per-execution DB (disabled for this run) |

**n8n fairness notes.** Single instance from the official image, tuned per
Expand All @@ -36,7 +36,7 @@ Cold start measured through its container, as officially distributed.
**Read both columns honestly.** After the five fixes, Vejas delivers in the
same order of magnitude as the category (~2.7k vs 3.4k/s saturated, both
sink-bound) while **persisting every hop**, starting ~30× faster, holding
~25× less memory, in a ~70× smaller binary. The remaining latency gap
~20× less memory, in a ~30× smaller binary. The remaining latency gap
(2 ms vs 1 ms) is the price of the stronger guarantee — two persisted
JetStream hops sit in the path.

Expand Down
2 changes: 1 addition & 1 deletion docs/COOKBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,7 @@ The agent's path, every step governed:
changes the baseline under the canary stops it with a reason — never a
silently stale diff.
4. On your word: **promote**. In a cluster, that publishes a version every
instance converges on — measured at 60 ms, lossless mid-burst — with an
instance converges on — measured at 20 ms, lossless mid-burst — with an
audit record (actor, from-hash → to-hash). Rollback is a promote to any
previous version, forward-only, previewable the same way.

Expand Down
2 changes: 1 addition & 1 deletion docs/book/src/concepts/agents-mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,5 @@ MCP tool — the platform grows its own tool surface as flows are written.
And the self-healing loop closes the circle: a dead letter → the agent
reads `vejas_dlq`, drafts a candidate, proves it on real traffic
(`vejas_time_travel`, `vejas_canary_start`), proposes with evidence → human
approves → cluster-wide promote in 60 ms → replay, from the panel in governed
approves → cluster-wide promote in 20 ms → replay, from the panel in governed
mode, `vejas_dlq_replay` otherwise.
2 changes: 1 addition & 1 deletion docs/book/src/guides/change-safely.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ double-sending them.

With `VEJAS_CLUSTER=1`, a local file write is refused — a promote instead
publishes a **version** into a shared JetStream KV overlay that every
instance converges on ([benchmarks](../reference/benchmarks.md): 60 ms
instance converges on ([benchmarks](../reference/benchmarks.md): 20 ms
convergence, lossless mid-burst). If a later git deploy moves the baseline
the promote was made against, the overlay is **evicted loudly** (git wins;
the promoted content is kept in version history to re-promote) — visible at
Expand Down
4 changes: 2 additions & 2 deletions docs/book/src/guides/clustering.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ survivor.
2026-09-22): everything delivered, the singleton timer ticked once per
second and not three times, and the two survivors shared the rest of the
work between them.
- Singleton failover: **~2.6 s graceful**, ~5.9 s crash (TTL-bound).
- Singleton failover: **at most 2 s graceful**, 4–5 s crash (TTL-bound).

([benchmarks](../reference/benchmarks.md).) Graceful shutdown rides
`SIGTERM` (a k8s rolling restart): the lease hands off, in-flight work
Expand All @@ -49,7 +49,7 @@ drains, the instance exits.
In cluster mode a local file write is **refused** (`409`) — a split where
one instance fixed a rule and the others did not is the worst failure for a
business surface. Change flows through GitOps, or through a **version** that
publishes cluster-wide and every instance converges on in **60 ms,
publishes cluster-wide and every instance converges on in **20 ms,
lossless** ([change safely](change-safely.md)).

## Scaling past one getter
Expand Down
2 changes: 1 addition & 1 deletion docs/book/src/guides/dlq-replay.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ runtime's user and reaches the same bus, far more than a proposal needs.
([time-travel](change-safely.md)), watch it shadow live traffic
(canary).
3. In governed mode it *proposes* with that evidence; you approve — the
promote fans out cluster-wide in 60 ms.
promote fans out cluster-wide in 20 ms.
4. Replay from the panel, or `vejas_dlq_replay` outside governed mode — the
dead messages pass under the new version, and their envelopes record the
transition.
Expand Down
2 changes: 1 addition & 1 deletion docs/book/src/guides/governed-mode.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ the `X-Approval-Token` header. The panel asks for the token when it needs it.
shows the evidence; a proposal with none is flagged
**"⚠ No evidence"** loud — approving blind is a deliberate act.
3. **On approve** — the change executes exactly as a normal promote would
(hot-reload, or a [cluster-wide version](change-safely.md) in 60 ms) and
(hot-reload, or a [cluster-wide version](change-safely.md) in 20 ms) and
is recorded.

## Two safeguards
Expand Down
2 changes: 1 addition & 1 deletion docs/book/src/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Three design commitments shape everything:
3. **Measured, not claimed.** Every number in these docs comes from a
reproducible benchmark in [`bench/`](reference/benchmarks.md): cold start
11 ms, 10 MB RSS under load, end-to-end p50 2 ms uncongested, every hop
persisted, cluster promote in 60 ms — lossless.
persisted, cluster promote in 20 ms — lossless.

## Where to go

Expand Down
6 changes: 3 additions & 3 deletions docs/book/src/reference/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@ yours.
|---|---|---|
| Cold start (spawn → healthz) | 11–13 ms | `bench/run.sh` |
| Runtime RSS under load | 10 MB (38 MB with 50 live flows) | `bench/flow-only.sh`, `bench/multi-flow.sh` |
| Binary / image | 11.2 MB / 118 MB | — |
| Binary / image | 11.2 MB / 40 MB to pull, 100 MB unpacked | release assets, `docker buildx imagetools inspect --raw` |
| e2e latency, uncongested (webhook→flow→sink) | **p50 2 ms, p99 3 ms** | `bench/paced.sh 20 15` |
| e2e paced sustained ~1 900/s | p50 14 ms, p99 36 ms, 20 000/20 000 | `bench/paced.sh 2000 15` |
| e2e saturated (32 conns) | ~4 900/s ingest, ~2 650/s delivered (sink-bound) | `bench/run.sh 15 32` |
| Isolated flow hop | 13 652/s (13 746/s aggregate over 10 flows) | `bench/flow-only.sh`, `bench/multi-flow.sh` |
| MQTT loopback, QoS 1 both ways, real mosquitto | 2 070 rt/s, 5 000/5 000 | `bench/broker-mqtt.sh 5000` |
| Cluster: instance kill -9 under load | 20 000/20 000, zero loss | `bench/cluster.sh` |
| Cluster: singleton failover | ~2.6 s graceful / ~5.9 s crash (TTL-bound) | `bench/cluster-gaps.sh` |
| Cluster-wide version promote | 60 ms convergence, lossless mid-burst | `bench/cluster-promote.sh` |
| Cluster: singleton failover | at most 2 s graceful / 4–5 s crash (TTL-bound) | `bench/cluster-gaps.sh` |
| Cluster-wide version promote | 20 ms convergence, lossless mid-burst | `bench/cluster-promote.sh` |

Every hop persisted in JetStream throughout — the guarantee never moved
while these numbers were earned. Methodology, the five ceilings that fell
Expand Down
4 changes: 2 additions & 2 deletions docs/launch/distribution-kit.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Title alternates (kept for reference):
>
> One Rust binary on NATS/JetStream, no other dependency. Measured, reproducible in
> bench/: cold start 11–13ms, 10MB RSS, end-to-end p50 2ms uncongested, every hop
> persisted, cluster promote 60ms lossless. Twenty-one connector recipes admitted
> persisted, cluster promote 20ms lossless. Twenty-one connector recipes admitted
> by CI (MQTT/RabbitMQ against real brokers; IBM MQ, Kafka).
>
> Young: it has run since August 2026 as the evidence-collection engine of a NIS2
Expand Down Expand Up @@ -82,7 +82,7 @@ questions come):

The friendliest audience — they'll appreciate the dogfooding. Angle: *"a platform
that uses JetStream as its entire substrate — transport, persistence, KV, leases,
audit."* Show the clustering + the 60ms lossless promote.
audit."* Show the clustering + the 20ms lossless promote.

## This Week in Rust (TWiR) — "Call for participation" / project spotlight

Expand Down
2 changes: 1 addition & 1 deletion site/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@ <h4>Business surface</h4>

<div class="facts">
<p>1 binary&nbsp;&nbsp;·&nbsp;&nbsp;1 dependency (NATS)&nbsp;&nbsp;·&nbsp;&nbsp;<b>0 builder UI</b>&nbsp;&nbsp;·&nbsp;&nbsp;nothing proprietary to export</p>
<p>measured: cold start 11–13&thinsp;ms&nbsp;&nbsp;·&nbsp;&nbsp;10&thinsp;MB RSS&nbsp;&nbsp;·&nbsp;&nbsp;p50 2&thinsp;ms, every event hop persisted&nbsp;&nbsp;·&nbsp;&nbsp;cluster promote 60&thinsp;ms, lossless&nbsp;&nbsp;·&nbsp;&nbsp;<a href="https://github.com/cpoder/vejas/tree/master/bench">the benchmarks</a></p>
<p>measured: cold start 11–13&thinsp;ms&nbsp;&nbsp;·&nbsp;&nbsp;10&thinsp;MB RSS&nbsp;&nbsp;·&nbsp;&nbsp;p50 2&thinsp;ms, every event hop persisted&nbsp;&nbsp;·&nbsp;&nbsp;cluster promote 20&thinsp;ms, lossless&nbsp;&nbsp;·&nbsp;&nbsp;<a href="https://github.com/cpoder/vejas/tree/master/bench">the benchmarks</a></p>
</div>

<section id="demo">
Expand Down
Loading