A reliable SQS on Redis. At-least-once delivery, a visibility timeout, retries with exponential backoff, and a dead-letter queue — built on Redis Streams consumer groups. For projects that already run Redis and don't want to operate a heavy SQS, Kafka, or RabbitMQ setup to have a simple queue.
- 📦 Zero runtime dependencies. Not "only a few" — literally none.
ioredisis a peer you almost certainly already have. Under 80 kB packed, CLI included. - 🔒 A real visibility timeout. One consumer owns a message for
retryOnIdleMs. Miss the window and another worker picks it up — the SQS bargain, on the Redis you already run. - ⚛️ Atomic redelivery. Every move is one server-side Lua script: copy forward, ack behind, indivisibly. No crash window where a message sits in both places, or neither.
- ♻️ Retries that actually back off. Exponential escalation, a cap, and jitter applied after the cap — so a recovering cohort spreads out instead of stampeding your database together.
- 💀 A dead-letter queue that keeps things. Never trimmed, no TTL, no cap. Undecodable messages stay readable,
redriveputs them back atomically, andpurgeis the only thing that ever deletes. - 🎯 Typed bodies, inferred. Zod, Valibot, ArkType — any Standard Schema.
msg.bodyis typed from the schema, so your types can't drift from your validator. - 🧰 An ops CLI in the box.
xstreamq depth,peek,dlq ls / redrive / purge,compact,health— human-readable or--json, no extra install. - 🛑 Graceful shutdown. Drains in-flight handlers, hands them an
AbortSignal, and interrupts blocked reads rather than waiting them out. - 🩺 Health you can wire to a probe.
healthy()checks the connection and every background loop — a consume loop that died silently is exactly what it exists to catch. - 🧪 300+ tests, against a real Redis. The whole suite runs on Node 20/22/24 and on both ioredis majors, and the Lua scripts are tested as the correctness core they are.
- 💣 Survives having its workers killed. 200,000 messages with a consumer SIGKILLed every 400ms — 102 hard kills, zero loss. The load harness is a Kotlin producer that writes the envelope itself, so it tests the published contract rather than the library against itself.
And what you don't get — stated up front, because a reliability library that oversells is worse than one that doesn't exist.
- This is at-least-once, not exactly-once: handlers must be idempotent (ideally)
- A handler that overruns the drain timeout cannot be cancelled (JavaScript can't).
- Retries are opt-in — no
retryOnIdleMs, no visibility timeout. - And it's built for a few hundred thousand messages a day up to a few thousand a second on one Redis — measured at ~8,000/s sustained with a trivial handler, though a handler doing real work will bind long before xstreamq does. Past that, buy a real broker.
Each of those has its own section below. None of them is a footnote.
If you already run Redis, a decent "queue" is four Redis primitives and one background loop away — a stream, a consumer group, the pending-entries list, and XACK. The primitives are the easy part; the reliability semantics around them are not, and that is what xstreamq owns so you don't stand up another piece of infrastructure to get them. xstreamq is built on the main premise that - things will fail or go wrong in production, so resilience and safety are the core principles.
npm install xstreamq ioredisioredis is a peer dependency — the assumption is that most projects already have one, and xstreamq can reuse the client you have already configured for TLS or sentinel. Nothing else: xstreamq has no runtime dependencies of its own, CLI included.
The ops CLI ships in the same package, so it needs no separate install:
npx xstreamq depth orders order-serviceRequires Node ≥ 20 and Redis ≥ 6.2 (lag reporting needs 7.0). Published as ESM and CJS, with types for both.
import { XStreamQ } from 'xstreamq';
const dq = new XStreamQ({ url: 'redis://localhost:6379' });
await dq.subscribe(
'orders',
async (msg) => {
await processOrder(msg.body);
},
{
group: 'order-service',
retryOnIdleMs: 30_000,
maxRetries: 5,
},
);
await dq.start();
await dq.publish('orders', { id: 1, total: 4200 });
// on SIGTERM
await dq.stop({ drainTimeoutMs: 10_000 });Pass an existing client instead of a URL with new XStreamQ({ redis }). xstreamq never closes a client it did not create.
publish appends an entry to a Redis stream. Each subscriber gets its own consumer group, so workers in one group share the work (add workers to scale out) and separate groups each get a full copy (fan-out).
The moment a worker reads a message it lands in the group's pending entries list — "delivered, not yet acknowledged". It leaves only on XACK:
- handler succeeds → acked → gone
- handler throws → deliberately not acked → stays pending
- worker crashes → never acked → stays pending
That is the whole basis of at-least-once delivery: a failed message stays put instead of vanishing. A background reclaimer then picks up anything pending longer than retryOnIdleMs, bumps its retry count, and redelivers it — and after maxRetries attempts, moves it to the dead-letter queue instead.
The reclaimer's redelivery is a single server-side Lua script, so the "copy to the retry stream" and "ack on the source" steps cannot be torn apart by a crash. There is no window in which a message exists in both places.
publish()
│
▼
┌───────────────────────────┐
│ {orders} │ the queue
└─────────────┬─────────────┘
│ XREADGROUP >
▼
┌───────────────────────────┐
│ PEL · group "svc" │ in flight — delivered to
│ delivered, un-acked │ exactly one consumer in
└───┬───────────────────┬───┘ the group
handler │ │ handler throws, or the
returns ▼ │ worker dies → no XACK,
XACK ✓ │ so it just stays put
done │
│ ...idle > retryOnIdleMs...
▼
┌─────────────────────────┐
│ reclaim.lua │ XADD + XACK as
│ ONE indivisible step │ one atomic op
└────┬───────────────┬────┘
rc + 1 <= maxRetries │ │ rc + 1 > maxRetries
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ {orders}.svc.retry │ │ {orders}.svc.dlq │
│ rc + 1, oid stamped │ │ NEVER trimmed, no TTL │
└────────────┬────────────┘ └────────────┬────────────┘
│ XREADGROUP > │
▼ │ waits for a human:
┌─────────────────────────┐ │
│ PEL · "svc-retry" │ │ dlqMessages() inspect
│ the same handler │ │ redrive() put back
└────────────┬────────────┘ │ purge() delete
│
│ fails again → reclaim.lua → back to the SAME retry
└─ stream, rc + 1 each time, until rc > maxRetries → DLQ
Everything below the pending list only exists if you set retryOnIdleMs — see the warning just below. Without it a failed message stops at the PEL and stays there.
Three things the picture is meant to make obvious:
- Nothing is ever deleted on the way through. A message moves by being copied forward and acked behind, in one atomic step, so there is no moment where it is in neither place — or in both.
- The retry stream loops back to itself, not to a
.retry.retry. Every failed attempt bumpsrcin the same place, so the retry chain never forks. - The DLQ is a terminus, not a stage. Nothing consumes it and nothing trims it. It stays exactly as it is until a human calls
redriveorpurge.
Two shortcuts are not drawn, to keep the main path clear. An entry with no body at all — which only a foreign producer can create — skips the retry chain and goes straight to the DLQ, since no handler could ever process it. And redrive puts a dead letter back on {orders} as a fresh publish: rc reset to 0, no oid, a brand-new id.
A subscriber with no retryOnIdleMs has no visibility timeout at all. There is no reclaimer for it, so a failed delivery sits in the pending list forever — not lost, but not retried either. This is deliberate (no retry config, no
background cost), and it is the sharpest default in the library. If you want "never drops the load", you want retryOnIdleMs.
await dq.subscribe('orders', handler, {
group: 'order-service',
retryOnIdleMs: 30_000, // the visibility timeout — see below
maxRetries: 5, // then it goes to the DLQ (null = retry forever, no DLQ)
backoffMultiplier: 2, // each attempt waits longer
backoffMaxMs: 300_000, // capped, then jittered
onDlq: async (body, originalId, count) => alert(body, originalId, count),
});retryOnIdleMs is the visibility timeout: how long one consumer gets to finish a message before another may take it. Un-acked for longer than that and the message is redelivered — to any worker in the group, across every replica you run. Set it to how long you are willing to let a single attempt hold a message.
Two things follow from that, and they are the same trade-off SQS makes. A handler slower than the timeout will see its message picked up elsewhere while it is still working, so set the timeout above your slowest realistic attempt — or use dedupeTtlMs so the second delivery is acked instead of re-run. And retries are opt-in: with no retryOnIdleMs there is no visibility timeout at all, so a failed message sits un-acked forever, neither retried nor dead-lettered.
Retried messages move to {orders}.<group>.retry; dead letters land in {orders}.<group>.dlq. Both are per-group, so each subscriber has its own DLQ to inspect and redrive.
The DLQ is never trimmed. A dead letter you can't read is worthless, so it grows without a length cap regardless of any other setting. The retry stream can be capped with retryMaxLen, but it defaults to no cap on purpose: trimming a retry entry that is still in flight would drop a message silently, so the default is to grow visibly under a failure storm instead.
stop({ drainTimeoutMs }) stops the read loops immediately (it doesn't wait out the blocking read) and gives an in-flight handler until the timeout to finish and ack.
A handler that overruns the drain timeout is not cancelled. JavaScript cannot cancel a promise. It keeps running to completion, orphaned, while shutdown proceeds — and the process may not exit until it settles.
What you get instead is an AbortSignal:
dq.subscribe('orders', async (msg, ctx) => {
await fetch(url, { signal: ctx.signal }); // stops promptly on drain
});A handler that passes ctx.signal down to its own I/O stops promptly. A handler that ignores it degrades to "keeps running". Either way at-least-once still holds: the message stays pending and gets redelivered.
If you have long-running handlers, read this section again before your first rolling deploy.
Any Standard Schema validator works — Zod, Valibot, ArkType — with no runtime dependency added:
import { z } from 'zod';
const Order = z.object({ id: z.number(), total: z.number() });
dq.subscribe(
'orders',
async (msg) => msg.body.total, // inferred as number
{ group: 'order-service', schema: Order },
);
await dq.publish('orders', { id: 1, total: 42 }, { schema: Order });
await dq.publish('orders', { id: 1 }, { schema: Order }); // ✗ compile errorThe type is inferred from the schema, so the handler's msg.body and the body publish accepts cannot drift from the validator. There is nothing to annotate by hand.
A body that fails validation on consume is treated as poison: it is left un-acked so the reclaimer routes it to the DLQ, rather than crash-looping your handler. For anything that isn't Standard Schema, pass a dump/load pair.
Handlers should be idempotent — at-least-once means "at least". msg.originalMessageId is stable across retries and is the natural dedupe key.
For a shortcut, opt into built-in dedupe with dedupeTtlMs, which records a marker after the handler succeeds and skips a repeat delivery within the window. Set it comfortably longer than your whole retry window.
await dq.depth('orders', 'order-service'); // stream, retry and DLQ sizes at once
await dq.pending('orders', 'order-service'); // in-flight, and which consumer holds it
await dq.lag('orders', 'order-service'); // undelivered backlog (Redis ≥ 7)
await dq.peek('orders', { count: 10 }); // read without consuming
await dq.dlqMessages('orders', 'order-service'); // triage the DLQ
await dq.redrive('orders', 'order-service', id); // put one back
await dq.redriveAll('orders', 'order-service'); // put them all back
await dq.purge('orders', 'order-service', id); // give up on one
await dq.purgeAll('orders', 'order-service'); // give up on all of them
await dq.compact('orders'); // trim what every group has finished with
await dq.registered(); // every (stream, group) xstreamq knows
await dq.healthy(); // is the connection up and are the loops alive?Only purge ever deletes a dead letter. A dead letter that cannot be republished is left in place and reported, never quietly dropped — redriveAll returns it under unredrivable.
The same operations are available from the CLI, which ships in the same package:
xstreamq ls # every (stream, group) it knows
xstreamq depth orders order-service # stream, retry and DLQ sizes
xstreamq pending orders order-service # in flight, and which worker has it
xstreamq peek orders --count 10 # read without consuming
xstreamq dlq ls orders order-service # triage the dead letters
xstreamq dlq redrive orders order-service # put them all back
xstreamq dlq purge orders order-service --yes # asks first without --yes
xstreamq compact orders --json # trim what everyone is done with
xstreamq health # exits 1 when unhealthyPoint it at a server with --url or $XSTREAMQ_URL. --json puts machine-readable output on stdout and keeps warnings on stderr.
xstreamq --help is the full reference — every command, every flag, and what each one does. It is the source of truth; the list above is a taste of it.
xstreamq's no-loss guarantees assume Redis itself doesn't lose data:
- AOF on,
appendfsync everysec - ideally a replica for failover
- idempotent handlers
The 359 unit and integration tests say the logic is right. They do not say what happens at 40,000 messages a second while workers are being killed. loadtest/ does.
It is a Kotlin load generator on JDK virtual threads publishing straight to Redis, with the real TypeScript library consuming, and a supervisor that SIGKILLs consumer processes on a timer. The producer deliberately does not import xstreamq — it reimplements the envelope from the open contract documented above, so a disagreement between the two shows up as a failed run instead of two sides agreeing on the same bug.
One Redis container, 12-core box, 128-byte bodies, loopback, a handler that does nothing.
| publish ceiling | ~44,000 msg/s, flat above 64 connections |
| drain ceiling (nothing publishing) | ~14,000 msg/s across 8 worker processes |
| sustained end-to-end | ~8,000 msg/s, p50 <1ms, p99 290ms, no backlog |
| 10,000 msg/s | marginal — a steady ~1.2s of lag |
| 12,000 msg/s | consumers fall behind, ~4s of lag, drains after the burst |
Publishing is bounded by Redis being single-threaded, and the sweep shows it cleanly: going from 64 to 256 connections leaves throughput flat at ~44k/s while p50 round-trip climbs 1.35ms → 5.69ms. Past the saturation point more connections buy latency, not capacity.
Take the consume numbers as a floor: they include an extra Redis round trip per message that the harness itself adds for kill-safe accounting, and a real handler doing real work will be the limit long before xstreamq is.
Nothing is lost when workers die. 200,000 messages, 6 consumer processes, one SIGKILLed every 400ms — 102 hard kills, zero loss. The retry stream held 6,292 reclaims against only 4,065 injected handler failures, so 2,227 messages were orphaned mid-flight and every one came back via the visibility timeout.
At-least-once has a measured price. Of those kill-orphaned messages, 1.44% ran their handler twice — killed in the window between the handler returning and XACK landing. Turning on dedupeTtlMs halves that to 0.69%, and the reason it only halves it is worth knowing: the marker is written after the handler and is itself a round trip, so it closes the marker→ack window and leaves the handler→marker window open. dedupeTtlMs narrows duplicate delivery. It does not remove it, and nothing here can.
Restarts spend the same retry budget as failures. xstreamq cannot distinguish "the handler failed" from "the worker died" — both are an un-acked entry going idle, and both cost one retry. With maxRetries: 1, zero injected failures, and a kill every 250ms, two messages were dead-lettered having never once had their handler complete, both stamped rc=2. Nothing was lost — they are in the DLQ and redrive puts them back — but a flappy deployment plus a low maxRetries will dead-letter healthy messages.
Retry routing is exact. At a 25% failure rate with maxRetries: 3, a message must fail four times to be dead-lettered: 0.25⁴ × 200,000 = 781 expected, 799 observed — about 0.6σ. Every dead letter carried rc=4 and a stamped oid.
Running it, and the full method and caveats, are in loadtest/README.md.
bun install
bun run redis:up # Redis on :6399 for the integration tests
bun run check # lua sync, lint, format, typecheck, tests
bun run buildTests run on Node (the runtime the package ships to) even though Bun is the package manager. Integration tests skip when no Redis is reachable — point them elsewhere with XSTREAMQ_TEST_REDIS_URL.
The Lua scripts in src/lua/ are the source of truth; bun run lua regenerates the committed src/lua.generated.ts from them, and CI fails if the two drift.
Design notes live in docs/design-docs/ — start with DESIGN.md.
MIT