queuelab

Deterministic discrete-event simulator for job queues: compare ordering, backoff, and retry policies on identical failure scenarios before you ship them.

The problem

Every backend with background work silently runs an experiment: what happens to tail latency and dead-letter volume when 5% of jobs fail transiently, a worker crashes mid-run, and your backoff doubles without a cap? Usually you find out in production, because nothing lets you compare "exponential vs equal-jitter backoff, FIFO vs shortest-job-first" on the same failure pattern.

queuelab makes that experiment local, deterministic, and one command long.

Key features

• Deterministic scenarios — seeded generation of arrivals, durations, and per-job failure scripts; two policies are compared on the exact same workload and failures
• Failure taxonomy — transient (fails, succeeds later), poison (never succeeds → dead-letter), timeout (killed at the timeout, retried), crash (worker goes down, job requeued)
• Pluggable ordering — fifo, lifo, sjf (shortest-job-first)
• Pluggable backoff — fixed, exponential (factor + cap), equal-jitter
• Real metrics — throughput, p50/p95 end-to-end latency, queue wait, dead letters, retry count distribution, worker utilization
• Compare mode — side-by-side matrix across ordering × backoff policies
• Zero dependencies — Python stdlib only

Terminal example

$ python -m cli compare --jobs 400 --workers 4 --fail-rate 0.12 --seed 7

scenario: 400 jobs, 4 workers, 12% failure rate, seed 7
(make-span seconds · p95 = end-to-end seconds · dead = dead-lettered jobs)

ordering  backoff      done  dead  retries   make-span    p50     p95   util
fifo      fixed         371   29      78      412.3s    58.1    190.4   0.91
fifo      exp           371   29      61      388.7s    52.9    171.2   0.92
fifo      eqj           371   29      58      381.9s    51.0    166.8   0.93
sjf       eqj           371   29      58      349.2s    41.7    150.3   0.95
lifo      eqj           371   29      58      377.4s    63.8    185.0   0.94

winner by p95: sjf/eqj  ·  identical done/dead across rows = same scenario

(Values are illustrative of the format; exact numbers come from the seeded run.)

Architecture summary

• src/scenario.py — seeded job generation (arrivals, durations, failure scripts)
• src/policies.py — ordering policies and backoff policies (pure functions)
• src/engine.py — discrete-event loop: workers, backoff requeue, crash restart, dead-lettering
• src/metrics.py — run metrics and percentile math
• src/cli.py — simulate / compare commands, JSON output

Requirements

• Python ≥ 3.10, no third-party packages

Installation

git clone https://github.com/Wsh7Ash/queuelab.git
cd queuelab

Quick start

PYTHONPATH=src python -m cli simulate --jobs 200 --workers 4 --seed 42
PYTHONPATH=src python -m cli compare --jobs 400 --workers 4 --fail-rate 0.12 --seed 7

Usage examples

# Poison jobs + aggressive failures, exponential backoff capped at 30s
PYTHONPATH=src python -m cli simulate --jobs 300 --workers 2 \
    --fail-rate 0.2 --poison-rate 0.05 --timeout 8 \
    --backoff exp --backoff-base 1.0 --backoff-cap 30

# FIFO vs SJF only, JSON for further analysis
PYTHONPATH=src python -m cli compare --policies fifo,sjf --backoffs eqj --json

# Same scenario, more workers
PYTHONPATH=src python -m cli simulate --jobs 500 --workers 8 --seed 9

Configuration

Flag	Meaning
--jobs N	number of jobs to generate
--workers N	worker count
--seed S	RNG seed (same seed ⇒ same scenario everywhere)
--arr-rate R	mean job arrivals per second
--dur-min / --dur-max	job duration range (s)
--fail-rate P	share of jobs whose first attempts fail transiently
--poison-rate P	share of jobs that never succeed
--timeout S	attempt timeout (0 disables timeouts)
--crash-rate P	share of attempts that crash the worker
--order P	fifo | lifo | sjf (simulate; compare takes a list)
--backoff P	fixed | exp | eqj (simulate; compare takes a list)
--backoff-base S	initial backoff delay
--backoff-cap S	maximum backoff delay
--max-attempts N	attempts before dead-letter (default 5)
--json	JSON output

Exit codes: 0 success · 2 usage error.

Testing instructions

PYTHONPATH=src python -m unittest discover -s tests

Project structure

queuelab/
├─ src/            scenario.py, policies.py, engine.py, metrics.py, cli.py
├─ tests/          engine/policy/metrics/CLI tests
├─ scripts/demo.py — produces results/ artifacts
├─ results/        demo output (report + comparison JSON)
├─ docs/           ARCHITECTURE.md, USAGE.md, PROJECT_SPEC.md
└─ .github/workflows/ci.yml

Limitations

• Single queue, homogeneous workers; no priorities, queues-per-topic, or rate limits yet
• Failure scripts are independent per job; correlated failures (one dependency down) are not modeled
• Attempt durations are simulated, not profiled from a real queue

Roadmap

• Per-topic queues and worker pools
• Correlated-failure mode ("dependency X down for 30s")
• CSV export of the compare matrix

Security notes

Pure local simulation. No network access, no subprocesses, no filesystem writes beyond the artifacts you redirect. Nothing here connects to a real queue system.

License

MIT