From 17ea7555a1c9a2c4d36ee291b6a00cf1ece64bff Mon Sep 17 00:00:00 2001 From: MaciejZiel Date: Wed, 14 Oct 2026 22:21:37 +0200 Subject: [PATCH] Rerun the benchmark counterbalanced on a quieter machine and update the numbers and caveats --- README.md | 55 +- bench/results.json | 1841 ++++++++++++++++++++++---------- docs/benchmarks-latency.svg | 81 +- docs/benchmarks-throughput.svg | 45 +- docs/benchmarks.md | 214 ++-- 5 files changed, 1452 insertions(+), 784 deletions(-) diff --git a/README.md b/README.md index cb81132..1efcdeb 100644 --- a/README.md +++ b/README.md @@ -449,51 +449,46 @@ What happens when… `make bench` runs a reproducible benchmark against a real PostgreSQL and, next to it, Celery on Valkey (the BSD-licensed Redis fork) with the same no-op handler, the same 2 KiB JSON payload and 32 job slots per worker process. Workers are separate processes -started through the real CLI. Each cell is repeated 5 times. The full method, every table, -the SKIP LOCKED explanation and the caveats are in [`docs/benchmarks.md`](docs/benchmarks.md); -raw per-run numbers are in [`bench/results.json`](bench/results.json). The benchmark script -now alternates which system runs first and pauses between them; the numbers here predate that -change (see the doc). +started through the real CLI. Each cell is repeated 6 times, with the order of the two +systems alternating and a 30 s cool-down between them. The full method, every table, the +SKIP LOCKED explanation and the caveats are in [`docs/benchmarks.md`](docs/benchmarks.md); +raw per-run numbers are in [`bench/results.json`](bench/results.json). Read the numbers with these caveats first: one 13th Gen Intel i5-13500H laptop (16 threads, 14 GiB RAM) running Postgres 17 with default settings, Valkey, the workers and the load -generator together, **and busy with unrelated work during the runs** (1-minute load average -between 2 and 38). Run-to-run spread is large, so cells are the median with the minimum and -maximum run in brackets. This is a first run: the order was fixed (pgqueue first, then -Celery), so it is not counterbalanced, and it has not been repeated on a quiet machine. +generator together. The machine was not dedicated: the 1-minute load average was between +0.8 and 3.6 before each drain measurement and peaked at 11 during the run. Cells are the +median with the minimum and maximum run in brackets. | Backlog of 20,000 jobs, jobs/s | pgqueue | Celery (thread pool, 32 threads, prefetch x4, early ack) | |---:|---:|---:| -| 1 worker process | 825 (725-1,399) | 349 (280-499) | -| 2 worker processes | 1,702 (1,427-2,528) | 722 (574-1,268) | -| 4 worker processes | 3,915 (2,444-5,475) | 1,765 (990-3,354) | -| 8 worker processes | 4,161 (2,589-8,142) | 3,422 (1,683-6,747) | +| 1 worker process | 1,323 (1,182-1,347) | 441 (400-445) | +| 2 worker processes | 2,788 (2,431-2,907) | 840 (725-1,000) | +| 4 worker processes | 5,267 (3,870-5,577) | 2,950 (1,279-3,003) | +| 8 worker processes | 10,103 (6,927-10,424) | 6,650 (2,292-6,693) | ![Backlog throughput by number of worker processes](docs/benchmarks-throughput.svg) What this does and does not show: -- The robust result is at 1 and 2 worker processes: pgqueue was about 2 times faster - (1.8 to 2.8 times) in all 5 runs. At 4 and 8 workers the claim is weaker. On the two - quietest runs (4 and 5, load about 3 to 6) the two were roughly even at 8 workers - (ratios 0.84 and 1.38) and pgqueue led by 1.21 and 2.16 at 4 workers. In the 4-worker - payload-size scenario Celery won some runs: at 2 KiB the ratios were 3.52, 2.23, 0.89, - 1.44 and 1.43, and at 256 B pgqueue lost 2 of 5 runs and its median was below Celery's - (2,998 against 3,163 jobs/s). Within a run pgqueue was always measured first and Celery - a few seconds after, so the order is not counterbalanced, and the load was not the same - for both halves. The absolute numbers moved by up to a factor of four with how busy the - machine was. +- pgqueue drained the backlog faster than this Celery configuration in all 6 runs at every + worker count: about 3 times at 1 and 2 worker processes, and about 1.3 to 1.9 times at 4 + and 8 in the typical runs. Three Celery cells (4 workers run 6, 8 workers runs 5 and 6) + were two to three times slower than its other runs for a reason that was not + investigated, which pulls the median ratio up. The order did not matter: pgqueue led + whether it ran first or second. The 3 times lead at 1 and 2 workers is largely a + property of one thread-pool process under the GIL against an asyncio worker. - The two are not equal in durability. pgqueue commits the claim and the completion of each job with an fsync; this Celery setup acknowledges early (`acks_late=False`, a crash loses in-flight tasks) on a Valkey without persistence. So the drain comparison is a durable queue against a non-durable one. - Enqueue is where Postgres pays for durability: a single `enqueue()` call (one fsynced - commit) took a median of 4.4 ms at 184 jobs/s for one client, against 1.7 ms and 566 + commit) took a median of 4.4 ms at 226 jobs/s for one client, against 0.8 ms and 1,251 jobs/s for a Celery publish to a Valkey without persistence. `enqueue_many` inserted - 1,000 jobs per call at 6,188 jobs/s (median; range 5,181 to 14,792). -- Pickup latency at 200 jobs/s into idle workers had a p50 of 4.5 to 7.3 ms for pgqueue - and 3.5 to 4.5 ms for Celery depending on the worker count; pgqueue's p99 was 15 to - 38 ms, Celery's 6 to 11 ms. The intervals differ: pgqueue's runs from `created_at` + 1,000 jobs per call at 14,162 jobs/s (median; range 9,413 to 15,559). +- Pickup latency at 200 jobs/s into idle workers had a p50 of 3.9 to 6.5 ms for pgqueue + and 3.4 to 3.6 ms for Celery depending on the worker count; pgqueue's p99 was 9 to + 22 ms, Celery's 4.1 to 4.7 ms. The intervals differ: pgqueue's runs from `created_at` (before the fsynced insert commit) to `started_at` (set in the claim statement), Celery's from just before `apply_async` serialises the message to the start of the handler. The ratios are not like-for-like. @@ -506,8 +501,8 @@ What this does and does not show: index per queue through a `LATERAL` subquery brought that to ~0.1 ms and doubled throughput. -Reproduce: `make bench` (needs Docker and uv; about an hour), then `make bench-chart` to -redraw the charts. `make bench-quick` checks the setup in a few minutes. +Reproduce: `make bench ARGS="--runs 6"` (needs Docker and uv; about 25 minutes), then +`make bench-chart` to redraw the charts. `make bench-quick` checks the setup in a few minutes. ## Design decisions and trade-offs diff --git a/bench/results.json b/bench/results.json index 9c49fbb..b699074 100644 --- a/bench/results.json +++ b/bench/results.json @@ -1,4 +1,5 @@ { + "schema_version": 2, "environment": { "cpu": "13th Gen Intel(R) Core(TM) i5-13500H", "logical_cpus": "16", @@ -6,14 +7,14 @@ "os": "Linux 7.0.0-38-generic", "python": "3.12.13", "postgres": "17.11 {'shared_buffers': '128MB', 'synchronous_commit': 'on', 'fsync': 'on', 'max_connections': '100'}", - "load_average_at_start": "38.62 28.42 17.01", + "load_average_at_start": "1.06 0.65 0.65", "broker": "valkey 7.2.4", "celery": "celery 5.6.3, redis-py 8.1.0", "persistence": "{'save': ''} appendonly={'appendonly': 'no'}", - "load_average_at_end": "4.66 4.80 7.46" + "load_average_at_end": "1.85 2.07 2.48" }, "parameters": { - "runs": 5, + "runs": 6, "jobs": 20000, "slots_per_worker": 32, "arrival_rate": 200, @@ -25,1868 +26,2566 @@ 8 ], "payload_bytes": 2048, - "quick": false + "quick": false, + "counterbalanced": true, + "cooldown_s": 30.0 }, "records": [ { "system": "pgqueue", "scenario": "enqueue", "run": 1, + "position": "first", "payload": 2048, "variant": "single", "jobs": 5000, - "seconds": 45.08915296100167, - "rate": 110.89141559888206, + "seconds": 22.220582092006225, + "rate": 225.01660754417105, "latency_ms": { - "p50": 5.584774498856859, - "p95": 19.54631574844825, - "p99": 66.51129696903809, - "max": 427.9835440029274, + "p50": 4.405134000990074, + "p95": 5.185386451921659, + "p99": 5.869424670672743, + "max": 31.20494399627205, "n": 5000.0 }, - "load_1min": "37.73" + "load_1min_before": "1.06", + "load_1min": "1.13" }, { "system": "celery", "scenario": "enqueue", "run": 1, + "position": "second", "payload": 2048, "variant": "single", "jobs": 5000, - "seconds": 8.862754460998985, - "rate": 564.1586960355002, + "seconds": 3.984219950987608, + "rate": 1254.9507962683135, "latency_ms": { - "p50": 1.6649319986754563, - "p95": 2.4693350449524587, - "p99": 4.84756686273613, - "max": 315.6045110008563, + "p50": 0.7658929971512407, + "p95": 1.0201154465903528, + "p99": 1.2520882723038085, + "max": 82.95009499124717, "n": 5000.0 }, - "load_1min": "36.85" + "load_1min_before": "0.75", + "load_1min": "0.77" }, { "system": "pgqueue", "scenario": "enqueue", "run": 1, + "position": "first", "payload": 2048, "variant": "ten", "jobs": 5000, - "seconds": 28.581017406002502, - "rate": 174.9412880924916, + "seconds": 14.96796544099925, + "rate": 334.04673599154194, "latency_ms": { - "p50": 51.301703500939766, - "p95": 100.73975665363832, - "p99": 153.63346406302298, - "max": 574.3456950003747, + "p50": 29.909154007327743, + "p95": 34.18318459080183, + "p99": 41.39343891016324, + "max": 154.34732800349593, "n": 5000.0 }, - "load_1min": "32.20" + "load_1min_before": "0.77", + "load_1min": "0.98" }, { "system": "celery", "scenario": "enqueue", "run": 1, + "position": "second", "payload": 2048, "variant": "ten", "jobs": 5000, - "seconds": 10.650425011001062, - "rate": 469.4648330780592, + "seconds": 6.853263473007246, + "rate": 729.5794214965407, "latency_ms": { - "p50": 18.792613998812158, - "p95": 37.44688920414774, - "p99": 61.80480238464952, - "max": 696.4193099993281, + "p50": 9.50618749629939, + "p95": 36.5536860008433, + "p99": 63.40394086219021, + "max": 366.98244500439614, "n": 5000.0 }, - "load_1min": "28.11" + "load_1min_before": "0.74", + "load_1min": "0.84" }, { "system": "pgqueue", "scenario": "enqueue", "run": 1, + "position": "only", "payload": 2048, "variant": "bulk", "jobs": 40000, - "seconds": 7.72024922700075, - "rate": 5181.179884724998, + "seconds": 2.9821228320070077, + "rate": 13413.263722969949, "latency_ms": { - "p50": 190.38571499913814, - "p95": 229.7530514515529, - "p99": 245.02151536231395, - "max": 251.8819960023393, + "p50": 69.23781200748635, + "p95": 94.50960769754602, + "p99": 109.78593462088611, + "max": 115.54221098776907, "n": 40.0 }, - "load_1min": "25.36" + "load_1min_before": "0.84", + "load_1min": "0.85" }, { "system": "pgqueue", "scenario": "drain", "run": 1, + "position": "first", "workers": 1, "payload": 2048, "jobs": 20000, - "rate": 825.2743848281829, - "load_1min": "22.02" + "rate": 1264.2594909195432, + "load_1min_before": "0.85", + "load_1min": "1.63" }, { "system": "celery", "scenario": "drain", "run": 1, + "position": "second", "workers": 1, "payload": 2048, "jobs": 20000, - "rate": 307.091147584142, - "load_1min": "20.19" + "rate": 443.69683353961636, + "load_1min_before": "1.13", + "load_1min": "1.35" }, { "system": "pgqueue", "scenario": "drain", "run": 1, + "position": "first", "workers": 2, "payload": 2048, "jobs": 20000, - "rate": 1426.7025988550924, - "load_1min": "19.97" + "rate": 2647.7063502050128, + "load_1min_before": "1.35", + "load_1min": "2.02" }, { "system": "celery", "scenario": "drain", "run": 1, + "position": "second", "workers": 2, "payload": 2048, "jobs": 20000, - "rate": 573.8004467968121, - "load_1min": "17.39" + "rate": 813.3528508355771, + "load_1min_before": "2.20", + "load_1min": "3.57" }, { "system": "pgqueue", "scenario": "drain", "run": 1, + "position": "first", "workers": 4, "payload": 2048, "jobs": 20000, - "rate": 2469.361782380255, - "load_1min": "14.28" + "rate": 3870.436218406955, + "load_1min_before": "3.57", + "load_1min": "4.78" }, { "system": "celery", "scenario": "drain", "run": 1, + "position": "second", "workers": 4, "payload": 2048, "jobs": 20000, - "rate": 990.1651978208564, - "load_1min": "15.43" + "rate": 2988.149717535591, + "load_1min_before": "3.09", + "load_1min": "3.21" }, { "system": "pgqueue", "scenario": "drain", "run": 1, + "position": "first", "workers": 8, "payload": 2048, "jobs": 20000, - "rate": 2588.8792343892146, - "load_1min": "19.10" + "rate": 10250.360471765896, + "load_1min_before": "3.21", + "load_1min": "3.51" }, { "system": "celery", "scenario": "drain", "run": 1, + "position": "second", "workers": 8, "payload": 2048, "jobs": 20000, - "rate": 1858.7040055432692, - "load_1min": "16.65" + "rate": 6651.826693301648, + "load_1min_before": "2.61", + "load_1min": "3.16" }, { "system": "pgqueue", "scenario": "pickup", "run": 1, + "position": "first", "workers": 1, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 7.343, - "p95": 17.358800000000002, - "p99": 56.35592, - "max": 145.712, + "p50": 6.664, + "p95": 12.802150000000001, + "p99": 20.9498, + "max": 74.276, "n": 3000.0 }, - "load_1min": "14.11" + "load_1min_before": "3.16", + "load_1min": "2.52" }, { "system": "celery", "scenario": "pickup", "run": 1, + "position": "second", "workers": 1, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 5.315542221069336, - "p95": 53.596627712249756, - "p99": 87.71420240402222, - "max": 137.86053657531738, + "p50": 3.3152103424072266, + "p95": 3.721010684967041, + "p99": 3.9258670806884766, + "max": 9.947538375854492, "n": 3000.0 }, - "load_1min": "13.86" + "load_1min_before": "1.53", + "load_1min": "1.40" }, { "system": "pgqueue", "scenario": "pickup", "run": 1, + "position": "first", "workers": 2, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 5.207000000000001, - "p95": 14.30675, - "p99": 52.77924, - "max": 262.881, + "p50": 4.4205000000000005, + "p95": 9.47525, + "p99": 12.660169999999999, + "max": 26.429, "n": 3000.0 }, - "load_1min": "15.25" + "load_1min_before": "1.40", + "load_1min": "1.68" }, { "system": "celery", "scenario": "pickup", "run": 1, + "position": "second", "workers": 2, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 4.410028457641602, - "p95": 8.52506160736084, - "p99": 10.952956676483154, - "max": 20.374298095703125, + "p50": 3.4422874450683594, + "p95": 3.754889965057373, + "p99": 3.9539361000061035, + "max": 5.481958389282227, "n": 3000.0 }, - "load_1min": "13.99" + "load_1min_before": "1.10", + "load_1min": "1.02" }, { "system": "pgqueue", "scenario": "pickup", "run": 1, + "position": "first", "workers": 4, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 4.472, - "p95": 8.7764, - "p99": 15.189280000000002, - "max": 49.061, + "p50": 4.092, + "p95": 8.8034, + "p99": 12.07727, + "max": 26.101, "n": 3000.0 }, - "load_1min": "18.60" + "load_1min_before": "1.02", + "load_1min": "1.44" }, { "system": "celery", "scenario": "pickup", "run": 1, + "position": "second", "workers": 4, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 4.246354103088379, - "p95": 8.142495155334473, - "p99": 10.999796390533447, - "max": 17.753124237060547, + "p50": 3.385782241821289, + "p95": 3.980743885040283, + "p99": 6.421833038330078, + "max": 7.683277130126953, "n": 3000.0 }, - "load_1min": "17.36" + "load_1min_before": "0.87", + "load_1min": "0.84" }, { "system": "pgqueue", "scenario": "pickup", "run": 1, + "position": "first", "workers": 8, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 4.4544999999999995, - "p95": 7.4091499999999995, - "p99": 10.244090000000002, - "max": 28.264, + "p50": 3.8445, + "p95": 5.8530999999999995, + "p99": 6.189, + "max": 30.694, "n": 3000.0 }, - "load_1min": "23.00" + "load_1min_before": "0.84", + "load_1min": "2.23" }, { "system": "celery", "scenario": "pickup", "run": 1, + "position": "second", "workers": 8, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 4.472017288208008, - "p95": 8.271324634552002, - "p99": 11.251399517059326, - "max": 26.91960334777832, + "p50": 3.5657882690429688, + "p95": 4.561960697174072, + "p99": 5.557565689086914, + "max": 6.857395172119141, "n": 3000.0 }, - "load_1min": "18.93" + "load_1min_before": "1.86", + "load_1min": "2.46" }, { "system": "pgqueue", "scenario": "drain-size", "run": 1, + "position": "first", "workers": 4, "payload": 256, "label": "small", "jobs": 10000, - "rate": 3498.9614549285675, - "load_1min": "18.85" + "rate": 5218.099246677529, + "load_1min_before": "2.46", + "load_1min": "2.46" }, { "system": "celery", "scenario": "drain-size", "run": 1, + "position": "second", "workers": 4, "payload": 256, "label": "small", "jobs": 10000, - "rate": 1028.8071280978897, - "load_1min": "17.19" + "rate": 3105.573555047941, + "load_1min_before": "1.74", + "load_1min": "2.06" }, { "system": "pgqueue", "scenario": "drain-size", "run": 1, + "position": "first", "workers": 4, "payload": 2048, "label": "medium", "jobs": 10000, - "rate": 3643.3321141515503, - "load_1min": "18.53" + "rate": 5453.799326173895, + "load_1min_before": "2.06", + "load_1min": "1.97" }, { "system": "celery", "scenario": "drain-size", "run": 1, + "position": "second", "workers": 4, "payload": 2048, "label": "medium", "jobs": 10000, - "rate": 1035.9953899817565, - "load_1min": "17.69" + "rate": 3012.0127078478044, + "load_1min_before": "1.20", + "load_1min": "1.20" }, { "system": "pgqueue", "scenario": "drain-size", "run": 1, + "position": "first", "workers": 4, "payload": 16384, "label": "large", "jobs": 10000, - "rate": 3132.2093842973727, - "load_1min": "15.35" + "rate": 4647.228462643379, + "load_1min_before": "1.20", + "load_1min": "1.41" }, { "system": "celery", "scenario": "drain-size", "run": 1, + "position": "second", "workers": 4, "payload": 16384, "label": "large", "jobs": 10000, - "rate": 984.2144138493258, - "load_1min": "19.98" + "rate": 2552.2354964547553, + "load_1min_before": "1.12", + "load_1min": "1.60" }, { - "system": "pgqueue", + "system": "celery", "scenario": "enqueue", "run": 2, + "position": "first", "payload": 2048, "variant": "single", "jobs": 5000, - "seconds": 24.20208390300104, - "rate": 206.5937801075057, + "seconds": 3.931971761994646, + "rate": 1271.6266297557424, "latency_ms": { - "p50": 4.386684999190038, - "p95": 7.335558452177793, - "p99": 9.644859913314576, - "max": 24.75346599385375, + "p50": 0.7702764924033545, + "p95": 1.0200462573266122, + "p99": 1.3103569310624152, + "max": 1.873966000857763, "n": 5000.0 }, - "load_1min": "17.52" + "load_1min_before": "1.60", + "load_1min": "1.55" }, { - "system": "celery", + "system": "pgqueue", "scenario": "enqueue", "run": 2, + "position": "second", "payload": 2048, "variant": "single", "jobs": 5000, - "seconds": 14.38342478100094, - "rate": 347.62235532419913, + "seconds": 22.007179126987467, + "rate": 227.1985869315021, "latency_ms": { - "p50": 2.87959200431942, - "p95": 4.34535954882449, - "p99": 5.902515582929482, - "max": 8.470221000607125, + "p50": 4.443391007953323, + "p95": 4.99411056007375, + "p99": 5.425022617273498, + "max": 24.614309004391544, "n": 5000.0 }, - "load_1min": "16.85" + "load_1min_before": "0.94", + "load_1min": "0.96" }, { - "system": "pgqueue", + "system": "celery", "scenario": "enqueue", "run": 2, + "position": "first", "payload": 2048, "variant": "ten", "jobs": 5000, - "seconds": 13.218141149001895, - "rate": 378.2680139088658, + "seconds": 6.317593824991491, + "rate": 791.4405608383244, "latency_ms": { - "p50": 25.435657498746878, - "p95": 28.870913250284502, - "p99": 33.9327484772366, - "max": 403.73103599995375, + "p50": 8.944565001002047, + "p95": 34.169657048914814, + "p99": 53.16304006773862, + "max": 206.10181899974123, "n": 5000.0 }, - "load_1min": "15.11" + "load_1min_before": "0.96", + "load_1min": "1.21" }, { - "system": "celery", + "system": "pgqueue", "scenario": "enqueue", "run": 2, + "position": "second", "payload": 2048, "variant": "ten", "jobs": 5000, - "seconds": 11.309189368999796, - "rate": 442.11833729707973, + "seconds": 11.991326415009098, + "rate": 416.9680506521518, "latency_ms": { - "p50": 16.215782503422815, - "p95": 57.99709665225237, - "p99": 89.74461567711842, - "max": 765.4800780001096, + "p50": 23.791327999788336, + "p95": 25.53958440112183, + "p99": 27.484948787023313, + "max": 139.0362020028988, "n": 5000.0 }, - "load_1min": "13.65" + "load_1min_before": "1.08", + "load_1min": "1.21" }, { "system": "pgqueue", "scenario": "enqueue", "run": 2, + "position": "only", "payload": 2048, "variant": "bulk", "jobs": 40000, - "seconds": 6.464510794001399, - "rate": 6187.629857021374, + "seconds": 2.682762027005083, + "rate": 14910.006775611862, "latency_ms": { - "p50": 153.09613199860905, - "p95": 244.95854519955174, - "p99": 258.40662220864033, - "max": 266.74574299977394, + "p50": 67.32783350162208, + "p95": 74.8176977467665, + "p99": 76.36229353141971, + "max": 76.54468600230757, "n": 40.0 }, - "load_1min": "12.02" + "load_1min_before": "1.21", + "load_1min": "1.19" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain", "run": 2, + "position": "first", "workers": 1, "payload": 2048, "jobs": 20000, - "rate": 725.4323605376657, - "load_1min": "11.77" + "rate": 437.9648932571535, + "load_1min_before": "1.19", + "load_1min": "1.57" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain", "run": 2, + "position": "second", "workers": 1, "payload": 2048, "jobs": 20000, - "rate": 279.97636674101545, - "load_1min": "10.39" + "rate": 1315.0251182736617, + "load_1min_before": "1.13", + "load_1min": "1.68" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain", "run": 2, + "position": "first", "workers": 2, "payload": 2048, "jobs": 20000, - "rate": 1701.982613279956, - "load_1min": "11.97" + "rate": 842.7868833661064, + "load_1min_before": "1.68", + "load_1min": "1.88" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain", "run": 2, + "position": "second", "workers": 2, "payload": 2048, "jobs": 20000, - "rate": 721.9893430575147, - "load_1min": "7.89" + "rate": 2749.528023159216, + "load_1min_before": "1.37", + "load_1min": "1.39" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain", "run": 2, + "position": "first", "workers": 4, "payload": 2048, "jobs": 20000, - "rate": 2443.879146019566, - "load_1min": "7.27" + "rate": 3002.868770399807, + "load_1min_before": "1.39", + "load_1min": "1.81" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain", "run": 2, + "position": "second", "workers": 4, "payload": 2048, "jobs": 20000, - "rate": 1205.9949418590977, - "load_1min": "10.56" + "rate": 5178.927154873519, + "load_1min_before": "1.25", + "load_1min": "1.47" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain", "run": 2, + "position": "first", "workers": 8, "payload": 2048, "jobs": 20000, - "rate": 3892.883482023774, - "load_1min": "13.64" + "rate": 6648.215884924776, + "load_1min_before": "1.47", + "load_1min": "1.58" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain", "run": 2, + "position": "second", "workers": 8, "payload": 2048, "jobs": 20000, - "rate": 1683.2564164182097, - "load_1min": "20.19" + "rate": 9916.11552233226, + "load_1min_before": "1.19", + "load_1min": "3.02" }, { - "system": "pgqueue", + "system": "celery", "scenario": "pickup", "run": 2, + "position": "first", "workers": 1, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 6.158, - "p95": 15.108899999999998, - "p99": 152.05146, - "max": 276.808, + "p50": 3.464937210083008, + "p95": 4.184079170227051, + "p99": 5.280892848968506, + "max": 10.369300842285156, "n": 3000.0 }, - "load_1min": "19.72" + "load_1min_before": "3.02", + "load_1min": "2.51" }, { - "system": "celery", + "system": "pgqueue", "scenario": "pickup", "run": 2, + "position": "second", "workers": 1, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 3.840327262878418, - "p95": 8.82648229598999, - "p99": 11.942691802978516, - "max": 21.192312240600586, + "p50": 6.3725, + "p95": 12.566300000000002, + "p99": 23.506700000000002, + "max": 79.619, "n": 3000.0 }, - "load_1min": "20.29" + "load_1min_before": "1.81", + "load_1min": "1.84" }, { - "system": "pgqueue", + "system": "celery", "scenario": "pickup", "run": 2, + "position": "first", "workers": 2, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 4.343, - "p95": 8.4795, - "p99": 58.6489, - "max": 165.783, + "p50": 3.5587549209594727, + "p95": 4.475581645965576, + "p99": 5.296525955200195, + "max": 6.313800811767578, "n": 3000.0 }, - "load_1min": "21.15" + "load_1min_before": "1.84", + "load_1min": "1.69" }, { - "system": "celery", + "system": "pgqueue", "scenario": "pickup", "run": 2, + "position": "second", "workers": 2, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 3.606081008911133, - "p95": 6.586277484893799, - "p99": 8.820066452026367, - "max": 15.003204345703125, + "p50": 4.391500000000001, + "p95": 10.4652, + "p99": 12.951749999999999, + "max": 34.056, "n": 3000.0 }, - "load_1min": "20.27" + "load_1min_before": "1.36", + "load_1min": "1.70" }, { - "system": "pgqueue", + "system": "celery", "scenario": "pickup", "run": 2, + "position": "first", "workers": 4, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 4.532, - "p95": 9.49195, - "p99": 15.620459999999998, - "max": 45.115, + "p50": 3.536701202392578, + "p95": 4.624462127685547, + "p99": 5.495924949645996, + "max": 7.063388824462891, "n": 3000.0 }, - "load_1min": "20.29" + "load_1min_before": "1.70", + "load_1min": "1.48" }, { - "system": "celery", + "system": "pgqueue", "scenario": "pickup", "run": 2, + "position": "second", "workers": 4, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 3.50797176361084, - "p95": 5.929052829742432, - "p99": 8.112523555755615, - "max": 13.669013977050781, + "p50": 3.932, + "p95": 6.45755, + "p99": 11.733730000000001, + "max": 25.424, "n": 3000.0 }, - "load_1min": "18.90" + "load_1min_before": "0.90", + "load_1min": "1.68" }, { - "system": "pgqueue", + "system": "celery", "scenario": "pickup", "run": 2, + "position": "first", "workers": 8, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 5.4835, - "p95": 12.36155, - "p99": 17.50569, - "max": 29.213, + "p50": 3.5921335220336914, + "p95": 4.738724231719971, + "p99": 5.784173011779785, + "max": 7.2021484375, "n": 3000.0 }, - "load_1min": "19.23" + "load_1min_before": "1.68", + "load_1min": "1.45" }, { - "system": "celery", + "system": "pgqueue", "scenario": "pickup", "run": 2, + "position": "second", "workers": 8, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 4.563808441162109, - "p95": 9.957492351531982, - "p99": 14.362025260925293, - "max": 28.88655662536621, + "p50": 3.799, + "p95": 5.85735, + "p99": 6.198040000000001, + "max": 26.165, "n": 3000.0 }, - "load_1min": "22.38" + "load_1min_before": "0.93", + "load_1min": "2.37" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain-size", "run": 2, + "position": "first", "workers": 4, "payload": 256, "label": "small", "jobs": 10000, - "rate": 2318.7867377223224, - "load_1min": "23.87" + "rate": 3080.7180711196197, + "load_1min_before": "2.37", + "load_1min": "2.79" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain-size", "run": 2, + "position": "second", "workers": 4, "payload": 256, "label": "small", "jobs": 10000, - "rate": 1159.339103356094, - "load_1min": "26.01" + "rate": 5347.571183584767, + "load_1min_before": "1.76", + "load_1min": "1.76" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain-size", "run": 2, + "position": "first", "workers": 4, "payload": 2048, "label": "medium", "jobs": 10000, - "rate": 2329.100608076651, - "load_1min": "25.17" + "rate": 3010.4783466546874, + "load_1min_before": "1.76", + "load_1min": "1.79" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain-size", "run": 2, + "position": "second", "workers": 4, "payload": 2048, "label": "medium", "jobs": 10000, - "rate": 1045.4916602035116, - "load_1min": "25.18" + "rate": 5348.0175687197225, + "load_1min_before": "1.08", + "load_1min": "1.64" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain-size", "run": 2, + "position": "first", "workers": 4, "payload": 16384, "label": "large", "jobs": 10000, - "rate": 2599.9615013736934, - "load_1min": "20.40" + "rate": 2570.9903140638953, + "load_1min_before": "1.64", + "load_1min": "2.01" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain-size", "run": 2, + "position": "second", "workers": 4, "payload": 16384, "label": "large", "jobs": 10000, - "rate": 992.3490402814351, - "load_1min": "15.07" + "rate": 4798.065721186009, + "load_1min_before": "1.29", + "load_1min": "2.44" }, { "system": "pgqueue", "scenario": "enqueue", "run": 3, + "position": "first", "payload": 2048, "variant": "single", "jobs": 5000, - "seconds": 34.49851580500399, - "rate": 144.93377130371368, + "seconds": 22.119133688000147, + "rate": 226.04863601473463, "latency_ms": { - "p50": 5.66179100133013, - "p95": 14.014201795362169, - "p99": 20.782233447243925, - "max": 541.6902500001015, + "p50": 4.4447284963098355, + "p95": 5.026520093088038, + "p99": 5.456768015719717, + "max": 24.699390996829607, "n": 5000.0 }, - "load_1min": "11.23" + "load_1min_before": "2.44", + "load_1min": "2.03" }, { "system": "celery", "scenario": "enqueue", "run": 3, + "position": "second", "payload": 2048, "variant": "single", "jobs": 5000, - "seconds": 8.841205862998322, - "rate": 565.5337153640655, + "seconds": 4.010029918994405, + "rate": 1246.8734899748204, "latency_ms": { - "p50": 1.7083099992305506, - "p95": 2.79489669483155, - "p99": 4.212858436003444, - "max": 10.707467001338955, + "p50": 0.7812469993950799, + "p95": 1.0271240484144073, + "p99": 1.2705230893334374, + "max": 2.1184090001042932, "n": 5000.0 }, - "load_1min": "10.57" + "load_1min_before": "1.29", + "load_1min": "1.27" }, { "system": "pgqueue", "scenario": "enqueue", "run": 3, + "position": "first", "payload": 2048, "variant": "ten", "jobs": 5000, - "seconds": 19.933400506000908, - "rate": 250.83527512000578, + "seconds": 12.851176984986523, + "rate": 389.06942187795596, "latency_ms": { - "p50": 38.08796299927053, - "p95": 52.84257829807757, - "p99": 100.80564835472615, - "max": 492.93822600157, + "p50": 24.331189997610636, + "p95": 34.90920330877998, + "p99": 42.618898894870654, + "max": 140.27189099579118, "n": 5000.0 }, - "load_1min": "12.33" + "load_1min_before": "1.27", + "load_1min": "1.28" }, { "system": "celery", "scenario": "enqueue", "run": 3, + "position": "second", "payload": 2048, "variant": "ten", "jobs": 5000, - "seconds": 7.887904863004223, - "rate": 633.8818845864829, + "seconds": 6.431791134993546, + "rate": 777.3884280533338, "latency_ms": { - "p50": 12.610375499207294, - "p95": 36.40313139658247, - "p99": 52.02377141395118, - "max": 263.5603800008539, + "p50": 9.459276501729619, + "p95": 33.66882719856221, + "p99": 53.03833416764974, + "max": 249.6415860077832, "n": 5000.0 }, - "load_1min": "13.58" + "load_1min_before": "0.78", + "load_1min": "0.87" }, { "system": "pgqueue", "scenario": "enqueue", "run": 3, + "position": "only", "payload": 2048, "variant": "bulk", "jobs": 40000, - "seconds": 6.934848261997104, - "rate": 5767.970471567429, + "seconds": 2.570782948998385, + "rate": 15559.462153576438, "latency_ms": { - "p50": 170.0141290020838, - "p95": 221.04751035149093, - "p99": 256.4960592490388, - "max": 260.2016149976407, + "p50": 64.11520550318528, + "p95": 66.80719259311445, + "p99": 69.86466591246426, + "max": 71.22016200446524, "n": 40.0 }, - "load_1min": "14.44" + "load_1min_before": "0.87", + "load_1min": "1.13" }, { "system": "pgqueue", "scenario": "drain", "run": 3, + "position": "first", "workers": 1, "payload": 2048, "jobs": 20000, - "rate": 741.5935921056312, - "load_1min": "15.74" + "rate": 1331.0328366605754, + "load_1min_before": "1.13", + "load_1min": "1.92" }, { "system": "celery", "scenario": "drain", "run": 3, + "position": "second", "workers": 1, "payload": 2048, "jobs": 20000, - "rate": 348.83336808476884, - "load_1min": "17.27" + "rate": 445.2475697833403, + "load_1min_before": "1.35", + "load_1min": "1.51" }, { "system": "pgqueue", "scenario": "drain", "run": 3, + "position": "first", "workers": 2, "payload": 2048, "jobs": 20000, - "rate": 1675.3718797694773, - "load_1min": "18.24" + "rate": 2906.6151586434394, + "load_1min_before": "1.51", + "load_1min": "1.51" }, { "system": "celery", "scenario": "drain", "run": 3, + "position": "second", "workers": 2, "payload": 2048, "jobs": 20000, - "rate": 662.4256528215456, - "load_1min": "11.96" + "rate": 836.9712536976522, + "load_1min_before": "0.98", + "load_1min": "1.54" }, { "system": "pgqueue", "scenario": "drain", "run": 3, + "position": "first", "workers": 4, "payload": 2048, "jobs": 20000, - "rate": 3914.6534252165925, - "load_1min": "11.47" + "rate": 5355.953096454652, + "load_1min_before": "1.54", + "load_1min": "3.06" }, { "system": "celery", "scenario": "drain", "run": 3, + "position": "second", "workers": 4, "payload": 2048, "jobs": 20000, - "rate": 1764.5011301271961, - "load_1min": "9.06" + "rate": 2972.6830652127546, + "load_1min_before": "2.06", + "load_1min": "2.01" }, { "system": "pgqueue", "scenario": "drain", "run": 3, + "position": "first", "workers": 8, "payload": 2048, "jobs": 20000, - "rate": 4160.502414186186, - "load_1min": "8.88" + "rate": 10022.349924953316, + "load_1min_before": "2.01", + "load_1min": "1.85" }, { "system": "celery", "scenario": "drain", "run": 3, + "position": "second", "workers": 8, "payload": 2048, "jobs": 20000, - "rate": 3422.487819796637, - "load_1min": "8.82" + "rate": 6678.601151553238, + "load_1min_before": "1.19", + "load_1min": "1.91" }, { "system": "pgqueue", "scenario": "pickup", "run": 3, + "position": "first", "workers": 1, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 10.039499999999999, - "p95": 21.312400000000004, - "p99": 38.15044, - "max": 186.268, + "p50": 6.560499999999999, + "p95": 10.21405, + "p99": 19.85114, + "max": 51.995, "n": 3000.0 }, - "load_1min": "7.44" + "load_1min_before": "1.91", + "load_1min": "1.72" }, { "system": "celery", "scenario": "pickup", "run": 3, + "position": "second", "workers": 1, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 3.507852554321289, - "p95": 4.242002964019775, - "p99": 5.659053325653076, - "max": 15.669584274291992, + "p50": 3.3903121948242188, + "p95": 3.6988377571105957, + "p99": 3.845694065093994, + "max": 25.56157112121582, "n": 3000.0 }, - "load_1min": "6.13" + "load_1min_before": "1.30", + "load_1min": "1.07" }, { "system": "pgqueue", "scenario": "pickup", "run": 3, + "position": "first", "workers": 2, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 6.327000000000001, - "p95": 14.788200000000002, - "p99": 22.34689, - "max": 166.036, + "p50": 4.3, + "p95": 7.5341499999999995, + "p99": 11.121149999999998, + "max": 26.0, "n": 3000.0 }, - "load_1min": "5.92" + "load_1min_before": "1.07", + "load_1min": "1.95" }, { "system": "celery", "scenario": "pickup", "run": 3, + "position": "second", "workers": 2, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 3.754854202270508, - "p95": 6.616246700286865, - "p99": 7.644095420837402, - "max": 14.815330505371094, + "p50": 3.3974647521972656, + "p95": 3.701460361480713, + "p99": 3.9916443824768066, + "max": 6.099224090576172, "n": 3000.0 }, - "load_1min": "6.09" + "load_1min_before": "1.40", + "load_1min": "1.30" }, { "system": "pgqueue", "scenario": "pickup", "run": 3, + "position": "first", "workers": 4, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 6.5104999999999995, - "p95": 14.164350000000002, - "p99": 20.85005, - "max": 255.088, + "p50": 3.957, + "p95": 5.898199999999999, + "p99": 8.26323, + "max": 27.506, "n": 3000.0 }, - "load_1min": "7.05" + "load_1min_before": "1.30", + "load_1min": "1.92" }, { "system": "celery", "scenario": "pickup", "run": 3, + "position": "second", "workers": 4, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 3.7631988525390625, - "p95": 5.845975875854492, - "p99": 7.403628826141357, - "max": 12.630224227905273, + "p50": 3.474712371826172, + "p95": 3.7689208984375, + "p99": 3.935384750366211, + "max": 5.607366561889648, "n": 3000.0 }, - "load_1min": "5.97" + "load_1min_before": "1.16", + "load_1min": "1.06" }, { "system": "pgqueue", "scenario": "pickup", "run": 3, + "position": "first", "workers": 8, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 7.151500000000001, - "p95": 20.5187, - "p99": 61.96403, - "max": 200.181, + "p50": 4.32, + "p95": 7.82025, + "p99": 12.35209, + "max": 27.732, "n": 3000.0 }, - "load_1min": "9.09" + "load_1min_before": "1.06", + "load_1min": "2.90" }, { "system": "celery", "scenario": "pickup", "run": 3, + "position": "second", "workers": 8, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 4.999041557312012, - "p95": 12.883663177490234, - "p99": 20.632174015045166, - "max": 40.76695442199707, + "p50": 3.4803152084350586, + "p95": 3.854060173034668, + "p99": 4.019520282745361, + "max": 7.066011428833008, "n": 3000.0 }, - "load_1min": "12.64" + "load_1min_before": "2.13", + "load_1min": "1.58" }, { "system": "pgqueue", "scenario": "drain-size", "run": 3, + "position": "first", "workers": 4, "payload": 256, "label": "small", "jobs": 10000, - "rate": 2563.7229062124393, - "load_1min": "20.38" + "rate": 5362.015037440192, + "load_1min_before": "1.58", + "load_1min": "1.58" }, { "system": "celery", "scenario": "drain-size", "run": 3, + "position": "second", "workers": 4, "payload": 256, "label": "small", "jobs": 10000, - "rate": 3466.183224889127, - "load_1min": "17.19" + "rate": 3094.224022303946, + "load_1min_before": "1.08", + "load_1min": "1.27" }, { "system": "pgqueue", "scenario": "drain-size", "run": 3, + "position": "first", "workers": 4, "payload": 2048, "label": "medium", "jobs": 10000, - "rate": 3014.8124813295, - "load_1min": "16.21" + "rate": 5272.437283410801, + "load_1min_before": "1.27", + "load_1min": "1.25" }, { "system": "celery", "scenario": "drain-size", "run": 3, + "position": "second", "workers": 4, "payload": 2048, "label": "medium", "jobs": 10000, - "rate": 3403.045913720026, - "load_1min": "13.76" + "rate": 2969.7731014230276, + "load_1min_before": "0.75", + "load_1min": "0.82" }, { "system": "pgqueue", "scenario": "drain-size", "run": 3, + "position": "first", "workers": 4, "payload": 16384, "label": "large", "jobs": 10000, - "rate": 1489.1287280372983, - "load_1min": "11.47" + "rate": 4930.39254239509, + "load_1min_before": "0.82", + "load_1min": "0.92" }, { "system": "celery", "scenario": "drain-size", "run": 3, + "position": "second", "workers": 4, "payload": 16384, "label": "large", "jobs": 10000, - "rate": 1921.1655508774245, - "load_1min": "9.79" + "rate": 2587.120765084173, + "load_1min_before": "0.70", + "load_1min": "1.25" }, { - "system": "pgqueue", + "system": "celery", "scenario": "enqueue", "run": 4, + "position": "first", "payload": 2048, "variant": "single", "jobs": 5000, - "seconds": 27.13840025999525, - "rate": 184.24077882624889, + "seconds": 4.035376275001909, + "rate": 1239.0418288806625, "latency_ms": { - "p50": 4.1135624996968545, - "p95": 11.65304429414391, - "p99": 18.08264980893, - "max": 61.1318070004927, + "p50": 0.7794535049470142, + "p95": 1.0700520397222135, + "p99": 1.398201466363389, + "max": 1.920638998853974, "n": 5000.0 }, - "load_1min": "7.84" + "load_1min_before": "1.25", + "load_1min": "1.23" }, { - "system": "celery", + "system": "pgqueue", "scenario": "enqueue", "run": 4, + "position": "second", "payload": 2048, "variant": "single", "jobs": 5000, - "seconds": 2.9728898409957765, - "rate": 1681.865211098854, + "seconds": 21.647871334003867, + "rate": 230.96959155268726, "latency_ms": { - "p50": 0.5128524971951265, - "p95": 0.927027744546649, - "p99": 1.1197004620044027, - "max": 3.1106969981919974, + "p50": 4.37775399768725, + "p95": 4.844997553300345, + "p99": 5.142568229930475, + "max": 24.631081003462896, "n": 5000.0 }, - "load_1min": "7.37" + "load_1min_before": "0.89", + "load_1min": "1.15" }, { - "system": "pgqueue", + "system": "celery", "scenario": "enqueue", "run": 4, + "position": "first", "payload": 2048, "variant": "ten", "jobs": 5000, - "seconds": 22.525252996994823, - "rate": 221.97308952166125, + "seconds": 6.496755737010972, + "rate": 769.6148974042236, "latency_ms": { - "p50": 30.761746998905437, - "p95": 145.85516980623652, - "p99": 266.00960627532913, - "max": 504.965855005139, + "p50": 9.066399499715772, + "p95": 35.58832805138081, + "p99": 58.56691555105499, + "max": 266.9524770026328, "n": 5000.0 }, - "load_1min": "7.85" + "load_1min_before": "1.15", + "load_1min": "1.14" }, { - "system": "celery", + "system": "pgqueue", "scenario": "enqueue", "run": 4, + "position": "second", "payload": 2048, "variant": "ten", "jobs": 5000, - "seconds": 5.4819459520003875, - "rate": 912.0848771184029, + "seconds": 12.286355330987135, + "rate": 406.9555100192825, "latency_ms": { - "p50": 7.774089001031825, - "p95": 29.739558155779378, - "p99": 46.220146818232024, - "max": 249.2208199982997, + "p50": 24.412803999439348, + "p95": 25.984695908118738, + "p99": 28.783915214444278, + "max": 147.1229860035237, "n": 5000.0 }, - "load_1min": "7.46" + "load_1min_before": "1.03", + "load_1min": "1.02" }, { "system": "pgqueue", "scenario": "enqueue", "run": 4, + "position": "only", "payload": 2048, "variant": "bulk", "jobs": 40000, - "seconds": 3.541166109003825, - "rate": 11295.714114707967, + "seconds": 3.026778737999848, + "rate": 13215.369692478, "latency_ms": { - "p50": 86.51759999702335, - "p95": 106.21537289698608, - "p99": 142.07703389125527, - "max": 164.0927870030282, + "p50": 74.33465499343583, + "p95": 82.4383848477737, + "p99": 111.48150702530984, + "max": 127.77236200054176, "n": 40.0 }, - "load_1min": "7.03" + "load_1min_before": "1.02", + "load_1min": "1.02" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain", "run": 4, + "position": "first", "workers": 1, "payload": 2048, "jobs": 20000, - "rate": 1085.7065487329562, - "load_1min": "5.75" + "rate": 432.6765026785005, + "load_1min_before": "1.02", + "load_1min": "1.29" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain", "run": 4, + "position": "second", "workers": 1, "payload": 2048, "jobs": 20000, - "rate": 483.1513060312756, - "load_1min": "3.55" + "rate": 1347.0507443548513, + "load_1min_before": "0.84", + "load_1min": "1.86" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain", "run": 4, + "position": "first", "workers": 2, "payload": 2048, "jobs": 20000, - "rate": 2193.0171114045147, - "load_1min": "3.92" + "rate": 852.4036182053226, + "load_1min_before": "1.86", + "load_1min": "2.08" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain", "run": 4, + "position": "second", "workers": 2, "payload": 2048, "jobs": 20000, - "rate": 1201.2801499033976, - "load_1min": "3.41" + "rate": 2825.644247127279, + "load_1min_before": "1.31", + "load_1min": "1.66" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain", "run": 4, + "position": "first", "workers": 4, "payload": 2048, "jobs": 20000, - "rate": 4068.2656057050654, - "load_1min": "3.36" + "rate": 2926.3684455504913, + "load_1min_before": "1.66", + "load_1min": "2.16" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain", "run": 4, + "position": "second", "workers": 4, "payload": 2048, "jobs": 20000, - "rate": 3354.2605324597284, - "load_1min": "3.08" + "rate": 5368.372415682088, + "load_1min_before": "1.45", + "load_1min": "1.38" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain", "run": 4, + "position": "first", "workers": 8, "payload": 2048, "jobs": 20000, - "rate": 5664.7663027829085, - "load_1min": "5.73" + "rate": 6692.826121044248, + "load_1min_before": "1.38", + "load_1min": "1.94" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain", "run": 4, + "position": "second", "workers": 8, "payload": 2048, "jobs": 20000, - "rate": 6747.352330724353, - "load_1min": "4.87" + "rate": 10424.173980125419, + "load_1min_before": "1.24", + "load_1min": "1.22" }, { - "system": "pgqueue", + "system": "celery", "scenario": "pickup", "run": 4, + "position": "first", "workers": 1, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 6.0975, - "p95": 14.7863, - "p99": 19.58638, - "max": 54.889, + "p50": 3.399491310119629, + "p95": 3.761577606201172, + "p99": 3.922250270843506, + "max": 12.563705444335938, "n": 3000.0 }, - "load_1min": "3.93" + "load_1min_before": "1.22", + "load_1min": "1.23" }, { - "system": "celery", + "system": "pgqueue", "scenario": "pickup", "run": 4, + "position": "second", "workers": 1, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 2.7168989181518555, - "p95": 3.2401204109191895, - "p99": 3.6842989921569824, - "max": 13.327360153198242, + "p50": 6.4045000000000005, + "p95": 10.5343, + "p99": 17.50354, + "max": 49.781, "n": 3000.0 }, - "load_1min": "3.35" + "load_1min_before": "1.03", + "load_1min": "1.38" }, { - "system": "pgqueue", + "system": "celery", "scenario": "pickup", "run": 4, + "position": "first", "workers": 2, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 3.8835, - "p95": 10.295049999999998, - "p99": 20.794640000000005, - "max": 273.911, + "p50": 3.3081769943237305, + "p95": 3.744173049926758, + "p99": 3.9611077308654785, + "max": 7.099390029907227, "n": 3000.0 }, - "load_1min": "3.60" + "load_1min_before": "1.38", + "load_1min": "1.71" }, { - "system": "celery", + "system": "pgqueue", "scenario": "pickup", "run": 4, + "position": "second", "workers": 2, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 2.800464630126953, - "p95": 3.2862067222595215, - "p99": 3.7076759338378906, - "max": 6.489753723144531, + "p50": 4.4045, + "p95": 6.1491, + "p99": 6.84835, + "max": 56.842, "n": 3000.0 }, - "load_1min": "3.00" + "load_1min_before": "1.18", + "load_1min": "1.62" }, { - "system": "pgqueue", + "system": "celery", "scenario": "pickup", "run": 4, + "position": "first", "workers": 4, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 3.79, - "p95": 5.4211, - "p99": 10.428310000000002, - "max": 98.603, + "p50": 3.506183624267578, + "p95": 3.8139939308166504, + "p99": 3.9649295806884766, + "max": 6.226539611816406, "n": 3000.0 }, - "load_1min": "3.05" + "load_1min_before": "1.62", + "load_1min": "1.22" }, { - "system": "celery", + "system": "pgqueue", "scenario": "pickup", "run": 4, + "position": "second", "workers": 4, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 2.7223825454711914, - "p95": 3.3565640449523926, - "p99": 3.9217662811279297, - "max": 7.006168365478516, + "p50": 3.9560000000000004, + "p95": 5.884200000000001, + "p99": 11.54553, + "max": 35.279, "n": 3000.0 }, - "load_1min": "2.34" + "load_1min_before": "0.89", + "load_1min": "1.53" }, { - "system": "pgqueue", + "system": "celery", "scenario": "pickup", "run": 4, + "position": "first", "workers": 8, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 3.836, - "p95": 9.85255, - "p99": 13.44224, - "max": 236.1, + "p50": 3.5789012908935547, + "p95": 3.906071186065674, + "p99": 4.052648544311523, + "max": 7.34400749206543, "n": 3000.0 }, - "load_1min": "4.18" + "load_1min_before": "1.53", + "load_1min": "1.40" }, { - "system": "celery", + "system": "pgqueue", "scenario": "pickup", "run": 4, + "position": "second", "workers": 8, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 2.0548105239868164, - "p95": 3.3158063888549805, - "p99": 3.6653661727905273, - "max": 7.615566253662109, + "p50": 3.8220000000000005, + "p95": 5.8012, + "p99": 6.198080000000001, + "max": 29.837, "n": 3000.0 }, - "load_1min": "3.57" + "load_1min_before": "1.32", + "load_1min": "2.14" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain-size", "run": 4, + "position": "first", "workers": 4, "payload": 256, "label": "small", "jobs": 10000, - "rate": 6369.74010590153, - "load_1min": "3.57" + "rate": 2996.1237511817426, + "load_1min_before": "2.14", + "load_1min": "1.89" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain-size", "run": 4, + "position": "second", "workers": 4, "payload": 256, "label": "small", "jobs": 10000, - "rate": 3736.738391084554, - "load_1min": "3.61" + "rate": 5179.721924006612, + "load_1min_before": "1.36", + "load_1min": "1.36" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain-size", "run": 4, + "position": "first", "workers": 4, "payload": 2048, "label": "medium", "jobs": 10000, - "rate": 4872.676759008726, - "load_1min": "3.89" + "rate": 3015.6141988466243, + "load_1min_before": "1.36", + "load_1min": "1.63" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain-size", "run": 4, + "position": "second", "workers": 4, "payload": 2048, "label": "medium", "jobs": 10000, - "rate": 3395.2207003329136, - "load_1min": "3.83" + "rate": 4128.726067625876, + "load_1min_before": "1.14", + "load_1min": "1.37" }, { - "system": "pgqueue", + "system": "celery", "scenario": "drain-size", "run": 4, + "position": "first", "workers": 4, "payload": 16384, "label": "large", "jobs": 10000, - "rate": 5484.933784534009, - "load_1min": "3.75" + "rate": 2545.7442053617233, + "load_1min_before": "1.37", + "load_1min": "2.15" }, { - "system": "celery", + "system": "pgqueue", "scenario": "drain-size", "run": 4, + "position": "second", "workers": 4, "payload": 16384, "label": "large", "jobs": 10000, - "rate": 2910.794568502165, - "load_1min": "3.27" + "rate": 4740.630003729839, + "load_1min_before": "2.03", + "load_1min": "2.51" }, { "system": "pgqueue", "scenario": "enqueue", "run": 5, + "position": "first", "payload": 2048, "variant": "single", "jobs": 5000, - "seconds": 18.10742730199854, - "rate": 276.1297845690173, + "seconds": 22.13041885200073, + "rate": 225.93336499584456, "latency_ms": { - "p50": 3.203447999112541, - "p95": 4.869761601366918, - "p99": 11.2776507774106, - "max": 159.1634470023564, + "p50": 4.439783995621838, + "p95": 4.999920049158391, + "p99": 5.848583834449528, + "max": 26.864833998843096, "n": 5000.0 }, - "load_1min": "3.21" + "load_1min_before": "2.51", + "load_1min": "2.06" }, { "system": "celery", "scenario": "enqueue", "run": 5, + "position": "second", "payload": 2048, "variant": "single", "jobs": 5000, - "seconds": 2.9161221949980245, - "rate": 1714.6057900373366, + "seconds": 3.958498314008466, + "rate": 1263.1052493582815, "latency_ms": { - "p50": 0.5053759996371809, - "p95": 1.0290260968758957, - "p99": 1.4984722039662302, - "max": 5.52466900262516, + "p50": 0.7726644980721176, + "p95": 1.0394057535449974, + "p99": 1.312658643728355, + "max": 1.9196560024283826, "n": 5000.0 }, - "load_1min": "3.19" + "load_1min_before": "1.25", + "load_1min": "1.15" }, { "system": "pgqueue", "scenario": "enqueue", "run": 5, + "position": "first", "payload": 2048, "variant": "ten", "jobs": 5000, - "seconds": 13.930296312006249, - "rate": 358.9299098893251, + "seconds": 13.166403723007534, + "rate": 379.75441929239855, "latency_ms": { - "p50": 27.764914997533197, - "p95": 32.50564004993066, - "p99": 42.038412287147366, - "max": 139.36303799710004, + "p50": 25.036985003680456, + "p95": 33.96863090310944, + "p99": 42.698799782665446, + "max": 143.48717899702024, "n": 5000.0 }, - "load_1min": "3.01" + "load_1min_before": "1.15", + "load_1min": "1.26" }, { "system": "celery", "scenario": "enqueue", "run": 5, + "position": "second", "payload": 2048, "variant": "ten", "jobs": 5000, - "seconds": 4.855084426999383, - "rate": 1029.8482086520955, + "seconds": 6.4210171819868265, + "rate": 778.6928236271862, "latency_ms": { - "p50": 7.09863800148014, - "p95": 25.529733846269664, - "p99": 42.25982627023768, - "max": 139.9783020024188, + "p50": 9.329667496785987, + "p95": 33.35556789461407, + "p99": 54.58410425839247, + "max": 224.5077300030971, "n": 5000.0 }, - "load_1min": "3.01" + "load_1min_before": "0.76", + "load_1min": "0.78" }, { "system": "pgqueue", "scenario": "enqueue", "run": 5, + "position": "only", "payload": 2048, "variant": "bulk", "jobs": 40000, - "seconds": 2.7041835950003588, - "rate": 14791.895074710967, + "seconds": 2.649129965007887, + "rate": 15099.296949699072, "latency_ms": { - "p50": 66.05182099883677, - "p95": 75.09466004848946, - "p99": 81.53902892052429, - "max": 83.13275300315581, + "p50": 64.8797675021342, + "p95": 72.67709805091727, + "p99": 98.5554059963033, + "max": 113.37714501132723, "n": 40.0 }, - "load_1min": "2.85" + "load_1min_before": "0.78", + "load_1min": "0.80" }, { "system": "pgqueue", "scenario": "drain", "run": 5, + "position": "first", "workers": 1, "payload": 2048, "jobs": 20000, - "rate": 1398.7215264581648, - "load_1min": "3.43" + "rate": 1334.0192011901192, + "load_1min_before": "0.80", + "load_1min": "1.35" }, { "system": "celery", "scenario": "drain", "run": 5, + "position": "second", "workers": 1, "payload": 2048, "jobs": 20000, - "rate": 499.2201708228078, - "load_1min": "3.04" + "rate": 443.2786682329031, + "load_1min_before": "1.11", + "load_1min": "1.32" }, { "system": "pgqueue", "scenario": "drain", "run": 5, + "position": "first", "workers": 2, "payload": 2048, "jobs": 20000, - "rate": 2527.7023693714336, - "load_1min": "3.44" + "rate": 2871.012315572162, + "load_1min_before": "1.32", + "load_1min": "1.75" }, { "system": "celery", "scenario": "drain", "run": 5, + "position": "second", "workers": 2, "payload": 2048, "jobs": 20000, - "rate": 1267.8315052941882, - "load_1min": "3.52" + "rate": 999.7393492929705, + "load_1min_before": "1.11", + "load_1min": "1.40" }, { "system": "pgqueue", "scenario": "drain", "run": 5, + "position": "first", "workers": 4, "payload": 2048, "jobs": 20000, - "rate": 5475.173831349194, - "load_1min": "3.96" + "rate": 5577.215971542038, + "load_1min_before": "1.40", + "load_1min": "2.65" }, { "system": "celery", "scenario": "drain", "run": 5, + "position": "second", "workers": 4, "payload": 2048, "jobs": 20000, - "rate": 2537.7084165343013, - "load_1min": "4.25" + "rate": 2903.704384718899, + "load_1min_before": "2.08", + "load_1min": "3.04" }, { "system": "pgqueue", "scenario": "drain", "run": 5, + "position": "first", "workers": 8, "payload": 2048, "jobs": 20000, - "rate": 8142.137886836757, - "load_1min": "4.07" + "rate": 10184.281504153545, + "load_1min_before": "3.04", + "load_1min": "3.44" }, { "system": "celery", "scenario": "drain", "run": 5, + "position": "second", "workers": 8, "payload": 2048, "jobs": 20000, - "rate": 5898.2492160144, - "load_1min": "5.22" + "rate": 2811.9102472554127, + "load_1min_before": "2.32", + "load_1min": "2.51" }, { "system": "pgqueue", "scenario": "pickup", "run": 5, + "position": "first", "workers": 1, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 9.539, - "p95": 20.44395, - "p99": 27.238259999999997, - "max": 50.666, + "p50": 7.071500000000001, + "p95": 19.2618, + "p99": 77.38679, + "max": 262.875, "n": 3000.0 }, - "load_1min": "4.80" + "load_1min_before": "2.51", + "load_1min": "3.16" }, { "system": "celery", "scenario": "pickup", "run": 5, + "position": "second", "workers": 1, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 3.3731460571289062, - "p95": 5.09103536605835, - "p99": 5.770459175109863, - "max": 7.795095443725586, + "p50": 3.784775733947754, + "p95": 4.486322402954102, + "p99": 5.013821125030518, + "max": 29.323577880859375, "n": 3000.0 }, - "load_1min": "4.01" + "load_1min_before": "5.12", + "load_1min": "6.44" }, { "system": "pgqueue", "scenario": "pickup", "run": 5, + "position": "first", "workers": 2, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 5.014500000000001, - "p95": 12.465549999999999, - "p99": 19.3887, - "max": 63.199, + "p50": 4.0275, + "p95": 12.4364, + "p99": 46.05915, + "max": 148.469, "n": 3000.0 }, - "load_1min": "5.72" + "load_1min_before": "6.44", + "load_1min": "7.50" }, { "system": "celery", "scenario": "pickup", "run": 5, + "position": "second", "workers": 2, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 2.4083852767944336, - "p95": 4.565095901489258, - "p99": 5.966072082519531, - "max": 8.34345817565918, + "p50": 3.9701461791992188, + "p95": 4.644668102264404, + "p99": 4.941718578338623, + "max": 11.057615280151367, "n": 3000.0 }, - "load_1min": "4.72" + "load_1min_before": "7.95", + "load_1min": "8.48" }, { "system": "pgqueue", "scenario": "pickup", "run": 5, + "position": "first", "workers": 4, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 5.398, - "p95": 10.129, - "p99": 12.63306, - "max": 28.3, + "p50": 3.9360000000000004, + "p95": 12.187099999999997, + "p99": 22.679460000000002, + "max": 112.539, "n": 3000.0 }, - "load_1min": "5.43" + "load_1min_before": "8.48", + "load_1min": "9.79" }, { "system": "celery", "scenario": "pickup", "run": 5, + "position": "second", "workers": 4, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 3.052234649658203, - "p95": 5.911779403686523, - "p99": 6.440732479095459, - "max": 10.005712509155273, + "p50": 4.209637641906738, + "p95": 4.640364646911621, + "p99": 4.852819442749023, + "max": 6.987810134887695, "n": 3000.0 }, - "load_1min": "4.51" + "load_1min_before": "9.09", + "load_1min": "9.00" }, { "system": "pgqueue", "scenario": "pickup", "run": 5, + "position": "first", "workers": 8, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 5.810999999999999, - "p95": 13.385250000000001, - "p99": 17.707310000000003, - "max": 90.209, + "p50": 3.9995000000000003, + "p95": 9.61645, + "p99": 11.856809999999998, + "max": 34.605, "n": 3000.0 }, - "load_1min": "6.13" + "load_1min_before": "9.00", + "load_1min": "10.67" }, { "system": "celery", "scenario": "pickup", "run": 5, + "position": "second", "workers": 8, "payload": 2048, "arrival_rate": 200, "jobs": 3000, "latency_ms": { - "p50": 4.0923357009887695, - "p95": 6.211698055267334, - "p99": 6.686742305755615, - "max": 8.281230926513672, + "p50": 4.21905517578125, + "p95": 4.6434760093688965, + "p99": 5.001344680786133, + "max": 7.065773010253906, "n": 3000.0 }, - "load_1min": "5.34" + "load_1min_before": "10.53", + "load_1min": "10.34" }, { "system": "pgqueue", "scenario": "drain-size", "run": 5, + "position": "first", "workers": 4, "payload": 256, "label": "small", "jobs": 10000, - "rate": 2997.8103418914275, - "load_1min": "5.34" + "rate": 3972.8213297068837, + "load_1min_before": "10.34", + "load_1min": "11.19" }, { "system": "celery", "scenario": "drain-size", "run": 5, + "position": "second", "workers": 4, "payload": 256, "label": "small", "jobs": 10000, - "rate": 3163.05426233278, - "load_1min": "5.75" + "rate": 1299.1231605864978, + "load_1min_before": "6.84", + "load_1min": "5.65" }, { "system": "pgqueue", "scenario": "drain-size", "run": 5, + "position": "first", "workers": 4, "payload": 2048, "label": "medium", "jobs": 10000, - "rate": 3260.9717188351497, - "load_1min": "5.61" + "rate": 3923.3060790062027, + "load_1min_before": "5.65", + "load_1min": "5.35" }, { "system": "celery", "scenario": "drain-size", "run": 5, + "position": "second", "workers": 4, "payload": 2048, "label": "medium", "jobs": 10000, - "rate": 2281.2304132437093, - "load_1min": "5.61" + "rate": 1319.9094294326148, + "load_1min_before": "3.43", + "load_1min": "3.95" }, { "system": "pgqueue", "scenario": "drain-size", "run": 5, + "position": "first", "workers": 4, "payload": 16384, "label": "large", "jobs": 10000, - "rate": 2931.023494566609, - "load_1min": "5.26" + "rate": 3369.3085786629135, + "load_1min_before": "3.95", + "load_1min": "3.35" }, { "system": "celery", "scenario": "drain-size", "run": 5, + "position": "second", + "workers": 4, + "payload": 16384, + "label": "large", + "jobs": 10000, + "rate": 1156.4275562503053, + "load_1min_before": "2.17", + "load_1min": "2.14" + }, + { + "system": "celery", + "scenario": "enqueue", + "run": 6, + "position": "first", + "payload": 2048, + "variant": "single", + "jobs": 5000, + "seconds": 7.020645347001846, + "rate": 712.1852412236207, + "latency_ms": { + "p50": 1.4192890012054704, + "p95": 1.7748755468346644, + "p99": 1.922907871339703, + "max": 3.4385350008960813, + "n": 5000.0 + }, + "load_1min_before": "2.14", + "load_1min": "2.12" + }, + { + "system": "pgqueue", + "scenario": "enqueue", + "run": 6, + "position": "second", + "payload": 2048, + "variant": "single", + "jobs": 5000, + "seconds": 23.584815123002045, + "rate": 212.00081382548333, + "latency_ms": { + "p50": 4.594713005644735, + "p95": 5.3703294579463545, + "p99": 10.585518453735858, + "max": 24.983657989650965, + "n": 5000.0 + }, + "load_1min_before": "1.83", + "load_1min": "1.55" + }, + { + "system": "celery", + "scenario": "enqueue", + "run": 6, + "position": "first", + "payload": 2048, + "variant": "ten", + "jobs": 5000, + "seconds": 6.831328874002793, + "rate": 731.9220157922609, + "latency_ms": { + "p50": 9.42957400548039, + "p95": 37.98709450129536, + "p99": 61.61140640833764, + "max": 233.6842000077013, + "n": 5000.0 + }, + "load_1min_before": "1.55", + "load_1min": "1.50" + }, + { + "system": "pgqueue", + "scenario": "enqueue", + "run": 6, + "position": "second", + "payload": 2048, + "variant": "ten", + "jobs": 5000, + "seconds": 12.490988674995606, + "rate": 400.2885704322968, + "latency_ms": { + "p50": 24.658403999637812, + "p95": 26.301378651260165, + "p99": 30.612151407985948, + "max": 236.68526699475478, + "n": 5000.0 + }, + "load_1min_before": "1.04", + "load_1min": "1.39" + }, + { + "system": "pgqueue", + "scenario": "enqueue", + "run": 6, + "position": "only", + "payload": 2048, + "variant": "bulk", + "jobs": 40000, + "seconds": 4.249429745002999, + "rate": 9413.027723787389, + "latency_ms": { + "p50": 104.06907149445033, + "p95": 120.93009269374306, + "p99": 138.39178000227548, + "max": 148.82607400068082, + "n": 40.0 + }, + "load_1min_before": "1.39", + "load_1min": "1.36" + }, + { + "system": "celery", + "scenario": "drain", + "run": 6, + "position": "first", + "workers": 1, + "payload": 2048, + "jobs": 20000, + "rate": 399.5773158203073, + "load_1min_before": "1.36", + "load_1min": "1.58" + }, + { + "system": "pgqueue", + "scenario": "drain", + "run": 6, + "position": "second", + "workers": 1, + "payload": 2048, + "jobs": 20000, + "rate": 1181.806325505982, + "load_1min_before": "0.95", + "load_1min": "1.36" + }, + { + "system": "celery", + "scenario": "drain", + "run": 6, + "position": "first", + "workers": 2, + "payload": 2048, + "jobs": 20000, + "rate": 725.0684676748871, + "load_1min_before": "1.36", + "load_1min": "1.65" + }, + { + "system": "pgqueue", + "scenario": "drain", + "run": 6, + "position": "second", + "workers": 2, + "payload": 2048, + "jobs": 20000, + "rate": 2431.406815290821, + "load_1min_before": "1.13", + "load_1min": "1.55" + }, + { + "system": "celery", + "scenario": "drain", + "run": 6, + "position": "first", + "workers": 4, + "payload": 2048, + "jobs": 20000, + "rate": 1278.5959688553264, + "load_1min_before": "1.55", + "load_1min": "2.65" + }, + { + "system": "pgqueue", + "scenario": "drain", + "run": 6, + "position": "second", + "workers": 4, + "payload": 2048, + "jobs": 20000, + "rate": 4126.257881279815, + "load_1min_before": "2.42", + "load_1min": "2.84" + }, + { + "system": "celery", + "scenario": "drain", + "run": 6, + "position": "first", + "workers": 8, + "payload": 2048, + "jobs": 20000, + "rate": 2291.842241321183, + "load_1min_before": "2.84", + "load_1min": "3.74" + }, + { + "system": "pgqueue", + "scenario": "drain", + "run": 6, + "position": "second", + "workers": 8, + "payload": 2048, + "jobs": 20000, + "rate": 6927.363746120775, + "load_1min_before": "2.58", + "load_1min": "4.42" + }, + { + "system": "celery", + "scenario": "pickup", + "run": 6, + "position": "first", + "workers": 1, + "payload": 2048, + "arrival_rate": 200, + "jobs": 3000, + "latency_ms": { + "p50": 3.441333770751953, + "p95": 4.196560382843018, + "p99": 5.9885382652282715, + "max": 15.123367309570312, + "n": 3000.0 + }, + "load_1min_before": "4.42", + "load_1min": "4.60" + }, + { + "system": "pgqueue", + "scenario": "pickup", + "run": 6, + "position": "second", + "workers": 1, + "payload": 2048, + "arrival_rate": 200, + "jobs": 3000, + "latency_ms": { + "p50": 6.0805, + "p95": 9.49515, + "p99": 31.535149999999998, + "max": 131.587, + "n": 3000.0 + }, + "load_1min_before": "2.93", + "load_1min": "2.87" + }, + { + "system": "celery", + "scenario": "pickup", + "run": 6, + "position": "first", + "workers": 2, + "payload": 2048, + "arrival_rate": 200, + "jobs": 3000, + "latency_ms": { + "p50": 3.5386085510253906, + "p95": 3.963959217071533, + "p99": 4.262971878051758, + "max": 7.498979568481445, + "n": 3000.0 + }, + "load_1min_before": "2.87", + "load_1min": "2.58" + }, + { + "system": "pgqueue", + "scenario": "pickup", + "run": 6, + "position": "second", + "workers": 2, + "payload": 2048, + "arrival_rate": 200, + "jobs": 3000, + "latency_ms": { + "p50": 4.346, + "p95": 6.579400000000001, + "p99": 21.62933, + "max": 122.281, + "n": 3000.0 + }, + "load_1min_before": "1.62", + "load_1min": "2.00" + }, + { + "system": "celery", + "scenario": "pickup", + "run": 6, + "position": "first", + "workers": 4, + "payload": 2048, + "arrival_rate": 200, + "jobs": 3000, + "latency_ms": { + "p50": 3.589510917663574, + "p95": 4.070258140563965, + "p99": 4.360263347625732, + "max": 6.865501403808594, + "n": 3000.0 + }, + "load_1min_before": "2.00", + "load_1min": "2.03" + }, + { + "system": "pgqueue", + "scenario": "pickup", + "run": 6, + "position": "second", + "workers": 4, + "payload": 2048, + "arrival_rate": 200, + "jobs": 3000, + "latency_ms": { + "p50": 3.951, + "p95": 8.332350000000002, + "p99": 13.150239999999998, + "max": 111.818, + "n": 3000.0 + }, + "load_1min_before": "1.42", + "load_1min": "2.31" + }, + { + "system": "celery", + "scenario": "pickup", + "run": 6, + "position": "first", + "workers": 8, + "payload": 2048, + "arrival_rate": 200, + "jobs": 3000, + "latency_ms": { + "p50": 3.5887956619262695, + "p95": 4.204761981964111, + "p99": 4.452309608459473, + "max": 7.3032379150390625, + "n": 3000.0 + }, + "load_1min_before": "2.31", + "load_1min": "2.14" + }, + { + "system": "pgqueue", + "scenario": "pickup", + "run": 6, + "position": "second", + "workers": 8, + "payload": 2048, + "arrival_rate": 200, + "jobs": 3000, + "latency_ms": { + "p50": 4.0325, + "p95": 9.8933, + "p99": 12.309199999999999, + "max": 33.193, + "n": 3000.0 + }, + "load_1min_before": "1.45", + "load_1min": "2.63" + }, + { + "system": "celery", + "scenario": "drain-size", + "run": 6, + "position": "first", + "workers": 4, + "payload": 256, + "label": "small", + "jobs": 10000, + "rate": 1428.0329029797897, + "load_1min_before": "2.63", + "load_1min": "3.15" + }, + { + "system": "pgqueue", + "scenario": "drain-size", + "run": 6, + "position": "second", + "workers": 4, + "payload": 256, + "label": "small", + "jobs": 10000, + "rate": 4450.789970340122, + "load_1min_before": "1.96", + "load_1min": "2.29" + }, + { + "system": "celery", + "scenario": "drain-size", + "run": 6, + "position": "first", + "workers": 4, + "payload": 2048, + "label": "medium", + "jobs": 10000, + "rate": 1336.8483103109368, + "load_1min_before": "2.29", + "load_1min": "2.36" + }, + { + "system": "pgqueue", + "scenario": "drain-size", + "run": 6, + "position": "second", + "workers": 4, + "payload": 2048, + "label": "medium", + "jobs": 10000, + "rate": 4183.252961210605, + "load_1min_before": "1.48", + "load_1min": "1.81" + }, + { + "system": "celery", + "scenario": "drain-size", + "run": 6, + "position": "first", + "workers": 4, + "payload": 16384, + "label": "large", + "jobs": 10000, + "rate": 1158.3480925630347, + "load_1min_before": "1.81", + "load_1min": "1.92" + }, + { + "system": "pgqueue", + "scenario": "drain-size", + "run": 6, + "position": "second", "workers": 4, "payload": 16384, "label": "large", "jobs": 10000, - "rate": 2714.6834962830817, - "load_1min": "4.66" + "rate": 3598.4237403941747, + "load_1min_before": "1.30", + "load_1min": "1.85" } ] } \ No newline at end of file diff --git a/docs/benchmarks-latency.svg b/docs/benchmarks-latency.svg index c3474bf..b1c9036 100644 --- a/docs/benchmarks-latency.svg +++ b/docs/benchmarks-latency.svg @@ -6,55 +6,54 @@ 0.0 -50.0 +25.0 -100.0 +50.0 -150.0 +75.0 -200.0 +100.0 milliseconds 1 2 4 8 worker processes - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + pgqueue p50 @@ -67,5 +66,5 @@ Celery p99 -Median of 5 runs, whiskers = min to max. 13th Gen Intel(R) Core(TM) i5-13500H; PostgreSQL, Valkey, workers and load generator share it. +Median of 6 runs, whiskers = min to max. 13th Gen Intel(R) Core(TM) i5-13500H; PostgreSQL, Valkey, workers and load generator share it. diff --git a/docs/benchmarks-throughput.svg b/docs/benchmarks-throughput.svg index ac2f2aa..8b204af 100644 --- a/docs/benchmarks-throughput.svg +++ b/docs/benchmarks-throughput.svg @@ -6,42 +6,41 @@ 0 -2,500 +5,000 -5,000 +10,000 -7,500 +15,000 -10,000 +20,000 jobs per second 1 2 4 8 worker processes - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + pgqueue (PostgreSQL) Celery (Valkey) -Median of 5 runs, whiskers = min to max. 13th Gen Intel(R) Core(TM) i5-13500H; PostgreSQL, Valkey, workers and load generator share it. +Median of 6 runs, whiskers = min to max. 13th Gen Intel(R) Core(TM) i5-13500H; PostgreSQL, Valkey, workers and load generator share it. diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 0667a22..a50a008 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -1,20 +1,16 @@ # Benchmarks What the numbers in the README come from, how to reproduce them, and how far to trust them. -Short version: they were taken on a laptop that was busy with other work, in a fixed -order (pgqueue first, then Celery), the run-to-run spread is large, and the honest summary -is a range, not a point. The only claim that holds up across all runs is the one for 1 and -2 worker processes; see [How to read this](#how-to-read-this). - -**The numbers below are the first run only**: order-fixed, on a loaded machine, schema 1 of -`bench/results.json`. The benchmark script has since been changed to counterbalance the -order and to record the load before and after each measurement (see -[Method](#method-schema-2)), but it has not been rerun yet, so no number here comes from the -new method. - -Run it yourself with `make bench` (about an hour with the defaults, plus the cool-down; -`make bench-quick` checks the setup in a few minutes and produces meaningless numbers). -The tables below are generated from [`bench/results.json`](../bench/results.json) by +Short version: 6 runs of every cell on one laptop with the order of the two systems +counterbalanced and a 30 s cool-down between them. pgqueue drained a 20,000-job backlog +faster than this Celery configuration in all 6 runs at 1, 2, 4 and 8 worker processes (about +3 times at 1 and 2 workers, 1.3 to 3.6 times at 4 and 8), and Celery was faster on single-job +enqueue and on pickup latency. The two are not equal in durability, and the machine was not +fully idle; see [How to read this](#how-to-read-this). + +Run it yourself with `make bench ARGS="--runs 6"` (about 25 minutes including the +cool-downs; `make bench-quick` checks the setup in a few minutes and produces meaningless +numbers). The tables below are generated from [`bench/results.json`](../bench/results.json) by `python -m bench.report` and the charts by `python -m bench.chart`; the numbers in the prose are taken from the same file. @@ -31,9 +27,9 @@ prose are taken from the same file. | Handler | no-op. This measures the queue, not your work | | Job payload | JSON object of about 2 KiB (identifiers, short strings and a free-text body, deterministic). A 256 B and a 16 KiB variant are in the payload table | | Concurrency | 32 job slots per worker process in both systems (asyncio tasks for pgqueue, threads for Celery) | -| Repetitions | 5 runs of every cell; in this first run the systems always ran in the same order, pgqueue first and Celery a few seconds later (not counterbalanced) | +| Repetitions | 6 runs of every cell; the order of the two systems alternates between runs (pgqueue first in runs 1, 3 and 5, Celery first in runs 2, 4 and 6) with a 30 s cool-down between them | -Cells below are the **median over the 5 runs, with the minimum and maximum run in +Cells below are the **median over the 6 runs, with the minimum and maximum run in brackets**. Raw per-run numbers, the load average at each measurement and the environment are in `bench/results.json`. @@ -76,124 +72,104 @@ not need because its job rows already hold them. Celery is charged that extra ro ### Enqueue - | Variant | System | jobs/s | p50 ms | p95 ms | p99 ms | |---|---|---:|---:|---:|---:| -| 1 client, 1 job per call | pgqueue | 184 (111-276) | 4.4 (3.2-5.7) | 11.7 (4.9-19.5) | 18.1 (9.6-66.5) | -| 1 client, 1 job per call | Celery + Valkey | 566 (348-1,715) | 1.7 (0.5-2.9) | 2.5 (0.9-4.3) | 4.2 (1.1-5.9) | -| 10 concurrent clients | pgqueue | 251 (175-378) | 30.8 (25.4-51.3) | 52.8 (28.9-145.9) | 100.8 (33.9-266.0) | -| 10 concurrent clients | Celery + Valkey | 634 (442-1,030) | 12.6 (7.1-18.8) | 36.4 (25.5-58.0) | 52.0 (42.3-89.7) | -| `enqueue_many`, 1,000 per call (latency is per call) | pgqueue | 6,188 (5,181-14,792) | 153.1 (66.1-190.4) | 221.0 (75.1-245.0) | 245.0 (81.5-258.4) | - +| 1 client, 1 job per call | pgqueue | 226 (212-231) | 4.4 (4.4-4.6) | 5.0 (4.8-5.4) | 5.7 (5.1-10.6) | +| 1 client, 1 job per call | Celery + Valkey | 1,251 (712-1,272) | 0.8 (0.8-1.4) | 1.0 (1.0-1.8) | 1.3 (1.3-1.9) | +| 10 concurrent clients | pgqueue | 395 (334-417) | 24.5 (23.8-29.9) | 30.1 (25.5-34.9) | 36.0 (27.5-42.7) | +| 10 concurrent clients | Celery + Valkey | 774 (730-791) | 9.4 (8.9-9.5) | 34.9 (33.4-38.0) | 56.6 (53.0-63.4) | +| `enqueue_many`, 1,000 per call (latency is per call) | pgqueue | 14,162 (9,413-15,559) | 68.3 (64.1-104.1) | 78.6 (66.8-120.9) | 104.2 (69.9-138.4) | ### Backlog throughput (jobs/s) - | Worker processes | pgqueue | Celery (thread pool, 32 threads, prefetch x4, early ack) | |---:|---:|---:| -| 1 | 825 (725-1,399) | 349 (280-499) | -| 2 | 1,702 (1,427-2,528) | 722 (574-1,268) | -| 4 | 3,915 (2,444-5,475) | 1,765 (990-3,354) | -| 8 | 4,161 (2,589-8,142) | 3,422 (1,683-6,747) | - +| 1 | 1,323 (1,182-1,347) | 441 (400-445) | +| 2 | 2,788 (2,431-2,907) | 840 (725-1,000) | +| 4 | 5,267 (3,870-5,577) | 2,950 (1,279-3,003) | +| 8 | 10,103 (6,927-10,424) | 6,650 (2,292-6,693) | ![Backlog throughput by number of worker processes, pgqueue and Celery](benchmarks-throughput.svg) ### Pickup latency at 200 jobs/s (ms) - | Worker processes | System | p50 | p95 | p99 | max | |---:|---|---:|---:|---:|---:| -| 1 | pgqueue | 7.3 (6.1-10.0) | 17.4 (14.8-21.3) | 38.2 (19.6-152.1) | 145.7 (50.7-276.8) | -| 1 | Celery + Valkey | 3.5 (2.7-5.3) | 5.1 (3.2-53.6) | 5.8 (3.7-87.7) | 15.7 (7.8-137.9) | -| 2 | pgqueue | 5.0 (3.9-6.3) | 12.5 (8.5-14.8) | 22.3 (19.4-58.6) | 166.0 (63.2-273.9) | -| 2 | Celery + Valkey | 3.6 (2.4-4.4) | 6.6 (3.3-8.5) | 7.6 (3.7-11.0) | 14.8 (6.5-20.4) | -| 4 | pgqueue | 4.5 (3.8-6.5) | 9.5 (5.4-14.2) | 15.2 (10.4-20.9) | 49.1 (28.3-255.1) | -| 4 | Celery + Valkey | 3.5 (2.7-4.2) | 5.9 (3.4-8.1) | 7.4 (3.9-11.0) | 12.6 (7.0-17.8) | -| 8 | pgqueue | 5.5 (3.8-7.2) | 12.4 (7.4-20.5) | 17.5 (10.2-62.0) | 90.2 (28.3-236.1) | -| 8 | Celery + Valkey | 4.5 (2.1-5.0) | 8.3 (3.3-12.9) | 11.3 (3.7-20.6) | 26.9 (7.6-40.8) | - +| 1 | pgqueue | 6.5 (6.1-7.1) | 11.6 (9.5-19.3) | 22.2 (17.5-77.4) | 76.9 (49.8-262.9) | +| 1 | Celery + Valkey | 3.4 (3.3-3.8) | 4.0 (3.7-4.5) | 4.5 (3.8-6.0) | 13.8 (9.9-29.3) | +| 2 | pgqueue | 4.4 (4.0-4.4) | 8.5 (6.1-12.4) | 12.8 (6.8-46.1) | 45.4 (26.0-148.5) | +| 2 | Celery + Valkey | 3.5 (3.3-4.0) | 3.9 (3.7-4.6) | 4.1 (4.0-5.3) | 6.7 (5.5-11.1) | +| 4 | pgqueue | 4.0 (3.9-4.1) | 7.4 (5.9-12.2) | 11.9 (8.3-22.7) | 31.4 (25.4-112.5) | +| 4 | Celery + Valkey | 3.5 (3.4-4.2) | 4.0 (3.8-4.6) | 4.6 (3.9-6.4) | 6.9 (5.6-7.7) | +| 8 | pgqueue | 3.9 (3.8-4.3) | 6.8 (5.8-9.9) | 9.0 (6.2-12.4) | 30.3 (26.2-34.6) | +| 8 | Celery + Valkey | 3.6 (3.5-4.2) | 4.4 (3.9-4.7) | 4.7 (4.0-5.8) | 7.1 (6.9-7.3) | ![Pickup latency by number of worker processes, pgqueue and Celery](benchmarks-latency.svg) ### Payload size, 4 workers, backlog throughput (jobs/s) - | Payload | pgqueue | Celery (thread pool, 32 threads, prefetch x4, early ack) | |---|---:|---:| -| 256 B | 2,998 (2,319-6,370) | 3,163 (1,029-3,737) | -| 2,048 B | 3,261 (2,329-4,873) | 2,281 (1,036-3,403) | -| 16,384 B | 2,931 (1,489-5,485) | 1,921 (984-2,911) | +| 256 B | 5,199 (3,973-5,362) | 3,038 (1,299-3,106) | +| 2,048 B | 4,728 (3,923-5,454) | 2,990 (1,320-3,016) | +| 16,384 B | 4,694 (3,369-4,930) | 2,549 (1,156-2,587) | -Load average (1 minute) sampled after each measurement: min 2.3, median 10.4, max 37.7. +One-minute load average sampled at each measurement: min 0.8, median 1.7, max 11.2. -Results file schema 1: the order was fixed (pgqueue first, then Celery) and one load value -was stored per measurement, taken after it. The rows labelled "Celery + Valkey" in the -enqueue and pickup tables are the same Celery configuration as in the column headers above. +Results file schema 2: every record has the load average before and after the measurement +and its position in the pair. The rows labelled "Celery + Valkey" in the enqueue and pickup +tables are the same Celery configuration as in the column headers above. ## How to read this -**The spread is as important as the median.** The laptop was also running unrelated work -(other benchmarks and containers) during these runs: the 1-minute load average was 38 when -the first run started and 4 to 5 when the last one finished, on a 16-thread CPU. The runs -that executed on a quieter machine are visibly faster. For example, 8 pgqueue workers -drained the backlog at 2,589, 3,893, 4,161, 5,665 and 8,142 jobs/s in runs 1 to 5, with -load averages between 19 and 4 at the time. A median of 4,161 jobs/s is therefore -a middle value of a noisy sample, not a property of the queue. Treat the figures as an -order of magnitude: low thousands of jobs per second per Postgres on a laptop, no-op -handler, default Postgres settings. - -**The comparison is weaker than it first looks.** In every run pgqueue was measured first -and Celery a few seconds after it, always in that order, so any systematic effect of the -order (a warm page cache, a checkpoint or autovacuum triggered by the previous -measurement, a load spike that happens to start between the two) is not counterbalanced. -The machine was also busy with unrelated work, so the two halves of a pair did not -necessarily see the same load. Nothing in this data can separate those effects from a real -difference; the changed benchmark script (see [Method](#method-schema-2)) alternates the -order and records the load, but it has not been rerun. - -What the first run does show, per number of worker processes (pgqueue over Celery, jobs/s -ratio per run, runs 1 to 5): +**Backlog throughput: pgqueue was ahead in every run.** Ratio of pgqueue over Celery, jobs/s, +runs 1 to 6: | Workers | Ratio in each run | Reading | |---:|---|---| -| 1 | 2.69, 2.59, 2.13, 2.25, 2.80 | pgqueue about 2 to 2.8 times faster in all 5 runs | -| 2 | 2.49, 2.36, 2.53, 1.83, 1.99 | about 1.8 to 2.5 times in all 5 runs | -| 4 | 2.49, 2.03, 2.22, 1.21, 2.16 | ahead in all 5 runs, but only 1.21 in run 4, one of the two quiet runs | -| 8 | 1.39, 2.31, 1.22, 0.84, 1.38 | mixed; Celery ahead in run 4 (6,747 against 5,665 jobs/s) | - -Runs 4 and 5 were the quietest (1-minute load roughly 3 to 6). On those two runs, 4 to 8 -workers were close to even: at 8 workers the ratios were 0.84 and 1.38, and at 4 workers -1.21 and 2.16. The roughly 2 times advantage holds at 1 and 2 worker processes in every run -including the quiet ones. That is the only claim here that is robust to the noise; at 4 and -8 workers the data supports "pgqueue was not slower in 9 of 10 runs, and possibly about even -on a quiet machine", not a ratio. With 5 runs per cell this is a pattern, not a -significance test. - -The payload-size scenario (4 workers) is not a win for pgqueue either. At 2 KiB the -per-run ratios were 3.52, 2.23, 0.89, 1.44 and 1.43, so Celery won run 3. At 256 B pgqueue -lost 2 of 5 runs (ratios 3.40, 2.00, 0.74, 1.70, 0.95) and its median was below Celery's -(2,998 against 3,163 jobs/s). At 16 KiB pgqueue lost run 3 (0.78) and the other four were -ahead. Do not read the payload table as pgqueue being faster at 4 workers. - -**Enqueue favours Celery, as it should.** A Valkey publish with persistence off is a memory -write; a pgqueue `enqueue()` is a durable commit. The gap on single-job enqueue is what -durability costs. If you enqueue inside a transaction you are committing anyway, the job -rides on that commit and adds one INSERT. If you can batch, `enqueue_many` amortizes the -commit across the batch. - -**Pickup latency is low for both at this rate.** With idle workers the median is a few -milliseconds for both systems. Comparing the two within each run, pgqueue's median was -between 1.0 and 2.9 times Celery's, over intervals that are not equivalent (see "What is -timed": pgqueue's includes a durable commit, Celery's does not). The tails differ more: pgqueue's p99 is tens of milliseconds, with single -outliers in the hundreds under load. Do not read the 1-worker row as a scaling effect; the -runs with a loaded machine dominate the maximum values. +| 1 | 2.85, 3.00, 2.99, 3.11, 3.01, 2.96 | about 3 times, very stable | +| 2 | 3.26, 3.26, 3.47, 3.31, 2.87, 3.35 | about 3.3 times, very stable | +| 4 | 1.30, 1.72, 1.80, 1.83, 1.92, 3.23 | 1.3 to 1.9 times; run 6 is 3.2 because Celery had a slow run | +| 8 | 1.54, 1.49, 1.50, 1.56, 3.62, 3.02 | about 1.5 times in runs 1 to 4; runs 5 and 6 are 3.6 and 3.0 because Celery had two slow runs | + +The order did not matter: pgqueue was ahead when it ran first and when it ran second. The +payload scenario (4 workers) agrees: pgqueue was ahead in all 6 runs at 256 B, 2 KiB and +16 KiB (ratios 1.7 to 3.1 at 256 B and 1.8 to 3.1 at 16 KiB). + +The drain figures at 4 and 8 workers have outliers on the Celery side: its 8-worker runs +were 6,648 to 6,693 jobs/s in runs 1 to 4 and 2,812 and 2,292 in runs 5 and 6, and its +4-worker run 6 was 1,279 against about 2,900 to 3,000 otherwise. I do not know why; the +load average before those runs (1.4 to 2.8) does not explain it. For that reason the ratio at +4 and 8 workers should be quoted as a range or as "about 1.5 times in the typical runs", not +as the median ratio, which is pulled up by the slow Celery runs. + +**The machine was not idle.** The 1-minute load average sampled around the measurements was +0.8 at minimum, 1.7 at median and 11.2 at maximum on a 16-thread CPU. Before each drain +measurement it was between 0.8 and 3.6. That is far quieter than a first run that was taken +with a load of 4 to 38, and the spread between runs is now small for most cells, but it is +not a dedicated machine, and the peak of 11 shows something else ran at times. + +**Single-job enqueue favours Celery, as it should.** A Valkey publish with persistence off +is a memory write; a pgqueue `enqueue()` is a durable commit. The median was 226 jobs/s for +pgqueue against 1,251 for Celery with one client, 395 against 774 with ten concurrent clients. +That gap is what durability costs. If you enqueue inside a transaction you are committing +anyway, the job rides on that commit and adds one INSERT. If you can batch, `enqueue_many` +amortizes the commit: 14,162 jobs/s with 1,000 jobs per call, which is higher than Celery's +single-job rate (Celery has no bulk publish, so there is no like-for-like number). + +**Pickup latency is low for both at this rate, and Celery's is lower.** With idle workers +the median is 3.9 to 6.5 ms for pgqueue and 3.4 to 3.6 ms for Celery. pgqueue's p99 is 9 to +22 ms against 4.1 to 4.7 ms for Celery. The intervals are not equivalent (see "What is +timed": pgqueue's includes a durable commit, Celery's does not), so read this as "both are in +single-digit milliseconds at the median", not as a pure speed comparison. At 1 worker +pgqueue's median is highest (6.5 ms) and it falls as workers are added. **Celery on one worker is slow here** (hundreds of jobs/s) because one process with a thread pool is limited by the GIL, while an asyncio worker spends its time waiting on the database. A prefork pool, or the gevent pool, would change this comparison. It was not -tried, so there is no number for it. +tried, so there is no number for it. The 3 times lead at 1 and 2 workers is largely this +effect. ## Why more workers help, and where SKIP LOCKED contention comes in @@ -209,10 +185,11 @@ batches, the head of the ready-index is full of locked entries, and a claimer ha over them to find free rows. The more workers and slots, the more locked entries sit at the head. Besides that, all workers insert and update rows of the same table and index pages, and every completion is its own UPDATE and commit. In this benchmark the backlog throughput -went from 825 jobs/s with one worker to roughly 4,000 with four, but the step from four to -eight workers added little at the median (3,915 to 4,161 jobs/s) while the best runs kept -growing; with a noisy machine and a 16-thread CPU shared with Postgres itself, a CPU or -disk limit and the claim-side skipping cannot be told apart from these runs. This +went from 1,323 jobs/s with one worker to 2,788 with two, 5,267 with four and 10,103 with +eight, which is close to linear at the median. The 4-worker cell has a wider range +(3,870 to 5,577) than the others. With a 16-thread CPU shared with Postgres itself, a CPU or +disk limit and the claim-side skipping cannot be told apart from these runs, and eight +workers is as far as this benchmark went. This benchmark did not instrument lock skipping, so the contention explanation above describes how the mechanism works, not a measured cause of this curve. If you want to measure it, compare `pg_stat_activity` wait events and the claim statement time (`pg_stat_statements`) @@ -223,10 +200,10 @@ across worker counts on a dedicated machine. * **One machine, one configuration.** Postgres, Valkey, the workers and the load generator compete for the same 16 threads and the same disk. On separate hosts the network adds latency and the CPU contention disappears. Neither is modeled here. -* **Noisy host.** See above. The load averages are stored with every measurement. +* **Not a dedicated machine.** See above. The load averages are stored with every measurement. * **The ratios are not like-for-like.** Besides durability, the pickup intervals differ in what they contain (see "What is timed"). -* **Five runs.** Enough to show the spread, not enough for confidence intervals. +* **Six runs.** Enough to show the spread, not enough for confidence intervals. * **No-op handler.** Real jobs spend time in their handler, which moves the bottleneck away from the queue. For handlers that take tens of milliseconds or longer, the queue is unlikely to be the limit at these rates. @@ -236,7 +213,8 @@ across worker counts on a dedicated machine. claim and every completion with an fsync; Valkey runs without persistence and Celery acknowledges early (`acks_late=False`), so a crash loses in-flight tasks. A Valkey with `appendonly yes`, or Celery with `acks_late=True`, would be slower; neither was measured. -* **Order not counterbalanced (first run).** pgqueue always ran first. See "How to read this". +* **Celery outliers.** Three of the Celery cells (4 workers run 6, 8 workers runs 5 and 6) were + two to three times slower than its other runs for a reason that was not investigated. * **Celery configuration.** One configuration (thread pool, 32 threads per process, prefetch multiplier 4, early ack, defaults otherwise). Other pools and settings were not measured. RQ was not benchmarked. @@ -249,12 +227,12 @@ across worker counts on a dedicated machine. ## Method (schema 2) -`bench/bench.py` was changed after the first run to remove the weaknesses described above. -None of the numbers in this document come from the changed script yet. +`bench/bench.py` counterbalances the order and records the load; the numbers in this document +come from a run of it. * **Counterbalanced order.** In every repeat, odd run numbers measure pgqueue first and - even run numbers measure Celery first. With the default 5 runs this is 3 against 2, so use - an even `--runs` (for example 6) if you want an exactly balanced design. + even run numbers measure Celery first. Use an even `--runs` (the published run used 6) + for an exactly balanced design. * **Cool-down between the two systems of a pair.** `--cooldown SECONDS` (default 30, forced to 0 with `--quick`) pauses between them so the checkpointer and autovacuum started by the first measurement can settle instead of landing in the second one. It also adds @@ -263,15 +241,13 @@ None of the numbers in this document come from the changed script yet. `load_1min` (after), the 1-minute load average from `/proc/loadavg`, and `position` (`first`, `second`, or `only` for `enqueue_many`, which has no Celery counterpart). * **Results file version.** `bench/results.json` has `schema_version: 2`, and `parameters` - gains `counterbalanced` and `cooldown_s`. A file without `schema_version` is schema 1. - The record fields of schema 1 are all still present, `load_1min` keeps its meaning, so - `bench.report` and `bench.chart` read both versions. The checked-in file is still the - schema 1 first run. + gains `counterbalanced` and `cooldown_s`. The checked-in file is the schema 2 run described + here (2026-10-14, 6 runs). An older file without `schema_version` is schema 1; `bench.report` + and `bench.chart` read both versions. -To rerun on a quiet machine: +To rerun: make bench ARGS="--runs 6" -The run time grows with the number of repeats (the 5-run first run took about an hour, -before cool-downs); check `uptime` first and do not run other jobs at the same time, otherwise the load columns will -just document the same problem again. +The published run took about 25 minutes. Check `uptime` first and do not run other jobs at +the same time, otherwise the load columns will just document the same problem again.