Skip to content

Benchmarks: correct overclaims and counterbalance the method - #15

Merged
MaciejZiel merged 6 commits into
mainfrom
bench-honest-claims
Oct 13, 2026
Merged

MaciejZiel merged 6 commits into
mainfrom
bench-honest-claims

Conversation

@MaciejZiel

Copy link
Copy Markdown
Owner

An independent review of #14 found the benchmark write-up overstated the comparison.

Text (README, docs/benchmarks.md)

  • Removes "under the same load" and "a few seconds apart, under similar load": the order was fixed (pgqueue first, then Celery), so it is not counterbalanced.
  • Restricts the robust claim to 1-2 workers. On the two quietest runs (4 and 5) 4-8 workers were roughly even (8 workers: 0.84 and 1.38). Payload-size scenario at 4 workers: Celery won some runs (2 KiB ratios 3.52, 2.23, 0.89, 1.44, 1.43; at 256 B pgqueue lost 2 of 5 and its median was below Celery's, 2,998 vs 3,163).
  • Labels the Celery column "thread pool, 32 threads, prefetch x4, early ack" and documents acks_late=False (at-most-once on receipt) against pgqueue's fsynced claim and completion.
  • Corrects the pickup-latency definitions (the ratios are not like-for-like), "factor of two to three" to "up to four", and the run time.
  • The existing numbers stay labelled as the first, order-fixed run on a loaded machine.

Method (bench/bench.py)

  • Each repeat alternates which system runs first.
  • New --cooldown (default 30 s, 0 with --quick) between the systems.
  • Each record stores position and the 1-minute load before and after. results.json is now schema_version 2 and stays backward compatible (load_1min kept); the checked-in file is unchanged schema 1.
  • Celery's early ack and prefetch defaults are spelled out in bench/celery_tasks.py (no behaviour change).

Only make bench-quick was run as a smoke test. The full benchmark was not rerun: the machine is loaded by other jobs. Rerun with make bench ARGS="--runs 6" on a quiet machine.

…n the benchmark

Odd runs start with pgqueue, even runs with Celery; a configurable pause (default 30 s) separates the two systems of a pair. Every record stores its position and the 1-minute load before and after. results.json gets schema_version 2 and keeps load_1min, so v1 readers still work.
…ncy per system, state the durability difference
…xtend the caveats, document the schema 2 method
… 1 and 2 workers, add durability and latency definitions
@MaciejZiel
MaciejZiel merged commit e7820d3 into main Oct 13, 2026
5 checks passed
@MaciejZiel
MaciejZiel deleted the bench-honest-claims branch October 13, 2026 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant