rt-latency-lab

Measuring real-time wakeup latency on Linux — and knowing when the measurement is worthless.

This is a cyclictest-alike. cyclictest is the reference tool and is what you should use for real work. This exists because a claim to understand real-time latency is worth more when the measurement is one you built and can defend than when it is a number copied from someone else's output.

It supplies the ground truth that safeedge has to caveat: every latency figure that runtime reports comes with a note saying the host could not support a trustworthy tail measurement. This is the repository that says what a trustworthy one would require.

---

The finding

Five runs of an identical command on this development host — SCHED_FIFO priority 80, mlockall, 1 kHz, 6 s each:

run	p50	p99	p99.9	max
1	26	196	600	3117
2	25	148	295	753
3	24	7308	27185	33156
4	25	203	14880	19856
5	25	178	8711	11442

Microseconds.

The median varies by 1.1×. The p99.9 varies by 92×.

That asymmetry is the point. A median that stable proves the measurement apparatus works — clock, loop, arithmetic, all fine. A tail that unstable proves something outside the process is intermittently stalling it, and nothing inside the guest prevents it: not SCHED_FIFO, not mlockall, not CPU pinning. The process is not running while it happens.

So: reproduce a tail figure or do not quote it. scripts/stability.sh tells you which situation you are in, and prints a verdict:

p50    ranged 24 .. 26 us  (1.1x)
p99.9  ranged 295 .. 27185 us  (92.2x)

VERDICT: the tail is not reproducible on this host.
The median is stable, so the measurement itself is fine -- but
something outside this process is stalling it intermittently.
Do not quote a single run's p99.9 from this machine.

---

The tool refuses to flatter you

rt-latency prints its conditions before its results, and says plainly when they do not support the numbers:

## Conditions

- Scheduling:         fifo (DENIED)
- CPU affinity:       not requested
- PREEMPT_RT kernel:  NO
- Virtualised:        YES
- Diagnostics:        pthread_setschedparam: Operation not permitted;

## Verdict

NOT TRUSTWORTHY as a characterisation of this code or this kernel.
  - Virtualised host: the hypervisor schedules the whole guest, so the
    tail is its scheduling decision rather than the guest kernel's.
  - Real-time scheduling was requested and denied, so this is a
    measurement of the time-sharing scheduler under whatever else is
    running.

The median may still be indicative. The tail is not.

Requesting SCHED_FIFO and being refused is common — RLIMIT_RTPRIO is 0 by default in many environments, including WSL2, where its hard limit is 0 and cannot be raised at all. A tool that does not read the policy back will report time-sharing latency as though it were real-time latency, and nothing in its output will hint at it.

---

Build and run

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build

./build/rt-latency --period-us 1000 --duration 10 --policy fifo --priority 80 --lock-memory

Real-time scheduling needs CAP_SYS_NICE and a non-zero RLIMIT_RTPRIO. Where the host cannot provide them, a container can:

docker build -t rtlab . && docker run --rm --cap-add=SYS_NICE --ulimit rtprio=99 --ulimit memlock=-1 -v "$PWD/results:/opt/rtlab/results" rtlab results 8

That is how the sweep below was produced on a machine whose kernel refuses SCHED_FIFO outright. It does not make the absolute numbers trustworthy — the host is still virtualised and the tool still says so — but it makes the relative comparison possible.

---

The sweep

scripts/sweep.sh runs a 2×4 grid: two scheduling policies against four load conditions. The grid is that shape on purpose. The interesting question is not "how low can latency go" — an idle machine with SCHED_FIFO always looks good — but which kinds of contention a real-time policy actually protects against.

Run	RT granted	p50	p99	p99.9	max
other-idle	no	76	282	437	4662
other-cpu	no	78	249	349	642
other-memory	no	81	254	368	759
other-cpu-memory	no	94	301	586	1025
fifo-idle	yes	26	270	8987	12969
fifo-cpu	yes	26	186	265	351
fifo-memory	yes	36	28517	107970	107970
fifo-cpu-memory	yes	34	411	879	2323

Microseconds. Italicised cells are hypervisor stalls, not measurements.

Read the medians; discard those tails. In the previous run of this exact sweep the stall landed on fifo-idle instead — 27 ms at p99.9. It moves between runs because it is not a property of any configuration on this table. It is the host pausing the whole VM, and it lands wherever it lands.

That is precisely why the stability check exists, and why the medians are the only rows worth reading here:

SCHED_FIFO roughly triples the median. 26 µs against 78 µs, consistent across both sweeps. That is the policy doing what it is for, and it reproduces.

CPU load and memory load are not the same thing. Under SCHED_FIFO, CPU contention barely registers — fifo-cpu is the cleanest row on the table, p99.9 of 265 µs — because the policy guarantees run time. Memory contention costs more at the median (36 µs against 26 µs) because cache is not a scheduled resource: no priority prevents a thread from taking misses when its working set has been evicted. That row is the one most benchmarks omit, and it is the one that decides whether a real deadline is met.

On a machine where the tails were trustworthy, that second effect is what the p99.9 column would show. Here it cannot, and saying so is more useful than publishing the number anyway.

The plot is a CDF on an axis linear in nines, not a histogram: a histogram renders the interesting region as an invisible sliver at the right-hand edge, while a nines axis gives p99.999 the same vertical space as p90.

---

No dependencies, anywhere

The plotter emits SVG directly, in about three hundred lines of pure Python.

That is not minimalism for its own sake. Real-time measurement happens on target hardware — a minimal install, often no network, frequently no Python packaging at all — and a plotting script that needs a pip install is a script that runs on the developer's laptop and measures the developer's laptop.

---

Further reading

docs/methodology.md is the substance: what makes a latency measurement valid, the seven things that quietly invalidate one, and an explicit list of what this tool does not do — no SCHED_DEADLINE, no interrupt-latency measurement, no reconfiguring the machine underneath you.

---

Licence

Apache-2.0.