Priority Inversion Lab

The CPU was busy. The urgent task wasn't running.

An interactive Nim 2.2.10 simulator for the scheduler causality hidden behind healthy utilization. It makes a high-priority waiter, low-priority lock owner, medium-priority preemption, priority donation, and deadline margin visible—and demonstrates how telemetry.sh can join those signals into one investigation.

What it demonstrates

Priority inversion begins when high-priority work needs a resource held by lower-priority work. The dangerous case becomes unbounded: medium-priority CPU work can preempt the low-priority owner, indirectly preventing the high-priority waiter from running.

A CPU graph sees useful work on the core. An urgent latency graph sees a deadline miss. Neither explains that a runnable medium task displaced the only task capable of releasing the lock.

The model replays one fixed-priority, single-core schedule through four strategies:

Strategy	Scheduler behavior	Constraint
Ordinary mutex	Owner retains its base priority while the urgent task blocks	Simple, but medium work can extend the urgent wait without a bound
Priority inheritance	The urgent waiter temporarily donates its priority to the owner	Runtime support and nested donation chains must be correct
Priority ceiling	The owner receives a configured ceiling when it acquires the lock	Predictable, with configuration cost and reduced concurrency
Short critical section	Only a state handoff remains under the lock	Requires snapshot, validation, or compensation logic

With an urgent arrival at 4 ms, 18 ms of low-priority lock work, six medium-priority tasks at 8 ms each, and a 30 ms response budget, the deterministic default model produces:

Strategy	Urgent lock wait	Inversion time	Urgent response	Deadline	CPU busy
Ordinary mutex	62 ms	48 ms	65 ms	missed by 35 ms	100%
Priority inheritance	14 ms	0 ms	17 ms	13 ms spare	100%
Priority ceiling	14 ms	0 ms	17 ms	13 ms spare	100%
Short critical section	0 ms	0 ms	3 ms	27 ms spare	100%

Every policy executes the same CPU work. The outcome changes because task identity, lock ownership, and effective priority change—not because utilization falls.

These are analytical results, not a benchmark of Nim, an operating system, or a real-time scheduler. Use the lab to form a hypothesis, then validate the actual service and runtime with scheduler and lock telemetry.

Run it

Requirements: Nim 2.2.10, a C compiler, Node.js, and curl. The application uses only Nim's standard library.

make check
make run

Open http://127.0.0.1:8080.

Or use the pinned Nim builder image:

docker build -t priority-inversion-lab .
docker run --rm -p 8080:8080 priority-inversion-lab

The multi-stage image compiles with nimlang/nim:2.2.10, copies the native binary into Debian Bookworm slim, and runs as the unprivileged 10001:10001 user.

API and CLI

The browser calls:

curl 'http://127.0.0.1:8080/api/simulate?low_critical_ms=18&medium_tasks=6'

Integer query parameters are normalized to safe model ranges:

• high_arrival_ms
• medium_arrival_ms
• low_critical_ms
• medium_tasks
• medium_cpu_ms
• high_cpu_ms
• deadline_ms
• short_lock_ms

Print the default result without starting a server:

make json

Telemetry recipe

Keep scheduling identity and lock ownership beside request latency:

scheduler.task.id
scheduler.priority.base
scheduler.priority.effective
lock.owner.task_id
lock.wait.duration_ms
lock.priority_donation
scheduler.preemption.reason
deadline.margin_ms

Useful investigation sequence:

1. Find urgent operations with negative deadline margin.
2. Split their duration into runnable, running, and lock-blocked time.
3. Join each lock wait to its owner task ID.
4. Compare the owner's base and effective priority while the waiter blocks.
5. Inspect which task ran when the owner was runnable but off-CPU.
6. Confirm donation reaches nested owners, or shorten the protected work.
7. Re-run the same workload and verify deadline margin—not only average CPU.

The urgent span, mutex wait, owner task, and scheduler slice are separate events. Collapsing them into a generic “CPU busy” symptom hides precisely the causal chain this lab exposes.

Model mechanics

The simulator advances a deterministic single-core scheduler in 1 ms ticks. The low-priority task acquires a lock at time zero. High- and medium-priority tasks arrive later. The scheduler always chooses the highest-priority runnable task; a blocked task is not runnable.

Policy semantics alter only the lock owner's effective priority or critical section boundary:

• ordinary mutex: the owner remains at P010;
• priority inheritance: a blocked P100 waiter boosts the owner to P100;
• priority ceiling: lock acquisition immediately boosts the owner to P100;
• short critical section: bounded state copying stays locked and the rest continues outside.

Intentional simplifications:

• there is one core and one lock;
• task priorities are fixed apart from modeled boosts;
• task CPU bursts do not block on I/O;
• context-switch and lock-operation overhead are omitted;
• medium tasks arrive one millisecond apart;
• priority ties use a stable task-class order;
• nested locks, interrupts, migration, and donation-chain depth are excluded.

References

• Linux kernel RT-mutex implementation design
• Linux kernel RT-mutex subsystem with priority inheritance
• Linux PREEMPT_RT theory of operation
• Nim 2.2.10 downloads
• Nim asynchronous HTTP server

Stack

• Nim 2.2.10 for the deterministic scheduler, JSON API, native server, and tests
• Nim standard library only; no web framework or package downloads
• semantic HTML, modern CSS, vanilla JavaScript, and Canvas 2D
• strict query normalization, security headers, deterministic fixtures, and a non-root multi-stage container
• unit, HTTP integration, and container lanes in GitHub Actions

License

MIT