Skip to content

tests: make the test runner and the timing-dependent tests robust - #524

Open
nbd168 wants to merge 9 commits into
ucode-lang:masterfrom
nbd168:fixes-tests
Open

nbd168 wants to merge 9 commits into
ucode-lang:masterfrom
nbd168:fixes-tests

Conversation

@nbd168

@nbd168 nbd168 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

This series changes only tests/. It fixes defects of the test runner
and removes the dependence of five tests on the speed of the host.

Test runner:

  • the work directory and the diff files are created below TMPDIR, and a
    TMPDIR that names no directory ends the run with an error
  • a test program that ends without a read of its code no longer ends
    the runner with SIGPIPE
  • the summary counts a file with several failed cases as one failure
  • the exit status is 1 for a run with failures; it was the number of
    failures, of which only the low 8 bits arrive

Tests:

  • 03_stdlib/55_clock accepts a late end of the sleep and bounds the
    realtime delta by the monotonic clock
  • 06_lib_uloop/13_process_function sends its signal when the child
    reports that its function runs
  • 06_lib_uloop/02_timer_lifetime and 03_resource_release run passes of
    the loop until the expected events arrived, for 10 s at most
  • 06_lib_uloop/11_task_messages lets the task wait for the ticks of the
    parent

tests/cram/test_runner.t is new and covers the runner changes.

Behaviour changes:

  • the runner returns 1 for a failed run, not the number of failures
  • the runner uses TMPDIR, which it ignored before

run_tests.uc creates the work directory of each test file as
/tmp/test.<pid> and the files for a diff in /tmp. Two uloop tests
create their own directories in /tmp. A run in an environment that
provides another temporary directory in TMPDIR writes outside of it.

Create the work directory and the diff files below the canonical path
of TMPDIR if it is set, and let the two tests create their directories
in the work directory. The messages of a test program hold the
canonical path of a source file, which does not match a TMPDIR that
ends in a slash, as on macOS, or that leads through a symbolic link.
End the run with an error if TMPDIR names no directory.

A syntax error shortens a source path of more than 48 bytes to "..."
and its last 45 bytes. With a longer work directory the runner no
longer found the directory in such a path, and the tests
04_modules/06_export_errors and 04_modules/14_circular_imports failed.
Replace the visible end of the directory in a shortened path as well.

Signed-off-by: Felix Fietkau <[email protected]>
run_testcase() writes the code of a test case to the standard input of
the test program. A program that ends without a read of its input, like
the one of 99_bugs/35_vm_callframe_double_free, whose module that -l
loads raises an error, can close the pipe before the runner writes. The
runner then gets SIGPIPE and ends with the rest of the tests not run. A
host under load shows this.

Handle SIGPIPE in the runner, so that the write fails and the runner
takes the exit code of the program as before. The runner does not
ignore the signal, because the test programs inherit an ignored signal.

Fixes: fbabec4 ("tests: replace test runner shell script with ucode implementation")
Signed-off-by: Felix Fietkau <[email protected]>
run_test() returns the number of failed test cases of a file, and the
summary adds that number to the count of failed files. A file with two
failed cases gives the line "Ran 1 tests, -1 okay, 2 failures".

Count a file once if any of its cases failed, like the shell script
that run_tests.uc replaced did.

Fixes: fbabec4 ("tests: replace test runner shell script with ucode implementation")
Signed-off-by: Felix Fietkau <[email protected]>
03_stdlib/55_clock requires that 240 ms to 260 ms pass on both clocks
around a sleep of 250 ms. A host under load ends the sleep later, and
the test fails.

Keep the lower limit for both clocks. Read the monotonic clock before
and after each read of the realtime clock, and require that the realtime
delta lies between the two monotonic deltas, with the tolerance of
10 ms. The upper limit of the monotonic delta itself becomes 60 s.

Fixes: 171402f ("lib: add date and time related functions")
Signed-off-by: Felix Fietkau <[email protected]>
06_lib_uloop/13_process_function sends a signal to the child 100 ms
after the fork and lets the function of the child sleep for 300 ms. If
the host does not run the parent for that time, the child ends first,
and the test prints "TERM: 0".

Let the function report through a pipe that it runs, and wait for the
parent on a second pipe. The parent sends the signal after the report
and lets the child go on afterwards, so a child that survived SIGTERM
still ends with code 0. The SIGUSR1 case runs further passes of the loop
before it prints the count, because a signal that reached the callbacks
of the parent is an event that can follow the exit of the child.

Fixes: 1ed0e1b ("uloop: allow a function to prepare the child in process()")
Signed-off-by: Felix Fietkau <[email protected]>
Two cases of 06_lib_uloop/02_timer_lifetime end the loop with a timer
of 20 ms or 30 ms and expect that a timer rearmed twice, or an interval
of 5 ms, fired before. On a host under load the end timer is due
before that, and the cases print "rearmed: 2" or "armed: false".

Run the loop in periods of 10 ms until the expected callbacks ran, for
10 s at most. The rearm case then runs the loop for a further period,
in which a timer that is still armed fires. The runs that expect no
tick keep their delay.

End each period from a timer. Without libubox commit 362951a2d96e
("uloop: fix uloop_run_timeout"), uloop.run() with a timeout returns
only when the loop ends.

Fixes: 115d308 ("uloop: release timers and handles once they are finished")
Fixes: 9863904 ("uloop: disarm an interval on set() without a value")
Signed-off-by: Felix Fietkau <[email protected]>
The cases of 06_lib_uloop/03_resource_release end the loop with a timer
of 50 ms or 500 ms. They expect that a process and a task ended, that a
closed pipe reported two hangups, or that an interval of 1 ms ticked
three times before. On a host under load the cases print "hangups: 1"
or "ticks: 2".

Run the loop in periods of 10 ms until the events arrived and the
descriptors of the handle and the task are closed, for 10 s at most.
Send the signal after the third tick. Run the loop for a further period
afterwards, in which a handle or an interval that is still registered
calls its callback again.

End each period from a timer. Without libubox commit 362951a2d96e
("uloop: fix uloop_run_timeout"), uloop.run() with a timeout returns
only when the loop ends.

Fixes: 115d308 ("uloop: release timers and handles once they are finished")
Signed-off-by: Felix Fietkau <[email protected]>
The first case of 06_lib_uloop/11_task_messages lets the task sleep for
500 ms between two messages and expects ten ticks of a 20 ms timer in
the parent in that time. A parent that the host does not run for a part
of that time counts fewer ticks.

Let the parent create a file after the tenth tick, and let the task
send its second message when the file exists. The task gives up after
10 s, so a loop that blocks between the messages still fails the case.

Also end the loops of all three cases after 60 s, and only if the last
message did not arrive, so that a slow host has the time to send the
large message of the second case.

Fixes: 1f54bd2 ("uloop: read task messages without blocking the event loop")
Signed-off-by: Felix Fietkau <[email protected]>
run_tests.uc passes the number of failed files to exit(). The exit
status holds the low 8 bits of that number only, so a run with 256
failed files ends with the status 0.

Return 1 for a run with failures.

Fixes: 41d33d0 ("tests: custom: return exit code if tests fails")
Signed-off-by: Felix Fietkau <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant