Generated test fixtures, faster robot pipeline - #330
Merged
Merged
Conversation
Two issues surfaced by the new test fixtures:
- --messageconfig patterns were inlined with str(list).replace("'", '"'),
so any quote in a pattern broke the JS string literal and left the
dashboard on the loading spinner. Use json.dumps and escape for the
single-quoted literal that data.js JSON.parses.
- Run metadata was deduplicated through a set, so its order changed per
process (hash randomisation). Use dict.fromkeys to dedupe in document
order; the old fixtures had no metadata so this never showed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Replace the 15 hand-made output.xml/log.html fixtures with 18 generated from two simulated projects (WebshopUI, WebshopAPI) whose behaviour is declared in tests/robot/resources/generator/libraries/profiles.py: persistent, flaky, broken-since and fixed-since tests, outage runs, an all-green and a pass+skip-only run, TRY/EXCEPT exceptions, feature-flag skips, duration trends and run/suite metadata, so every dashboard graph has data. Timestamps are shifted to a fixed schedule and durations scaled, so regenerating keeps run identities stable. The generator suites are not picked up by any runner; robot.toml gives RobotCode the python-path for their fake libraries. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Python unit tests pick fixtures by glob instead of file name. Robot CLI and database references and reference screenshots regenerated in the Docker image; obsolete overview screenshots removed. The timezone mask in the database comparison accepts missing microseconds and negative offsets, which the new fixtures use. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
CI robot job took ~4.7 min: ~110 s of container setup and 155 s of tests. Now 75 s of tests and a checkout + pip install: - Browser suites generate one shared dashboard per run (pabot lock) instead of parsing all 18 fixtures per test; each test still gets its own browser context. Cleanup lives in __init__.robot because pabot loads it without a resolvable path for Resource imports. - Chart animations are disabled through tests/robot/resources/ test_config.json (-j). The settings UI "duration" only scales the stagger delay in graph_config.js, Chart.js still animated 1 s, which the old Sleep 1s was covering. - Wait For Dashboard Idle replaces that sleep: Open Dashboard injects a test-only window.dashboard_is_idle() (resources/scripts/ dashboard_idle.js, unit-tested) that is true only when no spinner, overlay, modal/backdrop, jQuery fade or Chart.js animation is active and 50 ms have passed since. It must stay synchronous: Playwright's waitForFunction treats a returned Promise as truthy. - pabot runs 4 processes (ROBOT_PROCESSES overrides). Failed tests are rerun once and merged with rebot --merge, so a transient browser crash or timing race does not fail the pipeline. - The robot job runs in a prebuilt image published to GHCR by the new test-image.yml workflow (from the existing robot Dockerfile) instead of installing pip, the test requirements and rfbrowser every run. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Cross-platform replacement for the hard-coded command list in example.bat: derives tags, versions, timezones and custom filters from the generated fixtures and copies the result into example/. example.bat is now a one-line wrapper. Example dashboard, database and message config rebuilt from the new fixtures. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The Browser library installs its own Playwright (node side), which can be newer than the Playwright base image's, so rfbrowser init used to download a second full browser set next to the base image's three. Drop the base browsers, install only chromium for the library's version and flatten the result into a single layer so the deleted files and caches are really gone: 4.9 GB -> 2.3 GB, same fonts and system libraries so the reference screenshots are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A single flattened layer is downloaded and decompressed serially, which made the container start slower than with the larger layered image. Copy the filesystem in a few similar-sized layers instead so they are pulled in parallel, and push them zstd-compressed from test-image.yml. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #325 (
exceptions-tracking).Summary
tests/robot/resources/generator/replaces the 15 hand-madeoutput.xml/log.htmlfixtures with 18 generated from two simulated projects (WebshopUI,WebshopAPI).libraries/profiles.pydeclares what the data shows (persistent/flaky/broken-since/fixed-since tests, outage runs, an all-green and a pass+skip-only run, TRY/EXCEPT exceptions, feature-flag skips, duration trends, run/suite metadata), so every graph has data. Deterministic timestamps keep run identities stable across regenerations. Not part of any test runner;robot.tomlgives RobotCode the python-path for the fake libraries.--messageconfigpatterns containing quotes broke the inlined JS literal (dashboard stuck on the spinner) →json.dumps+ proper escaping.set) →dict.fromkeys, document order.tests/robot/resources/test_config.json(-j); the settings "duration" only scales the stagger delay, Chart.js still animated 1 s (the oldSleep 1swas covering that).Wait For Dashboard Idlereplaces sleeping: a test-onlywindow.dashboard_is_idle()injected fromtests/robot/resources/scripts/dashboard_idle.js(no product JS changed), true only when no spinner/overlay/modal/fade/Chart.js animation is active for 50 ms. Must stay synchronous — Playwright'swaitForFunctiontreats a returned Promise as truthy.rebot --merge) so a transient browser crash does not fail the run.ghcr.io/marketsquare/robotframework-dashboard-test-robot(existing robot Dockerfile, built by the newtest-image.ymlonmainwhenrequirements-test.txt/Dockerfile change, or manually). The image is chromium-only and flattened to a single layer (4.9 GB → 2.3 GB; same base, so reference screenshots are unchanged). First image pushed manually; the package is public.scripts/example.py(cross-platform) builds the example dashboard;example.batwraps it. Example, docs, CONTRIBUTING and agent skills updated.Verification
example/tests/robot/resources/outputs/log-*.html.Notes
database_output/keywords.txtaverages come from the CI image's Python 3.10; Python ≥ 3.12sum()rounds a fewx.xxx5values differently, so those references must be regenerated in Docker only.duration: settings.show.durationingraph_config.jsso the animation-duration setting means what it says.🤖 Generated with Claude Code