feat: rerun attempt history from merged outputs (#310) - #332
Open
timdegroot1996 wants to merge 17 commits into
Open
timdegroot1996 wants to merge 17 commits into
timdegroot1996 wants to merge 17 commits into
Conversation
Reimplements the exception-tracking feature from the stale PR #253 on a fresh branch off main (5 months of drift made the original diff conflict in 10 files), using its diff as the spec rather than resolving merge conflicts. - New ExceptionProcessor (processors.py): a ResultVisitor that tracks TRY/EXCEPT depth and records the message of the innermost failing keyword per TRY block (nested TRY, FOR-loops inside TRY, and keyword-wrapping- keyword cases all only count the leaf failure, not parents/ancestors). - New `exceptions` SQLite table (queries.py, database.py): created for both fresh and legacy databases, inserted per-run, included in get_data(), and cleaned up in _remove_run() — all defensively tolerant of a missing table for custom database classes. - New "Keyword Exceptions" graph (bar/timeline, keyword.js + graph_data/exceptions.js) and "Table Exception" (tables.js), wired through the existing graphMetadata-driven layout system — no changes needed to layout.js/graphs.js/settings.js since graph visibility and GridStack placement are fully data-driven from graphMetadata.js. - filter.js: filteredExceptions flows through the same timezone/filter/sort pipeline as the other filtered arrays. Validated end-to-end against a real Robot Framework suite exercising nested TRY/EXCEPT, a FOR loop inside TRY, and a keyword wrapping another failing keyword — exception counts matched hand-counted expectations exactly, the `exceptions` table and dashboard HTML embed the right data, and the bundled JS (extracted from the generated HTML) is syntax-clean with no duplicate function declarations. Tests: 30 new Python tests (ExceptionProcessor unit tests, database insert/get/remove-run round-trips including a missing-table legacy-DB case, dashboard placeholder embedding) and 10 new JS tests for get_exceptions_data() (bar/timeline aggregation, sorting, limits, aliasing). 340 Python / 266 JS tests passing overall. Docs: docs/graphs-tables.md, .github/skills/dashboard.md, CLAUDE.md, .github/copilot-instructions.md, README.md, setup.py long_description. Co-Authored-By: Claude Sonnet 5 <[email protected]>
# Conflicts: # .github/copilot-instructions.md # CLAUDE.md
The new Exceptions tile makes the keyword statistics section taller than the 1600px test viewport, so the page scrolled and the sticky navbar bled into the element screenshot. Raise the viewport height to 2000px and update the reference screenshot with the Linux-rendered result. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two issues surfaced by the new test fixtures:
- --messageconfig patterns were inlined with str(list).replace("'", '"'),
so any quote in a pattern broke the JS string literal and left the
dashboard on the loading spinner. Use json.dumps and escape for the
single-quoted literal that data.js JSON.parses.
- Run metadata was deduplicated through a set, so its order changed per
process (hash randomisation). Use dict.fromkeys to dedupe in document
order; the old fixtures had no metadata so this never showed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Replace the 15 hand-made output.xml/log.html fixtures with 18 generated from two simulated projects (WebshopUI, WebshopAPI) whose behaviour is declared in tests/robot/resources/generator/libraries/profiles.py: persistent, flaky, broken-since and fixed-since tests, outage runs, an all-green and a pass+skip-only run, TRY/EXCEPT exceptions, feature-flag skips, duration trends and run/suite metadata, so every dashboard graph has data. Timestamps are shifted to a fixed schedule and durations scaled, so regenerating keeps run identities stable. The generator suites are not picked up by any runner; robot.toml gives RobotCode the python-path for their fake libraries. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Python unit tests pick fixtures by glob instead of file name. Robot CLI and database references and reference screenshots regenerated in the Docker image; obsolete overview screenshots removed. The timezone mask in the database comparison accepts missing microseconds and negative offsets, which the new fixtures use. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
CI robot job took ~4.7 min: ~110 s of container setup and 155 s of tests. Now 75 s of tests and a checkout + pip install: - Browser suites generate one shared dashboard per run (pabot lock) instead of parsing all 18 fixtures per test; each test still gets its own browser context. Cleanup lives in __init__.robot because pabot loads it without a resolvable path for Resource imports. - Chart animations are disabled through tests/robot/resources/ test_config.json (-j). The settings UI "duration" only scales the stagger delay in graph_config.js, Chart.js still animated 1 s, which the old Sleep 1s was covering. - Wait For Dashboard Idle replaces that sleep: Open Dashboard injects a test-only window.dashboard_is_idle() (resources/scripts/ dashboard_idle.js, unit-tested) that is true only when no spinner, overlay, modal/backdrop, jQuery fade or Chart.js animation is active and 50 ms have passed since. It must stay synchronous: Playwright's waitForFunction treats a returned Promise as truthy. - pabot runs 4 processes (ROBOT_PROCESSES overrides). Failed tests are rerun once and merged with rebot --merge, so a transient browser crash or timing race does not fail the pipeline. - The robot job runs in a prebuilt image published to GHCR by the new test-image.yml workflow (from the existing robot Dockerfile) instead of installing pip, the test requirements and rfbrowser every run. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Cross-platform replacement for the hard-coded command list in example.bat: derives tags, versions, timezones and custom filters from the generated fixtures and copies the result into example/. example.bat is now a one-line wrapper. Example dashboard, database and message config rebuilt from the new fixtures. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The Browser library installs its own Playwright (node side), which can be newer than the Playwright base image's, so rfbrowser init used to download a second full browser set next to the base image's three. Drop the base browsers, install only chromium for the library's version and flatten the result into a single layer so the deleted files and caches are really gone: 4.9 GB -> 2.3 GB, same fonts and system libraries so the reference screenshots are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A single flattened layer is downloaded and decompressed serially, which made the container start slower than with the larger layered image. Copy the filesystem in a few similar-sized layers instead so they are pulled in parallel, and push them zstd-compressed from test-image.yml. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Brings the fixture generator, the regenerated output.xml fixtures, the faster robot pipeline and the example script onto this branch so they reach main together with the rerun feature. Conflict resolution: - 02_overview.robot: Webshop project names from fixture-generator, but without the per-project version filter tests that #322 removed (and their prj*V1_1.png references). - test_processors.py: the generated fixtures contain TRY/EXCEPT exceptions, so the exceptions assertion from fixture-generator wins. - Overview/keyword reference screenshots: fixture-generator versions; they are refreshed from a Docker run in the follow-up commit. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`robot --rerunfailed` + `rebot --merge` keeps only the final result of a re-executed test, but records every earlier attempt (status + message) in the test message as HTML. The dashboard now parses that history into a JSON `attempts` column on the tests table (oldest to newest) and stores the final attempt's message as the test message instead of the raw merge HTML that used to end up in the messages graph. Merged suites lose their start time in rebot; the earliest test start is used as a fallback so the runs/suites tables no longer show null. Front end: - Test section filter bar gets a Reruns select (Mark Reruns default / Final Result / First Attempt), persisted in settings.switch. Re-executed tests get a blue border in the Test Statistics timeline and scatter, Most Flaky, Recent Most Flaky, Most Failed, Recent Most Failed and Messages timelines; tooltips list every attempt with its message. First Attempt shows the status of the first try. Final Result behaves exactly as before. - Most Flaky counts a status change inside the attempt history as a flip, so a test that only passes on retry every night shows up. - Compare page gets its own Reruns select for the Tests graph. - Overview run cards show "Rerun: N (fixed M)" next to the status lines when tests were re-executed. - Stat widgets: Re-executed Tests, Recovered On Rerun, Failed All Attempts. Tables page: attempts column on the tests table. The Reruns control lives in the section filter bars because the graph headers cannot fit a third control at the default tile width; the bar row gap is an integer so a wrapped bar does not shift the sections below by a sub-pixel amount. Tests: parser/DB/processor unit tests, JS unit tests for the graph data modules, a robot test that merges two fixtures with rebot and checks the marked bars per view, and a robot test for the exceptions table. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
RERUNS in generate.py re-executes the failed tests of WebshopUI run 7 (one rerun) and WebshopAPI run 5 (two reruns) with robot --rerunfailed and merges the attempts with rebot --merge, so the fixtures carry an attempt history for the rerun feature. The merged output keeps the scheduled start as `generated`, so run identities and file names are unchanged. Fixtures regenerated; reference screenshots, database_output and the example dashboard refreshed from the Docker image. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
buildx attaches provenance/SBOM attestations by default, which turns the push into an image index whose children appear as "untagged" versions of the GHCR package. Deleting one of those in the package UI breaks the tagged image (manifest unknown on pull), which is what took the CI job container down on 2026-09-18. Disable the attestations so the package has exactly one version per push and nothing that looks deletable. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Any upload path works after `rebot --merge` (script, /add-outputs, admin page); only the listener does not, because it fires when the robot run ends, before the merged file exists. Also import `re` names directly in processors.py to match the other imports. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
robotframework 7.4.2 -> 7.5 and robotframework-browser 20.0.0 -> 20.4.0 (chromium 1234); the reference screenshots are unchanged. test-image.yml keeps the three newest package versions and deletes the rest after every push, so refreshing the image no longer leaves old versions behind. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #310
Also brings the
fixture-generatorbranch to main (merge commite09aa37): fixture generator, regenerated fixtures, faster robot pipeline,scripts/example.py.Problem
With
robot --rerunfailed, a test that fails on the first attempt and passes on the retry is invisible in the dashboard: only the final status of the merged output is stored. Consistent first-attempt failures are a real signal, not noise.Approach
rebot --mergealready records every attempt (status + message) of a re-executed test in the test message as HTML, and it is the workflow Robot Framework documents for--rerunfailed. The dashboard now reads that history instead of linking separate uploads: one run stays one run (a rerun output alone would show up as a 5-of-100-tests run in every graph), and no new CLI/server/listener surface is needed. Upload the mergedoutput.xmlonly — see the new Reruns docs page.What changed
Processing / storage
tests.attemptscolumn (JSON, first attempt → last), inline migration. The test message is the final attempt's message; previously the merge HTML (*HTML* <span class="merge">…) ended up in the messages graph.Dashboard
Rerun: 49 (fixed 22)line next to Passed/Failed/Skipped.attemptscolumn.Fixtures
RERUNSin the generator: WebshopUI run 7 (one rerun) and WebshopAPI run 5 (two reruns) are re-executed and merged, so the fixtures, reference screenshots and the example dashboard show the feature.Docs:
docs/reruns.md(+ sidebar, index card, README), graphs-tables, custom-database-class, generator README, CONTRIBUTING, testing skill.Tests
66 tests, 62 passed, 0 failed, 4 skipped— newValidate Dashboard Test Statistics Rerun View(merges two fixtures with rebot at runtime, asserts the marked bars per view on the Test and Compare pages) andValidate Compare Exception Table Base View; references refreshed from the Linux run.🤖 Generated with Claude Code