Skip to content

Report will_be_retried from the retry filter's decision - #1910

Open
Enceradeira wants to merge 2 commits into
mainfrom
fix/cucumber-ruby-1905-will-be-retried
Open

Report will_be_retried from the retry filter's decision#1910
Enceradeira wants to merge 2 commits into
mainfrom
fix/cucumber-ruby-1905-will-be-retried

Conversation

@Enceradeira

Copy link
Copy Markdown
Contributor

Description

Fixes #1905. Companion to cucumber/cucumber-ruby-core#345, and an alternative to #1907 / cucumber/cucumber-ruby-core#344.

With --retry 1 --retry-total 2 and three failing scenarios, the third scenario is missing from the HTML report. The TestCaseFinished message of that scenario claims willBeRetried: true, because the flag was computed from --retry alone, but the retry filter, honouring --retry-total, never retries it. The HTML formatter hides an attempt that will be retried and waits for one that never comes.

On main the messages are broken in two further ways, which #1907 does not address:

  • The runner is given the number of retries where it expects the number of attempts. With --retry 1 willBeRetried is never true, so a retried attempt is reported as a final result and the report shows the scenario twice.
  • The first attempt's TestCaseFinished is emitted after the retry, with the started id of the retry (fixed in the core PR).

This PR introduces Cucumber::RetryPolicy, shared by the retry filter and the runner. The filter asks it whether a failed test case is retried and records what it did, the runner asks the same question when it reports the outcome. will_be_retried? has no side effects, so both get the same answer.

A new scenario in features/docs/cli/retry_failing_tests.feature asserts the message values (attempt, willBeRetried, and that every testCaseFinished references the started id of the same attempt), which no existing test did.

Notes

Type of change

  • Bug fix

Checklist:

  • Tests have been added for any changes to behaviour of the code
  • New and existing tests are passing locally and on CI
  • bundle exec rubocop reports no offenses
  • RDoc comments have been updated
  • CHANGELOG.md has been updated

The TestCaseFinished message of a scenario claims it will be retried
based on --retry alone, ignoring the --retry-total circuit breaker, and
is off by one. The HTML formatter relies on that flag to pick the final
attempt of every scenario, so it drops or duplicates scenarios.

The new scenario lists every attempt reported by the message formatter
with its willBeRetried flag, for a run that hits --retry-total.

See #1905.
The runner reported will_be_retried from --retry alone, ignoring the
--retry-total circuit breaker of the retry filter. Once that limit was
hit, the TestCaseFinished message of a failed scenario still claimed it
would be retried, so the HTML formatter dropped the scenario from the
report. The value was also off by one, as the runner was given the
number of retries where it expected the number of attempts.

A RetryPolicy, shared by the retry filter and the runner, now owns the
decision. The filter asks it and records what it did, the runner asks it
when reporting the outcome of a test case.

Fixes #1905.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

retry-total conflicts with html-formatter when exhaustion limit hit

1 participant