Skip to content

Let a retry policy decide will_be_retried in the runner - #345

Open
Enceradeira wants to merge 2 commits into
mainfrom
fix/cucumber-ruby-1905-will-be-retried
Open

Let a retry policy decide will_be_retried in the runner#345
Enceradeira wants to merge 2 commits into
mainfrom
fix/cucumber-ruby-1905-will-be-retried

Conversation

@Enceradeira

@Enceradeira Enceradeira commented Sep 5, 2026

Copy link
Copy Markdown

Description

Alternative to #344, fixing cucumber/cucumber-ruby#1905 together with cucumber/cucumber-ruby#1910.

Since 19.0.0 the Runner emits the TestCaseFinished envelope and decides its will_be_retried field from a maximum number of attempts. The runner cannot know that on its own: whether a test case runs again is decided by whoever runs it again, which in cucumber-ruby is the retry filter honouring both --retry and --retry-total. #344 copies the --retry-total counter into the runner, which makes the reported symptom go away but keeps the retry decision in two places, and leaves two further defects in the message stream:

  • cucumber-ruby passes the number of retries where the runner expects the number of attempts. With --retry 1 the flag is never true, so a retried attempt is reported as a final result and the HTML report shows the scenario twice. With --retry 2 the second attempt is reported as final although a third one follows.
  • The runner broadcasts the test_case_finished event before emitting the TestCaseFinished envelope. The retry filter re-runs the test case from inside that event, so the envelope of the first attempt is emitted after the retry and carries the test_case_started_id of the retry. On main, a retried scenario produces two TestCaseFinished envelopes for attempt 2 and none for attempt 1.

The compatibility kit does not catch any of this because it only compares message keys, not values.

This PR:

  • replaces the max_attempts constructor argument with a retry_policy, an object answering will_be_retried?(test_case, result). With no policy, no test case is reported as retried.
  • emits the TestCaseFinished envelope before broadcasting the test_case_finished event, so a retry started from that event no longer reorders the stream.
  • adds specs for the envelope values and for the re-entrant retry case, which previously had no coverage.

Type of change

  • Bug fix
  • Breaking change (constructor argument of Cucumber::Core::Test::Runner), see upgrading_notes/20.0.0.md

Checklist:

  • Tests have been added for any changes to behaviour of the code
  • New and existing tests are passing locally and on CI
  • bundle exec rubocop reports no offenses
  • RDoc comments have been updated
  • CHANGELOG.md has been updated

@Enceradeira
Enceradeira force-pushed the fix/cucumber-ruby-1905-will-be-retried branch 2 times, most recently from 19900a2 to d98745a Compare September 5, 2026 14:51
The runner emits the TestCaseFinished envelope after broadcasting the
test_case_finished event. The retry filter in cucumber-ruby re-runs the
test case from inside that event, so the envelope of the first attempt
is emitted after the whole retry and carries the test_case_started id
of the retry instead of its own.

The runner also derives will_be_retried from a maximum number of
attempts, which it cannot know: whether a test case runs again is
decided by whoever runs it again, and the retry filter stops retrying
once --retry-total is exhausted. The runner has to ask a retry policy.

These specs fail until the runner emits the envelope before the event
and asks a retry policy.

Related to cucumber/cucumber-ruby#1905.
@Enceradeira
Enceradeira force-pushed the fix/cucumber-ruby-1905-will-be-retried branch from d98745a to 3419704 Compare September 5, 2026 14:55
The runner computed the will_be_retried field of the TestCaseFinished
envelope from a maximum number of attempts. It cannot know that on its
own: whether a test case runs again is decided by whoever runs it again,
which in cucumber-ruby is the retry filter honouring both --retry and
--retry-total. The runner now asks a retry policy instead.

The envelope is also emitted before the test_case_finished event. The
retry filter re-runs the test case from inside that event, so the
envelope of the first attempt used to be emitted after the retry, with
the test_case_started id of the retry.

Related to cucumber/cucumber-ruby#1905.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant