Skip to content

Flaky chaos test: executor loss mid-fetch surfaces as h2 broken pipe instead of a fetch failure #2811

Closed

Description

@andygrove

Describe the bug

stage_losing_input_after_partial_success_is_recovered (chaos-testing/tests/ha.rs, Scenario K) fails intermittently in CI. When the map-side executor is killed while a final-aggregate task is streaming its shuffle input, the task sometimes fails with a raw transport error instead of a fetch failure:

query must recover: Execution error: Job ... failed: Job failed due to stage 2 failed:
Task failed due to runtime execution error: DataFusionError(Shared(External(ArrowError(IpcError(
"Arrow error: Ipc error: code: 'Unknown error', message: \"h2 protocol error: error reading a body
from connection\", source: hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe,
error: \"stream closed because of a broken pipe\" }) })")))))

The scheduler treats this as an ordinary task failure and fails the job, so the stage is not re-run as the scenario expects.

To Reproduce

Flaky, so it needs a few runs of the chaos test:

cargo test -p ballista-chaos --test ha stage_losing_input_after_partial_success_is_recovered

Observed in CI:

Expected behavior

Losing the executor that holds the shuffle input should be reported as a fetch-partition failure so the scheduler re-runs the producing stage, whichever way the connection breaks (clean Unavailable, h2 reset, or mid-body broken pipe).

Additional context

#2202 made wrapped FetchFailed errors classify as fetch-partition errors, but a body-read failure on the Flight do_get stream arrives as ArrowError::IpcError wrapping a tonic/hyper error and appears not to be matched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions