You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Flaky chaos test: executor loss mid-fetch surfaces as h2 broken pipe instead of a fetch failure #2811
stage_losing_input_after_partial_success_is_recovered (chaos-testing/tests/ha.rs, Scenario K) fails intermittently in CI. When the map-side executor is killed while a final-aggregate task is streaming its shuffle input, the task sometimes fails with a raw transport error instead of a fetch failure:
query must recover: Execution error: Job ... failed: Job failed due to stage 2 failed:
Task failed due to runtime execution error: DataFusionError(Shared(External(ArrowError(IpcError(
"Arrow error: Ipc error: code: 'Unknown error', message: \"h2 protocol error: error reading a body
from connection\", source: hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe,
error: \"stream closed because of a broken pipe\" }) })")))))
The scheduler treats this as an ordinary task failure and fails the job, so the stage is not re-run as the scenario expects.
To Reproduce
Flaky, so it needs a few runs of the chaos test:
cargo test -p ballista-chaos --test ha stage_losing_input_after_partial_success_is_recovered
Observed in CI:
main push run 42113028346 (2026-11-05), test linux ballista
Losing the executor that holds the shuffle input should be reported as a fetch-partition failure so the scheduler re-runs the producing stage, whichever way the connection breaks (clean Unavailable, h2 reset, or mid-body broken pipe).
Additional context
#2202 made wrapped FetchFailed errors classify as fetch-partition errors, but a body-read failure on the Flight do_get stream arrives as ArrowError::IpcError wrapping a tonic/hyper error and appears not to be matched.
Activity