fix: pull the container image only once per process - #250
Merged
Conversation
Container.start() pulled the image from the registry on every single call. Since each test gets its own container, a full test run made 77 registry round trips for an image that was already local -- 91 of 156 seconds, 58% of the total runtime. That also made the suite flaky. Pull latency is normally ~1.1s, but outliers of up to 15.5s showed up in two of four measured runs, and pytest allows 30s for setup, test and teardown combined. Two runs failed with a setup time of exactly 30.01s, hitting different tests each time. Docker Hub allows 100 anonymous requests per hour and source IP, so a single run already consumes 77% of that budget. Remember which images were pulled in this process, so the first start pulls and every later one reuses the local image. Runtime drops from ~156s to ~64s, and the slowest setup from 15.78s to 1.84s. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01F3ADPwk11GHQF7BBwQHaPz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Container.start()callsdocker pullon every invocation. Since each test gets its own container via the fixture, a full suite run makes 77 registry round trips for an image that has been local since the first one.Measured across four instrumented runs (locally):
That last row matters: outside the suite the pull is perfectly stable, so this is a network effect, not a code path that got slower.
Why it made the suite flaky
pytest allows
timeout = 30for setup, test and teardown combined. A 15 s outlier eats half of that. Two runs failed with a setup time of exactly 30.01 s — that is the timeout firing, not a test misbehaving, which is why it hit a different test each time (test_cancelling_read_events_async, thentest_throws_error_for_invalid_subject).Docker Hub allows 100 anonymous requests per hour per source IP (verified via the
ratelimit-limitheader). One suite run consumes 77 of them, so a second run within the same hour exceeds the budget. CI runners share IP ranges, so the pressure there is higher still.The fix
Remember which images were already pulled in this process. The first
start()pulls, every later one reuses the local image. Images are still refreshed once per test run, so the semantics are unchanged.Result
The runtime gain is far bigger locally than in CI, because GitHub runners sit much closer to the registry — there each pull costs a fraction of the ~1.1 s seen locally. The point of the change is the same in both places though: the number of network operations that can stall drops from 77 to 1, and with it the exposure to rate limiting, which is worse in CI precisely because runners share IP ranges.
All 77 tests pass on 3.12, 3.13 and 3.14.
🤖 Generated with Claude Code
https://claude.ai/code/session_01F3ADPwk11GHQF7BBwQHaPz