fix(http): stop replaying POST on a gateway 5xx - #61
Conversation
HotdataClient passed its own `retries=` into Configuration, which replaces the generated SDK's retry policy wholesale. That policy listed POST in `allowed_methods` alongside a (502, 503, 504) forcelist, so an intermediary timing out a long request produced a silent, identical re-POST while the server was still working on the first one. For a non-idempotent call — a data load — the duplicate collides with the write lock the original holds and is refused, and the caller sees contention rather than the timeout that actually happened. The override is removed so the SDK's own default applies. It is the policy this wrapper was reaching for: hotdata._retry retries a pre-response connection reset on any method (the stale pooled socket case, where the server did no work) while leaving read timeouts and status retries idempotent-only. hotdata_framework/http.py existed only to build that policy and had no other callers, so it goes with it.
There was a problem hiding this comment.
The reasoning holds up: Configuration(retries=) replaces rather than merges, and the old policy paired a 5xx forcelist with POST in allowed_methods, which is a silent replay of a non-idempotent load. Dropping the override is the right fix, and nothing else in the repo imported hotdata_framework.http (it wasn't in __all__, isn't referenced in CONTRACT.md, and the removal is called out in the changelog). hotdata is locked at 0.8.0 in uv.lock, matching the floor the description reasons about, and the release-metadata check will pass since the version is unchanged.
Two things I could not verify and am not claiming: the three assertions in tests/test_retry_policy.py all describe hotdata 0.8.0's default_retry(), and that package isn't installed in the review checkout — so whether retries is a ConnectionResetRetry with status == 0 and no POST rests on the CI run, which had not reported when this review started. If Configuration's default were None instead, test_no_status_forcelist would fail on None.status_forcelist rather than pass vacuously, so the test does have a floor under it either way.
Non-blocking notes inline on the test comment wording and on the now-stale README bullet.
The test's ProtocolError with no cause is not a read timeout — those are ReadTimeoutError and never reach that branch. Say what the value actually is, and note that _is_connection_error is urllib3-private on an unpinned transitive dependency. README advertised configuring SDK retries; that is now the SDK's own default, which this package deliberately leaves alone.
* docs(changelog): fold the post-bump entries into 0.11.0 #61 merged after #59 set the version, so its notes landed under [Unreleased] while pyproject.toml already said 0.11.0. Publishing from that state would have tagged v0.11.0 with #61 code in it, release notes that never mention the change, and an [Unreleased] section describing things already shipped. That matters more than tidiness here, because #61 REMOVES a public module (hotdata_framework.http and default_http_retries). 0.11.0 would have dropped a public export with no note in its own section. The release script only checks that a matching version section exists, not that [Unreleased] is empty, so nothing would have caught it. Entries merged into the existing sections in Keep a Changelog order: the http removal joins session removal under Removed, and the POST fix goes under Fixed. [Unreleased] is now empty. No content edited, only relocated. * docs(changelog): name the referent, and fix the blank lines The http-module bullet said "the retry policy above", which was true where it sat under [Unreleased] -- directly below the Fixed entry describing it. Keep a Changelog puts Removed before Fixed, so after the fold the nearest thing above is the session removal and the retry policy is below. Exactly what "relocated, not edited" missed: a positional reference does not survive reordering. Now names what it points at. Also collapses two double blank lines -- between the Removed bullets, and between [Unreleased] and [0.11.0] -- to match every other gap in the file.
The bug
HotdataClientpassed its ownretries=intoConfiguration:Configuration(retries=...)replaces the SDK's policy rather than adding to it, and this one putsPOSTinallowed_methodsnext to a 5xx forcelist. So any intermediary with a request timeout — a gateway, a load balancer — answering 502/503/504 on a slow request causes urllib3 to silently re-send that request, up to three times.That is fine for a
GET. It is not fine for a load: the call is not idempotent, and the server is still working on the first one. The duplicate arrives while the original holds the table's write lock, is refused as a conflict within milliseconds, and the caller sees contention instead of the timeout that actually happened. Retry logic layered above then re-uploads and re-sends against a lock that is still held, so the whole load fails without the original ever having failed.The function's own docstring says it is for "transient connection failures (e.g. stale pooled sockets)" — a transport-layer concern. Catching response status codes was not the intent.
The fix
Drop the override so the generated SDK's default applies. That default is the policy this wrapper was reaching for, and
hotdata._retrystates the reasoning directly:and
ConnectionResetRetrystill retries a pre-response connection reset on any method includingPOST— the stale-pooled-socket case this wrapper was written for, where the connection was already gone when urllib3 tried to send and the server did no work. That behaviour is preserved; only the unsafe status/read retry onPOSTgoes away.Confirmed the floor already carries it —
hotdata0.8.0 (our pinned minimum) resolvesdefault_retry()tostatus=0, empty forcelist,allowed_methodswithoutPOST.Behaviour change to be aware of
A request that previously got three silent retries on a gateway 5xx now surfaces the 5xx to the caller on the first one. For a slow non-idempotent call that is the point — it fails cleanly instead of duplicating itself. For genuinely transient gateway blips on reads, there is currently no way to opt back in through this wrapper —
HotdataClient.__init__takes noretries=and buildsConfigurationitself. Adding a passthrough is a small, separate change; deliberately not bundled here, since the default this PR restores is the safe one.Removed
hotdata_framework/http.pyanddefault_http_retries(). The module existed only to build this policy, has no other callers in the repo, and is not inhotdata_framework.__all__. It predateshotdata._retry, which supersedes it. Called out under Removed in the changelog since this is a published package.Tests
tests/test_retry_policy.pypins the resulting policy rather than the absence of an argument, so reintroducing an override that is unsafe forPOSTfails here:status == 0POSTnot inallowed_methodsProtocolErroris not reclassifiedVerified they have teeth by running the previous policy against the same assertions — the first two fail. Full suite: 133 passed.
Lint is clean on the changed files;
client.py's import-ordering warning andtest_request_timeout.py's long line both already fail onmainand are left alone.