Repository navigation
Ask the original link for a fresh signed URL when one expires mid-download or while paused - #166
Open
Arpit-Khandelwal wants to merge 1 commit into
Open
Arpit-Khandelwal wants to merge 1 commit into
Arpit-Khandelwal wants to merge 1 commit into
Conversation
…nload or while paused A link that redirects to a short-lived signed URL (Hugging Face's resolve links, S3 presigned ones) failed the download with "This link no longer works" once the signature expired, although the link itself still worked. The download now keeps the link it was given (requestedUrl, on the start request and its state, so it's saved with it and kept in history). When a range request is answered 401, 403 or 410 and that link differs from the one in use, it is probed again once and checked exactly as Fix checks a new link (size, range support, bytes compared with what's on disk). Every stream then goes on with the fresh URL. If that fails, the download fails as before. A resume, including after a relaunch, gets another try. Fixes anmolkapil#160
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #160.
A link that redirects to a short-lived signed URL (Hugging Face
resolvelinks, S3 presigned URLs) failed the download with "This link no longer works" once the signature expired, mid-download or while paused, even though the pasted link still worked.What changes
requestedUrlon the start request and its state). The dialog sets it from the probe, which also covers links from the browser extension. It's saved in the manifest and kept in history. The manifest stays at version 7: the field is optional, so downloads saved before this load as they did and just don't refresh.requestedUrldiffers from the URL in use, that link is probed again and checked exactly the way Fix checks a link: same size, range support, and bytes compared with what's on disk. I moved those checks out ofreplaceLinkintoadoptLink, so Fix and this share one path. Every stream then carries on with the fresh URL, with no strikes counted.Why a 403 isn't counted as a connection limit when the refresh works:
concurrency.tsreads a 403 as "fewer connections, please". After a real expiry every new request is refused, and counting those would cut the stream count for minutes after each refresh. This only applies to downloads whose link redirects somewhere else; if the link redirects to the same URL, or the check fails, the 403 counts exactly as before.How I checked it
e2e/signedUrl.spec.tsruns against a local origin where/startredirects to?sig=1, and after "expiry"sig=1answers 403 to range requests while/startpoints atsig=2. It covers four cases: expiry mid-download, expiry while paused then resumed, expiry across a quit and relaunch, and the link itself returning 410 (which still errors, with no loop). Every completed download is checked byte for byte.mainall four fail; with this change they pass, 20/20 with--repeat-each=5.requestedUrlstill loads and completes.I didn't run a real hour-long Hugging Face or S3 download; the expiry is simulated by the local server.