K2GO-394 feat(dashboard): base-map downloads via the durable job engine - #559
Merged
Conversation
Add a "basemaps" job type and runner (sockets/maps-base.exec.ts). The app now
downloads the global map pmtiles through dash-node, with the server up, instead
of a blocking aria2c in the maps runrole. That in-proot aria2c wedged at CN:0 on
a full network drop and never recovered.
The runner mirrors kiwix.exec.ts: the canonical aria2 flag set plus the withRetry
outer reconnect loop. On a full interface loss aria2 exits (code 19, DNS) and the
loop re-runs it, which resumes via --continue. The poll shows "Reconnecting n/5".
It downloads the DIRECT pmtiles URL, not a .meta4 metalink. Metalink downloads
were the exact case aria2 could not recover. Files land in /library/www/maps, so
the role's "creates:" and .meta4 probe then skip the download. A cancel prunes the
partial, because aria2 preallocates the full size and the role would take that
partial as complete.
Reached through the generic surface (POST /api/basemaps/download; GET and POST
/api/basemaps/jobs/:id/{cancel,pause,resume,retry}). Kept separate from the FQR
"maps" type (tile-extract). Bumps the dashboard version to 1.3.2.
Device-verified on a fresh rootfs at 4 MB/s: cut wifi mid-download, retryAttempt
climbed 1..5 with the partial kept, then restore resumed from the partial to done.
…node Upstream patch 0003 for roles/maps/tasks/download_large_file.yml. On proot the blocking aria2c cannot recover a full mobile-radio drop; it wedges with no exit. Gate the stock download "when: not is_proot", so it is unchanged off proot. Add a "when: is_proot" step that asserts dash-node already placed the pmtiles at dest_path. dash-node downloads it first, with resume; the role then only post-processes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Base-map (pmtiles) downloads now run through dash-node, the dashboard durable job
engine, with the server up -- the same path ZIM downloads already use. This
replaces the blocking aria2c that the maps runrole ran in-proot, which wedged at
zero connections on a full network drop and never recovered.
Two areas, both separate from the controller app (that integration is a later PR):
Dashboard (
static/dashboard): a newbasemapsjob type and runner(
sockets/maps-base.exec.ts). It mirrors the proven kiwix runner -- thecanonical aria2 flag set plus the
withRetryouter reconnect loop -- anddownloads the DIRECT pmtiles URL into
/library/www/maps. It is reachedthrough the generic REST surface (
POST /api/basemaps/download, and the sharedcancel/pause/resume/retry). Dashboard version 1.3.1 -> 1.3.2.Maps role (
tools/upstream-patches/patches/0003-maps-download-is-proot-delegate.patch):gate the stock download
when: not is_proot(unchanged off proot) and,when: is_proot, assert dash-node already placed the file atdest_path. The rolethen only post-processes; its download tasks skip via
creates:.Why
On proot (Android) a blocking aria2 cannot recover a mobile-radio drop: it sits at
zero connections and never exits, so nothing retries. dash-node downloads with
--continueplus an outer reconnect loop -- the mechanism already proven for ZIMs.Verification
Device-verified on a fresh rootfs at 4 MB/s: a 978 MB terrain pmtiles, wifi cut
mid-download.
retryAttemptclimbed 1..5 ("Reconnecting n/5"), the partial waskept, and after restoring wifi the retry resumed from the partial via
--continueto done. Cancel prunes the partial, so the role does not take a preallocated
partial as complete.
To confirm before merge: that the maps role routes only pmtiles
(
expand_archive=false) throughdownload_large_file.ymlon proot, so theis_prootpresence assert never meets a file the app did not delegate to dash-node.