K2GO-394 feat(maps): resilient base-map download via dash-node - #562
Merged
Conversation
…ke (B.app foundation) The app-side of the resilient maps download. proot shares the host netns, so the in-proot aria2c the is_proot download task opens on 127.0.0.1:6810 is reachable here. - MapsDownloadProgress (pure domain): aria2's status + bytes -> phase, percent, ETA. - MapsDownloadRpc: ~1s poll of aria2.tellActive for the download bar, pause/resume, and shutdown-on-complete -- device-verified that aria2c with --enable-rpc does NOT exit on complete, so the blocking Ansible task needs the app to call aria2.shutdown. Tolerant of an absent RPC (a stock rootfs, or between files) -> reports idle, no crash. - MapsDownloadRepository: the one LiveData the progress screen observes. - MapsRunroleCommand: an overload writes the RPC handshake (maps_download_rpc_secret/ port) into local_vars, with a D2 hex-token guard; the 4-arg form stays RPC-free. Not wired into the install pipeline yet (next commit). No behavior change on its own.
…remove before merge) Test-only scaffolding so the is_proot download task (with --enable-rpc/--continue) can be exercised on device without waiting on a rootfs bake. InstallService copies the patched download_large_file.yml over the role before the runrole reads it. TEMPORARY: this asset + overlayMapsDownloadTask() must be removed before merge; the real role change ships as an upstream_patches patch (rootfs bake). Tracked so it leaves no trace. See the asset header.
…(B.app) Start the MapsDownloadRpc monitor when the maps runrole begins and stop it when it ends: it mints a per-run RPC secret (written to local_vars via MapsRunroleCommand), publishes the subordinate download bar to MapsDownloadRepository, and shuts aria2c down on complete so the blocking task returns. - Reconnection: the existing NetworkStateLiveData observer now also pauses the maps download when validated internet is lost and resumes it when it returns (aria2 --continue picks up the partial) -- Android drives it, not aria2's blind retry. - doPause/doResume route a maps operation through the RPC (relaxing the rootfs-only guard), so the notification/UI pause/resume reach the in-proot aria2c. - Cleanup on onProcessExit/onError/teardown. Degrades cleanly when the RPC is absent (stock rootfs): the monitor reports idle.
InstallService clears/posts the download progress from its background install thread, so setValue crashed (LiveData.setValue is main-thread-only). postValue is safe from any thread. Device-verified: a full maps install now writes the RPC handshake, the monitor drives the in-proot aria2c, and the app shuts it down on complete so the play advances (PLAY RECAP failed=0), with no crash.
…aps detail (B.ui) The honest second measure from the mockup: a thinner download bar under the phase spine in the "Preparing your maps" detail, fed by MapsDownloadRepository -- real percent, bytes, rate and ETA from the in-proot aria2c. A download-scope Pause/Resume toggles ACTION_PAUSE/RESUME, which the service routes to aria2 over RPC. Present only while a live download exists; on a stock rootfs (no RPC) the repository stays idle and the card stays hidden -- the screen is then the phase-spine-only Variant 3. Title string parked in strings_untranslated pending l10n at close.
The maps detail opened under the legacy key "maps" (not "mod:maps"), so it fell through to the observe-only branch and showed only Back -- no Cancel. Treat "maps" as the module it is in configureDetailBar, so the detail reuses the existing Cancel-while-running (the mockup's operation-level Cancel install, confirmed via the shared BrandDialog "Stop installing?") and Retry-on-failure the other modules have. No duplicated cancel logic. Device-verified: Back + Cancel show while maps runs; Cancel opens the Stop-install confirm.
…drop the app overlay Replaces the temporary app-side role overlay (assets/maps/download_large_file.yml + InstallService.overlayMapsDownloadTask, used only to de-risk on device) with the real carry: tools/upstream-patches/patches/0003-maps-download-is-proot-rpc.patch, applied at rootfs-build time by apply-upstream-patches.sh (idempotent, reverse-dry-run skip). The patch splits the maps download task by is_proot; the is_proot variant is resumable (--continue, no --allow-overwrite) and opens a loopback JSON-RPC (--enable-rpc --rpc-listen-all=false, secret/port from vars). Verified: forward-applies cleanly, result matches, reverse dry-run skips. The app keeps writing the RPC handshake and running the monitor; on a rootfs without the patch yet the monitor reports idle and the screen shows phase-only progress (no crash). No trace of the app-side overlay remains.
…he RPC race + shutdown Second-pass review fixes on B: - Reconnection no longer un-pauses a user's manual pause. A new mapsUserPaused owner (set on a user pause, cleared on a user resume / fresh run) gates the auto-resume, so the network only resumes a network-initiated pause -- the rootfs path's rule. - doPause/doResume/onValidatedNetworkReturned capture mapsRpc into a local before use, so a concurrent null from the install thread cannot NPE. - MapsDownloadRpc shutdown-on-complete gates on getGlobalStat numStopped>0 (plus a no-error check) instead of a non-empty tellStopped, so a momentarily empty tellStopped does not miss the shutdown and hang the task. - stop() nulls poll so callAsync's guard means "stopped".
Move k2go_maps_dl_title out of the WIP tracking list into per-locale feature files, so no localization debt ships with the resilient-download UI. Add strings_maps_download.xml in values/ (base English "Downloading map data") and in the 33 translation locales. Translations are machine-generated and marked for human review; this satisfies lint MissingTranslation and keeps the feature-file pattern used by strings_diskguard.xml. Revert strings_untranslated.xml to "No pending strings".
…t/K2GO-394-maps-resilient-download
…t/K2GO-394-maps-resilient-download
…t/K2GO-394-maps-resilient-download
…t/K2GO-394-maps-resilient-download
Add a maps catalog the app resolves layer selections against, so it can hand dash-node the file id to download while the box composes the mirror URL (the same split kiwix uses -- the app never holds the switnet host). One builder: tools/build_maps_catalog.py generates assets/maps_catalog.csv (group,level,file,bytes,date) from the mirror .meta4 pointers; the gradle refreshMapsCatalog task calls it (release / manual) and the weekly publish-catalogs workflow publishes it to Cloudflare. This replaces the inline refreshMapsSizes size task and its maps_sizes.csv -- one build path, no double construction. MapsCatalog reads the file name (fileFor) for the download id and keeps the size for the Choose screen.
…etire the RPC
The maps SETUP now downloads the selected base-map pmtiles through dash-node (the
durable job engine) -- the same resilient path as ZIMs -- before the runrole, so a
network drop recovers ("Reconnecting n/5") instead of wedging an in-proot aria2c.
InstallService.downloadMapsBasemapsThenRun resolves the selection to catalog file
ids (MapsBasemapSelection, a pure unit-tested rule), POSTs them via
RestContentClient("basemaps"), publishes progress to MapsDownloadRepository (the
subordinate download bar + Pause/Resume), and only on completion runs the runrole,
whose download tasks then skip via creates:. A download failure stops the batch,
mirroring the runrole onError.
Retires the aria2 JSON-RPC path: deletes MapsDownloadRpc, reverts MapsRunroleCommand
to the plain 4-arg build (no download handshake), and rewrites MapsDownloadProgress
from aria2 fields to the REST poll fields (percent, speed token, reconnect counter).
Device-verified: a maps install downloaded the selected pmtiles through dash-node
(exact catalog sizes, no leftovers), the runrole skipped them via creates:, and the
play finished failed=0; the bar, Pause/Resume and reconnect all work.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
The maps SETUP now downloads the selected base-map pmtiles through dash-node (the
in-server durable job engine) -- the same resilient path ZIM and Kolibri use --
instead of a blocking in-proot aria2c that wedged on a network drop. The download
shows a live progress bar with Pause/Resume and "Reconnecting n/5", survives a radio
drop (resumes via
--continue), and runs BEFORE the maps runrole; the runrole thenonly post-processes and its download tasks skip natively via
creates:.Pieces:
InstallService): for the maps module it resolves thelayer selection to catalog file ids (
MapsBasemapSelection, a pure unit-testedrule), POSTs them to
RestContentClient("basemaps"), drives the subordinatedownload bar + Pause/Resume from
MapsDownloadRepository, and runs the runroleonly once the download finishes. A download failure stops the batch like the
runrole onError.
tools/build_maps_catalog.py->assets/maps_catalog.csv,gradle
refreshMapsCatalog, apublish mapsline in the weekly catalogworkflow): the app resolves a layer to a pmtiles file id; the box composes the
mirror URL (the kiwix split, so the switnet host never enters the app). Replaces
the inline
refreshMapsSizessize task -- one builder, no double construction.MapsDownloadRpc, revertsMapsRunroleCommandto the plain 4-arg runrole (no download handshake), andmodels progress from the REST poll fields (percent, speed token, reconnect
counter).
The maps role itself is left unpatched -- dash-node placing the files ahead of the
role, plus the role's native
creates:skip, are what make it work. Seecontroller/docs/ADR-K2GO-394-maps-download-via-dashnode.mdfor why there is norole patch.
Why
On proot a blocking aria2 cannot recover a mobile-radio drop: it sits at zero
connections and never exits, so nothing retries and the SETUP hangs. dash-node
downloads with
--continueplus an outer reconnect loop -- the mechanism alreadyproven for ZIMs -- and the app can pause/resume it.
Verification
(
MapsBasemapSelection,MapsDownloadProgress,MapsRunroleCommand).pmtiles through dash-node at their exact catalog sizes with no
.aria2leftover,the runrole ran the clean 4-arg command and skipped every download via
creates:,and
PLAY RECAPfinishedfailed=0. The live bar, Pause/Resume and reconnectwere exercised on device.
Notes
basemapsrunner that composes the mirror URLfrom a file id) -- already merged.
expand_archive) is not delegated -- dash-node does notextract archives -- so it downloads in-proot as upstream does.