From 9a0b23c21d67d4441687d6b78509980a1fe94402 Mon Sep 17 00:00:00 2001 From: "Luis Guzman (AppDevForAll)" Date: Mon, 7 Sep 2026 09:51:12 -0600 Subject: [PATCH 1/2] K2GO-394 feat(dashboard): base-map downloads via the durable job engine Add a "basemaps" job type and runner (sockets/maps-base.exec.ts). The app now downloads the global map pmtiles through dash-node, with the server up, instead of a blocking aria2c in the maps runrole. That in-proot aria2c wedged at CN:0 on a full network drop and never recovered. The runner mirrors kiwix.exec.ts: the canonical aria2 flag set plus the withRetry outer reconnect loop. On a full interface loss aria2 exits (code 19, DNS) and the loop re-runs it, which resumes via --continue. The poll shows "Reconnecting n/5". It downloads the DIRECT pmtiles URL, not a .meta4 metalink. Metalink downloads were the exact case aria2 could not recover. Files land in /library/www/maps, so the role's "creates:" and .meta4 probe then skip the download. A cancel prunes the partial, because aria2 preallocates the full size and the role would take that partial as complete. Reached through the generic surface (POST /api/basemaps/download; GET and POST /api/basemaps/jobs/:id/{cancel,pause,resume,retry}). Kept separate from the FQR "maps" type (tile-extract). Bumps the dashboard version to 1.3.2. Device-verified on a fresh rootfs at 4 MB/s: cut wifi mid-download, retryAttempt climbed 1..5 with the partial kept, then restore resumed from the partial to done. --- static/dashboard/CHANGELOG.md | 1 + static/dashboard/package.json | 2 +- static/dashboard/routes.ts | 2 +- static/dashboard/server.ts | 1 + static/dashboard/sockets/jobs.ts | 2 +- static/dashboard/sockets/maps-base.exec.ts | 169 +++++++++++++++++++++ 6 files changed, 174 insertions(+), 3 deletions(-) create mode 100644 static/dashboard/sockets/maps-base.exec.ts diff --git a/static/dashboard/CHANGELOG.md b/static/dashboard/CHANGELOG.md index 809244ca2..ae2f80959 100644 --- a/static/dashboard/CHANGELOG.md +++ b/static/dashboard/CHANGELOG.md @@ -4,6 +4,7 @@ One line per version, newest first. Every REST-facing change bumps the version i (the app surfaces it via `/system/dashboard/update-check` and the "Update available" pill), so this file is the human record of what each bump enables. Keep entries short: `version - change (TICKET)`. +- **1.3.2** - Base-map downloads via the durable job engine (K2GO-394). New `basemaps` job type + runner (`sockets/maps-base.exec.ts`), reached through the generic surface (`POST /api/basemaps/download {items:[]}`, `GET /api/basemaps/jobs/:id`, `POST /api/basemaps/jobs/:id/{cancel,pause,resume,retry}`). It downloads the selected global map pmtiles (vector / satellite / terrain) with aria2 straight into `/library/www/maps`, reusing the EXACT kiwix mechanism -- the canonical aria2 flag set plus the `withRetry` OUTER reconnect loop -- so a full Wi-Fi drop (aria2 exits on DNS, code 19) recovers by re-running aria2 which resumes via `--continue`, surfaced as "Reconnecting n/5". This replaces the aria2c the maps runrole ran IN-PROOT, which could not recover a mobile-radio drop (it wedged with no exit): the app runs this job (server up) and the runrole then only post-processes, its download tasks SKIPPING via `creates:` (see the maps role's `is_proot` delegate patch). Two things learned the hard way and encoded here: it passes the DIRECT pmtiles URL, NOT a `.meta4` metalink (metalink downloads were the exact case aria2 could not recover), and it does NOT diverge the flags (an aggressive `--max-tries=1 --timeout=10` cut wedged worse; the kiwix values are load-bearing). Kept SEPARATE from the FQR `maps` type (tile-extract). Device-verified: cut Wi-Fi at 27% of a pmtiles -> "Reconnecting 5/5", partial kept -> restore -> resumed to done, file complete at dest_path. (K2GO-394) - **1.3.1** - Live firehose signal for the app-side backstop (K2GO-386, ADR-386 §6). New read-only `GET /system/disk-guard/firehose` returns `{ recurring, maxStreak, paths, lastTruncatedAtMs, now }`. The in-box guard (1.3.0) truncates a runaway `.log` every tick, so the disk may never go low -- but a recurring firehose means an off-proot orphan the box CANNOT stop; only an app-side reap can. This endpoint exposes the guard's LIVE in-memory streak state (never a parsed log line, so a restart-resolved firehose reports clean) as the app's SECOND reap trigger. `recurring` is `maxStreak >= 2` (a single `.log` refilled past the cap on at least two consecutive ticks); `lastTruncatedAtMs` (wall-clock, 0 if never) lets the app judge freshness. It is an ALERT only: the app re-probes live log growth before it reaps (confirm before acting). Localhost-only. (K2GO-386) - **1.3.0** - Proot log rotation, dash-node-triggered (K2GO-386, ADR-386). proot has no systemd/cron, so `/etc/cron.daily/logrotate` never runs — logrotate was installed but never triggered, and a service log (php-fpm, dash-node) could grow until the device hit ENOSPC. dash-node now runs, every 10 min (no work at boot; `timer.unref`), a firehose guard THEN `logrotate /etc/logrotate.conf`: the guard truncates any log past ~1 GiB in place first (a runaway ~GB/min that logrotate would otherwise copy — doubling disk + pegging CPU on a weak phone), so L2 never meets a firehose; a recurring firehose is flagged for the future app-side reap (ADR-386 §6). The K2Go-owned config `/etc/logrotate.d/k2go` (copytruncate + `size 100M`, proot-correct — no reopen signal — overriding the RPi-oriented nginx/php-fpm snippets and adding calibre-web + dash-node; kiwix has no log, kolibri self-rotates) is installed at deploy by `tools/setup-proot-logging.sh` (rootfs build + rebuild/dev-push), not at boot. Not a REST-surface change; the version bump is the delivery mechanism for the new dash-node behavior (no ansible role yet). (K2GO-386) - **1.2.12** - Dashboard-update card back end (ADFA-5339, Phase 1 server half). New read-only `GET /system/dashboard/rebuild/log`: the last ~200 lines of `/var/log/dash-rebuild.log`, for the card's expandable "Details" (no file yet = empty log, not an error). `POST /system/dashboard/rebuild` now accepts `{ site: true }`: it refreshes the served landing page in the SAME run via `site-updater.sh`, from the same clone the rebuild's git fetch+reset refreshes, in finalize AFTER the core swap verifies live — so the site matches the new source. The site is a separate, versionless artifact: it never touches the reported version, and a site failure is logged, never a rollback of the (already-verified) core update. Both localhost-only. (ADFA-5339) diff --git a/static/dashboard/package.json b/static/dashboard/package.json index 82d9088cc..8b25f9249 100644 --- a/static/dashboard/package.json +++ b/static/dashboard/package.json @@ -1,6 +1,6 @@ { "name": "dashboard-console", - "version": "1.3.1", + "version": "1.3.2", "description": "", "main": "index.js", "scripts": { diff --git a/static/dashboard/routes.ts b/static/dashboard/routes.ts index fdb1d86f0..5557eac9c 100644 --- a/static/dashboard/routes.ts +++ b/static/dashboard/routes.ts @@ -38,7 +38,7 @@ const ZIMS_DIR = '/library/zims/content/'; const KIWIX_INDEXER = '/usr/bin/iiab-make-kiwix-lib'; const ZIM_NAME_RE = /^[A-Za-z0-9._-]{1,150}\.zim$/; -const VALID_TYPES: JobType[] = ['kiwix', 'maps', 'books', 'kolibri']; +const VALID_TYPES: JobType[] = ['kiwix', 'maps', 'books', 'kolibri', 'basemaps']; function isType(t: string): t is JobType { return (VALID_TYPES as string[]).includes(t); } diff --git a/static/dashboard/server.ts b/static/dashboard/server.ts index 87fe409d0..b62995447 100644 --- a/static/dashboard/server.ts +++ b/static/dashboard/server.ts @@ -7,6 +7,7 @@ import helmet from 'helmet'; import { jobs } from './sockets/jobs'; import './sockets/kiwix.exec'; import './sockets/maps.exec'; +import './sockets/maps-base.exec'; import './sockets/books.exec'; import './sockets/kolibri.exec'; import { apiRouter } from './routes'; diff --git a/static/dashboard/sockets/jobs.ts b/static/dashboard/sockets/jobs.ts index 7cd743a0f..06ad91ed3 100644 --- a/static/dashboard/sockets/jobs.ts +++ b/static/dashboard/sockets/jobs.ts @@ -13,7 +13,7 @@ import fs from 'fs'; import path from 'path'; import { RollingLog, LogSlice } from './rolling-log'; -export type JobType = 'kiwix' | 'maps' | 'books' | 'kolibri'; +export type JobType = 'kiwix' | 'maps' | 'books' | 'kolibri' | 'basemaps'; export type JobPhase = | 'queued' | 'downloading' | 'indexing' | 'processing' // ADFA-4894 (control surface): 'paused' is a stopped-but-resumable state — like 'canceled' it diff --git a/static/dashboard/sockets/maps-base.exec.ts b/static/dashboard/sockets/maps-base.exec.ts new file mode 100644 index 000000000..f9d1098c9 --- /dev/null +++ b/static/dashboard/sockets/maps-base.exec.ts @@ -0,0 +1,169 @@ +// sockets/maps-base.exec.ts -- K2GO-394 +// +// Base-map runner for the durable job engine: download the selected GLOBAL map pmtiles +// (vector / satellite / terrain) with aria2, straight into the maps serve dir. This is a +// drop-in replacement for the aria2c the maps runrole used to run in-proot, which wedged +// at CN:0 on a full network drop and never recovered (K2GO-394): once the files are here, +// the runrole's `creates: dest_path` and the .meta4 size-probe both SKIP the download, so +// the role only post-processes (symlinks, maps-config.js). +// +// Kept SEPARATE from the FQR `maps` runner (maps.exec.ts / tile-extract.py) -- same engine, +// different concern. Reuses the PROVEN kiwix mechanism (see kiwix.exec.ts): the canonical +// aria2 flag set plus the withRetry OUTER loop that re-runs aria2 (which resumes via +// --continue) on a full-interface-loss exit (19 = DNS cannot resolve), surfaced on the poll +// as "Reconnecting n/5". No --lowest-speed-limit, on purpose (it turns a slow mobile link +// into a hard abort). +import { jobs, RunnerContext, CanceledError, PausedError, classifyStop } from './jobs'; +import { withRetry } from './net-retry'; +import fs from 'fs'; +import path from 'path'; + +// = maps_serve_path = dest_base_path in roles/maps; the runrole moves each pmtiles here. +const MAPS_DIR = '/library/www/maps'; +// The app passes one catalog_file_url per selected layer. Only plain https URLs (no shell is +// used -- spawn takes an argv -- so this guards SSRF/log-shape, not shell injection). +const SAFE_URL = /^https:\/\/[^\s'"`$<>|;()]+$/; + +// ADFA-4832 CANONICAL aria2 flag set -- an EXACT copy of kiwix.exec.ts (only -d differs). The kiwix +// runner recovers from a Wi-Fi drop (aria2 --max-tries=5 absorbs in-flight blips; a FULL interface +// loss makes aria2 exit and the withRetry OUTER loop re-runs it, resuming via --continue), so this +// mirrors it exactly rather than inventing a retry model. Notably NO --lowest-speed-limit (it turns +// a slow mobile link into a hard abort). If you change a flag here, change kiwix.exec.ts and +// controller/app/.../Aria2Manager.java too. +const ARIA2_ARGS: string[] = [ + '-d', MAPS_DIR, + '--continue=true', + '--allow-overwrite=true', + '--auto-file-renaming=false', + '--max-connection-per-server=4', + '--split=16', + '--follow-metalink=mem', + '--check-integrity=true', + '--console-log-level=warn', + '--summary-interval=1', + '--download-result=hide', + '--async-dns=false', + '--max-tries=5', + '--retry-wait=5', + '--timeout=60', + '--connect-timeout=15', + '-Z', + '-j', '5', +]; + +// The transient aria2 exit codes the OUTER loop re-runs (aria2 resumes via --continue): 1 unknown, +// 2 timeout, 6 network, 7 unfinished, 19 DNS, 29 HTTP 503. Terminal (not retried): 3/4 not found, +// 9 no space, 13 file exists. +const ARIA2_TRANSIENT_EXITS = new Set([1, 2, 6, 7, 19, 29]); +// 5 VISIBLE reconnect waits (3, 6, 9, 18, 36 s ~= 72 s total), surfaced as "Reconnecting n/5". +const RETRY_DELAYS = [3_000, 6_000, 9_000, 18_000, 36_000]; + +/** Convert an aria2 rate token ("34MiB", "512KiB", "1.2MB") to bytes/sec. */ +function parseRate(token: string): number { + const m = /^([\d.]+)\s*([KMGT]?i?B)?/i.exec(token); + if (!m) return 0; + const val = parseFloat(m[1]); + const unit = (m[2] || 'B').toUpperCase(); + const mult: Record = { + B: 1, KIB: 1024, MIB: 1024 ** 2, GIB: 1024 ** 3, TIB: 1024 ** 4, + KB: 1000, MB: 1e6, GB: 1e9, TB: 1e12, + }; + return Math.round(val * (mult[unit] ?? 1)); +} + +/** Remove the download artifacts aria2 leaves next to a completed pmtiles. */ +function cleanupMetadata(): void { + try { + for (const f of fs.readdirSync(MAPS_DIR)) { + if (/\.(meta4|aria2)$/.test(f)) fs.unlinkSync(path.join(MAPS_DIR, f)); + } + } catch { /* non-fatal */ } +} + +// Clean-on-cancel (mirrors kiwix.exec.ts): a canceled download must not leave a partial behind. aria2 +// preallocates the FULL file size, so a canceled partial looks complete to the maps role's +// `creates: dest_path` and to the is_proot presence assert -- the role would then serve a truncated +// pmtiles. Prune each requested file's partial and its .aria2/.meta4 siblings. A file with no .aria2 +// is already complete, so leave it. An ERROR keeps the partial on purpose (resume via --continue). +function cleanupPartials(files: string[]): void { + for (const f of files) { + try { + const control = path.join(MAPS_DIR, `${f}.aria2`); + if (!fs.existsSync(control)) continue; // no control file -> complete, do not delete it + fs.unlinkSync(control); + for (const sibling of [f, `${f}.meta4`]) { + const p = path.join(MAPS_DIR, sibling); + if (fs.existsSync(p)) fs.unlinkSync(p); + } + } catch { /* best-effort */ } + } +} + +const mapsBaseRunner: (ctx: RunnerContext) => Promise = async (ctx) => { + const urls = ctx.ids.map(String).filter((u) => u.length > 0); + if (urls.length === 0) throw new Error('no base-map URLs requested'); + for (const u of urls) if (!SAFE_URL.test(u)) throw new Error(`unsafe base-map URL: ${u}`); + // aria2 saves each URL under its basename in MAPS_DIR (no --out); the same basename the maps role + // expects at dest_path. Used to prune the right partial on cancel. + const files = urls.map((u) => path.basename(u)); + fs.mkdirSync(MAPS_DIR, { recursive: true }); + + ctx.throwIfCanceled(); + ctx.update({ phase: 'downloading', speed: 0, detail: files.join(', ') }); + + // Pass the DIRECT pmtiles URL (not .meta4), the same way the kiwix runner passes the .zim + // URL directly. aria2 downloads it into MAPS_DIR under its own basename -- exactly the runrole's + // dest_path, so the role's `creates:` then skips the download. (An explicit .meta4 metalink is + // what the in-proot runrole used, and metalink downloads are what wedged aria2 on a network drop + // -- K2GO-394; --follow-metalink=mem still honors a metalink the mirror serves on its own.) + try { + await withRetry(() => new Promise((resolve, reject) => { + const dl = ctx.spawn('/usr/bin/aria2c', [...ARIA2_ARGS, ...urls]); + const onData = (buf: Buffer) => { + const text = buf.toString(); + const re = /\((\d+)%\).*?DL:([^\s]+)/g; + let m: RegExpExecArray | null; + let lastPct = -1; + let lastRate = ''; + while ((m = re.exec(text)) !== null) { lastPct = parseInt(m[1], 10); lastRate = m[2]; } + if (lastPct >= 0) { ctx.reportRetry(0, 0); ctx.update({ phase: 'downloading', percent: lastPct, speed: parseRate(lastRate) }); } + }; + dl.stdout?.on('data', onData); + dl.stderr?.on('data', onData); + dl.on('error', reject); + dl.on('exit', (code, signal) => { + if (signal === 'SIGKILL' || ctx.isCanceled()) return reject(new CanceledError()); + if (code === 0) return resolve(); + // Carry the exit code so the outer loop tells a transient network failure (retry, + // resuming via --continue) from a terminal one (not found / no space). + const err = new Error(`aria2 exited with code ${code}`); + (err as { code?: number }).code = code ?? -1; + reject(err); + }); + }), { + delaysMs: RETRY_DELAYS, + tries: RETRY_DELAYS.length + 1, + signal: ctx.signal, + isCanceled: ctx.isCanceled, + isTransient: (e) => ARIA2_TRANSIENT_EXITS.has((e as { code?: number }).code ?? -1), + onRetry: ({ attempt, err }) => { + ctx.reportRetry(attempt, RETRY_DELAYS.length); + ctx.log(`[basemaps] reconnect ${attempt}/${RETRY_DELAYS.length} after: ${err instanceof Error ? err.message : String(err)}`); + }, + }); + } catch (e) { + // pause KEEPS the partial (+ .aria2) so resume continues via --continue; a real error also + // keeps it so a retry/reconcile resumes rather than restarting from zero. + const stop = classifyStop(ctx); + if (stop === 'paused') throw new PausedError(); + if (stop === 'canceled') { cleanupPartials(files); throw new CanceledError(); } + throw e; + } + + cleanupMetadata(); + ctx.update({ phase: 'done', percent: 100, speed: 0 }); +}; + +jobs.registerRunner('basemaps', mapsBaseRunner); + +export { mapsBaseRunner }; From 877f0254ad873790e4ceccfe538a0885654532db Mon Sep 17 00:00:00 2001 From: "Luis Guzman (AppDevForAll)" Date: Mon, 7 Sep 2026 09:51:12 -0600 Subject: [PATCH 2/2] K2GO-394 feat(maps): delegate the in-proot base-map download to dash-node Upstream patch 0003 for roles/maps/tasks/download_large_file.yml. On proot the blocking aria2c cannot recover a full mobile-radio drop; it wedges with no exit. Gate the stock download "when: not is_proot", so it is unchanged off proot. Add a "when: is_proot" step that asserts dash-node already placed the pmtiles at dest_path. dash-node downloads it first, with resume; the role then only post-processes. --- ...0003-maps-download-is-proot-delegate.patch | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 tools/upstream-patches/patches/0003-maps-download-is-proot-delegate.patch diff --git a/tools/upstream-patches/patches/0003-maps-download-is-proot-delegate.patch b/tools/upstream-patches/patches/0003-maps-download-is-proot-delegate.patch new file mode 100644 index 000000000..9790cc818 --- /dev/null +++ b/tools/upstream-patches/patches/0003-maps-download-is-proot-delegate.patch @@ -0,0 +1,36 @@ +Upstream-PR: not yet submitted +Upstream-Status: open +Applies-to: roles/maps/tasks/download_large_file.yml +Summary: On proot, delegate the base-map pmtiles download to dash-node (the dashboard's durable job engine) instead of running a blocking aria2c in-proot. The in-proot aria2 could not recover from a full mobile-radio drop -- it wedged with no exit (K2GO-394). dash-node downloads the pmtiles with its own resumable, reconnecting aria2 (--continue plus an outer reconnect loop -- the proven kiwix mechanism) into the maps serve dir BEFORE this role runs. So the original download task is gated `when: not is_proot` (unchanged off proot), and a new `when: is_proot` step asserts the file is already at dest_path (a missing file means the host skipped the dash-node step); the role then only post-processes. Carried for our build; WIP for upstream (K2GO-394). + +diff --git a/roles/maps/tasks/download_large_file.yml b/roles/maps/tasks/download_large_file.yml +index f8aa705..78a55cc 100644 +--- a/roles/maps/tasks/download_large_file.yml ++++ b/roles/maps/tasks/download_large_file.yml +@@ -140,6 +140,26 @@ + args: + executable: /bin/bash + creates: "{{ dest_path }}" ++ when: not is_proot ++ ++ # proot / Android (K2GO-394): dash-node -- the dashboard's durable job engine -- downloads the ++ # base-map pmtiles with its own resumable, reconnecting aria2 (--continue plus an outer reconnect ++ # loop) BEFORE this role runs. On proot this role only post-processes, so it must NOT download ++ # in-proot, where a blocking aria2 cannot recover from a mobile-radio drop (it wedges with no ++ # exit). The file is expected at dest_path already; fail early and clearly if the host skipped the ++ # dash-node step, rather than leaving a broken symlink for maps-update.py to trip on. ++ - name: "Base map {{ file_name }} must be present on proot (downloaded by dash-node)" ++ stat: ++ path: "{{ dest_path }}" ++ register: proot_basemap ++ when: is_proot ++ ++ - name: "Fail if dash-node did not place {{ file_name }} on proot" ++ assert: ++ that: proot_basemap.stat.exists ++ fail_msg: "On proot, {{ dest_path }} must be downloaded by dash-node before the maps role runs (K2GO-394)." ++ quiet: yes ++ when: is_proot + + rescue: + # We output summaries to a log file for the user's benefit,