feat: parallel vendor JS download with incremental caching

[Oaklight]

Summary

• Rewrite vendor.py to use ThreadPoolExecutor for concurrent downloads (8 workers), replacing the serial recursive approach
• BFS-style transitive dependency resolution: fetch all top-level packages in parallel, discover deps, fetch those in parallel, repeat
• PDF.js files download concurrently alongside ESM packages
• SHA-256 content hashing in manifest (v2) enables incremental updates — unchanged files are skipped on re-download
• Partial download failures don't abort; browser CDN fallback covers gaps
• Progress callback for CLI feedback (Downloading JS modules... 12/31)
• Add 26 unit tests covering helpers, dep discovery, import rewriting, parallel download, incremental caching, and partial failure

Benchmark: ~4s vs ~15s (3.6x speedup) on first-run vendor download.

Closes #76

Test plan

• pytest tests/test_vendor.py — 26 tests pass
• Fresh vendor download produces correct files with rewritten imports
• Incremental update skips unchanged files (verified via mtime)
• Partial failure continues and logs warning
• Pre-commit hooks pass (ruff, ruff-format, ty)
• Manual test: tinyleaf . in a fresh environment (delete ~/.config/tinyleaf/vendor/)

[clementine-oaklight]

LGTM ✅ — 架构改得干净，BFS + ThreadPoolExecutor 的拆法比老的递归 _download_esm 好很多。

逐项：

• BFS 层级并行：_discover_deps 和 _rewrite_imports 分离后，fetch 只做 I/O，dep discovery 和 rewrite 都在主线程跑，url_map/pkg_map 不需要锁。正确。
• 增量缓存：SHA-256 hash 比较 → 内容没变的文件跳过写入。v1 manifest 没有 sha256 字段，old_files.get(name, {}).get("sha256") 返回 None，graceful fallback 到全量写入。正确。
• Partial failure：单个包下载失败不 abort，all_failures 收集后 stderr 输出，浏览器 CDN fallback 兜底。correct for a dev tool。
• PDF.js 并行：和 ESM 包在同一个 executor 里并发，不需要等 ESM 先完成。正确。
• Tests：26 个 test 覆盖了 helpers、dep discovery、import rewriting、full download、incremental skip、partial failure、manifest 结构，mock 粒度合适。

没有 blocking issue。

两个 non-blocking：

1. Progress 跳跃 — PDF.js futures 在 BFS 循环结束后才 collect，所以 progress counter 会在 ESM 全部完成后突然跳。BFS 发现 transitive deps 时 total_est 也会涨，用户看到分母变大。不影响正确性，但 UX 上数字不完全线性。
2. vendor.py 直接 print(…, file=sys.stderr) — library 层直接输出到 stderr，比 raise/return failures 让 caller 决定怎么报要 opinionated 一些。当前只有 cli.py 一个调用者，没有实际问题，但如果以后被当 library 用会 surprise。

[elena-oaklight]

Clean refactor — the recursive-to-BFS transformation is well-structured, the incremental caching logic is correct, and the test coverage is solid. A few observations below, nothing blocking.

Architecture — Separating _discover_deps / _rewrite_imports from the download loop is a nice decomposition. The old recursive _download_esm mixed fetching, parsing, rewriting, and writing into one function; the new design makes each phase independently testable and keeps concurrent fetching on the I/O boundary where it belongs.

Thread safety — Confirmed that url_map/pkg_map mutation only happens in the main thread (inside the as_completed loop after futures resolve), not inside worker callables. _opener is set once before the executor block and only read by workers. All good.

Incremental hash correctness — The SHA-256 is computed on the rewritten ESM content (what's actually on disk), not the raw fetch content, so the cache correctly invalidates when url_map changes between runs (e.g. a new transitive dep alters import paths). PDF.js hashes are on raw binary. Both correct.

Minor nits inline.

[elena-oaklight]

Nit: PDF.js data gets downloaded, hashed, then (if unchanged) discarded — and the manifest loop below re-reads the file from disk and re-hashes it. You could stash the hash from here into a small dict (e.g. pdfjs_hashes[local_name] = file_hash) and look it up during manifest assembly to skip the redundant I/O + hash.

Not a correctness issue, just wasted work on incremental runs.

[Oaklight]

Fixed in 6f6be72 — PDF.js hash is now stashed in pdfjs_manifest during the download phase and reused directly in manifest building. No re-read from disk.

[elena-oaklight]

The print(..., file=sys.stderr) here bypasses the progress callback pattern established for the rest of the function. Consider either:

• Adding an optional on_warning callback, or
• Using warnings.warn() so callers can control filtering,
• Or at minimum documenting that the function writes to stderr on partial failure.

Not blocking — the current behavior is fine for the CLI use case.

[Oaklight]

Fixed in 6f6be72 — replaced print(stderr) with warnings.warn() so callers can control filtering.

[elena-oaklight]

Nit: the " " * 20 padding to clear the progress line works but is fragile if the progress string ever gets longer. \r + \033[K (ANSI erase-to-EOL) is more robust, or you could track the previous line length. Fine for now though.

[Oaklight]

Fixed in 6f6be72 — using \\033[K (ANSI erase-to-EOL) instead of space padding.

[elena-oaklight]

The time.sleep(0.05) makes this test sensitive to filesystem timestamp resolution. On some Linux filesystems (ext4 with 1s granularity), 50ms might not be enough. Consider 0.1 or comparing file content/hash instead of mtime. Unlikely to bite in CI but worth noting.

[Oaklight]

Fixed in 6f6be72 — bumped to time.sleep(0.1).

[milo-oaklight]

Review: parallel vendor JS download with incremental caching

Verdict: Approve ✅

Clean rewrite. The BFS-style dep resolution + ThreadPoolExecutor is a solid improvement over the recursive serial approach, and the separation of download → discover → rewrite is easier to reason about than the old recursive _download_esm.

What looks good

• Concurrency model is correct. _discover_deps mutates url_map/pkg_map, but only from the main thread inside as_completed loops — no concurrent mutation. _opener is set once before the executor starts; reads from worker threads are safe.
• BFS dep resolution guarantees complete url_map before rewriting. Import paths can only be rewritten after all transitive deps are known. The old recursive approach relied on depth-first ordering to achieve this; the new explicit two-phase (fetch-all → rewrite-all) is cleaner.
• Partial failure is handled correctly. Failed downloads are collected, logged to stderr, and don't crash the process. CDN fallback is documented.
• Manifest v2 upgrade path is correct. Old v1 manifests lack sha256, so old_files.get(name, {}).get("sha256") returns None → forced re-write on first run after upgrade.
• Test suite is thorough — 26 tests cover helpers, dep discovery, import rewriting, parallel download, incremental skip, and partial failure.
• CI green (lint + security).

Minor observations (non-blocking)

1. "Incremental caching" still re-fetches everything from CDN — only disk writes are skipped when hashes match. For the ~20 small ESM files this is fine, but the term "incremental" might set expectations of skipped network calls. A future enhancement could compare the local manifest first and skip _fetch entirely for unchanged specifiers (at the cost of not detecting upstream changes).

2. _rewrite_imports + _sha256 computed twice per ESM file — once in the "write files" phase, once in manifest building. Storing the rewritten content would avoid the double work. Negligible perf impact at current scale, but easy to clean up.

3. PDF.js binary re-read from disk for manifest — the binary data is available in memory during the as_completed loop but discarded before manifest building, then re-read via open(filepath, "rb"). Same minor redundancy.

4. Progress counter for PDF.js deferred — PDF.js futures are submitted first but results collected after all ESM BFS levels complete, so the progress callback won't reflect their completion until the ESM phase finishes. Cosmetic only.

None of these block merging. Nice work on the 3.6× speedup.