ADFA-5552: Optimize every media file in documentation.db (in place)

[alexmmiller]

Documentation Database — Media Optimization Report

Ticket: ADFA-5552 · Branch: fix/ADFA-5552

Summary

Every image in documentation.db was optimized in place, reducing the database file by 26% with no change to how any page references its media and no loss of any image.

Metric	Before	After	Reduction
Database file	249.1 MB	183.6 MB	−65.5 MB (−26.3%)
Optimizable image bytes	130.2 MB	65.7 MB	−64.5 MB (−49.5%)
Images optimized	—	1,081	—

Integrity verified afterward: PRAGMA integrity_check and foreign_key_check both pass, and all 1,192 image records reassemble and decode correctly.

Scope

Optimization was applied to every media file in the database, across all documentation sets (Android a/, IntelliJ i/, Kotlin k/, p/, Java j/) — not just the Kotlin website. It reuses the optimization pipeline from PR #24 (the Kotlin website's optimize_media.py) and applies it to the images already stored in the database.

Videos (mp4, quicktime) and favicons (x-icon) were intentionally left untouched — the pipeline does not re-encode video.

Savings by media type

Type	Images optimized	Bytes saved	Method
PNG	720	57.84 MB	pngquant + downscale
GIF	61	5.70 MB	per-frame downscale (animation preserved)
WebP	161	0.58 MB	re-encode + downscale
SVG	126	0.30 MB	Scour minification
JPEG	13	0.09 MB	re-encode + downscale
Total	1,081	64.51 MB

PNG downscaling is essentially the entire win. A further 109 images were already minimal and left as-is, and 14 were left untouched because re-encoding them would have increased their size (see "never enlarge" below).

Methods — exact settings

All raster images are downscaled with a Lanczos filter to a maximum width of 500px, preserving aspect ratio, and never upscaled (an image already ≤500px wide keeps its dimensions). Then, per format:

• PNG — quantized with pngquant (--quality 65-95, --speed 4, metadata stripped). pngquant is run at full resolution first (so its palette selection sees the original color detail), then the image is downscaled, then quantized again at the delivered size.
• JPEG — re-encoded at quality 82, progressive, with optimized Huffman tables.
• GIF — every frame downscaled individually; frame count, per-frame durations, and loop count are preserved so animations keep playing. Static GIFs are simply resized.
• WebP — re-encoded at quality 80, method 6 (maximum compression effort). Animated WebP is left untouched.
• SVG — minified with Scour: metadata/comments/editor cruft stripped, IDs shortened, styles converted to attributes, groups collapsed, and numbers rounded to 4 decimal places. SVGs are never rasterized to PNG (that would rename the file — see below).

Key design decision: optimize in place, no format changes

The Kotlin website pipeline (PR #24) converts images to WebP, which renames files (e.g. foo.png → foo.webp), and then rewrites the image references embedded in Kotlin's stored pages. That reference-rewriting step only understands Kotlin's page format.

The other documentation sets don't share that format. For example, an Android page references its media by absolute URL (https://developer.android.com/images/…), which has no literal link to the path the file is stored under (a/devsite/media/…) that a text substitution could follow. Renaming media outside Kotlin therefore can't be done safely by this tooling.

The solution is to optimize each image in place — same path, same file extension, same content type — so every reference, however it's written, keeps resolving to the same, now-smaller file. This is why the process never converts to WebP and never rasterizes an SVG. It trades a little additional savings (WebP would shrink things further) for correctness and uniform coverage across all doc sets.

Correctness and safety properties

• Never enlarges an image. A stored image is only replaced when the optimized result is actually smaller. Images that would grow on re-encode (several already-small animated GIFs) are left exactly as they were.
• Backup first. The database is copied to a timestamped file (documentation.db.backup-YYYYMMDD-HHMMSS) before any change.
• Single transaction. All updates happen in one transaction and are rolled back on any error; the file is then VACUUMed to reclaim freed space. A --dry-run mode does all the work and reports savings without writing anything.
• Handles chunked media. Large files are stored split across multiple 1 MB database rows. These are reassembled into the whole image before optimizing and re-split on write, agreeing exactly with how the server reads them back. (Verified: every reassembled image — including 39 continuation rows folded into their bases — decodes correctly.)
• Handles both database generations. Older databases store SVG/WebP as plain Brotli; newer ones compress them against a shared 256 KB Brotli dictionary (from the CompressionDictionary table). The tool detects which and, for dictionary databases, round-trips those rows through the brotli CLI using the dictionary read from the database itself. The decompress→optimize→recompress→decompress round-trip was confirmed lossless before any write.

Reproducing / running it

uv run --with-requirements requirements.txt scripts/optimize_db_media.py <path-to>.db --max-width 500

Add --dry-run to preview savings without modifying the database. Other tunables: --jpeg-quality, --webp-quality, --pngquant-speed, --svg-precision, --verbose. Requires the pngquant and brotli command-line tools on PATH.

Code: a single self-contained file — scripts/optimize_db_media.py. The database chunking protocol is inlined into it (rather than imported), so this PR adds exactly one file to main with no other in-repo dependencies.

A note on image width (worth a group decision)

A 500px max width was used uniformly, matching the Kotlin website's setting. For Android technical diagrams and screenshots this is fairly aggressive — most were originally wider. If preserving more detail in specific doc sets matters, the process can be re-run from the backup with a larger --max-width (globally or per doc set) at the cost of some of the savings above.

[alexmmiller]

Update: WebP conversion + reference updating, and a review pass

This PR now carries two scripts. optimize_db_media.py gained a --webp mode, and update_media_references.py repairs the links that conversion leaves behind.

What changed

optimize_db_media.py keeps its original in-place mode (same path, same extension — nothing renamed, so every reference keeps resolving) and adds --webp, which mirrors insert_optimized_media.py's own --webp behaviour: every static raster becomes WEBP, and an SVG still over --svg-rasterize-threshold after minifying is rasterized. A converted file is written to a new path, its content type set to the format actually produced, and the old row plus its chunk chain deleted. Animated GIFs stay GIFs (animated-WEBP encoding isn't implemented), and animated WEBP/APNG are left alone entirely.

update_media_references.py then fixes the dangling links, across every doc set.

Why it matches on filename

Conversion changes only the extension, and each doc set writes references differently — Android media/static_..._foo.gif, IntelliJ ./images/foo.gif, Kotlin a JSON-escaped /k/html/images/... — with no single form that maps onto Content.path. The filename is the one component every form shares. That's unambiguous in this database, verified before relying on it: no two images share a basename within a doc set, and no converted basename collides with an image left alone.

Worth recording: Android's /images/styles/disclosure_down.png-style URLs point at assets never stored in the database (no flattened equivalent exists — checked). Those are pre-existing missing assets, unrelated to this work, and are correctly left alone.

The rename map is a diff, not an inference

--before takes the pre-conversion copy and the map is the diff. The tempting self-contained alternative — "every stored X.webp implies a converted X.png" — was tried and rejected: it produced 4530 candidate names where only 745 files had actually converted, because images that were always WEBP fabricate renames. Each spurious entry would have rewritten links to unrelated external assets into dead WEBP links. --before is now validated as the same database lineage (dictionary bytes + non-image path set) so a mistyped backup can't silently rewrite 29k rows.

Review fixes (/code-review xhigh, 13 findings — 11 fixed, 2 deferred)

Correctness:

• .jpeg → .jpg rename in the no-rename mode. A file was renamed whenever its extension differed from its type's canonical one, so photo.jpeg was deleted and re-inserted as photo.jpg — in the mode documented as never renaming. optimize_media now returns the content type it produced, so only a genuine format change can rename.
• Chunk-row UNIQUE collision aborting the whole run. content_chunking notes guide.html and guide.html-1 are legal as separate pages, and owned_fragment_paths deliberately won't claim one. Inserting over it rolled back everything. clear_fragment_slots now frees the slots first and logs what it removes.
• target_path split the extension off the whole path, so a/v1.2/logo targeted a/v1.webp.
• templateId hardcoded to 0, against populate_db.insert_chunked_content's convention that fragments reuse the base row's languageID/contentTypeID/templateId.
• Reference matching anchored to a URL-ish delimiter, so a filename in prose (save it as copy.png) is no longer rewritten. 39 converted names are short enough for this to matter.
• Backup names take the next free suffix instead of failing on a same-second rerun.

Cleanup: images were decoded twice on the default path (the open image is now handed to the encoders); animated skips are counted separately from unhandled types; dead branch removed; flags that do nothing under --webp now warn.

Deferred, still open: streaming instead of fetchall() (memory), and parallelising the optimize loop the way load_android_json_db.py already does. Both are real, neither affects correctness, and both are sizeable enough that I'd rather land them separately.

Measured end-to-end on a 249 MB database

	Before	After
Database file	249.1 MB	178.4 MB (−28.4%)
Optimizable image bytes	130.2 MB	60.6 MB (−53.5%)
image/png	733 rows, 69.31 MB	20 rows, 0.02 MB
image/jpeg	13 rows, 0.28 MB	0
image/gif	139 rows, 55.27 MB	114 rows, 49.53 MB
image/webp	161 rows, 2.18 MB	906 rows, 8.14 MB
image/svg+xml	183 rows, 3.20 MB	183 rows, 2.90 MB
video (mp4/quicktime)	16.61 MB	16.61 MB (untouched)

745 files converted; 1699 references rewritten across 578 rows.

Verification: integrity_check and foreign_key_check pass; all 1192 images decode with 0 extension/format mismatches; 0 rows still link to a deleted filename; templateId preserved exactly (3507 non-zero, unchanged); the reference pass is idempotent (a second run finds 0). The conversion output was byte-identical to the pre-fix run — the fixes close latent holes this database doesn't happen to contain — and clear_fragment_slots reported 0 collisions. Tightening the reference anchor produced the identical 1699 references, so removing the prose risk cost nothing.

Notes

• Animated GIFs are now ~82% of remaining media (49.5 MB), untouched by design. Biggest remaining opportunity.
• 5 large SVGs fall back to minified-SVG instead of rasterizing when native cairo isn't installed (graceful, logged). ~2.9 MB of SVG remains, mostly one file.
• Both scripts declare their dependencies inline (PEP 723), so uv run scripts/<name>.py … works with no setup. pngquant and brotli are system CLIs — under --webp pngquant isn't needed at all.

[davidschachterADFA]

Why is text/plain relevant?