Estimate depth from the frame, then relight the scene with it: directional light, distance haze, a tilt-shift focus band, a little bloom. Old games come out looking like they were made now — and it runs every frame on a handheld.
Captured on an AYANEO Pocket FIT Elite at 120 fps. HD-2D relights the scene from an estimated depth map; ESPCN is the luminance super-resolution path, reconstructing at exactly the factor the screen draws.
The real-time pipeline lives in RetroAI-Scaler; the reasoning behind every decision, including the routes that were measured and rejected, is in implementation.md.
Training corpus is not distributed — please sort that out yourself. ESPCN needs none at all; it trains on procedurally generated vector scenes.
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -r requirements.txt
.venv/bin/python scripts/train_espcn.py --scales 6 # no data required
.venv/bin/python scripts/probe_bias.py
.venv/bin/python scripts/probe_shift.py
.venv/bin/python scripts/probe_smoothing.py
.venv/bin/python scripts/probe_latency.pyThe probes are the acceptance tests and none of them needs the handheld. Each prints named figures; these are the ones to read.
| script | figure | what it has to read |
|---|---|---|
probe_bias |
RAMP, peak-peak, BAND |
Near zero on inputs with no geometry. Anything reported there came from the weights rather than the picture. |
probe_shift |
net depth |
Exactly 0 at every shift, in the interior. The ~0.0006 left at the default margin is the convolutions' zero padding at the frame edge, not the network. |
probe_shift |
mip read vs sliding box |
The shading's own shift-variance, with the depth held perfect. sliding box is flat; an anchored mip read is a period-8 triangle. |
probe_smoothing |
estimator noise |
What temporal averaging actually has to suppress, measured on camera-still pairs only. |
probe_smoothing |
shimmer left / lag behind truth |
The two costs of each blend rate, so the rate is chosen against both rather than one. |
probe_latency |
per-frame residual, mean / worst frame |
How far the shading drifts from the picture between inference results. |
probe_shift is the one that matters most: a depth net that is not equivariant
to integer translation makes the lighting swim while the player walks, and no
amount of temporal filtering removes it — averaging a deterministic
oscillation trades its amplitude for lag at exactly one rate. Save arithmetic by
reducing --variant/blocks, never by raising --down.
models/ ships the exported ncnn pairs, ready to load.
GNU General Public License v3.0, same as the app that ships these
models — RetroAI-Scaler. The
weights in models/ are covered by it too.
The corpus and the game frames it came from are not.