Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
87e2a73
README: projects using FlyHigh, full list in docs/projects.md
MONZikWasTaken Nov 9, 2026
42134c2
Merge release into dev
MONZikWasTaken Nov 9, 2026
99d2b6c
Static protection keeps line art on pans; smoother pacing at x5, even…
MONZikWasTaken Nov 9, 2026
6e3d57f
Exact modes: a tick shows a source frame only where it lands on it
MONZikWasTaken Nov 9, 2026
a590894
ARCHITECTURE: wrap the planner paragraph
MONZikWasTaken Nov 9, 2026
cd33fd3
Tick lock: frames held to their whole number of ticks; Auto sizes by …
MONZikWasTaken Nov 9, 2026
dc67e11
v0.3.5
MONZikWasTaken Nov 9, 2026
2edbb40
Merge v0.3.5 into dev
MONZikWasTaken Nov 9, 2026
0d7ef18
Auto on the exact grid where no factor is even; the display's rate wi…
MONZikWasTaken Nov 9, 2026
d7c6667
Video mode: in-betweens on their own pair's span; CostSampler retries…
MONZikWasTaken Nov 9, 2026
9cec9df
options.md: Auto, the headroom and fill_display as they are now
MONZikWasTaken Nov 9, 2026
f52abbb
Auto: a penalty that does not lower the stall load is lifted (the sta…
MONZikWasTaken Nov 9, 2026
b77e494
v0.3.6
MONZikWasTaken Nov 9, 2026
542e290
Merge v0.3.6 into dev
MONZikWasTaken Nov 9, 2026
8dad44a
A change after a held frame is shown as drawn (animation on twos and …
MONZikWasTaken Nov 9, 2026
f4ab629
v0.3.7
MONZikWasTaken Nov 10, 2026
14beaf9
Merge v0.3.7 into dev
MONZikWasTaken Nov 10, 2026
6ef9ece
Browser module build remaps source paths
MONZikWasTaken Nov 11, 2026
d8028c1
Vulkan interop: frames from a host's own Vulkan device
MONZikWasTaken Nov 16, 2026
00fb03c
v0.4.0
MONZikWasTaken Nov 16, 2026
f590729
Merge v0.4.0 into dev
MONZikWasTaken Nov 16, 2026
d58c62d
v0.4.1: static protection by default, nearest-source regions as an op…
MONZikWasTaken Nov 17, 2026
8d2fe0f
Merge v0.4.1 into dev
MONZikWasTaken Nov 17, 2026
4efa6e0
Cascade pair stage behind FH_CASCADE (experiment)
MONZikWasTaken Nov 17, 2026
94003ff
Still test waits for three unchanged pairs (static variant)
MONZikWasTaken Nov 17, 2026
f1df4a1
v0.5.0: precise motion for large frames, the still test waits for thr…
MONZikWasTaken Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,28 @@ follow semantic versioning as described in [CONTRIBUTING.md](CONTRIBUTING.md).

## [Unreleased]

## [0.5.0] - 2026-11-18

### Added

- Precise motion for large frames: `SessionOptions::precise_motion`, `preciseMotion` in the browser module,
`FH_PRECISE=1` in the environment for the C API. Off by default. On frames above 1280 px on the long side the pair
stage finds the motion on a 400-line picture and refines it on an 800-line one (`internal` auto picks 1408 x 800 for
1080p) instead of one pass at 480 lines, with the same weights. Captured 1080p 120 fps video (UVG, 14 clips): 33.83
against 33.42 dB, LPIPS 0.159 against 0.165; synthetic 1080p pans and moving objects: 35.41 against 33.35 dB. It
costs GPU time: 1.94 against 1.06 ms per generated frame at 1080p on an RTX 5090. No effect on smaller frames or
with `fast_render`.

### Changed

- With static protection on (the default), the still test of the render trusts a pixel only after three unchanged
source pairs, like the protection itself: one pair in which a moving detail happened to cover the same place twice
no longer holds it still. On the realtime defaults: UVG 1080p 33.42 dB and LPIPS 0.165 (0.4.1: 33.23 and 0.173),
synthetic 1080p 33.35 dB (32.58); ground-truth triplets ATD 31.78, CS2 29.39, natural 38.16 dB (31.66, 29.32,
38.18). Local contrast of the in-between on UVG is 0.75 of the ground truth's (0.77): the held pixels were sharp
and often wrong. The age pass of static protection costs 0.08 to 0.10 ms more per pair on an RTX 4060 Ti; a
session without static protection is unchanged.

## [0.4.1] - 2026-11-17

### Changed
Expand Down
12 changes: 6 additions & 6 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ members = ["crates/flyhigh-options", "crates/flyhigh-engine", "crates/flyhigh-ca
default-members = ["crates/flyhigh-options", "crates/flyhigh-engine", "crates/flyhigh-capi", "crates/flyhigh-cli", "examples/player"]

[workspace.package]
version = "0.4.1"
version = "0.5.0"
edition = "2021"
license = "Apache-2.0"
repository = "https://github.com/getflyhigh/flyhigh-engine"
Expand Down
11 changes: 10 additions & 1 deletion crates/flyhigh-engine/src/api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,15 @@ pub struct Options {
}
impl Options {
pub fn new(width: u32, height: u32) -> Options {
let internal = if width.max(height) > 960 { Some(default_internal(width, height)) } else { None };
let internal = if width.max(height) > 960 {
Some(if crate::options::precise_default() && crate::options::precise_applies(width, height) {
crate::options::precise_internal(width, height)
} else {
default_internal(width, height)
})
} else {
None
};
Options {
width,
height,
Expand Down Expand Up @@ -148,6 +156,7 @@ impl Options {
linear_t: self.linear_t,
protection_mask: false,
near: crate::options::near_default(),
precise: crate::options::precise_default() && crate::options::precise_applies(self.width, self.height),
}
}
}
Expand Down
17 changes: 16 additions & 1 deletion crates/flyhigh-engine/src/gen_tex.rs
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,14 @@ pub struct TexOptions {
pub protection_mask: bool,
/// nearest-source regions (`NetOpts::near`, realtime mode)
pub near: bool,
/// precise motion (`SessionOptions::precise_motion`): the two-level pair stage; `iw` x `ih` is its fine level
pub precise: bool,
}
impl TexOptions {
/// defaults: k 1, internal size `default_internal`, realtime, repair `DEFAULT_REPAIR`, no cadence, model sampler, no static protection
pub fn new(w: u32, h: u32) -> TexOptions {
let (iw, ih) = crate::default_internal(w, h);
let precise = crate::options::precise_default() && crate::options::precise_applies(w, h);
let (iw, ih) = if precise { crate::options::precise_internal(w, h) } else { crate::default_internal(w, h) };
TexOptions {
w,
h,
Expand All @@ -86,6 +89,7 @@ impl TexOptions {
linear_t: false,
protection_mask: false,
near: crate::options::near_default(),
precise,
}
}
}
Expand Down Expand Up @@ -193,6 +197,15 @@ impl GenTex {
opts: TexOptions,
tune: Option<&crate::TuneTable>,
) -> Result<GenTex> {
// fast render has no cascade: precise motion set by the environment default falls back to the plain rung
let mut opts = opts;
opts.precise &= crate::options::precise_applies(opts.w, opts.h);
if opts.precise && opts.fast_render {
opts.precise = false;
if (opts.iw, opts.ih) == crate::options::precise_internal(opts.w, opts.h) {
(opts.iw, opts.ih) = crate::default_internal(opts.w, opts.h);
}
}
anyhow::ensure!((1..=3).contains(&opts.k), "k must be 1..3");
anyhow::ensure!(opts.repair <= 2, "repair must be 0..2");
anyhow::ensure!(opts.w >= 64 && opts.h >= 64, "frame size too small (min 64 x 64)");
Expand All @@ -215,6 +228,8 @@ impl GenTex {
decisive: crate::options::decisive_default() && opts.mode == TexMode::Realtime,
still: crate::options::still_default() && opts.mode == TexMode::Realtime,
near: opts.near && opts.mode == TexMode::Realtime,
cascade: crate::options::cascade_for(opts.precise).0,
cascade_iters: crate::options::cascade_for(opts.precise).1,
};
let mut sess = Vec::new();
for _ in 0..n_sess {
Expand Down
1 change: 1 addition & 0 deletions crates/flyhigh-engine/src/gpu/pipeline.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ fn on_static(gx: f32, gy: f32) -> bool {\n\
let hook = " // static hook\n";
let rule = " let sa = on_static(gxa, gya); let sb = on_static(gxb, gyb);\n\
var raw = 0.0;\n\
stillw = stillw * select(0.0, 1.0, abs(f32(age[y * pp.W + x])) >= STILL_RUN);\n\
if (sa && sb) { rk = 0.0; raw = 1.0; } else if (sa) { rk = 0.0; bwa = 0.0; bwb = 1.0; } else if (sb) { rk = 0.0; bwa = 1.0; bwb = 0.0; }\n";
let out_old = " outp[(c * pp.H + y) * pp.W + x] = f16(o[c]);";
let out_new = " let i = (c * pp.H + y) * pp.W + x;\n\
Expand Down
107 changes: 106 additions & 1 deletion crates/flyhigh-engine/src/model/graph.rs
Original file line number Diff line number Diff line change
Expand Up @@ -66,13 +66,118 @@ pub(crate) fn count_uses_unrolled(ops: &[Value], k: u32, uses: &mut HashMap<Stri
}
if op["op"].as_str() == Some("repeat") {
if let Some(b) = op["body"].as_array() {
for _ in 0..k {
for _ in 0..repeat_times(op, k) {
count_uses_unrolled(b, k, uses);
}
}
}
}
}
/// How often a repeat op runs its body: its own number, or the session's K where the graph says "K".
pub fn repeat_times(op: &Value, k: u32) -> u32 {
op["times"].as_u64().map_or(k, |n| n as u32)
}

fn renamed(op: &Value, suffix: &str) -> Value {
let mut op = op.clone();
if suffix.is_empty() {
return op;
}
for key in ["in", "out", "carry"] {
if let Some(list) = op.get_mut(key).and_then(Value::as_array_mut) {
for v in list {
if let Some(n) = v.as_str() {
*v = Value::String(format!("{n}{suffix}"));
}
}
}
}
if let Some(body) = op.get_mut("body").and_then(Value::as_array_mut) {
for b in body {
*b = renamed(b, suffix);
}
}
op
}

/// The cascade form of a graph: the pair stage runs whole at the coarsest of `levels` pyramid levels (each half the
/// size of the one before), then at every finer level the encoders run again and the fine block (the graph's repeat
/// body) continues from the flow, visibility and hidden state of the level below, upsampled: `fine_iters[i]` times
/// at the i-th level above the coarsest (the last entry repeats).
/// The coarse level keeps the motion range of a low internal size; the fine levels give the flow the precision of the
/// frame's own resolution, which a low internal size alone lacks (0.4 px of error at 1080p from a 480-line grid).
/// The mid stage is unchanged and reads the finest level under the graph's own names. `None` when the graph has no
/// repeat with carried tensors before its mid stage.
pub fn cascade(ops: &[Value], levels: u32, fine_iters: &[u32]) -> Option<Vec<Value>> {
use std::collections::HashSet;
let ms = mid_start(ops);
let rep = ops[..ms].iter().position(|o| o["op"] == "repeat")?;
let carry = names(&ops[rep], "carry");
if levels < 2 || carry.is_empty() || fine_iters.is_empty() {
return None;
}
let produced_before: HashSet<String> = ops[..rep].iter().flat_map(|o| names(o, "out")).collect();
// what a finer level needs from the ops before the repeat: the closure of the given reads, stopping at the
// carried tensors (they come from the level below) and at the graph inputs
let closure = |seed: &[String]| -> Vec<bool> {
let mut need: HashSet<String> =
seed.iter().filter(|n| produced_before.contains(*n) && !carry.contains(*n)).cloned().collect();
let mut keep = vec![false; rep];
for i in (0..rep).rev() {
if names(&ops[i], "out").iter().any(|o| need.contains(o)) {
keep[i] = true;
need.extend(names(&ops[i], "in").into_iter().filter(|n| !carry.contains(n)));
}
}
keep
};
let mut reads = HashMap::new();
count_uses(ops[rep]["body"].as_array()?, &mut reads);
let body_reads: Vec<String> = reads.into_keys().collect();
// the finest level also feeds every op after the repeat whose result is read at all
let mut used = HashMap::new();
count_uses(ops, &mut used);
let mut tail_reads = body_reads.clone();
for o in &ops[rep + 1..] {
if o["op"] == "render_pix" || names(o, "out").iter().any(|n| used.contains_key(n)) {
tail_reads.extend(names(o, "in"));
}
}
let sfx = |l: u32| if l == 0 { String::new() } else { format!("#{l}") };
let up = |src: String, dst: String| serde_json::json!({"op": "interpolate", "in": [src], "out": [dst], "scale": 2, "mode": "bilinear", "align_corners": false});
let mut out = Vec::new();
for l in 1..levels {
for img in ["A", "B"] {
out.push(serde_json::json!({"op": "interpolate", "in": [format!("{img}{}", sfx(l - 1))], "out": [format!("{img}{}", sfx(l))],
"scale": 0.5, "mode": "bilinear", "align_corners": false, "antialias": true}));
}
}
let top = levels - 1;
out.extend(ops[..=rep].iter().map(|o| renamed(o, &sfx(top))));
for l in (0..top).rev() {
let keep = closure(if l == 0 { &tail_reads } else { &body_reads });
out.extend(ops[..rep].iter().zip(&keep).filter(|(_, k)| **k).map(|(o, _)| renamed(o, &sfx(l))));
for n in &carry {
let (below, here) = (format!("{n}{}", sfx(l + 1)), format!("{n}{}", sfx(l)));
if n == "flow" {
// flow is in pixels of its own level: twice the grid, twice the value
out.push(up(below, format!("{here}:up")));
out.push(
serde_json::json!({"op": "scale", "in": [format!("{here}:up")], "out": [here], "factor": 2.0}),
);
} else {
out.push(up(below, here));
}
}
let mut r = renamed(&ops[rep], &sfx(l));
let step = (top - 1 - l) as usize;
r["times"] = Value::from(fine_iters[step.min(fine_iters.len() - 1)].max(1));
out.push(r);
}
out.extend(ops[rep + 1..].iter().cloned());
Some(out)
}

// the first op of the t-dependent (per generated frame) stage: everything from here on is rebuilt per mid in a split plan
pub fn mid_start(ops: &[Value]) -> usize {
ops.iter()
Expand Down
13 changes: 10 additions & 3 deletions crates/flyhigh-engine/src/net/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ pub fn build(ops: &[Value], env: &mut HashMap<String, Tensor>, plan: &mut Vec<St
let (res, steps): OpOutput = match name {
"repeat" => {
let body: Vec<Value> = op["body"].as_array().unwrap().clone();
for _ in 0..cx.k {
for _ in 0..crate::model::graph::repeat_times(op, cx.k) {
build(&body, env, plan, cx);
}
continue;
Expand Down Expand Up @@ -507,6 +507,13 @@ pub fn build_network_ex(
let (hp, wp) = (a.h, a.w);
let (ihp, iwp) = (ih.div_ceil(32) * 32, iw.div_ceil(32) * 32);
let internal = (iwp, ihp) != (wp, hp);
// cascade: as many of the requested levels as the network size divides into (the coarsest level needs /16)
let mut levels = if net.fast_render { 1 } else { net.cascade.max(1) };
while levels > 1 && (ihp % (16 << (levels - 1)) != 0 || iwp % (16 << (levels - 1)) != 0) {
levels -= 1;
}
let cascaded = (levels > 1).then(|| crate::model::graph::cascade(ops, levels, &net.cascade_iters)).flatten();
let ops: &[Value] = cascaded.as_deref().unwrap_or(ops);
let arena_was = if std::env::var_os("FH_NO_ARENA").is_some() {
gpu.arena.borrow_mut().take()
} else {
Expand All @@ -530,9 +537,9 @@ pub fn build_network_ex(
let static_p = StaticParams::new(net, wp, hp, wp, hp);
let static_age = net.protects().then(|| gpu.buf(&vec![0.0; (wp * hp) as usize]));
// the age pass's own counters (keep_static_age.wgsl): unchanged pairs, and those with the surroundings moving
let static_state = static_age.as_ref().map(|_| gpu.buf(&vec![0.0; (2 * wp * hp) as usize]));
let static_state = static_age.as_ref().map(|_| gpu.buf(&vec![0.0; (3 * wp * hp) as usize]));
if let Some(st) = &static_state {
env.insert("static:state".into(), Tensor { buf: st.clone(), c: 2, h: hp, w: wp, c0: 0, scale: 1.0 });
env.insert("static:state".into(), Tensor { buf: st.clone(), c: 3, h: hp, w: wp, c0: 0, scale: 1.0 });
}
// the host's mask (`SessionExt::write_mask`); the zero buffer fills the binding when the session has none
let mask = net.protection_mask.then(|| gpu.buf(&vec![0.0; (wp * hp) as usize]));
Expand Down
50 changes: 50 additions & 0 deletions crates/flyhigh-engine/src/options.rs
Original file line number Diff line number Diff line change
Expand Up @@ -230,6 +230,54 @@ pub struct NetOpts {
pub still: bool,
/// nearest-source regions after the render: where the warped frames disagree over a region, the nearer frame alone
pub near: bool,
/// pyramid levels of the pair stage (`model::graph::cascade`); 0 or 1 = the plain graph
pub cascade: u32,
/// fine iterations at each cascade level above the coarsest, coarse to fine
pub cascade_iters: [u32; 3],
}
/// The default of `SessionOptions::precise_motion`: off, FH_PRECISE=1 turns it on.
pub fn precise_default() -> bool {
std::env::var("FH_PRECISE").is_ok_and(|v| v.trim() == "1")
}
/// Whether precise motion applies to a frame: above 1280 px on the long side only, where its 800-line rung is
/// smaller than the frame. Measured at 1080p; smaller frames keep the plain pair stage.
pub fn precise_applies(w: u32, h: u32) -> bool {
w.max(h) > 1280
}
/// The internal size of precise motion when the host gives none: the 800-line rung (aspect kept, multiples of 32),
/// never above the frame. Half of it is the level the motion is found on, so the search range stays that of the
/// 400-line picture while the result is refined on the 800-line one.
pub fn precise_internal(w: u32, h: u32) -> (u32, u32) {
if !precise_applies(w, h) {
return (w, h);
}
let s = (800.0 / h as f64).min(1.0);
let rw = ((w as f64 * s / 32.0).round() as u32 * 32).max(64);
let rh = ((h as f64 * s / 32.0).round() as u32 * 32).max(64);
(rw.min(w), rh.min(h))
}
/// Pyramid levels and fine iterations of the pair stage: two levels with two iterations for precise motion, else the
/// plain graph. FH_CASCADE=<levels> and FH_CASCADE_K=<fine iterations per level, coarse to fine, comma separated>
/// override both for comparisons.
pub fn cascade_for(precise: bool) -> (u32, [u32; 3]) {
match std::env::var_os("FH_CASCADE") {
Some(_) => cascade_default(),
None => (if precise { 2 } else { 0 }, [2; 3]),
}
}
/// The comparison override of `cascade_for`.
pub fn cascade_default() -> (u32, [u32; 3]) {
let levels = std::env::var("FH_CASCADE").ok().and_then(|v| v.trim().parse::<u32>().ok()).unwrap_or(0);
let given: Vec<u32> = std::env::var("FH_CASCADE_K")
.map(|v| v.split(',').filter_map(|x| x.trim().parse().ok()).collect())
.unwrap_or_default();
let mut iters = [2; 3];
for (i, slot) in iters.iter_mut().enumerate() {
if let Some(&v) = given.get(i).or(given.last()) {
*slot = v;
}
}
(levels, iters)
}
/// The decisive blend: off by default (on screen its patches at a failed motion were no better than the blend's
/// fog); FH_BLEND=decisive turns it on for comparisons.
Expand Down Expand Up @@ -270,6 +318,8 @@ impl NetOpts {
decisive: false,
still: false,
near: false,
cascade: cascade_for(false).0,
cascade_iters: cascade_for(false).1,
}
}
/// The render's mode word (render.wgsl / render_tex.wgsl): bit 0 decisive blend, bit 1 still test, bit 2
Expand Down
Loading
Loading