Skip to content

present: optional MetalFX spatial upscaling (DM_METALFX) - #315

Closed
Xipzer wants to merge 2 commits into
seedds:mainfrom
Xipzer:metalfx
Closed

Xipzer wants to merge 2 commits into
seedds:mainfrom
Xipzer:metalfx

Conversation

@Xipzer

@Xipzer Xipzer commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

The game's image is stretched to the drawable by one bilinear sample in the present shader. On an iPad Pro 13" M4 that is 1920x1080 to 2752x1548, a 2.0x upscale per axis.

That single filter is the largest source of aliasing on a panel much larger than the game's output, ahead of the game's own render scale. MTLFXSpatialScaler reconstructs the same upscale instead.

Why spatial

It needs no motion vectors and no depth, so it costs nothing per game and works for every title through the shared present_encode_m4 path. newSpatialScalerWithDevice:compiler: (iOS 26.0+) returns an id<MTL4FXSpatialScaler> that encodes on the MTL4CommandBuffer the present already uses, so there is no Metal 3 bridging.

Only the whole-image case is taken. A cropped present keeps the bilinear path, as does any failure to build the scaler or its output texture.

Settings

  • DM_METALFX=1 turns it on
  • DM_METALFX_OUT=0.5..4.0 sets the output as a share of the present viewport. Above 1.0 reconstructs larger and the present samples it back down. Clamped to 8192 keeping aspect.

Both are in the launcher.

Measurements

Breath of the Wild 1.9.0, iPad Pro 13" M4. 60 fps held at 100 % with and without, GPU time per frame unchanged within run to run noise. Metal HUD confirms Scaling Spatial, input 1920x1080, target 2752x1548.

At DM_METALFX_OUT=2.0 the scaler runs 1920x1080 to 5504x3096 and 60 fps still held, though thermal state reached 2. The extra reconstruction is mostly averaged away by the present's single bilinear tap on the way back down, so 1.0 is the useful setting.

Two things worth knowing for this file

MetalFX.h pulls in <complex.h>, whose I macro collides with the local static const MTLVertexFormat I[] in vfmt_for_ct. Hence the #undef I after the import.

present_encode_m4's src parameter is objc_externally_retained, so ARC refuses to reassign it. The blit source and its dimensions are tracked in separate locals.

Not included

MTL4FXTemporalScaler and MTL4FXFrameInterpolator are both in the iOS 27 SDK. Both need per pixel motion vectors and depth, which means identifying the NVN render targets that carry them per game. That is a separate investigation.

The game's image is stretched to the drawable by one bilinear sample in the
present shader. On an iPad Pro 13" M4 that is 1920x1080 to 2752x1548, a 2.0x
upscale per axis, and it is the largest source of aliasing on a panel that much
larger than the game's output: larger than the game's own render scale.

MTL4FXSpatialScaler reconstructs the same upscale instead. It needs no motion
vectors and no depth, so it is per game free, and newSpatialScalerWithDevice:
compiler: gives an MTL4 scaler that encodes on the command buffer the present
already uses. Only the whole-image case is taken; a cropped present keeps the
bilinear path, as does any failure to build the scaler or its output texture.

DM_METALFX=1 turns it on, DM_METALFX_OUT=0.5..4.0 sets the output as a share of
the present viewport (above 1.0 reconstructs larger and the present samples it
back down). Both are in the launcher.

Measured on Breath of the Wild 1.9.0, iPad Pro 13" M4, 60 fps held at 100 %
either way, GPU time per frame unchanged within run to run noise. Metal HUD
confirms Scaling Spatial, input 1920x1080, target 2752x1548.

Two things worth knowing for anyone touching this file:
MetalFX.h pulls in <complex.h>, whose I macro collides with the local
static const MTLVertexFormat I[] in vfmt_for_ct, hence the #undef I.
present_encode_m4's src parameter is objc_externally_retained, so ARC refuses
to reassign it and the blit source has to be tracked in separate locals.
@v0lumehi

Copy link
Copy Markdown
Contributor

Thanks, and good to see the M4 iPad numbers: on an iPhone 17 Pro Max we measured 2–5 ms GPU for the same spatial pass, so the cost clearly varies by device. Present-level MetalFX is already in main, though: it landed with c13ddcf (from #215) as DM_PRESENT_UPSCALE=metalfx, on the settings screen as "Standard / MetalFX", with the Metal HUD's MetalFX rows hidden because they flicker. This branch predates that, like #316, so a rebase onto current main would show the overlap. If something here is missing from main's version, a small PR on top of it would be the way. On temporal upscaling: agreed that it needs per-game motion vectors and depth; I have an experiment that scales the 3D scene before the HUD for one game, and would be happy to compare notes.

Three additions on top of the spatial scaler, all of which work for every game
because none of them needs to know anything about the game.

Box downsample. The present shader took one bilinear sample, which reads 4 texels
whatever the reduction, so reconstructing above screen size and sampling back down
threw most of it away: DM_METALFX_OUT=2.0 looked identical to 1.0. The tap count
now follows the reduction ratio, capped at 4x4, so it is real supersampling.

Anisotropic filtering (DM_ANISO=2..16). The host already builds every sampler the
game uses and maxAnisotropy already flows through from nvnSamplerBuilderSetMaxAnisotropy.
The games ask for almost none, being built for a handheld, so ground and wall
textures blur at a glancing angle. This raises the floor: sampling only, no render
target, no allocation, and a sampler that asks for more keeps what it asked for.

Sharpening (DM_SHARPEN=0..1). Contrast adaptive, 5 tap cross, on the final picture.
Clamped to the neighbourhood min and max so it cannot ring or halo. A second
pipeline from the same library, selected per frame, so it costs nothing when off.

Measured on Breath of the Wild 1.9.0, iPad Pro 13" M4, all 60.00 fps, thermal nominal:

  render scale  MetalFX  scaler target  GPU
  100%          off      -              8.53 ms
  100%          100%     2752x1548      9.09 ms
  100%          200%     5504x3096     11.34 ms
  140%          off      -              8.38 ms
  140%          100%     2752x1548      8.94 ms
  140%          200%     5504x3096     11.77 ms

MetalFX at screen size costs 0.55 ms, at 200% output 2.8 to 3.3 ms. Worst case
11.77 ms of 16.67 ms.
@Xipzer

Xipzer commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, that is the answer I needed. I was working from an older main and did not see c13ddcf, so the scaler itself is a duplicate and should go.

Splitting what is left into small PRs on top of current main:

  1. nvn/sampler: optional anisotropic and trilinear filtering #335, already open: anisotropic and trilinear filtering. These were the sampler half of this branch and are independent of the present path, so they moved out first.
  2. Next: an output size for DM_PRESENT_UPSCALE with a real downsample. This is the part of the branch I think is still missing from main, and it is a correctness point rather than a feature: the present shader takes one bilinear sample, which reads 4 texels whatever the reduction. Reconstructing above screen size and sampling back down therefore threw nearly all of it away, so 200% output looked identical to 100%. With the tap count following the reduction ratio, capped at 4x4, it becomes real supersampling. On an M4 iPad, Breath of the Wild at 1920x1080 input: 8.53 ms off, 9.09 ms at screen size (2752x1548), 11.34 ms reconstructing to 5504x3096 and averaging down, all at 60.00 fps with a 16.67 ms budget.
  3. Then: contrast adaptive sharpening of the final picture, 5 tap cross, clamped to the neighbourhood min and max. A second pipeline from the same library, so it costs nothing when off.

I will close this branch once 2 and 3 are up rather than rebase it, since almost none of the original diff survives.

On the 2 to 5 ms you measured for the spatial pass on an iPhone 17 Pro Max against 0.55 ms on the M4 iPad: that gap is larger than the raw GPU difference, so part of it is probably the output size. The M4 iPad number is a 1920x1080 to 2752x1548 scale; an iPhone at 2868x1320 from 1280x720 handheld is a bigger ratio. I have the same phone and will post a matched pair.

On temporal: I dumped a full frame of Breath of the Wild with DM_NVN_PASSDUMP, 20,007 records. Every colour attachment at scene resolution is RGBA8Unorm, RG11B10Float, RGBA8Unorm_sRGB or R32Float. The only two channel target in the frame is a 256x256 RG8Unorm lookup table, so there is no velocity buffer to bind, and the game applies no sub-pixel jitter either. Depth32Float is there if reprojection is ever worth trying, but static-only vectors would look worse than spatial. I would be interested in your pre-HUD scaling experiment: that is the part I could not do without knowing which pass is the last 3D one.

@v0lumehi

Copy link
Copy Markdown
Contributor

Here's the pre-HUD scaling experiment, on the my-main integration branch of my fork, https://github.com/v0lumehi/nxnative/tree/my-main (it sits on #76 and a sampler LOD bias commit, so it doesn't apply to main on its own yet):

  • 24ef01b8: DM_PRESENT_UPSCALE=metalfx-scene. The profile names the draw that scales the scene into the output (scene_upscale, fragment shader keys). The replay runs that draw at the scene's size into a texture of its own, MetalFX's spatial scaler upscales it, and the result is copied into the target at the viewport. The HUD then draws on top at the screen's resolution. Scene-sized passes sample mipmapped textures with extra LOD bias.
  • a2eb59b8: what's still open with it in Breath of the Wild's note.
  • f343e42a: a small fix where it meets main.

Possibly relevant to #335: d67e7f27 passes the game's LOD bias through to the NVN and Vulkan samplers. Worth checking that the two don't overlap.

@Xipzer

Xipzer commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Closing in favour of two small PRs on current main, as suggested: #335 for the sampler side (anisotropic and trilinear filtering) and #336 for the present side (supersampled upscaling output and sharpening). The spatial scaler itself is already in main via c13ddcf, so nothing of this branch's original diff is left to rebase.

@Xipzer Xipzer closed this Oct 26, 2026
@Xipzer

Xipzer commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Correction to the numbers above: the present-side follow-up is #345, not #336.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants