From f791483fe1b6c528e2f71a487306f941d7abeed0 Mon Sep 17 00:00:00 2001 From: EtienneLescot Date: Wed, 21 Oct 2026 12:55:20 +0200 Subject: [PATCH 1/2] docs(effects): recipe for lifting real UI blocks out of the recording Agents read block rects from the DOM while recording, write them into the WGSL, hide the cursor and draw the pointer in the effect, moved with the block it sits on. No contract change: the contract 1 inputs already cover it. --- electron/mcp/effects-contract.ts | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/electron/mcp/effects-contract.ts b/electron/mcp/effects-contract.ts index 832e3d16ed..b3f1dcbb14 100644 --- a/electron/mcp/effects-contract.ts +++ b/electron/mcp/effects-contract.ts @@ -144,5 +144,25 @@ Parameters: depth (number, 0–1, default 0.6), tint (color, default "#000000"). return vec4(mix(c.rgb, params.tint.rgb, k), c.a); } +## Recipe: lift real UI blocks out of the recording + +To detach a card or a panel, bring it to the camera, zoom into it or explode the interface into 3D +cards. The compositor sees pixels, not the page, so the blocks come from you: + +1. **Read them while you record.** In the page you drive, take each block's + \`getBoundingClientRect()\` divided by the viewport size: fractions of the recorded frame. Scroll + before the take starts, so nothing moves while a block is lifted. +2. **Write them into the code** as constants: \`const CARD: vec4 = vec4(x, y, w, h);\`. + A block that moves during the take (a panel that opens, a card that slides) gets a few keyframes + mixed on \`ctx.time\`, never a long table: a large constant table slows every frame on some GPUs. + These fractions are points of the recording's surface (when the project crops the recording, + take them relative to the crop rectangle), so place them with \`fx_from_footage\` and they follow + the zoom, the layout and the 3D camera. +3. **Hide the cursor and draw the pointer yourself.** \`frame()\` already holds the pointer: a lifted + block would carry a frozen copy of it while the page keeps another. Turn it off with + \`setEditorSettings\` \`{ cursor: { show: false } }\`; \`ctx.cursor\` and \`ctx.clicks\` keep the + recording's pointer. Draw it last: over a lifted block, move it with that block's transform, so it + stays on the part of the interface the user points at; anywhere else, at \`ctx.cursor.xy\`. + Check the result with getFrames at the moments that matter. `; From 6fc219beae27c2fb1a20d5ee1eeb49af60c0406b Mon Sep 17 00:00:00 2001 From: EtienneLescot Date: Wed, 21 Oct 2026 15:19:00 +0200 Subject: [PATCH 2/2] docs(effects): convert rects for screen and window takes, require an editable cursor --- electron/mcp/effects-contract.ts | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/electron/mcp/effects-contract.ts b/electron/mcp/effects-contract.ts index b3f1dcbb14..8da5f1e2bb 100644 --- a/electron/mcp/effects-contract.ts +++ b/electron/mcp/effects-contract.ts @@ -150,8 +150,10 @@ To detach a card or a panel, bring it to the camera, zoom into it or explode the cards. The compositor sees pixels, not the page, so the blocks come from you: 1. **Read them while you record.** In the page you drive, take each block's - \`getBoundingClientRect()\` divided by the viewport size: fractions of the recorded frame. Scroll - before the take starts, so nothing moves while a block is lifted. + \`getBoundingClientRect()\`. Divided by the viewport size, it is a fraction of the recorded frame + only when the recording is the viewport itself. For a screen or window recording, convert it to + frame pixels first (times \`devicePixelRatio\`, plus the viewport's origin in the frame), then + divide by the frame size. Scroll before the take starts, so nothing moves while a block is lifted. 2. **Write them into the code** as constants: \`const CARD: vec4 = vec4(x, y, w, h);\`. A block that moves during the take (a panel that opens, a card that slides) gets a few keyframes mixed on \`ctx.time\`, never a long table: a large constant table slows every frame on some GPUs. @@ -160,7 +162,9 @@ cards. The compositor sees pixels, not the page, so the blocks come from you: the zoom, the layout and the 3D camera. 3. **Hide the cursor and draw the pointer yourself.** \`frame()\` already holds the pointer: a lifted block would carry a frozen copy of it while the page keeps another. Turn it off with - \`setEditorSettings\` \`{ cursor: { show: false } }\`; \`ctx.cursor\` and \`ctx.clicks\` keep the + \`setEditorSettings\` \`{ cursor: { show: false } }\`. This needs an \`editable-overlay\` recording + (the default): with \`cursor: "system"\` the pointer is drawn into the video and cannot be hidden, + so record again. \`ctx.cursor\` and \`ctx.clicks\` keep the recording's pointer. Draw it last: over a lifted block, move it with that block's transform, so it stays on the part of the interface the user points at; anywhere else, at \`ctx.cursor.xy\`.