Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion electron/ai-edition/deep-agent/service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ export const TOOL_DESCRIPTIONS: Record<string, string> = {
setAspectRatio:
"Set the shape of the exported video: 16:9 for X, YouTube or a slide, 9:16 for a vertical story or short, 1:1 or 4:5 for a square feed, auto to follow the recording, or one of the clips' own shapes (getCurrentDocument's settingsContext.originalAspectRatios). Annotations on the frame keep their place and shape. The same as setEditorSettings with aspectRatio alone.",
setEditorSettings:
'Change the project\'s look: pass only the settings to change, named as getCurrentDocument returns them under `settings`. Background: wallpaper (a bundled path from settingsContext.bundledWallpapers, a CSS colour or gradient, or a base64 PNG/JPEG data URL; never a file path or web URL; an embedded image reads back as a marker, leave the key out to keep it), wallpaperMotion (moves a gradient or image only), backgroundBlur 0–1. Frame: aspectRatio ("auto", a preset, or an originalAspectRatios token; a fixed format picked for the first time fills the frame with a window that follows the cursor, which formatFollowCursor turns off when settingsContext.formatFill is "available"), padding 0–100 %, borderRadius 0–64 px, shadowIntensity 0–1, frame (none, window, laptop, phone, monitor) and frameTheme (light, dark), motionBlurAmount 0–1, depthOfField (blurs 3D-tilted zooms only). Camera: webcamLayoutPreset, webcamMirrored, webcamReactiveZoom (shrinks on zoom), webcamMaskShape (square or rectangle, the camera\'s own proportions; an untouched roundness follows it), webcamRoundness 0–1, webcamSizePreset 15–60, webcamAnchor (a corner or an edge middle), webcamCropRegion (the framing: a square share of the picture, width = height ≥ 1/3) and webcamCropPan (where that window sits, 0–1 per axis, 0.5 centred), webcamBackgroundMode (none, transparent, blur, custom: needs settingsContext.cameraSegmentation ready), webcamBlurIntensity 0–1, webcamWallpaper (as wallpaper). Cursor, under `cursor`: show, autoHide, theme (settingsContext.cursorThemes), asArrow (the recorded cursor kinds drawn as the arrow instead), size 1.5–6, smoothing 0–1, motionBlur 0–1, clickBounce 0–2, clickImpact, clickSound (only heard when the take recorded clicks) and clickSoundGainDb −24–0, model3d. Also autoFocusAll (every zoom follows the cursor) and audioGainDb −12–12 dB, the master volume. A value out of range or a key not listed is refused with the reason, and nothing changes. The result lists what was applied, including what the editor changes with it, and `inactive` for any of those the current state leaves without effect.',
'Change the project\'s look: pass only the settings to change, named as getCurrentDocument returns them under `settings`. Background: wallpaper (a bundled path from settingsContext.bundledWallpapers, a CSS colour or gradient, or a base64 PNG/JPEG data URL; never a file path or web URL; an embedded image reads back as a marker, leave the key out to keep it), wallpaperMotion (moves a gradient or image only), backgroundBlur 0–1. Frame: aspectRatio ("auto", a preset, or an originalAspectRatios token; a fixed format picked for the first time fills the frame with a window that follows the cursor, which formatFollowCursor turns off when settingsContext.formatFill is "available"), padding 0–100 %, borderRadius 0–64 px, shadowIntensity 0–1, frame (none, window, laptop, phone, monitor) and frameTheme (light, dark), motionBlurAmount 0–1, depthOfField (blurs 3D-tilted zooms only). Camera: webcamLayoutPreset, webcamMirrored, webcamReactiveZoom (shrinks on zoom), webcamMaskShape (square or rectangle, the camera\'s own proportions; an untouched roundness follows it), webcamRoundness 0–1, webcamSizePreset 15–60, webcamAnchor (a corner or an edge middle), webcamCropRegion (the framing: a square share of the picture, width = height ≥ 1/3) and webcamCropPan (where that window sits, 0–1 per axis, 0.5 centred), webcamBackgroundMode (none, transparent, blur, custom: needs settingsContext.cameraSegmentation ready), webcamBlurIntensity 0–1, webcamWallpaper (as wallpaper). Cursor, under `cursor`: show, autoHide, theme (settingsContext.cursorThemes), asArrow (the recorded cursor kinds drawn as the arrow instead), size 1.5–6, smoothing 0–1, motionBlur 0–1, clickBounce 0–2, clickImpact, clickSound (only heard when the take recorded clicks) and clickSoundGainDb −24–0, model3d. Also autoFocusAll (every zoom follows the cursor), audioGainDb −12–12 dB, the master volume, and recordingAudioMuted, which silences the recording\'s own sound in the preview and the export while voiceovers and music keep playing. A value out of range or a key not listed is refused with the reason, and nothing changes. The result lists what was applied, including what the editor changes with it, and `inactive` for any of those the current state leaves without effect.',
addEffect:
"Write a visual effect as a shader and pose it over a span of the edited timeline (virtual seconds), or pose one of this project's effects again (effectId). Use it for a look the settings do not offer. `effect.code` is WGSL that defines `fn effect(uv: vec2<f32>) -> vec4<f32>`, called for every pixel; uv is 0..1 with (0,0) top-left. Available inside it: `frame(uv)` samples the composed picture; `footage(p)` samples the recording itself at its own resolution at point p of its surface (the space of `fx_to_footage`), transparent outside 0..1, with no cursor or webcam on it: read it for anything that magnifies, frame() for the rest; `params.<name>` reads each declared parameter; `ctx.time` is programme seconds; `ctx.region.x` goes 0 to 1 across the region (y its length in s, z seconds since it began); `ctx.resolution` and `ctx.aspect`; `ctx.cursor` (xy in uv, z = 1 when on screen); `ctx.clicks[0..3]`, the last four clicks, newest first (xy in uv where it happened, z seconds since, w = 1 when the slot holds one); `ctx.footage` and `ctx.camera`, the recording's and the webcam's rects (x, y, w, h in uv; camera zero-sized when absent) — all already placed through zoom and layout; `fx_to_footage(uv)` and `fx_from_footage(p)` map between output uv and the recording's own surface (0..1 across the recording) through the warp the compositor draws it with, zoom, layout and 3D camera included, and `ctx.footage_aspect` is its width over height: the effect works in whichever of the two spaces fits it; `fx_px(n)`, n pixels of a 1080p frame as a fraction of the height; `fx_hash(vec2<u32>)`, a random number in [0,1) identical on every GPU. Rules: the effect is a pure function of those inputs (no state between frames), so the preview and the export show the same pixels; reason in uv or fx_px, never raw pixels; every loop is a `for` with literal bounds (`for (var i = 0; i < 16; i++)`), at most 4096 iterations per pixel — the one hard rule. `params` declares up to 8 (recommended) parameters, each `{type: number|toggle|color|point, name, label, default}` (number also `min`, `max`); declare one only when moving it visibly changes the effect over its whole range. `target`: `image` (default: the picture under the annotations and captions), `text` (the annotations and captions alone, premultiplied alpha), `both` (everything); use text or both only when asked. One effect runs at a time: to combine two looks, write them in one shader. The result gives the cost tier, in frame() and footage() reads per pixel, with what a 1080p frame pays on a fast discrete GPU (RTX 4070 Ti) and on an integrated one (Apple M1): light ≤ 8, up to 0.06 ms and 1.3 ms; medium ≤ 32, 0.14 ms and 4.3 ms; heavy ≤ 128, 0.54 ms and 16 ms; very heavy beyond, 2 ms and 49 ms at 512 reads. It also gives a recommendation when the tier runs long: light anywhere, medium up to a minute, heavy up to 10 s, very heavy a few seconds. It is a recommendation, never a refusal. Code that does not compile is refused with the compiler's error at the line of your code: fix it and call again. Check the result with getFrames.",
setEffect:
Expand Down
2 changes: 2 additions & 0 deletions electron/ai-edition/editor-settings-tool.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ describe("setEditorSettings", () => {
borderRadius: 12,
frame: "laptop",
audioGainDb: -3,
recordingAudioMuted: true,
autoFocusAll: true,
cursor: { size: 2, theme: "glass-lens", show: true, asArrow: ["text"] },
});
Expand All @@ -80,6 +81,7 @@ describe("setEditorSettings", () => {
borderRadius: 12,
frame: "laptop",
audioGainDb: -3,
recordingAudioMuted: true,
autoFocusAll: true,
cursorTheme: "glass-lens",
});
Expand Down
2 changes: 2 additions & 0 deletions electron/ai-edition/editor-settings-tool.ts
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ export const setEditorSettingsArgs = z
webcamWallpaper: wallpaperArg("webcamWallpaper"),
webcamBlurIntensity: bounded("webcamBlurIntensity"),
audioGainDb: z.number().min(-AUDIO_GAIN_DB_LIMIT).max(AUDIO_GAIN_DB_LIMIT),
recordingAudioMuted: z.boolean(),
autoFocusAll: z.boolean(),
cursor: z
.object({
Expand Down Expand Up @@ -191,6 +192,7 @@ function settingsForModel(document: AxcutDocument) {
webcamWallpaper: elideDataUrl(s.webcamWallpaper),
webcamBlurIntensity: s.webcamBlurIntensity,
audioGainDb: s.audioGainDb,
recordingAudioMuted: s.recordingAudioMuted,
autoFocusAll: s.autoFocusAll,
cursor: {
show: s.cursorShow,
Expand Down
10 changes: 10 additions & 0 deletions src/components/ai-edition/ExportDialog.params.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,16 @@ describe("ExportDialog MP4 params", () => {
expect(at30).toMatchObject({ fps: 30, bitrate: expected(30) });
expect(at30?.bitrate).toBeLessThan(at60?.bitrate ?? 0);
});

it.each([false, true])("sets clip hasAudio when recording mute is %s", async (muted) => {
renderDialog({
...DOC,
legacyEditor: { ...DOC.legacyEditor, recordingAudioMuted: muted },
});
await exportMp4();
const clips = vi.mocked(exportMultiNative).mock.calls.at(-1)?.[0];
expect(clips?.[0]?.hasAudio).toBe(!muted);
});
});

describe("ExportDialog format settings", () => {
Expand Down
8 changes: 4 additions & 4 deletions src/components/ai-edition/ExportDialog.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ function revealExportedFile(filePath: string): void {
* each with its asset's screen file + camera file (falls back to the screen when a clip has
* no camera — the no-webcam layout is a later step) and its source trim. */
function buildNativeClipList(document: AxcutDocument): CompositorClipInput[] {
const recordingAudioMuted = getEditorSettings(document).recordingAudioMuted;
const assetById = new Map(document.assets.map((a) => [a.id, a]));
return resolveVisibleClips(document).flatMap((clip) => {
const asset = assetById.get(clip.assetId);
Expand All @@ -125,17 +126,16 @@ function buildNativeClipList(document: AxcutDocument): CompositorClipInput[] {
// the single canonical precedence used by every consumer (clip.probe → asset.duration
// → timeline-length guess). See `resolveClipSourceEndSec` for the full order.
const sourceEndSec = resolveClipSourceEndSec(clip, asset);
// ponytail: `hasAudio` stays optimistic for the same reason as in
// `buildSceneDescription` — nothing populates `asset.audio` yet, and the
// native side degrades cleanly on a stream-less file.
// `asset.audio` is not populated by the probe yet, so keep the current
// optimistic default unless the project explicitly mutes its recording audio.
return [
{
screenPath: asset.originalPath,
webcamPath: camera.path,
sourceStartSec: clip.sourceStartSec,
sourceEndSec,
webcamOffsetSec: camera.offsetSec,
hasAudio: true,
hasAudio: !recordingAudioMuted,
},
];
});
Expand Down
45 changes: 45 additions & 0 deletions src/components/ai-edition/RightPanes.audioPane.test.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
// @vitest-environment jsdom
import "@testing-library/jest-dom";
import { cleanup, fireEvent, render, screen } from "@testing-library/react";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { I18nProvider } from "@/contexts/I18nContext";
import { LOCALE_STORAGE_KEY } from "@/i18n/config";

const { setSettings } = vi.hoisted(() => ({ setSettings: vi.fn() }));

vi.mock("@/lib/ai-edition/store/useEditorSettings", () => ({
useEditorSettings: () => ({
settings: { audioGainDb: -6, recordingAudioMuted: true },
set: setSettings,
setLive: vi.fn(),
commit: vi.fn(),
hasDocument: true,
}),
}));

import { AudioPane } from "./RightPanes";

beforeEach(() => {
localStorage.clear();
localStorage.setItem(LOCALE_STORAGE_KEY, "en");
setSettings.mockClear();
});

afterEach(() => {
cleanup();
localStorage.clear();
});

describe("AudioPane reset button", () => {
it("resets output gain and unmutes recording audio", () => {
render(
<I18nProvider>
<AudioPane />
</I18nProvider>,
);

fireEvent.click(screen.getByRole("button", { name: "Reset audio" }));

expect(setSettings).toHaveBeenCalledWith({ audioGainDb: 0, recordingAudioMuted: false });
});
});
20 changes: 19 additions & 1 deletion src/components/ai-edition/RightPanes.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -3454,11 +3454,29 @@ export function AudioPane() {
onCommit={() => void commit()}
/>
</div>
<div
style={{
display: "flex",
alignItems: "center",
justifyContent: "space-between",
gap: 12,
marginTop: 12,
}}
>
<span>{ts("audio.muteRecording")}</span>
<Toggle
checked={settings.recordingAudioMuted}
disabled={!hasDocument}
ariaLabel={ts("audio.muteRecording")}
tooltip={ts("audio.muteRecordingHelp")}
onChange={(checked) => void set({ recordingAudioMuted: checked })}
/>
</div>
<button
type="button"
className={styles.secondaryBtn}
disabled={!hasDocument}
onClick={() => void set({ audioGainDb: 0 })}
onClick={() => void set({ audioGainDb: 0, recordingAudioMuted: false })}
>
{ts("audio.reset")}
</button>
Expand Down
32 changes: 32 additions & 0 deletions src/components/ai-edition/VirtualPreview.audio.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,38 @@ describe("applyPreviewAudioSettings", () => {
applyPreviewAudioSettings(null, [element], 0, 6.0206);
expect(element.volume).toBe(1);
});

it("mutes routed and direct-output recording elements without changing output gain", () => {
const graph = fakeGraph();
const primary = { volume: 0.2 } as HTMLAudioElement;
const supplemental = { volume: 0.2 } as HTMLAudioElement;
const directFallback = { volume: 0.2 } as HTMLAudioElement;
const fallbackGain = { gain: { value: 1 } } as GainNode;
applyPreviewAudioSettings(
graph,
[primary, supplemental, directFallback],
6.0206,
9.5424,
true,
{
routedElements: new Set([primary, supplemental]),
fallbackGainNodes: new Map([[supplemental, fallbackGain]]),
},
);

expect(graph.gain.gain.value).toBeCloseTo(2, 3);
expect(graph.voice.gain.value).toBe(0);
expect(primary.volume).toBe(1);
expect(supplemental.volume).toBe(1);
expect(fallbackGain.gain.value).toBe(0);
expect(directFallback.volume).toBe(0);
});

it("silences newly mounted direct-output sources before WebAudio is available", () => {
const element = { volume: 1 } as HTMLAudioElement;
applyPreviewAudioSettings(null, [element], 6.0206, 9.5424, true);
expect(element.volume).toBe(0);
});
});

describe("previewCeilingCurve", () => {
Expand Down
Loading
Loading