[umiyosh]

Closes #13

Why

Narration and BGM alone leave explainer videos feeling flat. Short sound effects on visible events (a card appearing, a chip arriving, a step lighting up) make them feel finished. The ElevenLabs Sound Effects API works on pay-as-you-go without a subscription, at about $0.002 per second of output.

Two findings from trying it shaped the design:

• Generated levels vary by 30 dB or more. A "soft UI pop" peaked at -32 dBFS while a "quick airy whoosh" peaked at 0 dBFS. Mixed as they come, one is inaudible and the other is louder than the narration. Every effect is leveled before mixing, against the narration. The first version leveled to a fixed -20 dB × 0.5, which put the effects 12–17 dB under the speech. A listener heard none of them, so the level was then picked by ear from mixes at +0, +6 and +12 dB.
• Claude cannot judge whether an effect sounds right, and neither could Gemini. Asked to listen to the two effects plus deliberate noise, a harsh tone and silence, gemini-3.8-flash got 10–12 of 18 right and called silence a "soft UI pop". gemini-3.1-pro-preview called the pop "silence" three times out of three. So effects are kept short and quiet, silence is detected mechanically, and a preview file lets a person listen to all of them.

What changed

• script.json: an optional sfx list per scene (at seconds from the scene start, prompt, duration 0.5–10 s with a default of 1.5, volume 0–1) and a script-wide sfxVolume (default 1). All values are validated as plain numbers, like the other values written into the filter graph.
• motion-render sfx <work> (new sfx.ts):
  • Calls POST /v1/sound-generation with mp3_44100_128, the best format pay-as-you-go allows. The provider call is one function, so it can be swapped for fal.ai.
  • Stores audio/sfx/<hash>.mp3 keyed by model, prompt and duration, so an unchanged effect is never paid for twice and identical effects are shared across scenes.
  • Measures each file with volumedetect. A peak below -60 dB is a failed attempt, retried with the same 90 s timeout and 3 attempts as Gemini.
  • Writes audio/sfx/preview.m4a, every effect in script order at its mixing level with a 0.6 s gap, and prints the order.
• build:
  • Stops before rendering if an effect was not made.
  • Places each effect at the film's actual scene start + at (placeSfx, like placeOnFilm), so effects follow the animation when narration moves scenes. Warns about an effect past its scene's end.
• Leveling (sfxTrack, narrationLevel): each effect is brought to 4 dB under the narration's average level and lowered by its volume. Its peak is never allowed past -1 dB. The narration level is measured from the scene WAVs, and -16 dB, Gemini TTS's usual level, is assumed before any narration exists.
• mix(): each effect is cut to duration, given its gain, faded at both ends and delayed into place, then joins the final amix before alimiter. A script without sfx produces exactly the same graph as before, which a test asserts.
• audio.ts: the retry loop moved into withRetries(), shared by Gemini and ElevenLabs. runFfmpeg and TIMEOUT_MS are exported for sfx.ts.
• cli.ts: the commands that take only a work dir (narrate, bgm, sfx) go through a table. This lowers main's complexity from 25 to 22, and the quality baseline is updated to match.
• SKILL.md:
  • Add effects only when ELEVENLABS_API_KEY is set.
  • Put them on visible events: 0–2 per scene, 0.5–1.5 s, soft prompts, the same prompt for the same kind of event.
  • Check stills at each effect's time.
  • Report the preview for a person to listen to.
  • Put 🎧:11.ai on the title scene. Audio made outside a paid subscription is non-commercial and must be credited in its title, per the ElevenLabs help center.
• Docs: a new docs/elevenlabs-key(.ja).md covering the pay-as-you-go top-up, a key restricted to Sound Effects, cost and licensing. The README (en/ja) gains rows in requirements, setup, commands, environment variables and troubleshooting.

Verification

• make check: lint, format, typecheck, tests with coverage, quality tests and the TypeScript quality gate all pass. 78 tests, including the Chromium render smoke tests run outside the sandbox. Coverage is 94% lines in total.
• End to end with the real APIs against the stand-in player (tests/fixtures), with scenes resized to the narration's timing.json (2.9 s and 6.9 s):

  • narrate, bgm and sfx all succeeded. sfx made 3 effects and the preview. A second sfx run made 0 and called no API.

  • Two films were built from the same audio, one with sfx and one without. The output says narration + BGM + 3 sound effects.

  • The difference between the two audio tracks, in 50 ms windows, starts at 1.00 s, 3.40 s and 5.45 s. The intended times were 1.0, 3.4 and 5.4 s. The chime's attack is soft, so it crosses the threshold one window late.

  • Effect level against the narration and BGM in the same window (RMS), after the leveling fix. These match the +6 dB mix the listener picked to within 0.1 dB:

    Effect	Narration + BGM	Effect	Gap
    pop (1.0–1.6 s)	-15.1 dB	-26.5 dB	-11.5 dB
    whoosh (3.4–4.2 s)	-14.0 dB	-21.3 dB	-7.3 dB
    chime (5.4–6.4 s)	-15.0 dB	-21.8 dB	-6.7 dB

  • Stereo peak is -0.17 dBFS with effects. No sample is over 0 dBFS.

• Listening: all three effects are audible in the +6 dB mix. The whoosh sounds slightly noisy, which is acceptable for now.

Not verified

• A real Claude Motion film. The check used the stand-in player, so whether Claude places at on the right animation moments, following SKILL.md, is untested.
• Whether 4 dB under the narration suits other voices, BGM and kinds of effect. It was picked by ear from one test video with three effects.
• The English UI labels in docs/elevenlabs-key.md ("Add credits", "Usage"). They are inferred from the Japanese UI and the ElevenLabs docs.
• Whether pay-as-you-go output is treated as "outside a subscription" for licensing. The help center does not mention pay-as-you-go, so the docs take the conservative reading.