An explainer video falls apart when the picture changes before or after the sentence. The easiest way to avoid it is to reverse the order: voice first, then scenes.
The steps
- Write one sentence per scene (see local AI voice).
- Generate the audio for each sentence and measure the seconds. Add half a second of pause at the end.
- Write each scene’s
t0andt1in the brief:t0is the sum of the previous durations,t1 = t0 + duration. - In the renderer,
renderAt(t)finds the scene witht0 <= t < t1and draws with local timet - t0. - Join the audio to the video with ffmpeg.
{ "scenes": [
{ "id": "title", "text": "You stole a car and the police are after you? Relax.", "t0": 0.0, "t1": 3.9 },
{ "id": "swap", "text": "Swap the car and change clothes.", "t0": 3.9, "t1": 6.7 }
] }
// inside renderAt(t)
const s = brief.scenes.find((x) => t >= x.t0 && t < x.t1) || brief.scenes.at(-1);
drawScene(s.id, t - s.t0, s.t1 - s.t0); // local time and scene length
ffmpeg -y -i out.mp4 -i voce.wav -c:v copy -c:a aac -shortest final.mp4
Word-by-word captions
Inside the scene you have local time u and length d. Split the sentence into words and give each one a slice of d in proportion to its letters. The word to highlight is the one whose slice contains u.
function activeWord(words, u, d) {
const total = words.reduce((n, w) => n + w.length + 1, 0);
let acc = 0;
for (let i = 0; i < words.length; i++) {
acc += (words[i].length + 1) / total * d;
if (u < acc) return i;
}
return words.length - 1;
}
It is not synced to the syllable, but at 24 frames per second the eye accepts it. If you need real precision there are alignment tools that give each word’s time; for short videos it is not worth it.
Final check
Look at three frozen moments (start, middle, end of a long scene): the text must not cover faces or important objects. Then check that audio and video lengths match with ffprobe.
See also puppet videos without any gameplay.
Frequently asked questions
Why should the voice drive the timing?
Because if you fix scenes at 5 seconds and the sentence lasts 7, the voice gets cut or lags behind. If the voice decides, scene and speech end together.
How do I do word-by-word captions?
Split the sentence into words and spread the scene's seconds in proportion to each word's letter count. It is not exact to the millisecond, but it is enough for the eye.