# Syncing scenes and voice in a video

> How to make each scene last as long as its spoken sentence: the voice sets the timing and the scenes adapt. With a word-by-word caption example.

Status: testato · Updated: 2026-11-20

**TL;DR:** Do not pick scene lengths first. Generate each scene's voice, measure how long it lasts and use that number as the scene length.

An explainer video falls apart when the picture changes before or after the sentence. The easiest way to avoid it is to reverse the order: voice first, then scenes.

## The steps
1. Write one sentence per scene (see [local AI voice](/en/guide/local-ai-voice-for-videos)).
2. Generate the audio for each sentence and measure the seconds. Add half a second of pause at the end.
3. Write each scene's `t0` and `t1` in the brief: `t0` is the sum of the previous durations, `t1 = t0 + duration`.
4. In the renderer, `renderAt(t)` finds the scene with `t0 <= t < t1` and draws with local time `t - t0`.
5. Join the audio to the video with ffmpeg.

```json
{ "scenes": [
  { "id": "title",   "text": "You stole a car and the police are after you? Relax.", "t0": 0.0,  "t1": 3.9 },
  { "id": "swap",    "text": "Swap the car and change clothes.",                     "t0": 3.9,  "t1": 6.7 }
] }
```

```js
// inside renderAt(t)
const s = brief.scenes.find((x) => t >= x.t0 && t < x.t1) || brief.scenes.at(-1);
drawScene(s.id, t - s.t0, s.t1 - s.t0);   // local time and scene length
```

```bash
ffmpeg -y -i out.mp4 -i voce.wav -c:v copy -c:a aac -shortest final.mp4
```

## Word-by-word captions
Inside the scene you have local time `u` and length `d`. Split the sentence into words and give each one a slice of `d` in proportion to its letters. The word to highlight is the one whose slice contains `u`.

```js
function activeWord(words, u, d) {
  const total = words.reduce((n, w) => n + w.length + 1, 0);
  let acc = 0;
  for (let i = 0; i < words.length; i++) {
    acc += (words[i].length + 1) / total * d;
    if (u < acc) return i;
  }
  return words.length - 1;
}
```

It is not synced to the syllable, but at 24 frames per second the eye accepts it. If you need real precision there are alignment tools that give each word's time; for short videos it is not worth it.

## Final check
Look at three frozen moments (start, middle, end of a long scene): the text must not cover faces or important objects. Then check that audio and video lengths match with `ffprobe`.

See also [puppet videos without any gameplay](/en/guide/puppet-videos-without-gameplay).

## FAQ

### Why should the voice drive the timing?
Because if you fix scenes at 5 seconds and the sentence lasts 7, the voice gets cut or lags behind. If the voice decides, scene and speech end together.

### How do I do word-by-word captions?
Split the sentence into words and spread the scene's seconds in proportion to each word's letter count. It is not exact to the millisecond, but it is enough for the eye.
