Filling gaps and generating per segment

The media processor can generate content to fill empty spans in your timeline or produce one asset per segment of a plan. This guide walks through both strategies, so you can automatically generate b-roll to plug timeline holes or create content that matches a structured outline.

Background on how the media processor works in general is in Media services. You may also want to reference the media processor reference for the full parameter list.

Filling empty spans with generated content

When you have a timeline level with clips placed here and there, the gaps between them sit empty. The gaps source strategy finds those empty spans and generates one clip per gap, fitted to fill that span exactly. This is useful when you want to fill b-roll breaks with generated video, or add ambient content between dialogue clips.

How it works

1. Set source: gaps and gap_level to the timeline level you want to scan (for example, broll).
2. Provide a prompt or text that describes what should fill each gap.
3. Set place: gaps to fit each generated clip exactly to its span.
4. The step scans the level, finds unfilled spans, sends one generation request per gap, and places each result to fill its span perfectly.

Step-by-step

First, make sure you have clips already placed on the level you want to fill. For example, if you have a broll timeline level with some b-roll clips that leave gaps, and you want to generate video to fill those gaps:

- id: broll_gaps
  processor: media
  input:
    - id: source
      file: prompts/gap-fill.txt  # text describing what to generate, reused for each gap
  params:
    preset: veo                    # use Veo to generate video
    source: gaps                   # scan for unfilled spans
    gap_level: broll               # the timeline level to scan
    place: gaps                    # fit each clip to its span
  output:
    timeline: broll
  cache: true

The source input should contain a reusable prompt or description that fits any gap. For example, a file named prompts/gap-fill.txt might say: "Abstract video montage of technology and motion". The step uses this same prompt for every gap it finds, generating clips that match the description and fit the available space.

For each empty span in the broll level:

• A generation request is sent with your prompt
• The service returns a clip (or a longer clip that the step trims to fit)
• The result is placed to exactly fill that gap
• All gaps are processed this way

Generating one asset per segment of a plan

When you have a structured outline or plan (a list of segments, each with its own content), the plan source strategy generates one media asset per segment. This is useful when your content is organized as discrete sections, chapters, or story beats, and each deserves its own generated material.

How it works

1. Create a step that outputs a segment plan (a list of segments, each with text, duration, metadata).
2. Set source: plan and wire the plan-producing step as your segment_plan input.
3. Provide a plan_content_type to describe what kind of content each segment should generate.
4. The step generates one asset per segment, and places them in order.

Step-by-step

Suppose you have a step that breaks a narrative into chapters, outputting a segment plan where each segment has a description and target duration. You can use that plan to generate themed content for each chapter:

- id: chapter_images
  processor: media
  input:
    - id: segment_plan
      step: chapter_planner         # a step that outputs a segment plan
  params:
    preset: imagen                  # generate still images
    vars:
      Model: imagen-4.0-generate-001
    source: plan                     # generate per segment
    plan_content_type: chapter_cover # describe what each asset is for
    place: chain                     # chain the results end to end
  output:
    timeline: covers
  cache: true

For each segment in the plan:

• The step builds a generation request that includes the segment text and the plan_content_type context
• The service generates an asset tailored to that segment
• The asset is placed on your timeline

The place strategy can be at_position, chain, or gaps, depending on how you want the assets arranged.

Worked example: filling b-roll gaps with Veo

Here is a complete example that fills empty b-roll spans with generated video. Suppose you have a script with narration on a timeline, and b-roll clips already placed at certain moments. You want to fill the silent gaps with generated video that matches the overall tone:

steps:
  - id: main_narration
    processor: audio_or_imported_narration
    input:
      - id: source
        file: narration.mp3
    output:
      timeline: main

  - id: broll_hand_placed
    processor: broll_place
    input:
      - id: clips
        file: broll
    output:
      timeline: broll

  - id: fill_gaps
    processor: media
    input:
      - id: source
        file: gap_prompt.txt  # A text file containing something like:
                              # "Cinematic landscape montage, slow motion nature footage"
    params:
      preset: veo
      source: gaps
      gap_level: broll
      place: gaps
    output:
      timeline: broll
    cache: true

When this pipeline runs:

1. Narration and hand-placed b-roll clips create the initial timeline and gaps.
2. The fill_gaps step scans the broll level for empty spans.
3. For each gap found, it sends a generation request with your prompt text to Veo.
4. Veo returns a clip; if it is longer than the gap, the step trims it.
5. The trimmed clip is placed to exactly fill its gap.

You can adjust the gap_prompt.txt file without changing the config; the generated clips will adapt to match any prompt you provide.

Filling gaps with a lip-synced avatar

When you want to fill gaps with a presenter speaking directly to the audience, you can use HeyGen to generate a lip-synced avatar that speaks exactly the narration for each gap. This is useful when your narration has pauses or gaps where you want a visual of a speaker, such as an interview-style video or a presenter-led explainer.

How it works

1. Set source: gaps and gap_level to the timeline level you want to scan.
2. Set preset: heygen-audio and enable gap_audio: true.
3. Wire your narration audio file as the source input.
4. Set place: gaps to fit each avatar clip to its span.
5. The step scans the level, extracts the narration audio for each gap, and generates one avatar clip per gap with lip-sync matching that audio segment.

Step-by-step

Suppose you have a timeline with a main narration track and some gaps where you want an avatar speaking. First, make sure your narration is placed on a timeline level (often main). Then create a step that scans a different level for gaps and fills them with avatar clips:

- id: avatar_gaps
  processor: media
  input:
    - id: source
      file: narration.mp3  # your full narration audio
  params:
    preset: heygen-audio    # use HeyGen for lip-synced avatar
    vars:
      AvatarID: "<your-avatar-id>"
    source: gaps            # scan for unfilled spans
    gap_level: broll        # the timeline level to scan for gaps
    gap_audio: true         # slice the narration by gap time
    place: gaps             # fit each avatar clip to its gap
  output:
    timeline: broll
  cache: true

For each empty span in the broll level:

• The step extracts the corresponding audio segment from your narration based on the gap's start and end time
• A generation request is sent to HeyGen with that audio segment
• HeyGen returns an avatar video with lip-sync matched to the audio
• The result is placed to exactly fill that gap

A re-run reuses a previously generated avatar clip for a gap unless the narration audio changed, which is detected by its file size and modification time. If you edit the narration in place in a way that happens to keep the exact same size and timestamp, the step will not notice the change; disable caching for the step or re-save the narration file to force regeneration.

Worked example: filling pauses with an avatar

Here is a complete example that fills empty spans with a lip-synced avatar speaking the narration. Suppose you have a main narration track, and you want an avatar to appear during the gaps in your b-roll level:

steps:
  - id: main_narration
    processor: audio_or_imported_narration
    input:
      - id: source
        file: narration.mp3
    output:
      timeline: main

  - id: broll_hand_placed
    processor: broll_place
    input:
      - id: clips
        file: broll
    output:
      timeline: broll

  - id: avatar_in_gaps
    processor: media
    input:
      - id: source
        file: narration.mp3  # same narration audio
    params:
      preset: heygen-audio
      vars:
        AvatarID: "<your-avatar-id>"
      source: gaps
      gap_level: broll      # scan the broll level for gaps
      gap_audio: true       # extract audio for each gap
      place: gaps           # fit the avatar to each gap
    output:
      timeline: broll
    cache: true

When this pipeline runs:

1. The narration is placed on the main level.
2. The hand-placed b-roll clips are placed on the broll level.
3. The avatar_in_gaps step scans the broll level for empty spans.
4. For each gap, it extracts the audio segment from narration.mp3 that corresponds to that timeline span.
5. HeyGen generates an avatar video with lip-sync matched to that audio.
6. The avatar clip is placed to exactly fill each gap in the b-roll level.

The avatar will speak only the narration content that falls within each gap, so the full narration remains audible and the avatar appears natural with perfect lip-sync.

Next

• Media services
• Bring your own media service
• media processor reference
• Timeline and levels