Footage Director Guide

The footage director pipeline automatically assembles a timeline from raw, uncut footage, with no manual edit points needed. Give it a folder of clips; it transcribes, analyzes quality, assembles a cut, and produces a unified transcript for downstream processors.

When to use

Use this pipeline when:

You have raw footage without pre-defined cut points
You want AI to select the best moments and arrange them
You're working with multi-clip footage (interviews, event footage, multi-camera)

For single-clip pipelines with a clear structure, a simpler generate_transcript → ai_agent workflow is often sufficient.

In the editor

This is four steps in a row, all from the Library under AI and Transcript, each one wired to the one above it:

analyze_footage, with a clips file input pointing at your raw footage.
footage_assembler, with an analysis step input pointing at step 1.
video_director, with a transcript step input.
assemble_transcript, with analysis pointing at step 1 and footage at step 2.

Only two settings really shape the result: style and target_duration_sec on the assembler, plus max_speaker_ratio, which decides how much of the finished video is you talking and how much is everything else.

Tick Review gate on footage_assembler. It is the step that decides which of your takes are worth using and in what order, and it is far cheaper to send it back for another pass at that point than after captions and b-roll have been built on its choices.

The first two steps take a while, because they watch and listen to everything you shot. After they run, the timeline is populated and you can preview it, adjust individual clips, and re-run any later step on its own.

The pipeline

analyze_footage → footage_assembler → video_director → assemble_transcript

Step 1: analyze_footage

Transcribes each clip, detects silence, and uses Claude Vision to assess quality, identify key moments, and suggest usage (backbone/broll/skip).

- id: analyze_raw
  processor: analyze_footage
  input:
    - id: clips
      file: raw-footage       # glob matching all raw clips
  params:
    analysis_mode: adaptive
    language: "en"
  cache: true

Use cache: true: analysis is expensive and never changes for the same clips.

Step 2: footage_assembler

AI agent that reviews the analysis and builds an initial timeline: backbone clips (speaker/presenter) and broll clips, based on quality scores and content.

- id: assemble
  processor: footage_assembler
  input:
    - id: analysis
      step: analyze_raw
  params:
    style: documentary
    prefer_original_audio: true
    max_speaker_ratio: 0.4
    target_duration_sec: 120
  output:
    timeline: main
  review: true
  cache: true

review: true is important here: check that the assembly looks reasonable before spending time on subsequent processing.

Step 3: video_director (optional)

Produces a segment plan: a structured assignment of time ranges to content types. Downstream processors (heygen_speaker, drawings_ai_placer, gap-filling broll) use this plan to know where to place their content.

- id: direct
  processor: video_director
  input:
    - id: transcript
      step: transcript
  params:
    backbone_step: assemble
    min_segment_sec: 3.0
  cache: true

Skip this step if you're not using heygen_speaker or drawings_ai_placer.

Step 4: assemble_transcript

Merges per-clip transcripts from analyze_footage with the assembled backbone clips to produce a unified, timeline-relative transcript.

- id: transcript
  processor: assemble_transcript
  input:
    - id: analysis
      step: analyze_raw
    - id: footage
      step: assemble
  params:
    backbone_level: main
  cache: true

The output is in the same format as generate_transcript, so use it anywhere a transcript step is required.

Full example

version: 1

project:
  name: "Auto Assembly"
  resolution: [1080, 1920]
  fps: 30

input:
  - id: raw-footage
    select: "footage/*.MP4"

timeline:
  levels:
    - name: main
      order: 1
      type: any
      backbone: true
    - name: broll
      order: 2
      type: video
      backbone: false
    - name: captions
      order: 3
      type: video
      backbone: false

pipeline:
  - id: analyze_raw
    processor: analyze_footage
    input:
      - id: clips
        file: raw-footage
    params:
      analysis_mode: adaptive
      language: "en"
    cache: true

  - id: assemble
    processor: footage_assembler
    input:
      - id: analysis
        step: analyze_raw
    params:
      style: documentary
      target_duration_sec: 90
    output:
      timeline: main
    review: true
    cache: true

  - id: transcript
    processor: assemble_transcript
    input:
      - id: analysis
        step: analyze_raw
      - id: footage
        step: assemble
    cache: true

  - id: direct
    processor: video_director
    input:
      - id: transcript
        step: transcript
    params:
      backbone_step: assemble
    cache: true

  - id: broll_ai
    processor: broll_veo_generate
    input:
      - id: transcript
        step: transcript
      - id: backbone
        step: assemble
    params:
      backbone_step: assemble
      max_generate: 5
      eval_prompt: >
        Accept: clear, focused shot without glitches or uncanny elements.
        Reject: blurry, morphing objects, impossible physics.
    output:
      timeline: broll
    cache: true

  - id: captions
    processor: captions
    input:
      - id: transcript
        step: transcript
      - id: backbone
        step: assemble
    output:
      timeline: captions
    cache: true

outputs:
  - type: fcpxml
    name: assembled

Tips

analyze_footage is the most expensive step. Always use cache: true.
Adjust max_speaker_ratio in footage_assembler to control how much talking-head vs. broll you get. Lower value = less presenter on screen.
video_director is only needed if you're using heygen_speaker or drawings_ai_placer downstream (or a media step reading the plan with source: plan).
Check the footage_assembler output with review: true before running expensive downstream steps like Veo generation.