Puffin Ship Docs

ai_agent

General-purpose Claude tool-use loop. Claude is given a prompt and a set of tools; it calls them iteratively until it's satisfied with the result.

When to use

ai_agent is the flexible workhorse for any editing task that requires reasoning about content. The most common use is a content cut: Claude reads a transcript, reasons about what to keep, and creates edit decisions for each segment.

In the editor

Add it from the Library (AI group), or with + add step and a search for ai_agent.

The step's Prompt is the whole configuration: it is the instruction the agent works to. Say what you want done, and name the inputs it should read. Add those under Inputs with ids you choose, as step or file, and refer to them in the prompt by those ids. A content cut, the commonest use, wires a transcript step input and a speaker file input, and asks for the segments worth keeping.

guidelines holds standing constraints that are not really instructions ("never cut mid-sentence", "keep it under two minutes"). suggest_ideas makes the run pause and offer you a few directions before it commits to one. context points at extra material you have uploaded for background.

Set Output Level to the level the clips it creates should land on, usually main.

Tick Review gate. This step reasons about your content and its answer varies, so seeing the cut before the rest of the run is built on it saves rebuilding everything afterwards.

The clips it decides on appear on that level in the run timeline, ready to preview and adjust. Where a dedicated step exists for what you want, prefer it; this one is for the jobs nothing else covers.

YAML Example

- id: ai_cut
  processor: ai_agent
  input:
    - id: transcript
      step: transcribe_main
    - id: speaker
      file: speaker-footage
    - id: reference         # optional .txt reference script
      file: transcript
  prompt: |
    You are an editor making a content cut. The transcript and silence intervals are
    already provided in the context block above. Call get_input("speaker") to get the
    source filename, then call create_edit_decision for each segment to keep.
    Use source=files[0], type="video", level="main". Snap in/out to silence boundaries.
  output:
    timeline: main
  review: true
  cache: true

Example with suggest_ideas and context:

- id: ai_cut
  processor: ai_agent
  input:
    - id: transcript
      step: transcribe_main
    - id: speaker
      file: speaker-footage
  params:
    suggest_ideas: true
    context:
      dir: context/          # e.g. context/product-notes.md, context/talking-points.txt
  prompt: |
    You are an editor making a content cut of a Redpanda product demo.
    Use get_input("transcript") for timestamps, get_input("speaker") for the source file.
    Create edit decisions for the best segments, trimming dead air and off-topic tangents.
  output:
    timeline: main
  review: true
  cache: true

Inputs

Any number of named inputs. Reference them in your prompt using get_input("<id>").

ID Source Description
(any) file or step Accessible via get_input() in Claude's tool calls

Params

Param Type Default Description
guidelines text none Additional instructions or constraints for the AI agent beyond the prompt.
context text none Background context injected into the system prompt. Supports text:, file:, dir:, files:.
suggest_ideas bool false Before running, generate 3–5 editing directions and ask the human to pick one. The chosen direction is injected into the agent's system prompt.
allowed_content_types list none Plan mode only: restrict the planner to these content types (e.g. [tts, clip]). If empty, all registered types are available.
mute_audio bool false Force Volume=0 on all produced edit decisions.
max_clip_sec duration (s) none Truncate long clips to this duration during analysis.

Available tools

All data (transcript, footage analysis, silence intervals, workspace files) is pre-injected into the first user message. The agent does not need to call fetch tools, just read the context block and call decision tools.

Tool Description
get_input(id) For media file inputs: returns the source filename. Transcript/analysis inputs are already in the context block.
get_timeline_state() List all edit decisions created so far in this step, plus backbone duration and remaining time.
create_edit_decision(source, type, level, in, out, ...) Add a clip to the timeline.
list_edit_decisions(level) Filter existing edit decisions by level.
apply_transcript_corrections(corrections) Find-replace on a transcript input.

Output

EditDecisions in most cases: whatever Claude chose to create via create_edit_decision. When apply_transcript_corrections is used, returns Analysis instead. The output type is dynamic (OutputTypeUnknown).

Notes

  • Uses Puffin Ship's built-in AI: no API key required.
  • The prompt is sent as the system message. Keep it specific and instructional.
  • review: true is strongly recommended: always verify the AI's editorial decisions before continuing.
  • cache: true is recommended: saves API cost on re-runs where nothing has changed.
  • Wire transcript, footage analysis, and workspace analysis as explicit input: entries. The processor pre-injects them into the context block automatically. You no longer need to call get_analysis or list_workspace_files in your prompt.

Production Planner Mode

Set mode: plan at the step level to have ai_agent produce a SegmentPlan instead of edit decisions. In this mode, the agent assigns content types and generation configs to every second of the timeline, which downstream processors execute.

- id: plan
  processor: ai_agent
  mode: plan
  input:
    - id: script
      step: write_script
    - id: footage
      file: "clips/*.mp4"
  params:
    guidelines: |
      - Open with the hero footage clip.
      - Use TTS narration for the middle section.
      - Close with a HeyGen avatar call-to-action.
  prompt: |
    Create a production plan for a 60-second promotional video.

Available Content Types

The planner's available content types are auto-discovered from registered SegmentCapability entries:

Content Type Processor Config Fields
clip broll_place file (required), in_sec, out_sec
tts elevenlabs_tts text (required), voice_id
veo broll_veo_generate prompt (required), style
heygen heygen_speaker script (required), avatar_id, voice_id

API-backed types (tts, veo, heygen) only appear when the corresponding API key is configured.

Wiring Downstream Processors

Each downstream processor reads the plan via a segment_plan input:

- id: narration
  processor: elevenlabs_tts
  input:
    - id: segment_plan
      step: plan
  params:
    voice_id: "21m00Tcm4TlvDq8ikWAM"

- id: broll_ai
  processor: broll_veo_generate
  input:
    - id: segment_plan
      step: plan
  output:
    timeline: broll

- id: presenter
  processor: heygen_speaker
  input:
    - id: segment_plan
      step: plan
  params:
    avatar_id: "default"
  output:
    timeline: main

- id: place_clips
  processor: broll_place
  input:
    - id: segment_plan
      step: plan
  output:
    timeline: broll

params.guidelines

Optional free-form instructions injected into the planner's system prompt to guide content type selection:

params:
  guidelines: |
    - Open with existing footage when available.
    - Use TTS for factual narration segments.
    - Only use HeyGen for the closing segment.

params.allowed_content_types

Restrict the planner to a specific subset of content types. Any create_segment call with a type not in this list will be rejected by the tool, forcing the LLM to use only the permitted types. The system prompt is also filtered to only show the allowed types.

params:
  allowed_content_types:
    - tts
    - clip

This is the recommended way to exclude AI-generated media (Veo, HeyGen) from a pipeline that only uses user-provided footage.

When web search is enabled on the platform, the agent gains a web_search tool that Claude calls autonomously to research topics, find recent news, or gather data. It's injected into every ai_agent step automatically, with no configuration needed on your end.