Puffin Ship Docs

Media services

The media processor is a single, general-purpose way to call an external media service, generation of speech, images, video, or a stock search, and place the result on your timeline. Instead of a separate processor per service, you describe the service you want in YAML, and media does the calling, generating, and placing.

Why one processor

Media services come and go, and every team has a favorite. Rather than shipping (and maintaining) a dedicated processor for each one, media reads a description of the service, called a spec, and does whatever that spec says: send this request, with this auth, and interpret the response this way. Adding support for a new service is a configuration change, not a code change.

Specs

A spec is a YAML description of a service: how to authenticate, what request to send, and how to read the response. It covers things like:

  • Transport: which endpoint to call and how (for example, a plain HTTP request)
  • Auth: where a service key comes from and how it is attached to the request
  • Request: the shape of the request body, filled in with your text and any variables you provide
  • Response: where in the response the generated asset (or a link to it) can be found

You never have to write a spec from scratch to get started, see Presets below, but when a service has no shipped preset, writing one inline gets you the same result. The bring-your-own-media-service guide walks through a complete example.

Services differ in how they hand back what they generated. Some return the media directly as bytes in the response body. Others return the media as encoded data inside a JSON response, for example a base64-encoded image alongside other fields. The spec's response says which kind to expect and, for the JSON case, exactly where in the response to find the data. A spec can also declare one or more post_process steps, local transforms applied after the asset comes back, such as turning a generated still image into a moving vertical clip before it is placed.

Some services, video generation especially, do not hand back the finished asset right away. For those, the spec describes a background job instead: the initial request starts the job, and the step checks in periodically until the service reports the media is ready, then downloads it. This happens automatically, a step using a job-based spec behaves like any other from the outside, it just takes longer to finish.

Presets

A preset is starter configuration for a specific, popular service. Choosing a preset fills in the spec for you, so you only need to supply the values that are yours: your service key, a voice or model id, any other request-specific variables. You can also override individual fields of a preset if you need to tweak something, like a default model, without writing the whole spec yourself.

Think of a preset as a spec someone already wrote for a well-known service. You are never limited to the services with presets: any service reachable over HTTP can be described with a spec, preset or not.

The stage model

A media step works in four stages:

  • Source: how the request is produced from your inputs. You can generate a single asset, one per segment of a plan, or one per empty span of a timeline level to fill gaps. The single strategy builds one request from your text input. The gaps strategy discovers unfilled spans in a timeline level and generates a clip to fit each span; its gap_audio variant treats the source input as a narration audio track instead of text, slicing out the portion of audio that matches each gap so, for example, an avatar can be generated speaking exactly that span rather than the whole script. The plan strategy reads a segment plan from an input step and generates one asset per segment.
  • Judge: an evaluation step that can accept or reject a generated asset before it is placed, the same idea used by Puffin Ship's b-roll generation today.
  • Place: how the resulting asset lands on your timeline. You can place it at a fixed position, chain it end to end with anything already placed, or fit each generated clip into the span it was created to fill.
  • Reuse: a media step remembers what it already generated and how the judge rated it, so a rerun does not regenerate or re-review an unchanged request.
inputs (text, transcript, ...)
        |
        v
     source        -- builds the request
        |
        v
  external service  -- called using your spec/preset
        |
        v
     [judge]        -- accept/reject the result
        |
        v
      place         -- lands the asset on your timeline
        |
        v
    reuse cache     -- skip re-generating a request seen before

The judge stage (optional) asks the assistant to review each generated asset and accept or reject it based on your criteria before placing it on your timeline. This is useful when you want only high-quality takes. For example, when generating video clips, you can reject clips that don't match your visual style, have unclear speakers, or miss key moments. When enabled, rejected assets are discarded; only approved takes reach the timeline. For generated video, the judge also trims the clip to its most impactful moment, keeping just the strongest sub-range rather than the whole take.

An optional prompt judge works one step earlier: it reviews the request before anything is generated and can reject or improve it, so a paid call is not spent on a prompt that does not fit your criteria.

How the pieces fit together

A media step, in plain terms:

  1. Take a preset, a spec, or both (a preset with your own overrides).
  2. Read your source input and any vars you supplied.
  3. Build the request the spec describes and call the service, using a key from your workspace secrets.
  4. Interpret the response the way the spec describes, to find the generated asset.
  5. Place the result on your output timeline.

See the media processor reference for the full parameter list, and the bring-your-own-media-service guide for a worked example that writes a spec from scratch.

Media services as b-roll generation tools

A configured media service can be offered to Puffin Ship's automatic b-roll agent as a tool it invokes to generate and place clips where it decides. When you set up a media provider in the auto_broll_placer step, the agent can call that service to fill timeline gaps or enhance your edit. The agent sees the service's spec and capabilities, decides when and where to use it, and handles the generation and placement the same way it judges quality and choreographs the other generation steps.

Next