Claude and ffmpeg as a video editor

TL;DR: just show me the prompt.

Claude can’t watch a video. It can look at pictures, and ffmpeg can turn any video into pictures. That one fact is enough to make a working video editor out of a Claude session and a free command-line tool, as long as the edit itself lives in a text file.

I cut the 51-second trailer for Hambone Alley on Steam this way, and our three-minute entry video for the Gloo AI Hackathon. Every command below has been run as written.

What ffmpeg is

ffmpeg is a free, open-source program that reads, converts, cuts and joins almost any kind of video or audio file. It has no window. You type a command, and it writes a file. It comes with ffprobe, which reports what’s inside a file.

Download it from ffmpeg.org, or install it from a terminal:

winget install Gyan.FFmpeg     # Windows
brew install ffmpeg            # macOS
sudo apt install ffmpeg        # Ubuntu or Debian

What you need

• ffmpeg and ffprobe on your path. Check with ffmpeg -version.
• A Claude session that can run commands on your machine: Claude Code, or the Code tab in the desktop app.
• Your footage in one folder.

The loop

1. Look. Claude turns each clip into contact sheets and reads them.
2. List. Claude writes the edit as a shot list: which clip, in and out times, in what order.
3. Preview. A small, fast render you can watch in a minute.
4. Nudge. You say what’s wrong. Claude edits the list and renders again.
5. Final. One full-quality render from the original files.

You do the watching and the deciding. Claude does the looking-up, the arithmetic and the typing.

How Claude sees footage

First, the facts about a clip:

ffprobe -v error -show_entries format=duration:stream=codec_name,width,height,r_frame_rate -of default=nw=1 clip.mp4

Then a contact sheet: one frame every two seconds, twenty to a page, so a page covers 40 seconds.

ffmpeg -i clip.mp4 -vf "fps=1/2,scale=384:-1,tile=5x4" sheet-%02d.jpg

Claude reads the sheets and can tell you where the good moments are. For a closer look at one moment, it pulls a single frame:

ffmpeg -ss 12.5 -i clip.mp4 -frames:v 1 still.png

One thing to note: a sheet at two-second steps finds a moment, not a cut point. To place a cut, have Claude pull stills a few tenths of a second apart around it.

The shot list is the edit

Keep the edit decisions in a file and commit it. Footage is replaceable. The decisions aren’t.

{
  "size": "1920x1080",
  "fps": 60,
  "music": "music/theme.mp3",
  "music_db": -6,
  "game_audio_db": -12,
  "fade_in": 0.5,
  "fade_out": 1.5,
  "shots": [
    { "type": "card", "text": "Hambone Alley", "dur": 2 },
    { "type": "clip", "src": "strike.avi", "in": 12.5, "out": 17.0 },
    { "type": "clip", "src": "shop.avi", "in": 40.0, "out": 43.0 }
  ]
}

Claude writes a small script that reads this file and runs ffmpeg. The script only ever reads the list, so a render can be repeated from the file alone. “Start that second shot half a second later” becomes a one-number change.

Cutting, joining and finishing

Cut each shot from the original file, re-encoding so the cut lands on the exact frame and every piece matches:

ffmpeg -ss 12.5 -to 17 -i clip.mp4 -vf "scale=1920:1080,fps=60" -c:v libx264 -crf 18 -pix_fmt yuv420p -c:a aac -b:a 192k -ar 48000 -ac 2 seg-01.mp4

A title card is a solid colour with text on it, plus silent audio so it joins cleanly:

ffmpeg -f lavfi -i "color=c=0x111111:s=1920x1080:r=60:d=2" -f lavfi -i "anullsrc=r=48000:cl=stereo" -vf "drawtext=text='Hambone Alley':fontcolor=white:fontsize=96:x=(w-text_w)/2:y=(h-text_h)/2" -shortest -c:v libx264 -crf 18 -pix_fmt yuv420p -c:a aac -b:a 192k seg-00.mp4

If the text comes out in an odd font, name one: add fontfile= with the path to a font file inside the drawtext options.

Join the pieces. Because they all share one format, this copies them without re-encoding:

# list.txt
file 'seg-00.mp4'
file 'seg-01.mp4'
file 'seg-02.mp4'

ffmpeg -f concat -safe 0 -i list.txt -c copy cut.mp4

Then music under the game audio, a fade at each end, and the flag that lets the file start playing before it has finished downloading:

ffmpeg -i cut.mp4 -i music.mp3 -filter_complex "[0:a]volume=-12dB[game];[1:a]volume=-6dB[music];[game][music]amix=inputs=2:duration=first:normalize=0,afade=t=in:d=0.5,afade=t=out:st=8:d=1.5[a];[0:v]fade=t=in:d=0.5,fade=t=out:st=8:d=1.5[v]" -map "[v]" -map "[a]" -c:v libx264 -crf 18 -pix_fmt yuv420p -c:a aac -b:a 192k -movflags +faststart final.mp4

The st=8 is where the fade-out starts: the length of the cut minus the fade. Claude works that out from ffprobe.

Check the loudness before you upload:

ffmpeg -hide_banner -nostats -i final.mp4 -af loudnorm=print_format=summary -f null -

What made it fast

• Preview small. Render previews at 960x540 with -preset ultrafast. Save 1080p for the final.
• Cache the pieces. Name each rendered shot by a hash of its settings. Nudging one cut then re-renders one shot, not the whole trailer.
• Always cut from the originals. Never from a preview or a proxy, or the preview’s quality ends up in the final.
• Zoom only on still things. A crop that pushes in on a score or a menu looks deliberate. On a moving camera it looks like a mistake.

The Hambone Alley trailer came out at 1920x1080, 60 frames a second, H.264 at CRF 18 with 192k AAC audio.

Where it stops

This is an editor for cuts, cards, music and fades. It won’t replace a timeline for colour work or anything you need to feel frame by frame, and you still have to watch every preview yourself. For a trailer or a submission video on a deadline, that was the right trade.

A prompt to start with

Paste this into a Claude session opened in your footage folder.

I want to cut a short video from the footage in this folder using ffmpeg.
You can't watch video, so work from stills.

1. Run ffprobe on every video file and give me a table: file, duration,
   resolution, frame rate, whether it has audio.
2. For each file, make contact sheets with
   ffmpeg -i <file> -vf "fps=1/2,scale=384:-1,tile=5x4" sheets/<name>-%02d.jpg
   then read the sheets and list the moments worth using, with timestamps.
3. Ask me what the video is for, how long it should be, and what must be in it.
4. Write the edit as shotlist.json: output size and frame rate, music file and
   levels, fades, and an ordered list of shots. A shot is a clip (src, in, out),
   a still (src, dur) or a card (text, dur).
5. Write a script that renders shotlist.json with ffmpeg. It must only read the
   list, cut every shot from the original files, cache each rendered shot by a
   hash of its settings, and have two modes: preview (960x540, ultrafast) and
   final (H.264, CRF 18, yuv420p, AAC 192k, faststart).
6. Render a preview and tell me where it is. Then wait for my notes.

When I give notes, change shotlist.json, not the script, and render again.
To place a cut exactly, pull single frames a few tenths of a second apart
and look at them. Never overwrite or delete my original footage. Before a
final render, tell me the total length and check the loudness.