There are good benchmarks for generating video and almost none for editing it. That gap matters, because the two jobs are not the same job. Generating asks a model to invent every pixel. Editing asks it to take footage somebody else shot and decide what to keep, where to cut, and how to make the pieces agree with each other.
So we built CutBench: nine families of editing task, media generated from a fixed seed so the right answer is known by construction, and scoring on the document the model actually produced rather than on the commands it claims to have run.
The board is at the bottom. It is not the interesting part. The interesting part is that we found seven bugs in our own measurement, and every single one of them produced a plausible number that we would have published. A benchmark does not usually fail loudly. It fails quietly, in a direction that looks like a result.
01 / FIELD NOTESWhy editing needs a different benchmark from generation
HeyGen's Code2Video is the closest reference point. It hands a model a creative brief and a deterministic rendering environment, then scores the result on five axes (engagement, prompt-intent, composition, temporal, craft) entirely through pairwise human-preference Elo, using a judge model trained on screened annotators.
It has to work that way. When a model invents every pixel there is no right answer to compare against. HeyGen report that scoring against their own human reference gives near-zero for almost every model, which is why they moved to pairwise comparison instead.
Editing is the opposite situation, and that is the opportunity. If you construct the footage, most of what editors care about has an exact answer. Where a cut lands relative to a beat is a number of milliseconds. Whether dead air was removed is measurable against a known list of silences. Whether dual-system audio is in sync is the error against an offset you applied yourself.
None of that is taste. So CutBench is objective first, and sends only the genuine residue, shot choice and rhythm and feel, to preference judging. The measurable parts get measured.
02 / FIELD NOTESTruth you can construct
Every fixture is generated from a fixed seed with numpy and ffmpeg, and ships with a manifest of what it contains. A drum loop knows its own beat and downbeat times. A talking-head take knows exactly where its silences are. Two camera angles of the same moment know the offset between them, because we applied it. Two cameras of the same scene know what each one reads, because the generator measures them when it writes them.
The rule we inherited from the rest of our test suite is the one that makes this work: a fixture whose right answer lives only in a test's head is not a fixture.
Nine variants per family, and they deliberately run in both directions. A negative sync offset where the second recorder started later. A subject travelling right to left. A camera that is darker but more saturated. Variants that all lean one way buy sample size without buying power, because a single memorised heuristic passes every one of them.
And the fixtures are verified against themselves. A separate pass measures the generated media back using methods that share no code with the generator: cross-correlation for the alignment fixtures, onset detection for the beat grid, column analysis for the pan, mean RGB for the colour targets. If the manifest and the media disagree, the fixture is wrong and the run is abandoned.
That cross-check earned its keep. More on it below.
03 / FIELD NOTESThe headline metric is not craft. It is whether the file is usable.
One of our own agent runs applied 53 of 54 commands. The ledger recorded 53 applied. The render job reported COMPLETED. Every colour command returned ok: true.
The exported video was black for 85 of its 92 seconds.
The agent had written brightness values of 0 to 5 on a scale where 100 is neutral, crushing almost every clip to nothing. Two clips survived, and the reason is worse than the bug: their values were negative, which made the CSS filter string invalid, so the browser discarded the entire grade. One silent failure was rescued by a different silent failure.
A benchmark that scored that run on command receipts would have called it a near-perfect pass. That is the failure CutBench exists to make impossible to score well on, so every task is gated on the rendered file: does it have picture, sound where required, the requested runtime, and no silent damage. Fail the gate and the task scores zero on everything else, however elegant the edit decision list looks.
The gate is calibrated against those two real exports. Both are 92.00 seconds, 1920x1080, identical mean audio level, and both were reported COMPLETED. The only thing that separates them is looking at the picture.
agent's own grade (brightness 0-5) FAIL black 88.0% frozen 85.1%
brightness corrected (~100) PASS black 0.5% frozen 0.0%04 / FIELD NOTESWhat the nine families ask for
Each family is a job editors would happily hand to someone else, and each has constructible truth. Assembly turns many shots into a cut of target length. Dead air strips the silences from a talking head. Sync lines up a separate audio recorder with camera reference. Multicam aligns three angles and cuts a program. Music cuts picture to a tempo. Reframe follows a subject from 16:9 into 9:16. Colour matches two cameras. Continuity finds planted faults. B-roll covers jump cuts.
Models are given the measurements a real editor would already have: detected clap times, a silence list, a motion track, a tempo readout, the mean colour of each camera. They are scored on whether they use them correctly. That line matters. Sync is told where the clap falls in each recording and must work out the shift itself, including its sign. Handing over the offset would turn the task into transcription.
We learned that boundary the hard way. An early version printed every beat time for the music task, and all nine models scored a perfect 1.00 with a median error of zero milliseconds. That is not a benchmark, it is a copying exercise.
05 / FIELD NOTESSeven bugs in our own measurement
Every one of these produced a number that looked like a result. Six were found because a single model scored implausibly badly, not because anything looked wrong in general. Only one was caught by the tooling itself.
Two inverted signs. Three models placed the scratch audio at exactly -0.750 seconds, which is correct: the clap sits at 2.000s in the camera audio and 2.750s in the scratch recording. Our scorer compared against positive 0.750 and marked all three 1500 milliseconds wrong. The multicam scorer had the same inversion, which is why all nine models sat at precisely 0.30 on that family. Both self-tests passed the entire time, because they encoded the same convention as the code they were testing.
A token cap. max_tokens of 2000 starved the reasoning models, which spent the whole budget thinking and emitted nothing. One model went from 0 of 3 to 3 of 3 purely from raising it.
A call that never happened. The hosted proxy returned a null message for one model on every variant of one task, which arrived as a clean zero. Scored at face value that publishes "this model fails multicam" when the model was never asked.
A rate limit scored as wrong answers. Same shape as the last one, except it accrued tokens and cost from the variants that did get through, so the guard that catches zero-everything missed it entirely.
The extractor that punished thinking out loud. This is the worst one. DeepSeek-R1 scored 0 of 9 across three tasks. It had answered every question correctly. It reasons in the open, so it wrote nearly ten thousand characters of correct working, quoting JSON fragments as it went, and finished with exactly the right answer. Our extractor took everything between the first brace and the last brace, swallowed the entire transcript, failed to parse, and recorded a zero. Re-run with balanced-span extraction, it scored 9 of 9. We were one push away from publishing "DeepSeek-R1 scores zero at video editing" on a public leaderboard. A model that shows its work is not a model that got the answer wrong.
A colour space. ffmpeg's signalstats reads luma as stored, and H.264 with yuv420p stores TV range, where black sits at 16 rather than 0. Our colour target is a ratio of two lumas, and a shared pedestal does not cancel out of a ratio. It skews it, and hardest when one side is dark. We wrote a target of 162.4 where the true figure is 195.9. This is the one the tooling caught by itself: the independent verifier re-measures in RGB, where the range has already been expanded, and the two numbers disagreed.
And the benchmark writing its own answers into the data. Continuity planted three faults in a timeline of shot_0 through shot_5 and named them dup_of_shot_1, flash_frame and orphan_audio. All nine models scored a perfect 1.00. The transcripts show why: the first and only command was a single delete of exactly those three identifiers. Nobody detected anything. The family measured whether a model can read three names off a list it was handed.
{"commands":[{"op":"deleteClips","clipIds":[
"dup_of_shot_1","flash_frame","orphan_audio"]}],"done":true}06 / FIELD NOTESThe pattern worth taking away
Four separate times in this project, a test agreed with the code instead of checking it. The two inverted signs. A dead-air fixture where the "correct" document was a clip trimmed to the right total length, which keeps every silence and throws away the last five seconds of speech. An assembly fixture where the "correct" document was four clips of identical length all stacked at time zero, which uses every shot and is not an edit.
In each case the test had been written from the same mental model as the implementation, so it could only ever confirm it. The fix is not more tests. It is tests that state the arithmetic independently, or a second measurement that shares no code with the first.
The other pattern: a family where every model scores full marks is sometimes an easy task and sometimes a scorer that cannot detect a bad answer. Assembly scored coverage alone, ignoring the order and the varied hold lengths the brief asks for in words. B-roll counted a jump cut as covered if any upper-track clip spanned that moment, which one long clip laid across the whole programme satisfies for every cut at once. That is not b-roll, it is replacing the interview. Both read 1.00 for everybody, and both were measuring almost nothing.
Tightening those three scorers moved the board more than tripling the sample size did.
07 / FIELD NOTESThe board
Nine models, 81 tasks, 729 runs. Accuracy is the mean over all nine families.
The gap between first and last is 4.0 sigma, which is a real separation. The gap between first and second is 0.4 sigma, which is not. The claim this board supports is that the top of it beats the bottom of it, not that anything wins.
That 4.0 has three steps behind it. At 18 tasks per model it was 1.6 sigma and the report printed NOT POWERED TO RANK in as many words. Tripling the instances took it to 2.6. Fixing the three scorers that could not detect a bad answer took it to 4.0. Only the first of those three steps was about sample size.
Watch nemotron in particular. It sat fifth on the lenient board at 0.82 and finished last at 0.68, because it was the model benefiting most from the scoring we had to fix. It was passing b-roll with a takeover and assembly with a metronomic dump.
model mean 95% CI
gpt-5.6-luna 0.90 [0.84, 0.96]
gemini-3.8-flash 0.88 [0.80, 0.95]
deepseek-flash 0.83 [0.75, 0.91]
gpt-5.6-sol 0.83 [0.75, 0.91]
qwen3.7-plus 0.81 [0.73, 0.89]
gemini-3.5-flash-lite 0.78 [0.70, 0.86]
grok-3 0.75 [0.67, 0.84]
qwen3.6-flash 0.70 [0.61, 0.80]
nemotron-3-super-120b 0.68 [0.59, 0.77]
1st vs 2nd: 0.4 sigma
1st vs last: 4.0 sigma08 / FIELD NOTESWhich tasks are actually doing the work
A family every model aces ranks nobody and quietly inflates every mean by the same amount, so the report prints the spread across model means for each one.
Eight of the nine now separate models. Reframe is the standout finding: the best of nine models follows a moving subject from 16:9 into 9:16 barely two thirds correctly, and the worst scores zero. That is the task closest to what people actually want from an editing agent, and it is the one nothing does well.
Music is still a ceiling. Given a tempo and a first downbeat, every model puts its cuts on the grid. We would rather say that than pad the board with it.
family lo hi spread
multicam 0.31 1.00 0.69
reframe 0.00 0.67 0.67
broll 0.39 1.00 0.61
sync 0.51 0.89 0.38
assembly 0.79 1.00 0.21
continuity 0.85 1.00 0.15
deadair 0.89 1.00 0.11
colour 0.90 1.00 0.10
music 0.95 1.00 0.05 <- ceiling, ranks nobody09 / FIELD NOTESWhat we are not claiming
The deliverable gate has not run on these numbers. It is measured on the rendered file and the render pass is not built, so the accuracy figures above are ungated by our own standard. They are document-level scores and labelled as such throughout.
A separate set of three tasks, run on a hosted benchmark platform against ten frontier models including Claude and GPT-6, came back almost entirely solved: sync, reframe and colour were 3 of 3 for nearly every model. Those same families separate models sharply here. The difference is that the hosted version asks for a number with the analysis handed over, and this version asks for a working document. Frontier models do the arithmetic perfectly and then have to express it.
That is worth stating plainly because it is the more useful finding: on structured editing, the arithmetic is not the hard part.
Captions is omitted entirely, because it needs real intelligible speech and shaped noise will not transcribe.
And the benchmark is deliberately not scored through our own production planner. That carries thousands of tokens of house style and workflow guidance, and benchmarking through it would measure our prompt engineering while calling it a model's editing ability. Code2Video makes the same choice for the same reason.
10 / FIELD NOTESReproduce it, or disagree with it
The fixtures regenerate byte-identically from a fixed seed, the prompt every model receives is a file in the repository rather than an implementation detail, and the scorers are separate from the runner so a scoring bug can be fixed and re-applied without paying for the model calls again.
That last property is why six of the seven bugs above cost us a re-score rather than a re-run, and why a disputed score can be argued about from the stored trace instead of re-litigated by re-running.
If you take one thing from this: build the second measurement. The only bug here that our tooling caught on its own was the one where two independent methods had to agree on the same number. Every other one was caught by a human noticing that a particular model had scored implausibly badly, which is a detection method that does not scale and does not work at all when the wrong number happens to look reasonable.
Common questions
Which model should I use for AI video editing, based on this?
The top of the board beats the bottom of it by 4 sigma, so gpt-5.6-luna and gemini-3.8-flash are genuinely ahead of qwen3.6-flash and nemotron on these tasks. But the gap between first and second is 0.4 sigma, so there is no winner here, and nothing on this board does reframing well.
Why is the benchmark gated on the rendered file rather than on the edit?
Because a run can apply 53 of 54 commands, report COMPLETED, return ok on every colour command, and still export a video that is black for 85 of its 92 seconds. That happened to us. Command receipts describe intent; only the file describes what a person would actually receive.
Is this measuring the models or measuring Vidmoat?
The models. The prompt is a small published file, not our production planner, and the environment is our documented command schema, the same thing any developer gets from the API. Benchmarking through our real planner would measure our prompt engineering and mislabel it as model ability.
Why publish the bugs?
Because they are the transferable part. Anyone building a benchmark will hit the same class of problem, and the specific failure mode is not obvious: not a crash, not an outlier, but a plausible number in a direction that reads as a finding. We nearly published a zero for a model that had answered everything correctly.
What would make the ranking sharper?
More families that discriminate, rather than more instances. Tripling the sample size moved first-versus-last from 1.6 sigma to 2.6; fixing three lenient scorers moved it from 2.6 to 4.0. The scoring was the binding constraint, not the statistics. Beyond that, the render pass, which would put the deliverable gate in front of every score.