A note is not a prompt · FinalGrade
FINALGRADE
Research

A note is not a prompt

Making a good shot from scratch and taking a revision are different skills. Why every dimension is tested twice: Dailies and Notes.

On set, the first take is rarely the one that's printed. The director watches it and gives a note: "same thing, but slower." The crew changes that one thing and nothing else. Taking a note well is most of the job.

AI video models are usually tested on the first take only: here is a prompt, make a shot. But a filmmaker using these tools spends most of their time on the second step, trying to change one thing about a clip they almost like. That's where the frustration lives. Change the move, and the actor's wardrobe changes too. Warm the light, and the room rearranges.

Two tests for every dimension

So FinalGrade tests every dimension two ways:

  • Dailies. A fresh shot from a written setup. Can the model make the shot as directed?
  • Notes. A source clip plus one note, the way a director gives it on set. Can the model change only what was asked, and hold everything else?

Two rules keep the Notes test honest. Source clips are rotated across labs, and no model is ever asked to revise its own footage. And no source clip is reused across Notes tests, so evaluators aren't comparing against something they've already seen.

The notes themselves are written exactly as they would be said: "Split light him." "Go handheld." "Rack focus to the cup, let her go soft." No extra description, no reference images.

Revisions are much harder

The first finding is simply how much harder revisions are.

Figure 1How often the direction was delivered

Camera

Fresh shots
42%
Revisions
22%

Lighting

Fresh shots
45%
Revisions
25%
Share of clips ranked above the grader's line, all models, round 1.

Across all models, evaluators judged that the direction was delivered on 43% of fresh shots and 23% of revisions. The quick yes/no checks tell the same story: 58% passed on fresh shots, 53% on revisions. And on revisions, the "nothing else changed" check passed only 58% of the time. In more than four out of ten edits, the model changed something it wasn't asked to.

And the rankings flip

The second finding is that the two tests produce different leaderboards.

Figure 2Fresh shots and revisions are different races
Fresh shot (Dailies)Revision (Notes)
Wan3655Seedance5056Grok Imagine4251Kling6651Gemini Omni57370255075100
Camera and lighting combined, round 1. Score: 100 if a model's clip is always ranked first, 0 if always last. Five models support revisions; LTX and Hailuo were tested on fresh shots only. Preliminary, 3 graders.

The model that ranked first on fresh shots dropped to the middle of the pack on revisions. A model that ranked near the bottom on fresh shots ranked near the top at taking notes. Another went the other way, from third on fresh shots to last on revisions.

These are different skills. Generating a shot rewards a model's sense of what a good image looks like. Revising one rewards restraint: understanding which part of the image the note is about and leaving the rest alone. Some models have one and not the other.

By craft

The gap between fresh shots and revisions holds in both crafts we've tested:

Craft Fresh shots delivered Revisions delivered Revisions where something else changed
Camera 42% 22% 44%
Lighting 45% 25% 38%

Camera revisions were the most likely to disturb the rest of the shot. That makes sense: a camera note changes the frame itself, and models seem to treat a new frame as license to re-render everything in it.

What a good revision looks like

When evaluators did judge a revision delivered, it had three properties. The note was executed precisely, not approximately. The subject, wardrobe, set and framing held, so the revised clip could be cut against the original. And the change looked like it was made on set, with physically plausible light and movement, not painted on afterward. That combination is rare today, and it is the one professionals need most.

What this means for leaderboards

A single leaderboard that blends the two would hide all of this. It can also be unfair: not every model supports revisions, and a model scored only on fresh shots shouldn't be ranked alongside models scored on both as if they had taken the same test.

That is why FinalGrade reports fresh shots and revisions separately on every benchmark page, and marks models that were tested on fresh shots only. For a production team choosing a tool, the Notes number is often the one that matters most.

Early results from round 1. Numbers will move as more evaluators finish.

Keep reading