A note is not a prompt
Making a good shot from scratch and taking a revision are different skills. Why every dimension is tested twice: Dailies and Notes.
On set, the first take is rarely the one that's printed. The director watches it and gives a note: "same thing, but slower." The crew changes that one thing and nothing else. Taking a note well is most of the job.
AI video models are usually tested on the first take only: here is a prompt, make a shot. But a filmmaker using these tools spends most of their time on the second step, trying to change one thing about a clip they almost like. That's where the frustration lives. Change the move, and the actor's wardrobe changes too. Warm the light, and the room rearranges.
Two tests for every dimension
So FinalGrade tests every dimension two ways:
- Dailies. A fresh shot from a written setup. Can the model make the shot as directed?
- Notes. A source clip plus one note, the way a director gives it on set. Can the model change only what was asked, and hold everything else?
Two rules keep the Notes test honest. Source clips are rotated across labs, and no model is ever asked to revise its own footage. And no source clip is reused across Notes tests, so evaluators aren't comparing against something they've already seen.
The notes themselves are written exactly as they would be said: "Split light him." "Go handheld." "Rack focus to the cup, let her go soft." No extra description, no reference images.
Revisions are much harder
The first finding is simply how much harder revisions are.
Camera
Lighting
Across all models, evaluators judged that the direction was delivered on 43% of fresh shots and 23% of revisions. The quick yes/no checks tell the same story: 58% passed on fresh shots, 53% on revisions. And on revisions, the "nothing else changed" check passed only 58% of the time. In more than four out of ten edits, the model changed something it wasn't asked to.
And the rankings flip
The second finding is that the two tests produce different leaderboards.
The model that ranked first on fresh shots dropped to the middle of the pack on revisions. A model that ranked near the bottom on fresh shots ranked near the top at taking notes. Another went the other way, from third on fresh shots to last on revisions.
These are different skills. Generating a shot rewards a model's sense of what a good image looks like. Revising one rewards restraint: understanding which part of the image the note is about and leaving the rest alone. Some models have one and not the other.
By craft
The gap between fresh shots and revisions holds in both crafts we've tested:
| Craft | Fresh shots delivered | Revisions delivered | Revisions where something else changed |
|---|---|---|---|
| Camera | 42% | 22% | 44% |
| Lighting | 45% | 25% | 38% |
Camera revisions were the most likely to disturb the rest of the shot. That makes sense: a camera note changes the frame itself, and models seem to treat a new frame as license to re-render everything in it.
What a good revision looks like
When evaluators did judge a revision delivered, it had three properties. The note was executed precisely, not approximately. The subject, wardrobe, set and framing held, so the revised clip could be cut against the original. And the change looked like it was made on set, with physically plausible light and movement, not painted on afterward. That combination is rare today, and it is the one professionals need most.
What this means for leaderboards
A single leaderboard that blends the two would hide all of this. It can also be unfair: not every model supports revisions, and a model scored only on fresh shots shouldn't be ranked alongside models scored on both as if they had taken the same test.
That is why FinalGrade reports fresh shots and revisions separately on every benchmark page, and marks models that were tested on fresh shots only. For a production team choosing a tool, the Notes number is often the one that matters most.
Early results from round 1. Numbers will move as more evaluators finish.