Edit models change what they weren't asked to
Give a model one note and it rewrites the shot. In early grading, evaluators flagged unrequested changes on a third to nearly half of all edits.
A cinematographer's note is surgical. "Same shot, move the key to the other side" means exactly that: the actor, the wardrobe, the frame and the room stay put. Only the light moves.
That is what makes notes useful. A director can build a shot over several takes, keeping what works and changing what doesn't. A tool that can't hold the rest of the shot can't be directed this way. Every note becomes a gamble.
What we measured
In FinalGrade's Notes test, a model gets an existing clip and one note, written the way it would be said on set. Evaluators see the source clip next to every model's revision. For each revision, they answer the usual direction checks plus one more: other than the note, did anything else change?
What we found
In our first round, every model struggled with this.
Evaluators marked "something else changed" on 32% to 48% of edits, depending on the model. They judged that the note itself was delivered in only 18% to 30% of revisions. For every model, an unrequested change was more likely than a delivered note.
Across all models, the "nothing else changed" check passed on 58% of revisions. Put the other way: in roughly four out of ten edits, the model altered part of the shot it was told to leave alone.
What the notes describe
Evaluators don't just tick a box. They write defect notes, marked on the frame at the moment and place the problem appears. Round 1 produced 98 defect notes across both tests, and they describe the same failures again and again.
- Objects that appear, vanish or change between the source and the edit: cars and lights added to the background, details on a set changing.
- Camera moves nobody asked for: a note about handheld feel that produces a pan, or a lighting note that also reframes the shot.
- Jitter, glitches and artifacts: ghosting, frozen frames, posterized faces, jittery motion.
- Unnatural motion and physics: movement no camera operator or actor could produce.
- Focus that ignores the scene: subjects artificially sharpened, backgrounds blurred or sharpened regardless of depth.
- Light that moves or comes from the wrong place: the sun angle shifting, a light moving when the note didn't ask for it.
What it looks like note by note
The unrequested changes depend on the kind of note.
- Camera notes were the most disruptive. Asked simply to "go handheld," models often added a pan or a reframe that nobody asked for. Asked for a slow push in, some re-rendered the background or the subject's position.
- Lighting notes often moved more than the light. Asked to make the light hard "same position," models moved the source, changed the background or added sources.
- Focus notes frequently produced a new shot instead of a new focus plane: the composition shifted, or the subject was sharpened artificially rather than the background going soft.
What a passing revision requires
For a revision to pass on FinalGrade, it has to clear two bars at once:
- The note is delivered, precisely, as judged against the checks for that direction.
- Everything else holds: the subject, wardrobe, set, framing and lighting not mentioned in the note.
A revision that delivers the note and changes the wardrobe fails. A revision that holds everything but doesn't deliver the note also fails. Only the combination is usable on a real production.
Why this is the gap between a demo and a tool
To a casual viewer, many of these edits still look fine as standalone clips. To a professional, they're unusable: a shot that changes in ways you didn't ask for can't be cut against the rest of the scene. If the wardrobe changes, the scene breaks continuity. If the camera drifts, the shot no longer matches its coverage.
The models are treating a note as a fresh render, and that is the gap between a demo tool and a production tool.
What labs can do with this
The fix starts with measuring it separately. A model's fresh-shot quality and its ability to hold a shot under revision are different properties, and FinalGrade reports them separately. Partners receive every flagged change, marked on the frame, so their teams can see exactly what moved.
Early results from a panel still in progress. Numbers will move as more evaluators finish.