Edit models change what they weren't asked to · FinalGrade
FINALGRADE
Research

Edit models change what they weren't asked to

Give a model one note and it rewrites the shot. In early grading, evaluators flagged unrequested changes on a third to nearly half of all edits.

A cinematographer's note is surgical. "Same shot, move the key to the other side" means exactly that: the actor, the wardrobe, the frame and the room stay put. Only the light moves.

That is what makes notes useful. A director can build a shot over several takes, keeping what works and changing what doesn't. A tool that can't hold the rest of the shot can't be directed this way. Every note becomes a gamble.

What we measured

In FinalGrade's Notes test, a model gets an existing clip and one note, written the way it would be said on set. Evaluators see the source clip next to every model's revision. For each revision, they answer the usual direction checks plus one more: other than the note, did anything else change?

What we found

In our first round, every model struggled with this.

Figure 1On revisions, unrequested changes outpace the requested one
Something else changedTook the note
Gemini Omni
48%
18%
Grok Imagine
48%
30%
Wan
48%
22%
Seedance
35%
28%
Kling
32%
18%
Revisions (Notes test), camera and lighting combined, round 1. Share of graded edits where graders marked that something other than the note changed, and share ranked above the grader's line. Preliminary, 3 graders.

Evaluators marked "something else changed" on 32% to 48% of edits, depending on the model. They judged that the note itself was delivered in only 18% to 30% of revisions. For every model, an unrequested change was more likely than a delivered note.

Across all models, the "nothing else changed" check passed on 58% of revisions. Put the other way: in roughly four out of ten edits, the model altered part of the shot it was told to leave alone.

What the notes describe

Evaluators don't just tick a box. They write defect notes, marked on the frame at the moment and place the problem appears. Round 1 produced 98 defect notes across both tests, and they describe the same failures again and again.

Figure 2What the 98 defect notes describe
Objects appear, vanish or change
19
Camera moves nobody asked for
16
Jitter, glitches, artifacts
14
Unnatural motion or physics
12
Focus or sharpening wrong
11
Light moves or comes from the wrong place
5
Other
21
Defect notes from round 1, grouped by keyword into one theme each. Notes are written in the graders' own words and marked on the frame.
  • Objects that appear, vanish or change between the source and the edit: cars and lights added to the background, details on a set changing.
  • Camera moves nobody asked for: a note about handheld feel that produces a pan, or a lighting note that also reframes the shot.
  • Jitter, glitches and artifacts: ghosting, frozen frames, posterized faces, jittery motion.
  • Unnatural motion and physics: movement no camera operator or actor could produce.
  • Focus that ignores the scene: subjects artificially sharpened, backgrounds blurred or sharpened regardless of depth.
  • Light that moves or comes from the wrong place: the sun angle shifting, a light moving when the note didn't ask for it.

What it looks like note by note

The unrequested changes depend on the kind of note.

  • Camera notes were the most disruptive. Asked simply to "go handheld," models often added a pan or a reframe that nobody asked for. Asked for a slow push in, some re-rendered the background or the subject's position.
  • Lighting notes often moved more than the light. Asked to make the light hard "same position," models moved the source, changed the background or added sources.
  • Focus notes frequently produced a new shot instead of a new focus plane: the composition shifted, or the subject was sharpened artificially rather than the background going soft.

What a passing revision requires

For a revision to pass on FinalGrade, it has to clear two bars at once:

  1. The note is delivered, precisely, as judged against the checks for that direction.
  2. Everything else holds: the subject, wardrobe, set, framing and lighting not mentioned in the note.

A revision that delivers the note and changes the wardrobe fails. A revision that holds everything but doesn't deliver the note also fails. Only the combination is usable on a real production.

Why this is the gap between a demo and a tool

To a casual viewer, many of these edits still look fine as standalone clips. To a professional, they're unusable: a shot that changes in ways you didn't ask for can't be cut against the rest of the scene. If the wardrobe changes, the scene breaks continuity. If the camera drifts, the shot no longer matches its coverage.

The models are treating a note as a fresh render, and that is the gap between a demo tool and a production tool.

What labs can do with this

The fix starts with measuring it separately. A model's fresh-shot quality and its ability to hold a shot under revision are different properties, and FinalGrade reports them separately. Partners receive every flagged change, marked on the frame, so their teams can see exactly what moved.

Early results from a panel still in progress. Numbers will move as more evaluators finish.

Keep reading