What a crowd can't see · FinalGrade
FINALGRADE
Research

What a crowd can't see

Lens character, motivated light and focus fall-off rarely register with general raters. The details that make an image professional are the ones least likely to be measured today.

Some mistakes everyone can see: a hand with six fingers, a car that melts into a wall, a face that changes identity mid-shot. Those get fixed fast, because any rater catches them, and every crowd-preference leaderboard punishes them.

The mistakes that separate a professional image from an amateur one are quieter.

Figure 1Four details a crowd misses and a professional catches immediately
Lens characterNice blur either way.Round bokeh means it isn't anamorphic.
Motivated lightWarm, flattering face.The only lamp is on the right. Wrong.
Focus fall-offLooks cinematic.Blur doesn't grow with distance. Fake.
ContinuityTwo good shots.They can't cut together.
Illustrations of failure types graders flag in written notes.

Four details

Lens character. An anamorphic lens gives oval bokeh and horizontal flares, and a particular way of stretching the background. A model that's asked for anamorphic and returns round bokeh has failed, but most viewers will never notice. They see a nice blurry background either way.

Motivated light. A face lit warmly from the left, in a room whose only lamp is on the right, looks fine to a stranger. To a gaffer it's wrong the moment they see it. Motivation is what makes lighting believable: the audience doesn't consciously check it, but they feel it when it's off.

Focus fall-off. A shallow depth of field should fall off in a way that matches the lens and the distance. Objects far behind the subject should be softer than objects just behind it. Fake blur, applied evenly to everything behind the subject, reads as "cinematic" to a crowd and as fake to a cinematographer.

Continuity. The key light moves between a wide and its reverse, or the sun jumps sides. Nobody in a crowd study is watching for that. Every script supervisor is. Two shots that each look good but can't be cut together are useless in an edit.

Why it matters what gets measured

What gets measured gets improved. Model training follows the feedback it's given. If the only judges are general raters, models will keep getting better at the things anyone can see, and stay unreliable at the things professionals need.

The details above are not niche. They are what a director of photography is hired for. They are also what makes a shot usable in a real production, rather than impressive in a demo.

Can experts actually agree on this?

The obvious objection is that these details are subjective. If they were, a panel of experts would be no better than a crowd.

Our bet is that experts, judging one dimension at a time, agree on these details far more than chance. Early grading supports it.

Figure 2Experts converge far more than chance
30%of questions: graders independently picked the same best clip
7%expected if they were picking at random
Round 1, 30 questions, 3 graders, blind to the model and to each other.

In our first round, evaluators working independently and blind picked the same best clip on 30% of questions, against 7% if they were choosing at random. Agreement was highest on camera movement and lighting, and lowest on focus and depth of field, which tells us where our own prompts need to be sharper. We go deeper into the numbers in Do experts agree?

Why these details are hard for models

Part of the reason is training data. Most video used to train models is described in everyday language: "a woman by a window," "a man walking down a street." Almost none of it is labeled with the lens, the lighting setup or the focus distance, so models rarely learn the connection between those terms and what they look like. They learn what a pleasant image looks like, without learning how it was made.

The other part is evaluation. If the feedback a model gets comes from people who can't see these details, nothing pushes the model to get them right.

How we test each detail

Detail FinalGrade dimension Example direction What evaluators check
Lens character Camera: Lens character Anamorphic, oval bokeh, horizontal flares Bokeh shape, flare direction, background stretch
Motivated light Lighting: Motivation Motivate it from the window Key direction matches the source, color matches the source
Focus fall-off Camera: Depth of field Very shallow focus, the lights behind him fall to large soft discs Blur increases with distance, the subject's eyes are sharp
Continuity Lighting: Lighting continuity Same setup on the reverse Key side, ratio and color hold across the cut

What we're building toward

Each of the four details above maps to a dimension in FinalGrade's rubrics: lens character and depth of field in Camera, motivation and continuity in Lighting. Round 1 covered depth of field and focus. Lens character, motivation and continuity are being added in later rounds, graded the same way.

Keep reading