Do experts agree? Early signal from our first panel · FinalGrade
FINALGRADE
Methodology

Do experts agree? Early signal from our first panel

A benchmark built on expert judgment only works if experts converge. Early grading says they do, more than chance, and shows where the prompts need work.

The question we get most often is a fair one: isn't this just taste? If three cinematographers look at the same clips, won't they rank them three different ways?

If they did, an expert benchmark would be no better than a crowd. So we measure agreement on every question, and we publish it.

How we measure agreement

Evaluators graded every question independently, blind to the model and to each other. For each question, we compare each pair of evaluators' rankings and compute a rank correlation: 0 means their rankings are no more alike than chance, 1 means they are identical. We also check a simpler question: did they pick the same best clip?

The headline numbers

Figure 1Experts converge far more than chance
30%of questions: graders independently picked the same best clip
7%expected if they were picking at random
Round 1, 30 questions, 3 graders, blind to the model and to each other.
  • They picked the same best clip on 30% of questions, against 7% if they were choosing at random.
  • The closest pair of evaluators correlated at 0.54, on a scale where 0 is chance and 1 is identical. Across the whole panel the average was 0.34: moderate agreement, not unanimity, and real signal.
  • Agreement was higher on fresh shots than on revisions (0.42 versus 0.26), and higher on camera movement (0.42) and lighting (0.38) than on focus and depth of field (0.21).
  • The quick yes/no checks tracked the rankings (correlation 0.50). Two different ways of measuring the same clip pointed the same way.
Figure 2Where agreement is strongest

By test type

Fresh shots
0.42
Revisions
0.26

By control

Camera movement
0.42
Lighting
0.38
Focus and DOF
0.21
Closest pair of graders
0.54
Panel average
0.34
Checks vs rankings
0.50
Rank correlation, round 1. "Checks vs rankings" is the correlation between how many yes/no checks a clip passed and where it was ranked.

Question by question

Averages hide a lot. Here is agreement on each of the 30 questions in round 1.

Figure 3Agreement on each of the 30 questions
FRESHREVISION GRADERS DISAGREEGRADERS AGREE PANEL AVERAGE 0.34 Deep focus: bring the office behind her into focus too. (-0.80)Make it hard light. Same position. (-0.27)Throw the background out of focus, keep him sharp. (-0.17)Go wide open: keep him sharp, let the glass in front and the bar behind fall off. (-0.13)Split light him. (-0.07)Slow pull back from him to reveal the study (0.02)Locked off on the barber, then a whip pan to the customer waiting by the door, landing clean and holding (0.05)Rack focus from the near player to the far player as he moves a piece (0.10)Slow push in on him. (0.10)Razor-thin focus: only the bolt head and fingertips are sharp (0.12)Slow push in on him and rest on a frame where he fills it. (0.30)Pan with him. (0.30)Loop light him. (0.30)Hard daylight through the blinds (0.36)The work light as an underlight (0.43)Very shallow focus, the lights behind him fall to large soft discs (0.43)Focus preset on the near lockers where she stops; she runs into focus (0.48)Late-afternoon sun as a hard rim from the window behind him, no key on his face (0.48)Rack focus to the cup, let her go soft. (0.50)Late afternoon. Hard low sun through the window panes as a cookie pattern across her, three-quarter backlight, bounce off the wooden floor as the only fill (0.52)Handheld, following her from behind at the same distance (0.57)Firelight only (0.59)Make it golden hour: low warm sun pouring through the window behind her, rim light on her hair, long shadows across the table. (0.60)Deep focus, sharp front to back (0.64)Locked off, she walks into the frame (0.67)Go handheld. (0.73)Slow tilt up from her hands to her face. (0.73)Tracking alongside him as he carves the wall (0.77)Light him Rembrandt. (0.90)Shift focus to the shelves behind him, let him go soft. (0.93)-1-0.500.51
Mean pairwise rank correlation per question, round 1. 0 is chance, 1 is identical rankings. The questions on the left are being rewritten for the full round.

Most questions sit well to the right of zero, and fresh-shot questions cluster higher than revisions. A handful sit at or below zero, where evaluators ranked the clips differently.

What disagreement tells us

The low spots are useful. The questions where evaluators disagreed most were ones where the direction was ambiguous or the clips were nearly identical: a request for "deep focus" on an existing clip, a hard light that "holds position," a background thrown out of focus. In those cases, every model's clip often looked alike, or the direction allowed more than one reasonable reading.

Those prompts get rewritten before the full round. Disagreement among experts is a finding, and often it points back at the question, not the evaluators.

Pairs of evaluators

Agreement wasn't uniform across the panel. The closest pair of evaluators correlated at 0.54. The third evaluator agreed less with each of the other two, at 0.29 and 0.25. That isn't necessarily a problem: a different eye on the same clips is part of why panels have several members. It is a reason to read that evaluator's notes on the questions where they diverged, and to add more evaluators per question so no single view dominates.

A second instrument

Rankings aren't our only measure. On every clip, evaluators also answer three yes/no checks written for that direction. The checks are concrete ("the light falls off at the line of the nose") and leave little room for taste. In round 1, clips that passed more checks were ranked higher, with a correlation of 0.50. When two independent ways of measuring point the same way, that is good evidence the benchmark is measuring the work and not the evaluator.

Why moderate agreement is the right result

Perfect agreement would be suspicious. It would suggest the questions are too easy, or that evaluators are anchoring on each other. Real expert judgment has some spread, especially on close calls. What matters is that experts converge well above chance and that the convergence is strongest where the direction is clearest. Both are true.

What changes for the full round

  • Five evaluators per question, up from about 2.7, so each result rests on more judgments.
  • Rewritten questions where agreement was low, starting with deep focus, hard light that holds position, and throwing the background out of focus.
  • More calibration on the dimensions with the lowest agreement, focus and depth of field, before the round opens.

What happens next

Round 1 had about 2.7 evaluators per question. The full round targets five. With more evaluators per question, single disagreements matter less, ranges narrow and the scores stabilize. We'll publish the full agreement numbers when the round closes, including where experts disagree.

Early results from a panel still in progress.

Keep reading