Do experts agree? Early signal from our first panel
A benchmark built on expert judgment only works if experts converge. Early grading says they do, more than chance, and shows where the prompts need work.
The question we get most often is a fair one: isn't this just taste? If three cinematographers look at the same clips, won't they rank them three different ways?
If they did, an expert benchmark would be no better than a crowd. So we measure agreement on every question, and we publish it.
How we measure agreement
Evaluators graded every question independently, blind to the model and to each other. For each question, we compare each pair of evaluators' rankings and compute a rank correlation: 0 means their rankings are no more alike than chance, 1 means they are identical. We also check a simpler question: did they pick the same best clip?
The headline numbers
- They picked the same best clip on 30% of questions, against 7% if they were choosing at random.
- The closest pair of evaluators correlated at 0.54, on a scale where 0 is chance and 1 is identical. Across the whole panel the average was 0.34: moderate agreement, not unanimity, and real signal.
- Agreement was higher on fresh shots than on revisions (0.42 versus 0.26), and higher on camera movement (0.42) and lighting (0.38) than on focus and depth of field (0.21).
- The quick yes/no checks tracked the rankings (correlation 0.50). Two different ways of measuring the same clip pointed the same way.
By test type
By control
Question by question
Averages hide a lot. Here is agreement on each of the 30 questions in round 1.
Most questions sit well to the right of zero, and fresh-shot questions cluster higher than revisions. A handful sit at or below zero, where evaluators ranked the clips differently.
What disagreement tells us
The low spots are useful. The questions where evaluators disagreed most were ones where the direction was ambiguous or the clips were nearly identical: a request for "deep focus" on an existing clip, a hard light that "holds position," a background thrown out of focus. In those cases, every model's clip often looked alike, or the direction allowed more than one reasonable reading.
Those prompts get rewritten before the full round. Disagreement among experts is a finding, and often it points back at the question, not the evaluators.
Pairs of evaluators
Agreement wasn't uniform across the panel. The closest pair of evaluators correlated at 0.54. The third evaluator agreed less with each of the other two, at 0.29 and 0.25. That isn't necessarily a problem: a different eye on the same clips is part of why panels have several members. It is a reason to read that evaluator's notes on the questions where they diverged, and to add more evaluators per question so no single view dominates.
A second instrument
Rankings aren't our only measure. On every clip, evaluators also answer three yes/no checks written for that direction. The checks are concrete ("the light falls off at the line of the nose") and leave little room for taste. In round 1, clips that passed more checks were ranked higher, with a correlation of 0.50. When two independent ways of measuring point the same way, that is good evidence the benchmark is measuring the work and not the evaluator.
Why moderate agreement is the right result
Perfect agreement would be suspicious. It would suggest the questions are too easy, or that evaluators are anchoring on each other. Real expert judgment has some spread, especially on close calls. What matters is that experts converge well above chance and that the convergence is strongest where the direction is clearest. Both are true.
What changes for the full round
- Five evaluators per question, up from about 2.7, so each result rests on more judgments.
- Rewritten questions where agreement was low, starting with deep focus, hard light that holds position, and throwing the background out of focus.
- More calibration on the dimensions with the lowest agreement, focus and depth of field, before the round opens.
What happens next
Round 1 had about 2.7 evaluators per question. The full round targets five. With more evaluators per question, single disagreements matter less, ranges narrow and the scores stabilize. We'll publish the full agreement numbers when the round closes, including where experts disagree.
Early results from a panel still in progress.