How we grade · FinalGrade
FINALGRADE
Methodology

How we grade

Every model's clip side by side, labels hidden. Graders rank them, draw the line between delivered and not, and write the notes a lab can act on.

A single score out of ten tells you very little. Two people rarely mean the same thing by a 7, and the same person rarely means the same thing by a 7 on Monday and on Friday. So FinalGrade never asks an evaluator to score a clip on its own. We ask them to compare.

Figure 1One question, from prompt to score
01PromptOne direction, in the craft's terms
02Every modelSame prompt, public interface, no tuning
03Blind, side by sideLabels hidden, order shuffled
04Rank and lineBest to worst, then delivered or not
05Checks and notes3 yes/no checks, notes on the frame
06Score100 if always first, 0 if always last
Several graders answer every question independently. They never see which model made a clip or how other graders answered.

The setup

Every model receives the same prompt, through its public interface, with no tuning per model. We don't rewrite prompts to suit a model, and we don't cherry-pick generations.

The evaluator sees every model's result for that question side by side, with labels hidden and the order shuffled. A fresh-shot question shows seven clips. A revision question shows five, plus the source clip being revised. Evaluators never see which model made which clip.

The ranking and the line

For the one direction the question tests, the evaluator ranks the clips from best to worst. Then they draw a line: clips above it delivered the direction, clips below it didn't.

Figure 2What a grader does on one question
SCENE: A MAN IN HIS SIXTIES SITS BY A WINDOW IN A QUIET STUDY · DIRECTION: "SLOW DOLLY IN" CRANK 1100 PTSFRANK 283 PTSARANK 367 PTSGRANK 450 PTSDRANK 533 PTSERANK 617 PTSBRANK 70 PTS DELIVERED THE DIRECTIONDID NOT DELIVERTHE LINE
Clips are labelled by letter only. Rank sets the score; the line sets whether a clip "took the note." Checks and notes are added per clip.

That line is what lets us say not just which model did best, but whether any of them did the job. It matters more than it sounds. In round 1, the clip an evaluator ranked first was above their line 87% of the time. In the other 13%, the best clip still wasn't good enough. On 11 of 82 graded answers, the evaluator judged that no model delivered the direction at all.

The checks

For each clip, the evaluator answers three quick yes/no checks written for that direction. For "split light him": one half of his face is lit and the other is black; the light falls off at the line of the nose; the key comes from 90 degrees to the side. On revisions they answer one more: other than the note, did anything else change?

The checks and the rankings are two independent ways of measuring the same clip. In round 1 they agreed (correlation 0.50 between checks passed and rank). That is evidence the instrument is measuring something real, not just taste.

The notes

Evaluators write improvement and defect notes, marked directly on the frame at the moment and position they describe. Round 1 produced 369 written notes: 271 improvement notes and 98 defect notes. The notes matter as much as the ranking. They tell a lab what to change, not just that something is off.

The panel

Several evaluators rank every question independently. They don't see each other's answers and don't discuss the work. They are working professionals in the craft they judge, paid for their time, and always anonymous. They take their time: the median clip was watched for about 45 seconds, often several times over.

A worked example

Take a single revision question from round 1. The source clip shows a man at a desk, lit softly from the front. The note is four words: "Split light him."

The evaluator sees the source clip and five revisions, labeled only by letter, in a shuffled order. They watch each clip as many times as they need. For each one, they answer four checks: is one half of his face lit and the other black; does the light fall off at the line of the nose; does the key come from 90 degrees to the side; and, other than the note, has nothing else changed? They mark anything that went wrong directly on the frame, for example the moment his expression freezes or a lamp appears that wasn't in the source.

Then they rank the five clips from best to worst at delivering split light, and draw their line. In round 1, split light as a revision was delivered in 7% of graded clips, so on most answers the line sat at or near the top.

Why ranking beats rating

People are much better at comparing than at scoring in the abstract. Asked to rate a clip from 1 to 10, two experts can agree completely on its quality and still give it a 6 and an 8. Asked which of two clips is better, they agree far more often. Ranking also forces a decision on close calls, where ratings tend to cluster in the middle. And because every model is compared on the same question by the same evaluator, differences in how strict each evaluator is cancel out.

The score

A clip ranked first scores 100 and last scores 0, with the places in between spread evenly. Scores are averaged across questions and evaluators. Because every question averages to 50, the score reads directly as how often a model beats the field.

We report a 95% range for every score, computed by resampling questions. When ranges overlap, we say the models are statistically tied rather than pretend the order is meaningful.

We also report how closely evaluators agreed, per question. Where working professionals don't converge on a question, we flag it and rewrite it rather than trust the number.

What we exclude

The test's designer grades too, to check the instrument, but those answers are excluded from every published score. Practice questions are excluded. A model is only compared on the questions it actually took: models that don't support revisions are scored on fresh shots only, and marked that way.

Frequently asked questions

Do evaluators know which models are being tested? They may know which models exist, but they never know which model made which clip. Labels are hidden and the order is shuffled per question.

Is every model run with the same settings? Every model gets the same prompt through its public interface, with default settings unless the prompt itself specifies something like an aspect ratio. We don't tune prompts per model.

What happens when a model refuses or fails to generate? The clip is recorded as a failure for that question and ranked accordingly.

How many evaluators grade each question? Round 1 averaged about 2.7 per question. The full round targets five.

What's published

Rankings, ranges and methodology are public. Clips, notes and the detailed rubric are shared with partners under confidentiality. Evaluators' names are never shared.

Keep reading