Who grades the work · FinalGrade
FINALGRADE
Panel

Who grades the work

Working professionals at the top of their craft. Invitation only, always anonymous, opt in to every round, free to work with anyone.

A benchmark is only as good as the people judging it. So the most important decision we make is who sits on the panel.

Figure 1The round 1 panel, by the numbers
3graders in round 1, more joining
19 to 25years in the craft
Local 600International Cinematographers Guild members on the panel
494blind clip rankings
369written notes, marked on the frame
Round 1, camera and lighting. Graders: cinematographers and camera operators with feature, series and commercial credits.

Working professionals only

Every evaluator works in the craft they judge, with credits on features, series or commercial work. Camera is evaluated by cinematographers and camera operators. Lighting is evaluated by lighting professionals and cinematographers. Nobody grades a craft they don't practice.

Panels are invitation only. Evaluators are invited on their credits and their standing with peers, not through an open sign-up. The round 1 panel brought 19 to 25 years of experience each.

Why seniority matters

Senior professionals see things others don't. They have stood next to the monitor for thousands of takes. They know the difference between a 35mm and a 50mm at a glance, and they know which mistakes a production can live with and which ones force a reshoot. That judgment is exactly what crowd ratings can't supply.

How the panel is protected

The way a panel is run matters as much as who is on it.

Figure 2How a clip travels, and what never does
Model labsAI CLIPS ONLY FinalGradeLABELS REMOVED The panelBLIND, INDEPENDENT CLIPSSHUFFLED RANKS, NOTESRESULTS PANELIST NAMES, FOOTAGE OR LIKENESS: NEVER SHARED
Graders never see which model made a clip. Labs never receive a grader's name. No grader's own work is used.

Always anonymous. Evaluators are never named publicly and their names are never shared with labs, partners or clients. What we publish is the panel's judgment, not anyone's identity.

Opt in, every round. No one is committed beyond the round they accept. Panelists see what they'll be grading and decide whether to take part.

Paid properly. Panelists are paid for their time at a published day rate, the same for everyone, within seven days. Calibration time is paid too. See how panelists are paid.

Free to work with anyone. Evaluating for FinalGrade doesn't restrict who panelists work with, including the companies whose models they judge.

Only machine output. Every clip graded is AI-generated. No panelist's own footage, likeness or past work is used.

Blind and independent. Evaluators never see which model made which clip, and they don't see or discuss each other's answers. Where their judgments diverge, we report that.

How evaluators are selected

Selection follows four steps:

  1. Invitation or application. Most evaluators are recommended by current panelists or by peers in their craft. A small number apply directly.
  2. Credits review. We review credits and experience against the bar for the craft: current work at a senior level on features, series or major commercial projects.
  3. Calibration. A short, paid set of practice questions in Playback, our grading app. Calibration confirms that an evaluator's notes are specific enough to act on. Calibration answers never count toward results.
  4. First round. Evaluators see what a round involves before they accept it.

Independence

Evaluators may work with AI companies, including those whose models we test. We ask them to tell us about current engagements, but it doesn't disqualify them: grading is blind, so they can't favor a model they can't identify. Their answers are compared against other evaluators', and divergence is reported, not hidden.

What the panel produced in round 1

In the first round, the panel produced 494 blind clip rankings and 369 written notes, each marked on the frame at the moment it describes. Evaluators took the time the work needs: the median clip was watched for about 45 seconds.

Joining a panel

If you work at the top of camera, lighting, design, costume, sound, score or stunts and want to be considered for a future round, apply to join a panel.

Keep reading