Who grades the work
Working professionals at the top of their craft. Invitation only, always anonymous, opt in to every round, free to work with anyone.
A benchmark is only as good as the people judging it. So the most important decision we make is who sits on the panel.
Working professionals only
Every evaluator works in the craft they judge, with credits on features, series or commercial work. Camera is evaluated by cinematographers and camera operators. Lighting is evaluated by lighting professionals and cinematographers. Nobody grades a craft they don't practice.
Panels are invitation only. Evaluators are invited on their credits and their standing with peers, not through an open sign-up. The round 1 panel brought 19 to 25 years of experience each.
Why seniority matters
Senior professionals see things others don't. They have stood next to the monitor for thousands of takes. They know the difference between a 35mm and a 50mm at a glance, and they know which mistakes a production can live with and which ones force a reshoot. That judgment is exactly what crowd ratings can't supply.
How the panel is protected
The way a panel is run matters as much as who is on it.
Always anonymous. Evaluators are never named publicly and their names are never shared with labs, partners or clients. What we publish is the panel's judgment, not anyone's identity.
Opt in, every round. No one is committed beyond the round they accept. Panelists see what they'll be grading and decide whether to take part.
Paid properly. Panelists are paid for their time at a published day rate, the same for everyone, within seven days. Calibration time is paid too. See how panelists are paid.
Free to work with anyone. Evaluating for FinalGrade doesn't restrict who panelists work with, including the companies whose models they judge.
Only machine output. Every clip graded is AI-generated. No panelist's own footage, likeness or past work is used.
Blind and independent. Evaluators never see which model made which clip, and they don't see or discuss each other's answers. Where their judgments diverge, we report that.
How evaluators are selected
Selection follows four steps:
- Invitation or application. Most evaluators are recommended by current panelists or by peers in their craft. A small number apply directly.
- Credits review. We review credits and experience against the bar for the craft: current work at a senior level on features, series or major commercial projects.
- Calibration. A short, paid set of practice questions in Playback, our grading app. Calibration confirms that an evaluator's notes are specific enough to act on. Calibration answers never count toward results.
- First round. Evaluators see what a round involves before they accept it.
Independence
Evaluators may work with AI companies, including those whose models we test. We ask them to tell us about current engagements, but it doesn't disqualify them: grading is blind, so they can't favor a model they can't identify. Their answers are compared against other evaluators', and divergence is reported, not hidden.
What the panel produced in round 1
In the first round, the panel produced 494 blind clip rankings and 369 written notes, each marked on the frame at the moment it describes. Evaluators took the time the work needs: the median clip was watched for about 45 seconds.
Joining a panel
If you work at the top of camera, lighting, design, costume, sound, score or stunts and want to be considered for a future round, apply to join a panel.