Research, methodology and perspective on how AI video measures up to the people who make film.
A benchmark built on expert judgment only works if experts converge. Early grading says they do, more than chance, and shows where the prompts need work.
Read the report →Lens character, motivated light and focus fall-off rarely register with general raters. The details that make an image professional are the ones least likely to be measured today.
Read the report →Give a model one note and it rewrites the shot. In early grading, evaluators flagged unrequested changes on a third to nearly half of all edits.
Read the report →Making a good shot from scratch and taking a revision are different skills. Why every dimension is tested twice: Dailies and Notes.
Read the report →Working professionals at the top of their craft. Invitation only, always anonymous, opt in to every round, free to work with anyone.
Read the report →Camera and Lighting launch first. Five more crafts are on the way, from production design to stunts, each graded by the people who do the work.
Read the report →Key side, ratio, color temperature, motivation and practicals. Graded in the terms a gaffer uses on set.
Read the report →13 dimensions, from shot size to lens character. One example prompt for each, written the way a DP would say it.
Read the report →A prompt that tests everything at once tells a lab nothing it can act on. Isolating one control turns a failure into a fixable note.
Read the report →Every model's clip side by side, labels hidden. Graders rank them, draw the line between delivered and not, and write the notes a lab can act on.
Read the report →Crowd ratings and automated scores can't tell a casual viewer from a DP. So labs never see where their models fail working professionals.
Read the report →Professionals reword prompts until they hit the model's phrasing. The tools should learn the craft's vocabulary, not the other way around.
Read the report →Full per-question results, the clips and the panel's written notes, shared under confidentiality, or an evaluation of an unreleased model. Experts always stay anonymous.