HeyGen launches a benchmark for AI video that can grade bad taste

Why it matters

Coding agents already produce valid interfaces and animations. HeyGen is betting that a specialized visual judge can teach them the harder part: timing, composition and taste.

HeyGen, the AI video company founded by Joshua Xu (@joshua_xu_) and Wayne Liang, has released Code2Video, a benchmark designed to measure whether coding agents can produce motion graphics that look intentional rather than merely compile.

https://x.com/HyperFrames_/status/2102084939026247684

The benchmark report published September 18th was written by Abhay Zala, Miao Yang, Wenbo Zhu, Bin Liu and the Kaggle team. HyperFrames (@HyperFrames_) promoted the release on September 21st, arguing that software-engineering benchmarks do not capture the visual judgment required to turn code into watchable video.

Xu started HeyGen after working on advertising, machine learning and camera products at Snap. The Code2Video release extends that original bet on lowering video-production costs into a narrower technical question: whether an AI agent that already knows HTML, CSS and JavaScript can also learn pacing, hierarchy and motion.

Code that runs can still make a bad video

Code2Video contains 168 human-authored motion-graphics briefs covering common sections of product-launch videos, including hooks, product introductions, feature demonstrations, social proof and calls to action. An agent receives a brief, writes a composition and renders it through HyperFrames, HeyGen's open-source HTML-to-video framework.

HyperFrames treats HTML as the editable source for a video and uses timing attributes and seekable animation runtimes to render frames deterministically. The project had roughly 52,000 GitHub stars on September 21st, giving HeyGen a meaningful developer distribution channel for an approach that turns video production into a coding task.

The benchmark targets the gap between functional output and competent motion design. HeyGen's examples include text colliding during transitions, titles clipping at frame edges, animations freezing before a scene finishes and layouts that satisfy the written brief while looking small or inert.

Those failures are hard to capture with a code linter, which can establish that a composition executes without deciding whether it looks good. HeyGen also found that general-purpose vision-language models and established video metrics could prefer clips that human raters rejected.

In one comparison, five human evaluators unanimously selected one generated clip while a vision-language model panel favored the other. HeyGen says its traditional CLIPScore and FVD measurements barely separated the pair, even though the losing clip placed its interface low and partly outside the frame.

HeyGen built a judge for its own test

HeyGen trained a separate judge model to compare two videos against the same prompt across five axes: engagement, prompt intent, composition, temporal quality and craft. The judge produces a preference probability for each axis rather than one blended quality score.

According to HeyGen, the judge agreed with human preferences on 82% of 1,464 evaluation samples, compared with 75% for the general-purpose vision-language models it tested. HeyGen also says agreement rose above 95% when the judge expressed stronger confidence. Those figures are company-reported, and the report does not yet provide the model architecture or enough training detail to reproduce the judge independently.

That limitation matters because HeyGen controls most of the evaluation stack. It created HyperFrames, supplied the agent skill used to generate the videos and trained the model that grades the results. Code2Video therefore measures performance inside HeyGen's method for programmatic video production. It does not establish a universal ranking for every video agent, renderer or creative workflow.

Kaggle gives the project a stronger audit trail. HeyGen said every reported model run executed on Kaggle, which stores the raw traces and logs and will maintain the public Code2Video leaderboard as new models are added. HeyGen also released the design briefs, reference compositions and judge API through the partnership.

The models follow instructions and still lose to people

HeyGen ran the agents across all 168 prompts and used round-robin pairwise comparisons to produce Elo rankings. The leading models were tightly grouped, with the top four separated by 12 Elo points and overlapping confidence intervals, according to the report. Open-weight models also finished within 50 Elo points of the leader.

The wider result was less flattering. Nearly every model scored close to zero when compared directly with the human-authored reference videos. The agents generally reproduced requested text, colors and scene order, then fell short on composition, animation timing and finish.

The benchmark team found recurring errors: text arranged for its final position collides while moving, elements animate at the same time instead of forming a visual sequence, and models reduce the scale of uncertain compositions until the output is technically readable and visually flat.

Wenbo Zhu, HeyGen's agent research lead and a Code2Video co-author, previously led AI research at OpusClip and worked on machine-learning products at ByteDance. His current remit spans HyperFrames and HeyGen's Video Agent, placing the benchmark directly inside HeyGen's product-development loop rather than alongside it as an academic exercise.

HeyGen plans to use the judge as a feedback signal for agents that review and refine their own work. That is the larger commercial bet behind Code2Video: a grading model that can reliably identify weak timing or composition could reduce the human editing that still separates an agent-generated draft from a usable launch video.