RoboDojo puts generalist robot policies through a tougher sim-and-real test

Why it matters

RoboDojo gives embodied AI teams a harder public yardstick for the gap between lab demos and reliable robot manipulation, especially on long-horizon and real-world tasks.

Tianxing Chen (@MarioChan2002) and a 44-author robotics group have released RoboDojo, a new open benchmark for testing generalist robot manipulation policies across simulation and physical robots, after evaluating 30 policies and finding that current systems remain far from reliable on real manipulation tasks.

Chen framed the release bluntly in a thread on X: "We evaluated 30+ frontier embodied AI models." The result, he wrote, was that current generalist robot policies are "still far from robust real-world manipulation." The accompanying arXiv paper, first submitted on July 5th and revised on July 8th, gives the claim more force: RoboDojo reports that the best average simulation success rate among the listed policies was 8.80%, while a human expert teleoperation baseline reached 76.03%.

Chen is not approaching this as a one-off benchmark release. His personal site lists a run of recent robotics benchmark and manipulation projects, including RoboTwin 2.0, RMBench and G3Flow, and identifies him as an organizer and technical leader for the 2025 RoboTwin Dual-Arm Collaboration Challenge. On the RoboDojo paper, Chen is listed as a co-first author and co-project leader, with the author roster spanning MMLab@HKU, UC Berkeley, Tsinghua University, Peking University, Stanford, MIT, UNC, Princeton, CMU, NUS, NTU, CUHK, Imperial College, SJTU, Northwestern, HKUST (GZ), Zhejiang University and Yale.

That background matters because RoboDojo is aimed at a specific failure mode in robot learning: benchmark results that look strong inside narrow evaluation setups and then fail once the robot has to deal with memory, long horizons, visual ambiguity, precise contact, or physical-world variation. The paper argues that many existing evaluations rely on simple, short-horizon, or skill-narrow tasks, and that testing only in simulation misses deployment problems while testing only on real robots is costly and hard to reproduce.

RoboDojo tries to close that gap with 42 simulation tasks across five dimensions: generalization, memory, precision, long-horizon execution and open-vocabulary instruction following. The benchmark documentation lists tasks such as solving equations, stacking blocks by language, inserting tubes, playing a xylophone, depositing coins and classifying objects by language. The real-world side adds 18 physical tasks across three robot embodiments, ARX X5, Piper and Piper X, including disassembling LEGO, capping a pen, hanging mugs, inserting a charger, making bread and packing objects into a backpack.

The headline result is the size of the gap. On the simulation leaderboard captured in the paper, Hy-Embodied-0.5-VLA led the listed policies with an average 13.07 score and 8.80% success rate. Spatial Forcing followed at 12.38 and 8.04%. Pi0.5 scored 11.41 with a 6.91% success rate. X-VLA scored 10.13 with a 6.52% success rate. The human teleoperation baseline scored 80.42 with a 76.03% success rate. In the paper's real-world table, Pi0.5 averaged 22.9 score and 12.8% success across the three real robot setups, while several models collapsed to near-zero performance on at least one embodiment.

Those figures need careful reading. RoboDojo is not claiming to measure every possible robot policy in every possible commercial setting. It is a benchmark designed by its authors, with tasks, scoring and hardware choices that define the contest. The useful point is narrower and sharper: under one standardized sim-and-real protocol, the evaluated policies struggled most when the task required a chain of dependent actions, sparse memory, open-ended instruction following or precision contact. The paper says real-world testing exposed instability and safety-critical behavior that aggregate success scores do not fully capture.

The infrastructure is part of the story. RoboDojo uses Isaac Sim for heterogeneous parallel simulation and adds RoboDojo-RealEval, a remote real-world evaluation system intended to standardize hardware setup, workspace layout, lighting, scene resets, evaluation protocol and deployment interface. The team also released benchmark code and XPolicyLab, a framework for policy development, deployment and evaluation. Chen said the group used XPolicyLab to reproduce more than 30 models and build the public leaderboard.

The remote real-world evaluation piece is the most consequential part if it holds up. Robotics groups have long had a measurement problem: sim benchmarks are cheap and repeatable, while real-robot benchmarks are expensive, fragile and easy to overfit to local lab setups. RoboDojo's wager is that a shared remote physical testbed can make real-world manipulation results easier to compare without asking every team to maintain the same arms, cameras, lighting and reset procedure.

Chen also drew a line around governance. In the X thread, he said RoboDojo is operated by AI MMLab Club, described as a non-profit organization, with academic partners worldwide. He also said the benchmark, reproduced code and checkpoints are open-sourced and that there is "no commercial involvement." That is a direct response to an increasingly crowded embodied AI race in which benchmark placement can become product marketing, recruiting material and investor evidence. A shared academic leaderboard gives the project credibility only if submissions, protocols and reproduced baselines stay inspectable.

For robotics founders building generalist manipulation systems, RoboDojo is useful because it makes weakness legible. A low score on a single broad benchmark does not decide whether a robot company has a product. It does show where demo-friendly policies break: remembering a hidden state, recovering from partial failure, completing multi-step instructions and making contact-rich movements without drifting. The next wave of embodied AI companies will have to sell reliability, not benchmark novelty, and RoboDojo puts a harder public number on how far the field still has to go.