Exclusive: RoboRank, the 'LeetCode for roboticists,' will open evaluation environments and run a public scoreboard

Why it matters

Robotics lacks widely accepted, public leaderboards. If RoboRank can standardize safe, reproducible evals and attract contributions, it could accelerate iteration the way benchmarks did for LLMs.

Most readers have not encountered RoboRank yet. The project, which bills itself as a LeetCode for roboticists at roborank.dev, is being positioned as a public scoreboard for robotics methods. Founder Ben Caunt (@BdcauntBen) laid out the plan to RuntimeWire in an interview, framing it as neutral infrastructure for reproducible, citable benchmarks similar to what catalyzed progress in large language models.

https://x.com/BdcauntBen/status/2058715554832101715

The product ambition is straightforward: a place where researchers and teams can submit agents and methods, run against shared tasks, and compare results in the open. The engineering work to make that safe and reliable is not. Caunt said the hardest unsolved problem so far was not visualization but secure execution of arbitrary user code. Integrating with Rerun for visualizations went smoothly, but executing submissions raised concerns about privilege escalation and unintended access to sensitive data or host resources. The first line of defense is containerization: RoboRank runs submissions in Docker to sandbox processes and constrain filesystem and network access. Caunt is evaluating a move to a managed sandbox such as E2B to add another layer of isolation and operational simplicity, and he wants future iterations to be language agnostic so contributors are not locked to a single runtime.

The near-term architectural change is to separate the environments from the core web app and open-source them. Today, those environments encapsulate the tasks, assets, and evaluation logic for a given benchmark. By decoupling and publishing them, Caunt expects outside researchers to contribute new setups and models across domains, with the strongest contributions curated and upstreamed back into the platform. In practice, that means environments can evolve in public, tests can be inspected and reproduced locally, and there is a clear boundary between the site that runs rankings and the code that defines what is being measured.

Longer term, RoboRank is aimed at data-driven robotics, with benchmarks designed for VLAs and world models. Caunt's working assumption is that many of the most interesting advances will come from methods that learn from data at scale. That raises a separate constraint: compute economics. He sketched three workable patterns under discussion: constrain tasks to favor simpler models that can be evaluated cheaply; ask users to perform training themselves and only submit artifacts for inference; or allow uploads of pretrained models for side-by-side evaluation. Each path trades off ease of comparison, cost, and fairness. Constraining tasks reduces variance and enables more participants but might bias the field toward smaller models. Offloading training to users keeps platform costs down but complicates reproducibility unless training recipes and datasets are tightly specified. Accepting pretrained uploads broadens participation but increases the burden on the evaluation harness to standardize interfaces and prevent leakage of privileged weights or data.

Why build this at all? "Reproducible, public, citable benchmarks were the key way the public tracks the development of LLMs. If we expect the same strides in robotics to occur, we need a scoreboard for it," Caunt said. The scoreboard metaphor is literal in his design. Rather than treat results as one-off papers or videos, RoboRank is meant to hold the line on shared tasks, keep a running tally of methods, and make deltas over time visible. Caunt argued that a neutral infrastructure layer creates space to test what should be compared in robotics: models alone, robots and controllers, environment definitions, or end-to-end stacks that include hardware. Those questions are not abstract. The community is already discussing hardware-in-the-loop tests on robots like SO-101 and Unitree Go2 in RoboRank's Discord, and Caunt said part of the platform's job is to provide a path from simulated evaluations to on-robot checks without compromising safety or repeatability.

The emphasis on security and isolation runs through the design. Docker-based sandboxes give each submission a constrained execution context, and the team is layering in resource limits and strict I/O policies to prevent lateral movement between jobs. A managed sandbox such as E2B would add pre-hardened sandboxes and reduce the blast radius if something goes wrong during execution. Making the system language agnostic aligns with the open environment goal: contributors can define tasks and baselines in the tools that make the most sense for their subdomain while adhering to a stable interface contract for evaluation and scoring.

Open-sourcing the environments also addresses the social side of benchmarking. Versioned, inspectable tasks lower suspicion about cherry-picked scenarios and make it possible to cite exact environment versions alongside scores. It creates a venue for the inevitable debates about what is fair to measure and how to grade progress. Caunt's approach is to keep the core platform opinionated about submission protocols and security, while letting the community iterate on task design in the open. The best contributions will be curated and integrated into the first-party leaderboard so the public-facing scoreboard stays coherent even as the long tail of experimental environments grows.

Caunt's timeline starts with breaking out the environment codebase and publishing it, then iterating on the submission API and sandbox. The end state he described is a layered system: public, composable environments; a hardened execution layer for user code; and a visible, longitudinal leaderboard that tracks VLAs, world models, and other methods as they improve. If he can pull it off, RoboRank becomes less a single site and more a shared yardstick for robotics work produced across labs and startups, with results that are easier to trust, reuse, and cite.