New robot policy runs on an RTX 5090, beating OpenVLA-OFT while using 0.7% of its size

Jeff Cui shows that a frozen ViT backbone plus a tiny transformer can outperform a 7B‑parameter VLM in robot control.

By · Published

Why it matters

Patch Policy proves that dense visual features from a frozen ViT can replace massive VLMs for robot control, dramatically cutting compute costs and latency while improving performance.

A robotic system, with key components illustrating the efficient AI model's architecture. (Exploded-view technical diagram)

A lean visual policy beats a 7B‑parameter VLM

Jeff Cui (@jeffacce) announced on July 22, 2026 a new robot‑control method he calls Patch Policy. The approach freezes a state‑of‑the‑art Vision Transformer (ViT) pretrained on large‑scale image data, feeds every patch token to a lightweight transformer policy, and adds a simple block‑causal mask. The result, Cui writes, “beats OpenVLA‑OFT with 0.7% of its params, and trains on a 5090.”

The claim is backed by an arXiv preprint (arXiv:2607.18236) co‑authored with Gaoyue Zhou (@GaoyueZhou), Ada Langford (@LangfordAd91020), Bowen Tan (@bowentan_), Yann LeCun (@ylecun) and Lerrel Pinto (@LerrelPinto). The paper details experiments on a suite of simulated and real‑world manipulation tasks, comparing three families of visual policies:

  1. Global‑pooled features (e.g., ResNet or ViT CLS token) – the historic baseline.
  2. OpenVLA‑OFT, a 7 billion‑parameter vision‑language model fine‑tuned for control.
  3. Patch Policy, the new method using dense patch embeddings from a frozen ViT and a small transformer head.

Performance gap

Across the benchmark, Patch Policy improves task success by 40% over global‑pooled policies and by 18% over OpenVLA‑OFT when both are trained on the same tasks. The tiny policy contains only about 0.7 % of the total parameters of OpenVLA‑OFT, yet inference runs at roughly 10 ms latency – six times faster than the VLM baseline – without any hardware‑specific optimization.

Training on consumer hardware

Cui emphasizes that the entire pipeline fits on a single NVIDIA RTX 5090 GPU. “It trains on a 5090,” he notes in the first tweet of the thread. The training cost, therefore, is comparable to that of many academic labs and far below the multi‑GPU clusters typically required for 7 B‑parameter models.

Why dense patches matter

The thread argues that prior robot‑learning pipelines either pooled visual features globally, trained vision models from scratch, or hauled around massive VLMs. By contrast, dense patch features retain spatial detail that is crucial for precise manipulation. A side‑by‑side video in the thread shows a global‑pooled policy failing to locate a 2 mm‑tolerance cable, while the Patch Policy inserts the cable cleanly.

Encoder comparison

Cui’s team benchmarked five frozen encoders as control representations. DINOv2 and WebSSL consistently topped the leaderboard, suggesting that modern self‑supervised vision models provide the most useful dense features for control. The rankings remained stable across tasks and policy heads, reinforcing the importance of the visual backbone.

Open‑source rollout

The authors have opened a project website (patch‑policy.github.io) and a GitHub repo (https://github.com/gaoyuezhou/patch_policy) where code will be released “soon.” Supplemental videos demonstrate the robot inserting a cable, unplugging mid‑rollout, and repeating the maneuver – all with the lightweight policy.

Implications for robot‑learning research

Patch Policy challenges the prevailing belief that scaling model size is the primary driver of better robot control. By showing that off‑the‑shelf dense representations can substitute for massive VLMs, the work may shift investment toward better pretraining of vision backbones and more efficient policy architectures. It also lowers the entry barrier for labs without access to large compute clusters, potentially accelerating experimentation in dense‑perception robotics.

Sources

Reader comments

Conversation for this story loads after sign-in.