XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representations

Overview of XR-1. In XR-1, we introduce the Unified Vision-Motion Codes (UVMC), a discrete latent representation that jointly encodes visual dynamics and robotic motion. XR-1 adopts a three-stage training paradigm to enable precise low-level control across diverse robots and tasks.

In Stage-1, we learn the Unified Vision-Motion Codes (UVMC) through a dual-branch VQ-VAE architecture. The visual encoder and motion encoder jointly encode visual observations and action sequences into a shared discrete codebook. This unified representation bridges the semantic gap between human demonstrations and robot actions, enabling effective knowledge transfer across diverse robot embodiments and environments.

In Stage 2, after extracting the Unified Vision-Motion Code (UVMC) from the dual-branch VQ-VAE as the supervision signals, we integrate this representation into the policy learning process to enhance low-level action prediction. We pretrain the policy using large-scale cross-embodiment robotic datasets XR-D (158k trajectories and 69.1M frames).

In Stage 3, after large-scale UVMC-guided pretraining, the model develops strong capabilities in extracting unified vision-motion knowledge and generating precise low-level actions.

Experimental Setup. We evaluate XR-1 across six robot embodiments(Tien Kung 1.0/2.0, Single-/Dual-Arm UR-5e, Dual-Arm Franka, and AgileX Cobot Magic 2.0), covering more than 120 manipulation tasks with over 14,000 rollouts.

We conducted evaluations on bimanual collaboration, dexterous manipulation, fluid/deformable object handling, contact-rich interactions, and dynamic environments. Our XR1 model was compared against baseline methods including RDT, π0.5, π0, GR00T-N1.5, and UniVLA. We selected the best baseline (π0.5) for comparative demonstration videos. Demonstration videos are shown below:

We conducted few-shot learning experiments on the Dual-ArmUR-5e and Tien Kung 2.0 robotic systems, performing comparative analysis against single-task baseline methods including ACT and DP. The experimental results and comparative evaluation are presented as follows:

TK2-PlaceCircuit

Main experiment: 6 different embodiments, with 20 tasks per embodiment, comparing results against RDT, π0.5, π0, GR00T-N1.5, and UniVLA baselines.

Description: We provide pre-training logs to demonstrate the robustness of our hierarchical representation. Stable convergence: both the vision reconstruction loss and the action reconstruction loss decrease rapidly in the early stage of training and then converge to a stable plateau. Despite the high diversity of the heterogeneous pre-training data (Ego4D, OXE, Robomind, and XR-D), this steady convergence indicates that UVMC can effectively capture high-level dynamics. Codebook vitality: we further monitor the Active Code Number, defined as the number of unique codes activated within each batch on each GPU. The vision codes stabilize at approximately 30, while the motion codes reach a healthy plateau of approximately 50. This trend shows that our pre-training avoids codebook collapse and preserves strong representational capacity. The absence of sudden drops in code usage further confirms that the discrete latent space is consistently utilized to encode complex cross-modal features.

Limitations: One limitation of UVMC is that it uses images sampled every k frames, which may miss intermediate process information. Using all consecutive frames could provide richer vision-dynamic features, but would also introduce substantial computational overhead due to the large number of visual tokens. A promising future direction is therefore to learn finer-grained latent features while selectively retaining the most informative tokens and removing redundant ones.

Description: To validate that the joint representation remains effective after fine-tuning, we provide qualitative future-frame reconstructions for tasks such as DUR-FindTapeBasket and DUR-SweepTrash on our project website. The key result is that the predicted UVMC tokens are essential for reconstruction quality. Our UVMC decoder is intentionally 4x smaller than the encoder (Table 5), so successful reconstruction cannot rely on decoder memorization and must instead come from the information carried by the vision-motion codes. We therefore compare reconstruction using predicted vision codes, zeroed codes, and randomized codes. We find that zeroed or randomized codes completely fail to recover scene dynamics, while the predicted UVMC tokens enable the lightweight decoder to reconstruct high-fidelity future states. This directly confirms that the predicted tokens preserve the joint visual-dynamic information required for sequential manipulation.