Embodied AI startup X Square Robot has released X2Real, a simulation benchmark designed to measure how robot foundation models will perform in the physical world, and to keep growing its test set as models improve.
The company frames the problem plainly. Real-robot evaluation is expensive and slow, while simulation is fast but often unrealistic and easy for models to overfit. X2Real is built around three principles: faithfulness, effectiveness and fairness.
For faithfulness, the team recorded trajectories from physical robot arms, replayed the same control commands in simulation and tuned joint responses until the two matched, which the company says yields millimeter-level trajectory agreement. Visual appearance and physical properties such as contact were calibrated as well. Supported embodiments include the single-arm Franka, the dual-arm ArtiXon Arm-6A, the ARX R5 and the Quanta X1 mobile manipulator. On general manipulation tasks, X Square Robot reports a 0.96 correlation between simulated and real-world task success rates and 0.99 for progress scores. Those figures come from the company's own testing on that task subset.
For effectiveness, X2Real includes 44 tasks across 10 categories. Four cover reasoning: generalization, visual understanding, language understanding and memory. Five cover manipulation: fine manipulation, bimanual coordination, tool use, dynamic objects and mobile manipulation. A separate challenge set, including block stacking, pressing buttons in a color sequence and solving the Tower of Hanoi, targets precision, long-horizon memory and planning. The company says most current models still struggle there.
For fairness, appearance, task setup and embodiment conditions are randomized independently. Models post-train on a dataset of nearly 300 hours collected in a uniform green-screen studio, kept separate from the evaluation scenes. The benchmark runs on the Mana simulation ecosystem, built on NVIDIA Isaac Lab-Arena, which packages scenes, robots and task steps as reusable modules.
AI agents drive two loops. One writes new tasks and scenes from a constrained menu of options, with automatic checks removing invalid setups. The other breaks each task into a graph of subtasks, judges each node from object poses, contacts and grasp states, and generates targeted new tests where a model fails.
A unified protocol called Policy Space translates differences in input formats, action representations and inference timing. Developers run inference locally and connect to the platform, which handles scoring and trajectory replay online without a local simulation setup.
SEE ALSO: X Square Robot's TwinDEX Learns a 24-Step Chemistry Experiment · Zhejiang University's EMem-Bench Tests Embodied Memory