Ai2 ToyBlocks Tests Robot Learning, Generalization and Sim to Real Transfer
Ai2's open ToyBlocks simulator automates robot training data, exposes weak generalization, and transfers Molmo 2 plus ManiFlow to a Franka FR3 arm zero shot.
Summary
On November 6, 2026, Ai2 introduced FoundationWorlds and its first open testbed, ToyBlocks, a tabletop simulation where a robot arm manipulates labeled blocks. Built on MolmoAct 2, MolmoBot and MolmoSpaces, it combines Nvidia’s Isaac Lab reinforcement learning, motion planning and automatic task generation, avoiding teleoperation. Hundreds of plans run on one GPU at up to 24 times real time; randomized geometry and appearance test generalization, while metrics track success, speed, force and progress.
Initial tests found ManiFlow, π0.5 and MolmoAct 2 generalized poorly. Molmo 2 and ManiFlow split planning from execution. Claude Opus 5.5 solved 30 of 30 stacking tasks from text and 30 of 30 using camera feedback, but search and spell scored 63%, with 13% look alike and 15% orientation errors. Self written skill descriptions raised success from 24% to 63%, versus 36% for expert descriptions. Across world models trained on 900 trajectories for each of four skills, half successful, the 5 billion parameter Wan 2.2 led; Astra mishandled contact and LeWM drifted over full trajectories. Adding 100 generated trajectories lifted unseen region policy success from 19% to 78%.
Models trained only in ToyBlocks transferred zero shot to a Franka FR3 with a Robotiq gripper across 30 scenes containing 4 to 10 known blocks. ManiFlow scored 80.0% in simulation and 81.1% physically; Molmo 2 plus ManiFlow scored 55.5%, 111 of 200, versus 50.0%, 15 of 30, requiring more trials. Ai2 released assets, generators, data, two models, code and validation sets: 85,336 long horizon demonstrations containing about 6 to 12 skills each, nearly 1,600 hours and 144 million frames. A larger dataset, more skills, broader benchmarks and integrated reinforcement learning are due within weeks; two arm, dexterous hand and pile manipulation are planned.
Positives
- 24 times real time data generation lets hundreds of simulation plans execute in parallel on one GPU.
- 81.1% physical success for ManiFlow slightly exceeded its 80.0% simulation result after zero shot transfer.
- 100 Wan 2.2 generated trajectories raised policy success in an unseen region from 19% to 78%.
- 63% success with self written skill descriptions surpassed 36% with human expert descriptions and 24% with defaults.
- 85,336 demonstrations provide nearly 1,600 hours and 144 million frames for open robot learning research.
- 30 of 30 stacking tasks were solved by Claude Opus 5.5 from both text descriptions and closed loop camera observations.
Risks & concerns
- ManiFlow, π0.5 and MolmoAct 2 generalized poorly despite large scale pretraining.
- 63% search and spell performance leaves more than one third of tasks unsolved or incorrectly classified.
- 13% look alike letter errors and 15% orientation errors exposed persistent perception and spatial reasoning weaknesses.
- 50.0% physical success for Molmo 2 plus ManiFlow trailed 55.5% in simulation and covered only 30 real episodes.
- Astra struggled with physical contact, while LeWM accumulated drift across full trajectories.
- Integrated reinforcement learning is absent from the current package and is planned for a later release.