NVIDIA Cosmos 3 Adds Four Open Models for Robot and Driving Simulation
NVIDIA adds four open Cosmos 3 Nano models for interactive robot and driving simulations, with synchronized camera, LiDAR and action control, under open terms.
Summary
On November 5, 2026, NVIDIA added four open weight, post-trained Cosmos3-Nano checkpoints, each with 16B trainable parameters and OpenMDW1.1 terms permitting commercial and noncommercial use. Cosmos3-Nano-Transfer-Auto generates offline synthetic driving data across up to 11 synchronized camera views, including fisheyes, plus joint RGB and LiDAR, using per-camera World Scenario Map controls, text, optional initial images and aligned HD-map range maps. Cosmos3-Nano-Sim-Auto causally distills it into few-step streaming chunks driven by rolling history and updated controls. Closed-loop use requires a simulation runtime and WSM renderer.
Cosmos3-Nano-Sim-Transfer converts simulator depth, segmentation, edges, blur and prompts into photorealistic RGB while the simulator retains physics and ground-truth state. Cosmos3-Nano-Sim-Bimanual directly predicts observations from robot or camera actions for YAM and AgiBot two-arm systems controlled by policies or joysticks. Its mixed-action schema uses 59 values padded to 64, with coordinate, rotation, gripper, validity and domain conventions. T transitions require T+1 observations. Transfer-Auto can be adapted to new camera, LiDAR and driving domains, then causally trained and distilled into Sim-Auto. All four checkpoints include post-training recipes.
Cosmos 3’s Mixture-of-Transformers backbone links text and visual networks. Temporal-causal attention, key/value caching and compressed latents generate chunks from controls and history while masking future observations. After the first image, each latent step represents four RGB frames, with Transfer predicting one step per chunk and optimized Bimanual predicting two. Training progresses through bidirectional adaptation, teacher forcing, four-step consistency distillation and self-forcing on generated history. Sim-Bimanual requires Linux, an NVIDIA GPU, Docker and NVIDIA Container Toolkit. vLLM-Omni was tested on one GB200, not as a minimum-memory specification. Fidelity for policy learning and evaluation still needs measurement, and short runs prove execution, not convergence.
Positives
- Four 16B-parameter Cosmos3-Nano checkpoints provide open weights and post-training recipes for commercial and noncommercial projects under OpenMDW1.1.
- Transfer-Auto generates synchronized RGB and LiDAR across up to 11 camera views, expanding synthetic driving coverage across sensor configurations and environmental conditions.
- Sim-Auto uses few-step causal generation, rolling history and updated controls to support fast streaming and extended interactive driving simulations.
- Sim-Transfer preserves simulator physics while converting depth, segmentation, edges and blur controls into photorealistic RGB observations.
- Sim-Bimanual supports action-conditioned two-arm simulation for both YAM and AgiBot embodiments using policies or human-operated joysticks.
Risks & concerns
- Closed-loop Sim-Auto operation requires a separate simulation runtime and World Scenario Map renderer to update controls from driving inputs.
- One GB200 powered the published vLLM-Omni test, but NVIDIA provides no minimum-memory specification for deployment.
- Generated-history errors can compound across later frames, making self-forcing necessary and long-rollout reliability an unresolved challenge.
- Sim-Bimanual requires strict 59-value control schemas padded to 64, calibrated coordinate conventions and T+1 observations for T transitions.
- Policy-learning and evaluation fidelity remains unproven, while successful short runs confirm execution rather than model convergence.