NVIDIA Cosmos3-H-Dreams Runs Real Time Surgery Simulation on One GPU
NVIDIA's Cosmos3-H-Dreams generates controllable gallbladder surgery video in real time on one RTX PRO 6000, but remains an unvalidated research tool.
Summary
On October 20, 2026, NVIDIA introduced Cosmos3-H-Dreams, which it describes as the first real time generative surgical simulator trained on clinical cholecystectomy recordings with live kinematic control. It learns synchronized video and robot movements from CMR Surgical’s Versius platform, generating action conditioned tissue motion, reflections, smoke and occlusion on one NVIDIA RTX PRO 6000 GPU. The project advances Cosmos-H-Dreams, a Cosmos-Predict2.5 system for dVRK tabletop suturing, into clinical scenes without explicitly modeling geometry, deformation or physical interactions.
Cosmos3-H-Dreams specializes Cosmos3-H-Embodied-Nano, a 16 billion parameter model pretrained on Open-H-Embodiment, using 832 by 480 video aligned with actions at 10 Hz. Its 28 action dimensions cover poses and grippers for a left Cadiere grasper and right monopolar hook, plus energy and thumbstick controls. NVIDIA extends teacher training from 24 to 72 future frames, representing 2.4 to 7.2 seconds, then applies causal teacher forcing, Self-Forcing and a learned critic to reduce rollout errors. Separately trained four step and two step checkpoints trade visual quality for speed, while FlashDreams and adapted FadeMem accelerate streaming and consolidate attention history. VR controlled demonstrations begin from one image and produce different trajectories. Public GitHub and Hugging Face resources support compatible datasets and controllers. Planned work includes full procedure coverage, surgeon evaluation, more instruments, numerical simulation links and autonomous policies, but visual realism does not establish biomechanical accuracy, and the system is not validated for patient care.
Positives
- One RTX PRO 6000 runs Cosmos3-H-Dreams interactively through FlashDreams accelerated streaming inference.
- 28 action dimensions provide control over two instrument poses, grippers, energy settings and thumbsticks.
- Four step and two step checkpoints let developers balance generated image quality against inference speed.
- Public GitHub, Hugging Face and training resources support adaptation to compatible datasets, actions and controllers.
- VR recorded actions produce distinct continuations from the same initial frame, testing control response and unseen trajectories.
Risks & concerns
- Visual realism cannot prove biomechanical accuracy, tissue mechanics or suitability for clinical decisions.
- Two controlled instruments limit the simulator’s representation of richer surgical coordination, interaction and occlusion.
- Autoregressive errors can progressively distort anatomy or instrument geometry despite Self-Forcing and FadeMem mitigation.
- The faster two step checkpoint sacrifices visual quality for reduced inference time.
- Cosmos3-H-Dreams remains a research simulator and has not been validated for patient care.