Nvidia released Cosmos3-H-Dreams on October 20, 2026, introducing an interactive generative simulation system trained on clinical cholecystectomy procedures, according to an announcement published on Hugging Face. The model generates in-body surgical video that responds in real time to live kinematic control, running on a single Nvidia RTX PRO 6000 GPU.
The release builds upon the earlier Cosmos-H-Dreams system, which used Cosmos-Predict2.5 to simulate tabletop suturing tasks on the dVRK platform. Cosmos3-H-Dreams shifts from tabletop experiments to clinical procedures by learning visual dynamics from synchronized video and kinematics recorded on the CMR Surgical Versius robotic platform. Nvidia described it as the first real-time generative surgical simulator trained on clinical cholecystectomy data to support live kinematic control.
Surgical instruments operating inside the human body introduce physical complexities absent from tabletop setups. Small tool adjustments stretch tissue under traction, shift internal reflections, create occlusions, and produce smoke that alters visibility. Generative simulators must capture these anatomical responses across extended sequences without drifting into geometric distortions or altering instrument shapes over time.
Model Training
Cosmos3-H-Dreams builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter foundation model pretrained on the Open-H-Embodiment dataset. Researchers trained a bidirectional teacher model on curated clinical cholecystectomy recordings formatted at an 832-by-480 resolution. Training paired video with 28 action dimensions aligned at 10 Hz, covering pose and gripper states for both arms, thumbstick axes, thumbstick buttons, and energy controls. The training data isolated two active tools: a left Cadiere grasper and a right monopolar hook.
Developers adapted the teacher model first across 24 future frames, or 2.4 seconds, before extending the training horizon to 72 frames, representing 7.2 seconds at 10 Hz. Distillation then converted the bidirectional teacher into a causal student capable of generating video in short blocks. Training incorporated causal teacher forcing followed by Self-Forcing, which trained the student on its own generated context under the guidance of the teacher and a learned critic.
FlashDreams supplies the runtime library for two released student checkpoints. One checkpoint runs four denoising steps per generated block, while a separately trained second checkpoint uses two denoising steps to achieve faster inference times at the expense of visual quality. To manage memory during extended rollouts, FlashDreams integrates an adapted FadeMem module that consolidates the key-value cache.
Research Scope
Demonstrations showed the system generating diverging surgical scenes conditioned on actions recorded through a virtual reality headset. Nvidia stated that Cosmos3-H-Dreams remains a research simulator rather than a validated system for patient care. Stated future research directions include expanding coverage across all phases of cholecystectomy, evaluating anatomical accuracy with surgeons, supporting more than two instruments, coupling world action models for autonomous surgical policies, and combining generative video with numerical physics simulations.
