NVIDIA reported that developers are combining frontier artificial intelligence models with its Omniverse software libraries to convert simulation ideas into functioning applications. Under the demonstrated workflows, engineers guide AI agents through natural-language prompts, evaluate generated scenes, and use Omniverse libraries to handle physics calculations, rendering, and sensor modeling.
Frank DeLise, an Omniverse product manager at NVIDIA, used the GPT-6 Astra model to turn a SimReady warehouse and humanoid robot into an interactive simulator. The setup provides first-person and third-person camera views to evaluate automated warehouse tasks before deployment. DeLise instructed Astra to link Omniverse libraries for physics (ovphysx), scene updates (ovstage), rendering (ovrtx), and user interface creation (ovui). He also paired Astra with the simready-foundation library to establish the physical scene, allowing the model to generate the supporting animation and application code.
Simulation technology manager Doyub Kim instructed Astra to construct Zero to Alpamayo, a reusable autonomous-driving test environment based on Market Street in San Francisco. Kim directed the agent to map the workflow and connect asset creation, traffic systems, Omniverse RTX sensor simulation, and Alpamayo driving logic in stages. A separate experiment using Cosmos3-Nano altered weather and lighting conditions across recorded simulation footage. That setup allowed Kim to compare how the driving model reacted to identical road scenarios under altered environmental conditions.
Ashley Reid, an RTX sensor validation specialist at NVIDIA, directed Astra and Claude Fable 5 agents to evaluate simulated sensor fidelity against physical real-world recordings. The agents compared ovrtx camera and raw lidar data with recorded physical benchmarks over approximately three days. Reid guided an iterative workflow in which the agents built two digital twins from scratch and refined two existing ones. The models adjusted OpenUSD scenes to correct missing geometry, materials, and objects based on measured sensor discrepancies.
Robotics and vehicle testing
Omniverse engineering and product lead Tae Kim guided Astra using sports videos and natural-language instructions to build Robo Olympics. The project tests simulated Unitree G1 humanoid robots performing athletic actions under physical constraints. Astra developed robot controllers and refined them across physics trials. The system used the Newton Physics Engine to simulate movement, the open-source NVIDIA Warp framework to accelerate math calculations, and ovrtx to render virtual-camera views. In one experiment, the simulated robot cleared a single hurdle in 64 out of 100 trials, providing data to improve control timing.
Jens Jebens, senior product manager for OpenUSD at NVIDIA, directed Astra to examine automated mechanical disassembly by modeling a vehicle suspension inside PTC Onshape. Jebens then configured the assembly in NVIDIA Isaac Sim. Astra measured physical clearances around the parts and designed a custom wrench tool capable of reaching the suspension bolts. Jebens reported that the simulated robot successfully removed a suspension component, linking computer-aided tooling design directly to robot policy training.
Space and environment capture
Engineering director Nic Johns used Astra to transform NASA 3D assets into a browser-accessible International Space Station application featuring telemetry data. Johns assembled the base model using a single text prompt, then used a follow-up prompt to rotate the scene so Earth appeared on its daytime side. The technical workflow used Blender for initial asset preparation alongside Omniverse libraries for rendering (ovrtx), scene runtime execution (ovstage), and web streaming (ovstream).
Chirag Majithia, from the Isaac engineering applications team, instructed Astra to convert stereo camera captures into an editable OpenUSD testing studio. The pipeline combined PyCuSFM, FoundationStereo, and nvblox for scene reconstruction, supplemented by Blender-authored assets. Astra assembled the environment and deployed USD Content Agents to configure object movement and physical interactions. Majithia then ran collision and contact tests in Isaac Sim to refine the movement of doors and drawers within the captured space.
