Automating realistic simulation from real-world data

Challenge

Across sectors such as automotive, rail, maritime, logistics, and agriculture, simulation plays a vital role in developing, testing, and validating autonomous systems, operator training, and process monitoring. However, despite major advances in visual realism and physics accuracy, one critical bottleneck remains: the creation of realistic and diverse simulation scenarios. This process still relies heavily on manual work to model 3D environments and script both normal and edge-case situations, limiting the range of test conditions. As a result, rare but safety-critical events often go untested. Current methods also struggle to capture the complexity and unpredictability of real-world dynamics or to automatically convert real sensor or video data into usable simulation content. This creates a persistent gap between virtual testing and real-life operational performance.

Project goals

The Video2Scenario project aims to:

• Automate the generation of 3D simulation scenarios directly from real-world video footage, reducing the need for manual modeling and scripting;
• Enhance existing simulation platforms (e.g. CARLA, MISSIM, AirSim, NVidia Omniverse, Unreal, Unity) rather than developing new ones, ensuring broad applicability;
• Apply advanced 3D scene understanding—including video segmentation, object and actor recognition, spatial mapping, and motion estimation—to reconstruct consistent 3D environments;
• Leverage Generative AI to enrich and vary scenarios by adding semantic details, creating alternative behaviors, and generating edge cases such as near misses or failures;
• Ensure interoperability and reuse across simulation tools through open standards like OpenSCENARIO (for dynamic aspects) and USD (for static assets);
• Provide a scalable scenario variation engine capable of automatically generating “what-if” cases for extensive testing coverage;
• Deliver a low-effort, cost-effective workflow that accelerates simulation-based development and improves the representativeness of virtual testing compared to real-world conditions.