Closing the sim-to-real gap with PD Replica

At Parallel Domain, our mission is to push the boundaries of simulation for developing and testing machine perception models. This blog post outlines how our new technology, PD Replica, significantly improves simulation realism and variety. This reduces the domain gap between simulation and the real-world, ultimately unlocking a new level of performance and utility using Parallel Domain simulation.

Traditional CG environment vs. PD Replica: What’s the difference?

Traditionally, Parallel Domain has relied on procedural content generation to create detailed CG environments for simulation. These environments, which feature everything from cityscapes to back yards, have been widely adopted by leading companies across automotive and drones, helping to accelerate development in production applications as well as push the boundaries of AI research. However, this method of content generation struggles to reproduce the nearly infinite scale, variety, and detail of the real world.

Now, with the introduction of PD Replica, we’ve added the power of real-world scene reconstruction into our simulation suite. Unlike procedural worlds, which are generated programmatically, PD Replica automatically reconstructs simulation environments solely from image and video input. Not only does this generate environments nearly indistinguishable from the detail of the real world, it also enables a significantly larger scale and variety. If you can take photos of a location, you can now simulate it. This ultimately makes it easier for models to bridge the gap between simulation and real-world applications.

Evidence of sim-to-real gap closure

One of the primary goals of simulation is to serve as an augmentation, and in some cases a stand-in, for real-world testing and training. However, for this to be effective, the domain gap has to be small enough for the AI systems to be able to generalize from the simulated domain to the real.

In order to test whether PD Replica provides a reduced domain gap over traditional simulation, we chose a specific task to benchmark: parking spot detection. This task is well suited for such a comparison because it specifically focuses the perception system on features of the surrounding world, while being a critically important task for many of our automotive customers. By focusing on these static features, like the layout of parking spots, rather than dynamic elements like moving vehicles or pedestrians, we can evaluate how well these environments represent the real world.

With parking spot detection, small differences in lighting, textures, or even the angle of the camera can make or break the performance of a machine perception model. In our experiment, we generated two training datasets, one from PD Replica, and the other using our traditional procedural CG environments. We trained a model on each of these two datasets, and then benchmarked them on a global automotive company’s real-world testing dataset. In our experiments, models trained on PD Replica environments showed a 31.4% improvement over those trained on procedural data in mean average precision (mAP). 

What makes PD Replica stand out?

Replica environments aren’t just photorealistic; they’re based on actual scenes captured across diverse global locations. This diversity matters because real-world data is messy and complex. Parking lots in Tokyo don’t look the same as in San Francisco. Regional variations can be very significant, especially across parking slot sizes, slot shapes, line paint colors, conditions, and styles. These differences can make or break model performance.

By capturing these nuances in our Replica environments, we can simulate to generate data that is not only more realistic but also more diverse. This directly leads to better performance when these models are applied in the real world, which is especially important for zero-shot sim2real transfer—the ability of a model trained solely in simulation to perform accurately in the real world without any additional training.

Key Findings

• Increased Performance: Models trained on PD Replica data achieved a 31.4% improvement in parking spot detection accuracy compared to procedural environments.
• Realism Over Complexity: Despite the absence of night-time scenes in the PD Replica dataset, it outperformed procedural data with nighttime images. This highlights the power of realism over adding artificial complexity.
• Flexibility & Scalability: PD Replica makes it easy to generate environments from any geographic region, giving users the ability to tailor training data to their specific needs without time-consuming manual setup.

The Road Ahead

While PD Replica offers unparalleled realism, procedural generation still has its place. For scenarios that require highly specific layouts or features—like testing corner cases that are rare in real life—procedural environments provide the level of control and customization that remains essential. However, for general training and testing, especially for tasks that need to reflect real-world conditions, PD Replica sets a new standard.

As machine perception technology advances, the ability to test and train models in environments that closely mirror reality will be key to accelerating the development of smarter, safer systems. With PD Replica, we’re closing the gap between simulation and reality, helping to make autonomous systems more reliable and ready for the challenges of the real world.

Are you interested in seeing how PD Replica can enhance your machine perception models? Get in touch with us for a demo and learn more about how we can help you train, evaluate, and test your systems using the most realistic synthetic data available today.