[Question] Generating 3D Terrain Maps from Depth Images

[Marsak24]

Hello everyone,
I’m working on generating 3D terrain maps using robot-mounted cameras in simulation. There’s a paper titled “Neural Scene Representation for Locomotion on Structured Terrain” by David Hoeller et al., in which a neural network constructs a 3D map from depth images and feeds it into a locomotion policy.
I’d like to replicate this setup using Isaac Sim (or Isaac Lab—still figuring out which would be more appropriate). So far, I’ve used MobilityGen to collect some data (robot pose and depth images). I still need to add my custom robot, but the goal is to use this data as input to a neural network that reconstructs a 3D map from noisy sensor readings.
Since the original paper used Isaac Gym, I assume I’ll need Isaac Lab rather than just Isaac Sim. Could you please guide me on the best way to approach this? Particularly in terms of tool choice (Isaac Sim vs. Isaac Lab), and whether there are any relevant examples or recommended workflows?
Any help would be greatly appreciated!
Marwah

[RandomOakForest]

Thank you for your interest in Isaac Lab and Sim. This is a great post for our Discussions section. I will move it there for follow up. In the meantime, here is a summary that may help.

Choosing Between Isaac Sim and Isaac Lab

Isaac Sim is NVIDIA's high-fidelity robotics simulator built on Omniverse, ideal for designing, simulating, and testing robots in visually and physically realistic environments. It supports synthetic data generation, software-in-the-loop testing, and can be extended for custom workflows¹. However, for advanced robot learning tasks—especially those involving reinforcement learning (RL), policy training, and large-scale data collection—Isaac Lab is more suitable.

Isaac Lab is built on top of Isaac Sim and is specifically designed for robot learning research. It provides modular tools for RL, imitation learning, procedural terrain generation, and direct integration with neural network training frameworks. Isaac Lab is the recommended choice for end-to-end workflows where simulated sensor data is used to train and evaluate policies, as in the "Neural Scene Representation for Locomotion on Structured Terrain" paper²³.

Summary Table: Isaac Sim vs. Isaac Lab

Feature/Use Case	Isaac Sim	Isaac Lab
High-fidelity simulation	Yes	Yes (inherits Sim)
Synthetic data generation	Yes	Yes
Robot learning (RL, imitation)	Limited	Yes (core focus)
Procedural terrain generation	Basic	Advanced
Prebuilt RL environments	No	Yes
Custom RL policy integration	Manual	Streamlined
Neural network training interface	Indirect	Direct

Replicating the Neural Scene Representation Workflow

1. Simulation Platform

• Use Isaac Lab: Since the original paper used Isaac Gym (now deprecated), and Isaac Lab is its successor for RL and learning workflows on top of Isaac Sim, Isaac Lab is the right tool for your project⁴²³.

2. Procedural Terrain Generation

• Isaac Lab provides utilities for procedural terrain generation, including complex terrains like stairs, slopes, and pyramids. You can control terrain difficulty, structure, and randomness for curriculum learning or robust policy training⁵.
• Terrain generation is handled via configuration classes (e.g., TerrainGenerator, HfPyramidStairsTerrainCfg). These classes let you create terrains as height fields or meshes, which can be imported into the simulator for both robot interaction and ground-truth extraction⁵.

3. Sensor Data Collection

• Isaac Lab supports attaching multiple sensors (RGB, depth, LiDAR, etc.) to your robot. You can collect synchronized sensor data and robot poses, just as you did with MobilityGen²³.
• For ground-truth 3D geometry (needed for supervised learning or evaluation), you can extract mesh or point cloud data directly from the simulation stage. This is typically done by traversing the USD scene graph and querying mesh vertices, then transforming them to world coordinates⁶.

4. Custom Robot Integration

• You can import your custom robot into Isaac Lab using URDF, MJCF, or USD formats. Isaac Lab provides examples and documentation for integrating custom robots and configuring their sensors and actuators²⁷.
• Once imported, you can define observation spaces (e.g., depth images, proprioceptive states) and action spaces for RL or supervised learning workflows.

5. Neural Network Training and 3D Map Reconstruction

• Isaac Lab is designed to interface with popular RL and deep learning libraries (e.g., skrl, RLlib, rl_games)².
• For 3D scene reconstruction, you can collect depth images and poses, then train a neural network (e.g., a 4D fully convolutional network on point clouds, as in the referenced paper⁸) to reconstruct the terrain. Isaac Lab's data collection and ground-truth extraction tools streamline this process⁶.
• You can implement closed-loop training, where the reconstructed map is fed into a locomotion policy, and the policy's actions are executed in simulation for further data collection and policy improvement⁴⁹.

6. Recommended Workflow

1. Set up Isaac Sim and Isaac Lab on your system¹²³.
2. Import your custom robot and configure sensors (depth, RGB, etc.)⁷.
3. Use Isaac Lab’s terrain generation utilities to create diverse, procedurally generated terrains⁵.
4. Collect sensor data and ground-truth meshes for each simulation step⁶.
5. Train your neural network for 3D scene reconstruction using the collected data.
6. Integrate the reconstructed map as input to your locomotion policy, and train/evaluate the policy in closed-loop simulation⁴⁹.
7. Benchmark and validate your approach using Isaac Lab’s evaluation tools and, if needed, transfer to Isaac Sim for additional scenario testing or visualization⁴.

Additional Resources

• Isaac Lab documentation and API references: For terrain generation, robot import, and data extraction⁵³.
• NVIDIA forums and office hours: For troubleshooting and best practices¹².
• Example projects and YouTube tutorials: For step-by-step guidance on custom robots and RL integration⁹⁷.

Conclusion

Isaac Lab is the recommended platform for your project, as it provides the necessary tools for procedural terrain generation, sensor data collection, ground-truth extraction, and seamless integration with neural network training workflows. Isaac Sim remains foundational for high-fidelity simulation, but Isaac Lab extends it for advanced robot learning and research—precisely matching the requirements of your 3D terrain mapping and locomotion policy project⁴²³.

References

Footnotes

1. https://developer.nvidia.com/isaac/sim ↩ ↩² ↩³

2. https://developer.nvidia.com/isaac/lab ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

3. https://isaac-sim.github.io/IsaacLab/main/source/setup/ecosystem.html ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

4. https://developer.nvidia.com/blog/advanced-sensor-physics-customization-and-model-benchmarking-coming-to-nvidia-isaac-sim-and-nvidia-isaac-lab/ ↩ ↩² ↩³ ↩⁴ ↩⁵

5. https://isaac-sim.github.io/IsaacLab/main/source/api/lab/isaaclab.terrains.html ↩ ↩² ↩³ ↩⁴

6. https://forums.developer.nvidia.com/t/how-to-access-ground-truth-environment-geometry-in-isaaclab/330944 ↩ ↩² ↩³

7. https://www.youtube.com/watch?v=0voBBEOCQuQ ↩ ↩² ↩³

8. https://arxiv.org/abs/2206.08077 ↩

9. https://www.youtube.com/watch?v=Y9r7UyWpIAU ↩ ↩² ↩³

[Marsak24]

Thank you for the detailed reply and the summary. it’s very helpful in showing the full picture.
I’m still not very experienced with Isaac Lab and feeling a bit lost with some of the steps you mentioned, especially how to use the terrain generator and how to collect data.
I’ve collected some depth images and poses using Isaac Sim with MobilityGen, is it possible to reuse that data, or would you recommend starting over in Isaac Lab?
Could you please point me to a minimal example or walkthrough that shows how to:
Generate procedural terrain (I read the documentation but wasn’t sure how to actually use it)
Collect synchronized depth and pose data
Train a model from this data (where would the neural network part go in the workflow? I’ve used Isaac Lab before for RL locomotion, but this feels quite different.)
I’d really appreciate a concrete starting point to build from. Thanks again!

[youwyu]

Hello, you may want to take a look at our repository at https://github.com/youwyu/Adaptive-Diffusion-Terrain, we currently support terrain elevation generation from (partially observed) depth map. We will also release a new version with Consistency Models.

[Marsak24]

Thank you @youwyu . I am trying to do the entire thing from scratch. I have collected some point cloud data, preprocessed and voxelized them. For now I need to figure out how to get the depth ground truth from the stage?, I got that there are multiple meshes in the stage, I did collect the assets but I am still stuck with how to get point cloud from the stage.