Sensor Simulation

Generating simulated readings from cameras, lidar, IMUs, force sensors, and other sensors inside a simulator.

Sensor simulation means computing what each sensor “should read” based on the state of a virtual scene. Camera images and depth maps come from rendering; lidar and radar cast virtual rays or use ray tracing to compute distance; IMU (inertial measurement unit) readings come from a rigid body's acceleration and angular velocity; and contact forces and joint torques come straight from the physics solver. A good implementation also adds the flaws real sensors have, such as noise, distortion, latency, and missing depth values. NVIDIA Isaac Sim groups sensors into camera-based sensors, RTX ray-traced sensors (lidar, radar, acoustic), and physics-based sensors (contact, IMU, and so on). These are exactly the readings a policy sees, so if they diverge too much from the real sensor, they become a source of sim-to-real gap.

ExampleIsaac Sim provides ray-traced RTX lidar and radar simulation; robosuite v1.5 has also added sensor models.

Also called

Camera/Lidar/IMU Simulation, Sensor Modeling

Related

Rendering · Ray Tracing · Tactile Simulation · Sim-to-Real Gap (Reality Gap) · Simulation Fidelity · Visual Randomization

Sources

Isaac Sim 文档：Sensors (Chinese)
Choi et al., On the use of simulation in robotics (PNAS 2021)