Depth-to-Color Alignment

Reprojects a depth map into the color camera’s viewpoint so the two images’ pixels line up one to one.

In an RGB-D camera, the depth sensor and the color lens sit in different physical positions with different intrinsics and viewpoints, so the same pixel in each image doesn’t correspond to the same point if they’re simply overlaid. Alignment works by using the depth camera’s intrinsics to back-project each depth pixel into a 3D point, using the extrinsics between the two sensors to transform that point into the color camera’s coordinate frame, and then projecting it onto the color image plane using the color camera’s intrinsics. The resulting depth map matches the color image’s resolution and viewpoint, so a box detected or a mask segmented on the color image can look up depth directly and produce a colored point cloud. Intel RealSense SDK’s rs2::align and the ROS 2 driver’s align_depth.enable parameter do exactly this. The result is a computed approximation: changing viewpoint requires resampling, and occluded regions produce holes or misalignment, especially visible around object edges.

ExampleLaunching the RealSense driver in ROS 2 with align_depth.enable turned on publishes an extra topic, /camera/camera/aligned_depth_to_color/image_raw; a grasping program that boxes a cup in the color image can look up depth at the same pixel coordinates in this depth map, then back-project to get the cup’s 3D position.

Also called

Depth Registration, RGB-D Alignment

Related

Depth Camera · Camera Intrinsics · Camera Extrinsics · Projection / Back-Projection · Point Cloud · RealSense Depth Camera (D435i / D405)

Sources

librealsense rs-align example (IntelRealSense GitHub)
realsense-ros README (align_depth.enable)