Multi-sensor Time Synchronization / Timestamp Alignment

Aligning readings from cameras, encoders, IMUs, and other sensors onto one shared clock so simultaneous data lines up correctly.

A robot's cameras, joint encoders, force sensors, and IMU each run at their own frequency and with their own latency — a camera's latency, from exposure, encoding, and transmission, is usually larger than a joint reading's. Time synchronization puts all this data on one shared timeline: on the hardware side, a shared trigger signal can make multiple cameras expose at the same instant, or Precision Time Protocol (PTP, IEEE 1588) can align device clocks to sub-microsecond precision; on the software side, messages get paired by matching their timestamps to the closest match, as with ROS's message_filters, which offers both exact and approximate matching. For imitation learning, misalignment means the model ends up learning from mismatched observation-action pairs.

ExampleUMI first measures the latency of each data stream — camera, gripper width, and so on — separately, then aligns every observation to whichever stream has the largest latency, usually the camera; at deployment time it issues action commands slightly early to cancel out execution latency.

Also called

Timestamp Alignment, Time Sync, Hardware Sync

Related

Precision Time Protocol · Multi-Sensor Fusion · Camera-IMU Calibration · Observation-Action Pair · ROS Bag · Control Latency

Sources

Universal Manipulation Interface (arXiv 2402.10329, HTML)
Wikipedia: Precision Time Protocol
ros2/message_filters 文档 (Chinese)