Sensors

Overview

A robot knows only what its sensors tell it, and every sensor lies in its own characteristic way. Gyroscopes drift, cameras blur, depth cameras return nothing on glass, lidar points smear when the robot moves, and force sensors change their zero as they warm up. Good robot software is built around those errors rather than surprised by them.

This hub is for engineers choosing sensors, writing drivers or debugging an estimator that will not converge. It covers what each sensor measures, how it fails, and the calibration that turns raw readings into numbers in a known frame at a known time. The estimation side is in /topics/state-estimation/ and the vision side in /topics/robot-perception/.

Key ideas

Proprioceptive sensors: measuring the robot itself

Encoders measure joint and wheel angles. They are usually the most accurate sensor on the robot, and their errors come from the mechanism around them: backlash, wheel slip and compliance. See /topics/actuators-and-drives/.

An IMU combines a three-axis gyroscope, a three-axis accelerometer and often a magnetometer. The gyroscope measures angular velocity. The accelerometer measures specific force, not acceleration: REP 145, the ROS convention for IMU drivers, notes that a device at rest with its z axis up should report +g on z, in m/s². Integrating the gyroscope gives orientation that drifts with the bias. Double-integrating the accelerometer gives position that drifts much faster, which is why IMUs are fused with something else. Their noise and bias instability are characterised with an Allan variance plot from a long static recording.

Force/torque sensors measure the six components of force and torque, usually with strain gauges in a wrist-mounted body. They drift with temperature and must be zeroed with the tool’s weight compensated in the current orientation. Joint torque sensors do the same per joint and make torque-controlled arms possible.

Exteroceptive sensors: measuring the world

Cameras are cheap and information-rich. Rolling-shutter sensors read rows at different times, which distorts fast motion, so robots that move quickly prefer global-shutter cameras. Depth cameras use stereo matching, structured light or time of flight. All three struggle with transparent, shiny or very dark surfaces, and active infrared designs can wash out in sunlight.

Lidar measures range by timing light pulses, giving accurate geometry independent of lighting. Spinning units take each point at a slightly different time, so a moving robot sees a distorted scan unless the motion is compensated. Zhang and Singh describe this as what makes lidar odometry hard. Radar has much lower angular resolution, but it measures relative velocity directly and keeps working in dust, rain and fog.

Tactile sensors cover fingertips and skins with resistive or capacitive arrays, or use a camera behind a deformable gel to image the contact patch, as in GelSight. They report what vision cannot see at the moment of grasp: contact location, slip and shear.

Calibration: intrinsic, extrinsic and time

Calibration has three parts, and fusion fails if any is wrong.

• Intrinsic: the sensor’s own model. For a camera, the focal length, principal point and lens distortion, usually estimated from views of a checkerboard by the method of Zhang (2000). For an IMU, biases, scale factors and axis misalignment.
• Extrinsic: where the sensor sits relative to base_link. This becomes a static transform in the robot description. Camera-to-gripper (hand-eye) calibration is a common special case.
• Temporal: the offset between the sensor’s clock and the robot’s. A 10 ms offset on a camera turning at 1 rad/s is a 0.01 rad (0.57°) orientation error. Furgale, Rehder and Siegwart (2013) estimate time offsets and spatial transforms together.

Where to start

/learn/beginner/06-sensors-and-noise/ models noise, bias and drift in code and shows why averaging does not remove bias. /learn/beginner/02-frames-and-rotations/ covers the frames that extrinsic calibration fills in, and /topics/spatial-math/ the optical frame convention.

Tools and platforms

Common mistakes

• Trusting the timestamp the driver assigned on arrival rather than the time the measurement was taken.
• Forgetting that an accelerometer at rest reads +g on its upward axis, then subtracting gravity with the wrong sign.
• Measuring extrinsics with a ruler and never refining them. Small rotation errors become large position errors at range.
• Calibrating a camera at one focus or zoom setting and using it at another.
• Leaving a force/torque sensor’s zero from the morning in place all day.

Going deeper

• Z. Zhang, A flexible new technique for camera calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence 22(11), 2000.
• P. Furgale, J. Rehder and R. Siegwart, Unified temporal and spatial calibration for multi-sensor systems, IROS 2013. The method behind Kalibr.
• REP 145: Conventions for IMU Sensor Drivers. Its status is draft at the time of writing.
• Richard Hartley and Andrew Zisserman, Multiple View Geometry in Computer Vision, 2nd edition, Cambridge University Press, 2004. The reference for camera models.
• W. Yuan, S. Dong and E. H. Adelson, GelSight: high-resolution robot tactile sensors for estimating geometry and force, Sensors 17(12), 2017.