Robot perception
How a robot turns camera and depth data into object poses and geometry it can act on. Detection and segmentation, 6D pose estimation for manipulation, point-cloud processing and 3D reconstruction.
Overview
Robot perception turns sensor data into the quantities a robot acts on: where an object is in the robot’s frame, which way up it is, which surface it rests on, and what is in the way. It shares its tools with computer vision, but the output is different. A vision system succeeds when it labels an image correctly. A robot’s perception succeeds when the gripper closes in the right place, which needs metric geometry in base_link at a known time.
This hub covers perception for manipulation and scene understanding: detection and segmentation, 6D pose, point clouds and reconstruction. Mapping and localisation are in /topics/slam-and-mapping/. For the neural network side of vision, convolutional networks, transformers and how detectors are trained, see the AI/ML Field Guide’s computer vision hub.
Key ideas
From pixels to a pose in the robot frame
Every manipulation pipeline ends with a chain of transforms. A detector or segmenter finds the object’s pixels. The depth image and the camera intrinsics lift those pixels into 3D points in the camera’s optical frame (z forward, see /topics/spatial-math/). A pose estimator fits a model of the object. Finally the camera’s extrinsic calibration, from hand-eye calibration for a wrist camera, moves the result into base_link. Each step adds error. An extrinsic error is especially costly because it shifts every object by the same amount, so averaging over detections never removes it. See /topics/sensors/.
Detection, segmentation and 6D pose
Detection draws boxes around objects of known classes. Segmentation labels pixels, either by class or by object instance. Promptable segmenters such as Segment Anything and its video successor SAM 2 return a mask for an object given a click or a box, without training on that object, which makes them useful for picking unfamiliar items.
A grasp needs more than a mask. It needs the object’s 6D pose: position and orientation. Model-based estimators fit a known CAD model to the image and depth. PoseCNN (Xiang and colleagues, RSS 2018) is an early learned example. FoundationPose (Wen and colleagues) handles novel objects at test time given either a CAD model or a few reference images, and also tracks pose across frames. The BOP benchmark compares 6D pose methods on shared datasets and is the place to check claims.
Point clouds
A depth camera or lidar produces a point cloud, carried in ROS 2 as sensor_msgs/PointCloud2. A classic tabletop pipeline runs in a few steps. Downsample to a voxel grid so later steps run at a steady rate. Remove statistical outliers. Fit the dominant plane with RANSAC and drop it, which removes the table. Cluster what remains into objects. Each cluster can then be matched to a model or passed straight to a grasp planner.
Registration aligns two clouds. Iterative closest point (ICP), from Besl and McKay (1992), alternates between matching nearest points and solving for the best rigid transform. It converges only from a good initial guess, so pipelines use a learned or feature-based estimate first and ICP to refine.
3D reconstruction
Reconstruction fuses many depth views into one model. Volumetric methods store a truncated signed distance function (TSDF) in a voxel grid, averaging each new depth image into it, following Curless and Levoy (1996). KinectFusion showed this running in real time with camera tracking from the model itself. A surface mesh is extracted from the TSDF when needed. For manipulation, moving a wrist camera around the object fills in surfaces that a single view cannot see.
Where to start
/learn/beginner/02-frames-and-rotations/ covers the frame changes every pipeline ends with, and /learn/beginner/06-sensors-and-noise/ the noise in depth data. The project /projects/pybullet-pick-and-place/ closes the loop from an object pose to a grasp in simulation.
Tools and platforms
| Tool | What it is for | Licence | Platform |
|---|---|---|---|
| OpenCV | Image processing, camera models, feature detection and classical vision | Apache-2.0 | /platforms/opensource/ |
| Open3D | Point-cloud processing, registration and TSDF reconstruction in Python and C++ | MIT | /platforms/opensource/ |
| PCL | C++ point-cloud filtering, segmentation and registration, with ROS integration | BSD-3-Clause | /platforms/ros/ |
| SAM 2 | Promptable segmentation in images and video | Apache-2.0 | /platforms/opensource/ |
| FoundationPose | 6D pose estimation and tracking of novel objects | NVIDIA licence, non-commercial use only | /platforms/nvidia/ |
| Ultralytics YOLO | Object detection and segmentation training and inference | AGPL-3.0, or a paid enterprise licence | /platforms/opensource/ |
Common mistakes
- Reporting pose accuracy in pixels or on a benchmark, then discovering the hand-eye calibration is off by 5 mm.
- Using the depth image without aligning it to the colour image. They come from different sensors with different intrinsics.
- Forgetting the optical frame convention and placing every object behind the camera.
- Checking the licence after building the product. AGPL-3.0 and non-commercial research licences both constrain shipping.
- Running ICP from a poor initial guess and trusting the converged result.
Going deeper
- P. J. Besl and N. D. McKay, A method for registration of 3-D shapes, IEEE Transactions on Pattern Analysis and Machine Intelligence 14(2), 1992.
- R. B. Rusu and S. Cousins, 3D is here: Point Cloud Library (PCL), ICRA 2011.
- B. Curless and M. Levoy, A volumetric method for building complex models from range images, SIGGRAPH 1996.
- R. A. Newcombe and colleagues, KinectFusion: real-time dense surface mapping and tracking, ISMAR 2011.
- B. Wen, W. Yang, J. Kautz and S. Birchfield, FoundationPose: unified 6D pose estimation and tracking of novel objects, arXiv 2023, CVPR 2024.
- A. Kirillov and colleagues, Segment Anything, arXiv 2023.
- The AI/ML Field Guide’s computer vision hub for detection and segmentation models, and the FPGA Guide’s image and video hub for camera pipelines built in hardware, from MIPI CSI-2 input to ISP stages.