Manipulation and grasping

Overview

Manipulation is the work of changing where objects are and how they are arranged: picking, placing, inserting, opening, wiping, folding. Grasping is the step that sets up stable contact between the robot and the object. This is where kinematics, perception, motion planning and control all meet. It is also where most failures in a pick cell come from: a missed grasp, a slip during transport, or a collision on approach. This hub is for engineers building a pick-and-place cell or a research setup who want the ideas behind the libraries.

Key ideas

Contacts, friction and closure

A fingertip pressing on a surface can push along the contact normal. With Coulomb friction it can also resist a sideways force up to μ times the normal force, where μ is the friction coefficient. The set of forces one contact can apply is therefore a cone around the normal, the friction cone, with half-angle arctan(μ). A grasp has force closure when its contacts together can resist any small external force and torque on the object. For a two-finger planar grasp the test is short: the line joining the two contacts must lie inside both friction cones. Grasp quality metrics score how large a disturbance a grasp can resist for a given grip force.

Analytic grasping needs the object’s shape, its mass and the friction coefficient. Outside a lab those are rarely known well, which is why most grasp planners in use today learn from data.

Learned grasp planning

Learned planners map a depth image or point cloud to a ranked list of gripper poses. Bohg et al. (2014) sort the field by whether the object is known, familiar or unknown. Dex-Net 2.0 (Mahler et al., 2017) trained a grasp quality network on 6.7 million synthetic point clouds labelled with analytic metrics. According to the authors, it planned grasps in 0.8 s with 93% success on eight known objects of awkward shape, over more than 1,000 physical trials. Grasp Pose Detection (ten Pas et al., 2017) samples full 6-DoF grasp candidates in a point cloud and classifies each one. The output of any of these is a set of candidate poses, not a motion. A planner must still find a collision-free way to reach one of them.

Pick and place as a pipeline

A pick-and-place task is a chain of stages: detect the object, generate grasp candidates, filter them by inverse kinematics and collision, approach, close the gripper, lift, transport, place and retreat. Each stage can fail, and the stages constrain one another. A grasp is useless if the arm cannot also reach the place pose while holding the object that way. MoveIt Task Constructor plans the whole sequence as connected stages, so that incompatible choices are rejected before the robot moves. Production cells add checks after each stage, such as the measured gripper width or vacuum level, and a retry policy.

Contact-rich tasks and end effectors

Inserting a peg, mating a connector or wiping a surface means pushing against something stiff. Under pure position control a 1 mm position error against a rigid part can produce a large contact force. These tasks use impedance control or admittance control, which make the arm behave like a spring and damper, often with a wrist force-torque sensor. The ros2_controllers repository includes an admittance controller and a force-torque sensor broadcaster for this.

Choose the simplest end effector that does the job. A parallel-jaw gripper needs clearance on two opposite sides. A suction cup needs a flat, fairly non-porous surface. Soft and compliant fingers tolerate pose error. Multi-fingered hands can reorient an object in the hand, but they bring many more actuators and much harder sensing. OpenAI (2018) trained in-hand reorientation policies for a physical multi-fingered hand entirely in simulation, with randomised friction and appearance, and transferred them to the real hand.

Where to start

Work through Forward kinematics and Inverse kinematics first, since every grasp pose ends as an IK query. Simulate an arm in MuJoCo shows contact and friction in a physics engine. Then build the PyBullet pick-and-place project, which runs the full pipeline above on a simulated arm with a primitive gripper. The Jacobian entry explains how forces at the fingertips map to joint torques.

Tools and platforms

Common mistakes

• Trusting the camera-to-robot calibration without checking it. A few millimetres of hand-eye error turn a centred grasp into a fingertip grasp.
• Planning the grasp and the place pose separately, then finding that the arm cannot reach the second while holding the object as grasped.
• Closing the gripper on position alone. Check width or force after closing so that an empty grasp is detected.
• Using position control for insertion. Add compliance and limit force.
• Setting friction in simulation to a default and expecting the same grasp success on the real part.
• Ignoring the approach path. Many failures are collisions with neighbouring objects on the way in.

Going deeper

• Mason, Toward Robotic Manipulation, Annual Review of Control, Robotics, and Autonomous Systems, 2018.
• Lynch and Park, Modern Robotics, Cambridge University Press, 2017, chapter 12 on grasping and manipulation, with course material from Northwestern.
• Tedrake, Robotic Manipulation: Perception, Planning, and Control, MIT course notes, free online.
• Bohg, Morales, Asfour and Kragic, Data-Driven Grasp Synthesis: A Survey, IEEE Transactions on Robotics, 2014.
• Mahler et al., Dex-Net 2.0, 2017, and ten Pas et al., Grasp Pose Detection in Point Clouds, 2017.
• OpenAI et al., Learning Dexterous In-Hand Manipulation, 2018.