Robotics Field Guide
Topic hub

Robot learning

Learning robot controllers from data. Behaviour cloning and action chunking, reinforcement learning in simulation, vision-language-action models, and the open datasets and tools such as Open X-Embodiment and LeRobot.

Updated 14 Nov 2026 · robot learning, imitation learning, behaviour cloning, ACT, Diffusion Policy, reinforcement learning, vision-language-action, VLA, Open X-Embodiment, DROID, LeRobot

Overview

Robot learning replaces a hand-written controller with a policy learned from data. The policy maps what the robot senses, such as camera images and joint states, to what it does next, such as joint targets or gripper commands. Two kinds of data dominate. Demonstrations, usually collected by teleoperation, drive imitation learning. Reward in simulation drives reinforcement learning. Large pretrained vision-language-action models now combine both with web-scale data. This hub stays on the robot side: data collection, action spaces, control rates and transfer. For the machine learning itself, see the AI/ML Field Guide.

Key ideas

Imitation learning from demonstrations

Imitation learning in its simplest form, behaviour cloning, is supervised learning on pairs of observation and action recorded while a person teleoperates the robot. Its classic weakness is compounding error. A small mistake takes the robot to a state the demonstrations never visited, where the policy has no guidance, and errors grow from there. Two recent designs reduce this. ACT, from Zhao et al. (2023), predicts a chunk of future actions at once instead of one step. According to the authors it learned six fine bimanual tasks on low-cost arms, such as slotting a battery, with 80 to 90% success from 10 minutes of demonstrations. Diffusion Policy, from Chi et al. (2023), generates actions by iterative denoising, which copes with demonstrations that solve a task in more than one way. Mandlekar et al. (2021), with the robomimic framework, studied which choices matter when learning from offline human demonstrations.

Teleop demosimitation learningSim plus rewardreinforcement learningCross-robot dataVLA pretrainingPolicyobservation to actionRobotarms, grippersactionsimages,joint statesfine-tune on demosfrom the target robotloop runs at afixed control rate
Three sources of data for a robot policy. Teleoperated demonstrations feed imitation learning, simulation with a reward feeds reinforcement learning, and cross-robot and web data pretrain vision-language-action models that are then fine-tuned on the target robot. The deployed policy runs in a loop at a fixed control rate.

Reinforcement learning for control

Reinforcement learning improves a policy by trial and error against a reward. On real hardware that is slow, wears the robot and can be unsafe, so most robot RL runs in simulation and then transfers. Locomotion is the clearest success: the legged robots hub covers quadruped policies trained in minutes on one GPU. In manipulation, OpenAI (2018) trained in-hand reorientation on a multi-fingered hand entirely in simulation with randomised physics. Two things decide success more than the algorithm. One is reward design, since a policy will exploit any loophole in the reward. The other is the sim-to-real gap, covered in the simulation hub. The general theory, PPO and offline RL are on the AI/ML Field Guide’s reinforcement learning hub.

Vision-language-action models and their data

A vision-language-action (VLA) model starts from a vision-language model pretrained on web data and is trained further to output robot actions. Brohan et al. (2023) named the category with RT-2, which represented actions as text tokens. OpenVLA (Kim et al., 2024) is an open 7B-parameter VLA trained on 970,000 real robot demonstrations. Its authors report that it beat the closed 55B-parameter RT-2-X by 16.5 percentage points in task success across 29 tasks, and that it can be fine-tuned on consumer GPUs. π0 (Black et al., 2024) adds a flow-matching action head to a pretrained vision-language model and was trained on single-arm, dual-arm and mobile manipulators.

These models depend on shared datasets. Open X-Embodiment (2023) pooled data from 22 robots across 21 institutions, covering 527 skills. DROID (Khazatsky et al., 2024) holds 76,000 demonstration trajectories, or 350 hours, collected in 564 scenes. LeRobot from Hugging Face stores data in its LeRobotDataset format (Parquet plus MP4) on the Hugging Face Hub, and implements ACT, Diffusion Policy and several VLAs. Treat published success rates as results on the authors’ robots and tasks. Evaluate on your own robot, with your own cameras, before you rely on any of them.

Where to start

Get comfortable with the robot side first. Sensors and noise explains what the policy sees, and Simulate an arm in MuJoCo gives you an environment to collect data in. The PyBullet pick-and-place project is a scripted baseline to compare a learned policy against. The robot table lists the SO-101 arm that LeRobot supports. Background on neural networks is on the AI/ML Field Guide’s deep learning hub.

Tools and platforms

Tool What for Licence
LeRobot Datasets, recording, training and deployment of imitation, RL and VLA policies Apache-2.0
robomimic Imitation learning from offline demonstrations MIT
Diffusion Policy Reference code for the Diffusion Policy paper MIT
OpenVLA Open VLA training and fine-tuning code MIT
openpi π0 family models and fine-tuning code Apache-2.0
Isaac Lab GPU-parallel RL and imitation learning environments BSD-3-Clause and Apache-2.0
MuJoCo Playground RL environments and training on MuJoCo Apache-2.0

Common mistakes

  • Collecting demonstrations that solve the task in many inconsistent ways, then blaming the model.
  • Recording at one control rate and deploying at another, or with a different camera position.
  • Measuring success on the training scenes. Hold out objects, positions and lighting.
  • Writing a reward that the policy can satisfy without doing the task.
  • Assuming a VLA works zero-shot on a new robot. Plan to collect demonstrations for fine-tuning.
  • Skipping a safety layer. A learned policy still needs joint limits, velocity limits and a stop.

Going deeper

Latest news on this topic

Recent papers

  • Camera equalisation of an 11.5x drop in environmental illuminance: four imitation-learning policies on an SO-101 arm

    S. Cowling · arXiv cs.RO · digest

    The author trained four imitation-learning policies (pi0.5, ACT, Diffusion Policy and SmolVLA) on one shared set of 50 pick-and-place demonstrations on a physical SO-101 arm, then ran each ten times at about 520 lux and ten times at about 45 lux. Success was 20 of 40 in normal light and 19 of 40 in dim light, a difference whose 95% interval runs from minus 23 to plus 19 points, with no sign that the policies differed in sensitivity. The recordings explain why: with automatic exposure and white balance left on, mean image brightness fell by only 3% on the front camera and 19% on the wrist camera. The practical advice is to lock and report camera settings and compare image statistics with the training data, rather than quoting room lux. The sample is small, and an accidental repeat of one block matched its success count but only 6 of 10 placements.

  • How Much of Your VLA Is kNN

    D. Grachev, D. LaRocque, O. Moured · arXiv cs.RO · digest

    Looking up the nearest training demonstrations and replaying their action chunks, with no policy training at all, reached 91.7% success on the LIBERO simulation benchmark, where vision-language-action models now report more than 95%. Under LIBERO-Plus perturbations, retrieval degraded less than any of the policies the authors evaluated. Across nine public VLAs, they find that most of what fine-tuning does is turn the vision-language backbone into a better key for retrieval, while the actions stay close to the demonstrations. They also propose Local Action Consistency, an offline measure of whether nearby observations lead to similar future actions, which tracks retrieval success. All results are on LIBERO's simulated Franka tasks in a paper accepted to a CoRL 2026 workshop, so add a retrieval baseline to your own evaluation before crediting a policy with generalisation.

  • BlinkMimic: Training a Humanoid Motion Tracker in Minutes on a Single GPU

    F. Chen · arXiv cs.RO · digest

    BlinkMimic trains a single whole-body tracking policy for all 40 LAFAN1 motion clips on a simulated Unitree G1 in 3.4 minutes on one RTX 4090, averaged over three seeds and measured until a stated return threshold is met with no failed clip. It builds on earlier work that trains the policy by backpropagating through a world model fitted to parallel simulation data, adding a step size that drops once training has progressed, resets triggered by failures of the deployed policy, and GPU graph capture of each whole iteration. A leave-one-out ablation assigns each component to iteration cost, iterations needed to fit the motions, or iterations needed to clear the last failures. There is no hardware test, so treat it as a fast loop for developing tracking policies rather than evidence of transfer to a real robot. Code and per-seed results are public.