Advanced · 14
AI with a body: robots that learn
Why folding laundry is harder than passing exams, and how robots are starting to learn from simulation, demonstrations and language models.
8 min read · Tap underlined words for a definition
Swipe up to begin
live in text. Robots live in the messy physical world, where things slip, lights change and a mistake can break a glass. AI that senses and acts in the world is called embodied AI.
Sense, plan, act
Perception: turn camera images, depth readings and touch sensors into an understanding of what is where.
Planning: decide what to do, from "pick up the cup" down to a path for the arm.
Control: send precise, rapid commands to the motors, correcting many times a second as things move.
Moravec's paradox
In the 1980s, Hans Moravec and others noticed something odd: computers find chess and algebra easy but struggle with things a toddler does, like walking or grasping a toy.
Moravec's paradox has a likely explanation. Evolution spent hundreds of millions of years polishing our senses and movement, so they feel effortless. Abstract reasoning is recent, so it only feels hard to us.
According to Moravec's paradox, which is hardest for a robot?
Where robots get their data
Chatbots learn from the internet. There is no internet of robot movements, so data is precious.
Simulation: robots practise millions of times in a virtual world. The gap between simulation and reality is the sim-to-real problem. One fix is domain randomisation: vary colours, friction and lighting in the simulator so the real world looks like just another variation.
Learning from demonstrations: a person guides the robot or controls it remotely, and the robot learns to imitate those movements. This is called imitation learning.
Vision-language-action models
A vision-language-action model, or VLA, starts from a model trained on images and text from the web, then is trained further on robot data. It takes in a camera view and an instruction, and outputs robot actions, often written as special .
Google DeepMind's RT-2, in 2023, showed this lets robots use web knowledge, for example following instructions about objects they never saw during robot training.
- Data is scarce and slow to collect, one real movement at a time.
- Mistakes are physical: they can break things or hurt people, so safety matters more.
- Control must happen in real time; a robot cannot pause to think mid-grasp.
- Every home and warehouse is different, so robots must cope with endless variety.
What is the sim-to-real problem?
Before you drift off
- Embodied AI must perceive, plan and control in the real world.
- Moravec's paradox: everyday movement is harder for machines than abstract thought.
- Simulation, demonstrations and VLA models are how robots are catching up.
Terms in this lesson
Go deeper, free and open
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control · Brohan et al., Google DeepMind
- Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World · Tobin et al.
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models · Open X-Embodiment Collaboration · Shared open robot dataset
- LeRobot · Hugging Face · Open-source robot learning library
That's the lesson
AI with a body: robots that learn · 8 min
Spotted a mistake? Suggest an edit
Previous: Distillation and the rise of small models