How we teach a robot a movement from video: from footage to a kick in simulation

On the podcast “I to się liczy”, Adrian Andrzejczyk and I talked about agents that form hypotheses and test them on their own. From there I moved on to robots. I talked about the software I am building: it is meant to translate video into a robot skill.

The whole conversation is on YouTube (in Polish). This part starts at 40:34, and we came back to the details after the 50th minute.

This is the sixth post from that conversation. Here I lay out the process step by step and say where we stood at the beginning of September.

The goal: any video in, a file for the robot out

The idea is this: you take any video from YouTube, put it into the system, and at the end you get a file you can load onto the robot. That is the goal, eventually. At the beginning of September it did not work like that yet.

As an example I took footage of Messi from an Argentina national team training session, a free kick. I wanted the robot to be able to perform that movement: the run-up to the ball, the kick and so on. I do not show the footage itself here, because it is someone else's material. I show what comes out of it.

Why all this? The robot should have moves loaded that you simply trigger. You cannot put a complex move like a somersault together by hand, motor by motor, because the robot will fall over. What you control a robot with: a joystick, a phone app or code, down to individual motors, I described with the robot dog.

Step 1: the person in 3D

You have a film of a player running across the pitch, and you want the robot to do exactly the same. First you have to map the person. You capture all their joints and limbs and turn that into a 3D model.

For us, GEM-X from NVIDIA does this. It reconstructs a person's motion in 3D from single-camera footage, on a SOMA skeleton. For a quick first check of the footage we use MediaPipe from Google, which tracks 33 body landmarks.

In early September we took the motion of the arms, legs and torso from the free kick footage. We were not yet reconstructing the path the player runs across the pitch. That is why the person in the left panel moves on the spot.

Step 2: retargeting, because a robot has different joints

Now comes what is called retargeting. The person's 3D model has to be translated into the robot's 3D model. The robot has different joints, and they bend differently. You cannot carry it over one to one, because then the robot will not perform the same action. You have to map the human motion onto the robot's motion.

On top of that, robots differ from one another. They have different numbers of points that bend, and different angles. We have two humanoids: a Unitree G1 and an EngineAI T800. According to their makers, the G1 is 132 cm tall and the T800 173 cm. So the same human motion has to be carried over to each of them separately.

We carry the kick from the footage over to a G1 in its 29 degrees of freedom configuration. We use the open library GMR (General Motion Retargeting) for that. It transfers human motion to more than a dozen humanoid models, the G1 among them.

Step 3: a pretrained controller, fine-tuned in simulation

Only then comes a series of simulations. The robot does not have to learn to stand from scratch. There are base models, open and closed, and you build on top of them. What training from scratch looks like, with thousands of robots wobbling until they get it right, I wrote about in the post on humanoids. The free kick numbers I show below do not come from training from scratch.

Our base is SONIC from NVIDIA. It is a model that gives a humanoid a core set of motor skills, learned from a large body of human motion data. For the free kick we fine-tune only a small part of it. We run the result in the MuJoCo physics simulator. The fact that the robot did not fall over means nothing yet. What counts is whether it repeated the movement.

We check every version in ten runs, each with a different random seed. The accuracy thresholds are the same before fine-tuning and after it. The same principle works with agents: the yardstick is frozen before the first attempt, and only the changes that score better on it stay.

Where we stood at the beginning of September

The robot reproduced the movement, ran reasonably well, but still wobbled a bit. On the retargeting alone it ran well. In simulation it still sometimes fell over, or the kick was not complete, depending on the sample it learned from.

In numbers, as of 6 September: the pretrained controller without fine-tuning did not pass the accuracy check in any of the ten runs. After fine-tuning it passed in three, and in the other seven the errors were mostly in the amplitude of the left shoulder's roll. In both versions all ten runs reached the end of the clip, but that was not always so: an earlier fine-tuned version reached the end in only four runs out of ten.

One simple movement on the spot, standing while moving the arms, the G1 performed in simulation in ten runs out of ten. That movement had its own controller, trained separately for it, not built on SONIC. Dynamic movements such as the kick did not meet the same criteria yet. We refined the whole process on the free kick, so three out of ten says nothing about how the system will handle a new recording. On two new, public recordings we had not yet got a single movement that passed the check.

Then there is the ball. The robot more or less repeated the player's motion, but there is no ball in this simulation. For it to really kick one, you need a separate task in which the simulation knows where the ball is.

Everything I show here happens in simulation. What the robot learns there works in the digital world. Then you still have to load it onto the robot and see whether it plays out the same way in the physical world. It often turns out that a lot of fixes are still needed.