🏓 Taught SO-101 my ping pong ball trick. For now, in simulation.
In the previous post, I showed how SO-101 juggles a ball in simulation. It used a standard algorithm: formulas and a controller written by hand. Then I wanted to remove my code entirely and test pure RL: drop the ball, the robot makes random movements, drops it - loses, juggles - gets a reward.
In real life, I have a favorite ping pong ball trick that I show everyone: I juggle, juggle, and then the paddle makes a vertical loop around the ball - rises above it, descends on the other side, and catches it from below. So I wondered: could I teach this to SO-101?
RL learned to juggle the ball by itself, after several tens of millions of steps in simulation. But first, it found a loophole: it held the ball almost on the paddle, and the bounce counter increased. I had to require a minimum of 18 cm upwards. The trick was even funnier: I gave a reward for the angle around the ball, the counter showed 308 degrees, but in the video, the paddle just waved side to side. Then the trick started to count, but it turned out to be a "slap catch" - the ball flew one and a half meters to the side.
My main conclusion: in RL, almost all the time goes to the reward model, not the network. Any reward I wrote, RL performed in the way that was most convenient for it, not how I intended. And the real loop is such a rare event that the robot simply doesn't stumble upon it with random movements, the reward almost never triggers, and there's nothing to learn from. A combination worked: first, brute force movements proved that the loop was physically possible with these servos, then the robot imitated the found movements, and only then did RL refine them.
The hardest part was stitching it all together: regular bounce - loop - regular - loop. On pure policies, a maximum of 3 loops in a row. With look-ahead, where before each flight the robot plays out several options in a copy of the simulation and chooses the working one, it managed 37 in a row and 86% successful loops. I must admit: this is an ideal world model, the real robot doesn't have it.
But the most interesting thing for me in this experiment wasn't even RL. Most iterations were done by an AI agent while I slept: it watched videos and metrics, found the next loophole in the reward, came up with a hypothesis, changed the rules, and launched new training. It also found that some results were counted with a bug and suggested look-ahead itself. I set the goal and occasionally adjusted the direction. Essentially, this is a self-improving approach: the system itself looks for new ideas to test next and tests them.
This is exactly what I want to talk about next week, on November 10: how I taught my AI agents to improve themselves every day.
The video shows all the stages in order.
#robotics@rvnikita_blog #so101@rvnikita_blog #rl@rvnikita_blog #ai@rvnikita_blog #agents@rvnikita_blog