Returning to the post about how models will soon be able to control robots on their own: I decided to test Fable 5.2 on a virtual SO-101 in the new MuJoCo 4.
The setup is as follows: the model only sees images from cameras and readings from its motors, and it moves the arm with commands like "move the gripper to point X, Y, Z" and "squeeze the gripper." It is not told where the objects are.
On simple tasks like "find the cube and put it in the basket," it performs well: 5 out of 5 different layouts, about 17 commands, and a couple of minutes for each. It calculated the scale itself using the grid on the floor.
With more complex tasks like tying a shoelace with two hands, not so much. It managed to lift the end of the shoelace and cross the ends, but hasn't tied the knot yet.
But it burns through tokens like crazy: one cross with the shoelace took 7.5 minutes, 28 commands, and 2.6 million tokens, about $1.85 at API prices. Each image from the cameras remains in context, and with each step, there are more of them.
We continue to observe.
#robotics@rvnikita_blog #so101@rvnikita_blog #claude@rvnikita_blog #anthropic@rvnikita_blog #mujoco@rvnikita_blog