Video-to-sim-to-real

Robots learn many skills by practising in a computer simulation first, because a virtual robot can try thousands of times without breaking anything. Building that simulation by hand usually takes engineers a long time.

In a video-to-sim-to-real setup, an AI model starts from a plain video of a person doing the task. It works out where things are and how the body moves, writes the simulation itself, practises there, and then hands the result to the real robot.

Google used the term for its Gemini 4 Argon demo in 2026, where a humanoid robot learned to throw cornhole bags this way. The robot still needed a few real throws to adjust, because no simulation matches the real world exactly.