Someday soon, you’ll encounter a humanoid robot on the street, at a store, or even in your home. It will offer to help you, and hold out a friendly hand. Or open its back door, asking you to hop in. Will you accept?
If you’re not quite ready yet, I don’t blame you. After watching The Terminator, or simply watching a car driving down the street sans driver, this future we’ve been promised for decades all of a sudden feels a little scary. Can we trust these things to drive our kids to school, help with the dishes, or even go to war for us?
As Tesla, Waymo, Meta and a new crop of humanoid robotics companies, like Figure, all race toward dominance, we need confidence that our new AI robots know what they’re doing. Setting them all loose in the real world to collect data doesn’t feel like an option when we read headlines left and right about gruesome fatalities from automotive AI. So where will our autonomous technology collect the data it needs to construct its models? The real world isn’t it.
We believe the answer is right in front of us: the virtual worlds we know as games.
Video games have long been recognized as powerful tools for learning, training and reasoning. Research has shown that games enhance cognitive skills, improve problem-solving abilities and accelerate multimodal learning processes in humans. The military has leveraged games like World of Warcraft and other simulation platforms to prepare soldiers for combat, honing their strategic thinking and teamwork skills in a low-risk environment. Similarly, educational institutions have adopted game-based learning to engage students and enhance retention.
These examples highlight the efficacy of simulations in developing both cognitive and practical skills. We believe we can apply the same principles to the neural networks and AI systems that will serve as the brains of our future robots. But how?
Before diving into immersive simulated worlds, simply ingesting gameplay footage has proven to be a valuable training data set. This footage serves as a rich repository of trial-and-error experiences, enabling reasoning and task execution training. By analyzing these recordings, AI systems can observe a multitude of strategies, mistakes and successes, enabling them to learn optimal actions and avoid pitfalls. This process mirrors reinforcement learning, where agents iteratively refine their behavior and adaptability based on feedback from their actions.
Beyond processing game footage, game environments themselves provide multi-modal training grounds where AI learns to integrate vision, sound and even simulated haptic feedback. Just as a self-driving car AI learns to process video, lidar and radar simultaneously, robotic AIs must absorb diverse sensory inputs from their simulated worlds. While the ability to navigate a virtual city in Grand Theft Auto doesn’t immediately translate to driving in Manhattan, there still are valuable inputs for training. By combining reinforcement learning, we can create robots that not only think but also move, adapt and respond like humans. Games may just be the perfect place for AI agents and humans to meet and learn from one another.
While entertainment-based games have been instrumental in AI research, the most effective environments for robotics training are high-fidelity simulation platforms designed specifically for real-world learning. Meta’s Habitat 3.0, NVIDIA’s Isaac Sim and CARLA, an autonomous driving simulator, are all examples of game-like virtual sandboxes that mimic reality with unprecedented accuracy.
How will all these worlds, scenarios, and challenges be created? Until recently, this would’ve required developers to hand-code thousands or even millions of worlds to train in. But with the rise of new generative systems like Genie 2.0 from Google or Oasis from Decart.AI, improved real-time rendering with cloud GPUs, and interactive generative physics systems that closely approximate the real world, we now stand on the cusp of simply generating them en masse.
While virtual environments dramatically reduce physical risk, they introduce new challenges—namely, the massive energy demands of training AI at scale. Advances in efficient model architectures and reinforcement learning techniques, however, are making this tradeoff increasingly more palatable.
By embracing game worlds as a viable method for training agentic robots, innovators may be setting the stage for a new generation of physical AI systems that are smarter, safer and better equipped to navigate the complexities of our physical world. This goes beyond robotics to any application where real-world experience is needed, from training therapy AIs to AI-driven tutors personalizing student learning for any situation. With the right approach, we’ll speed our way there without any of the collateral damage we fear most.
