NVIDIA Unveils New Features for Generative AI in Robotics, Isaac Lab, and Humanoid Robots
Big announcements at the Conference for Robot Learning (CoRL) this week, with NVIDIA pushing further into intelligent robotics.
NVIDIA has announced a wealth of new features for roboticists at the Conference for Robot Learning (CoRL) this week, including general availability of its Isaac Lab robot learning framework, new workflows for humanoid robots in its Project GR00T, and an open source tokenizer that runs 12 times faster than the competition.
"Robotics developers can greatly accelerate their work on AI [Artificial Intelligence]-enabled robots, including humanoids, using new AI and simulation tools and workflows," says NVIDIA's Spencer Huang of the company's announcements. "The line-up includes the general availability of the NVIDIA Isaac Lab robot learning framework; six new humanoid robot learning workflows for Project GR00T, an initiative to accelerate humanoid robot development; and new world-model development tools for video data curation and processing, including the NVIDIA Cosmos tokenizer and NVIDIA NeMo Curator for video processing."
The Cosmos tokenizer, to focus on that first, is an open source tool that breaks down images and videos into tokens for use with machine learning models. It offers, the company says, "exceptionally high compression rates" and runs up to 12 times faster than its closest rivals — while NeMo Curator, which offers video processing curation, runs around seven times faster than "unoptimized pipelines."
The biggest news, though, is the general availability of Isaac Lab 1.2, an open source robot learning framework built atop NVIDIA's Omniverse platform and designed to provide developers with a way to train robot policies at scale. It covers any type of robots, including humanoid and quadrupedal as well as collaborative robots designed to work alongside humans, and is claimed to deliver training for complex movements and interactions — something which appears to be resonating with the industry, with companies and organizations from Agility Robotics and Boston Dynamics to Berkeley Humanoid and Unitree Robotics having already adopted the platform.
Elsewhere, Project GR00T — NVIDIA's initiative to provid accelerated libraries, foundation models, and data pipelines for robotics — has been expanded with new workflows specifically focused on humanoid robots. These workflows include: GR00T-GEN, for building generative AI-powered 3D environments; GR00T-Mimic, for motion and trajectory generation; GR00T-Dexterity, for manipulation; GR00T-Control, for whole-body control; GR00T-Mobility, for locomotion and navigation; and GR00T-Perception, for multimodal sensing.
"Humanoid robots are the next wave of embodied AI," claims NVIDIA's Jim Fan, senior research manager for embodied AI, in support of the new features. "NVIDIA research and engineering teams are collaborating across the company and our developer ecosystem to build Project GR00T to help advance the progress and development of global humanoid robot developers."
Another big announcement at the event was a partnership between NVIDIA and Hugging Face, with the companies collaborating to combine the latter's LeRobot AI platform with NVIDIA's AI, Omniverse, and Isaac technologies — with a live demonstration of the LeRobot software stack running on NVIDIA's Jetson Orin Nano platform. "Combining [the] Hugging Face open source community with NVIDIA’s hardware and Isaac Lab simulation has the potential to accelerate innovation in AI for robotics," claims LeRobot principal research scientist Remi Cadene.
Isaac Lab is available on the NVIDIA Developer site now, with the source code published to GitHub under the permissive BSD three-clause license; the Project GR00T workflows are "coming soon," Huang says, with technical information available on NVIDIA's blog; and the Cosmos tokenizer is available on GitHub and Hugging Face now under the permissive Apache 2.0 license, with NeMo Curator to follow by the end of the month. Those working on humanoid robots, meanwhile, can apply to join Nvidia's developer program.