NVIDIA Opens Cosmos 3, a Foundation Model for the Physical World

NVIDIA spent the last few years selling the shovels for the AI gold rush. At its GTC Taipei keynote on May 31 it handed developers a new one. Cosmos 3 is an open foundation model built not for chatbots but for machines that have to move through the physical world: robots, self-driving cars, and the cameras watching a factory floor.

The company is calling it the first fully open "omnimodel" for physical AI. The idea behind that label is that a single system handles text, images, video, ambient sound and physical actions, rather than bolting separate models together. Under the hood, NVIDIA pairs a reasoning transformer with a separate generation transformer in what it calls a mixture-of-transformers design. The reasoning half works out how objects interact and how things move, and the generation half then produces video or a sequence of actions for a robot to follow.

Why a model for the physical world matters

Most of the AI that reached the public over the past three years lives in a text box. Physical AI is harder. A robot arm or an autonomous vehicle has to cope with messy lighting, obstacles that were not in the plan, and the unforgiving fact that a wrong move has consequences you cannot delete. Training those systems usually means gathering enormous amounts of real-world data, which is slow and expensive.

NVIDIA's pitch is that Cosmos 3 can generate realistic synthetic scenarios to train and test robots, shrinking work that used to take months down to days. "The Cosmos 3 family of open, frontier omnimodels gives developers a generational leap in ability to build robots, autonomous vehicles and vision AI," chief executive Jensen Huang said in the launch announcement. Self-praise from a chip vendor is worth a pinch of salt, but the direction of travel is clear enough.

Three sizes, two available now

The release comes in three flavours. Cosmos 3 Super is the heavyweight, meant for post-training robotics and autonomous-vehicle models that need the highest physics accuracy. Cosmos 3 Nano trades some of that fidelity for speed, generating video and action reasoning in a fraction of a second. A third version, Cosmos 3 Edge, is promised later for real-time inference on devices at the edge of the network. Super and Nano are out now on build.nvidia.com, Hugging Face and as NVIDIA NIM microservices, with the weights published openly.

NVIDIA also announced a Cosmos Coalition of robotics labs and model makers building on the release, among them Agile Robots, Black Forest Labs, Generalist, LTX, Runway and Skild AI. Open weights matter here for the same reason they did when MiniMax put its M3 model into the open: it lets smaller teams build on frontier work without paying a toll at someone else's API.

Whether Cosmos 3 lives up to the demos is something the robotics community will sort out over the coming months. What is already clear is that the contest in physical AI used to be about who held the best closed model. Now it is just as much about who is willing to give the technology away.