New AI application inflates flat 2D images into realistic 3D models; artists in all fields can easily add new objects to their mockups without 3D modeling or rendering.
Nvidia researchers today have unveiled a 3D model – a revamped Knight Rider KITT car – produced from a 2D image by a new deep learning engine, GANverse3D; the new model includes the mesh, textures, and information that allows you to animate the object automatically.
Developed by the Nvidia AI Research Lab in Toronto, the GANverse3D application inflates flat images into realistic 3D models that can be visualized and controlled in virtual environments. This capability could help architects, creators, game developers and designers easily add new objects to their mockups without needing expertise in 3D modeling, or a large budget to spend on renderings.
A single photo of a car, for example, could be turned into a 3D model that can drive around a virtual scene, complete with realistic headlights, taillights, and blinkers. By combining the new model with NVIDIA Omniverse - announced during CEO Jensen Huang’s GTC keynote on Monday - the research team revamped KITT into a car for the 21st century.
To generate a dataset for training, the researchers harnessed a generative adversarial network, or GAN, to synthesize images depicting the same object from multiple viewpoints — like a photographer who walks around a parked vehicle, taking shots from different angles. These multi-view images were plugged into a rendering framework for inverse graphics, the process of inferring 3D mesh models from 2D images.
Once trained on multi-view images, GANverse3D needs only a single 2D image to predict a 3D mesh model. This model can be used with a 3D neural renderer that gives developers control to customize objects and swap out backgrounds.
When imported as an extension in the NVIDIA Omniverse platform and run on Nvidia RTX GPUs, GANverse3D can be used to recreate any 2D image into 3D — like the crime-fighting car KITT, from the popular 1980s Knight Rider TV show.
Previous models for inverse graphics have relied on 3D shapes as training data.Instead, with no aid from 3D assets, “We turned a GAN model into a very efficient data generator so we can create 3D objects from any 2D image on the web,” said Wenzheng Chen, research scientist at Nvidia and lead author on the project.
“Omniverse allows researchers to bring exciting, cutting-edge research directly to creators and end users,” said Jean-Francois Lafleche, deep learning engineer at Nvidia. “Offering GANverse3D as an extension in Omniverse will help artists create richer virtual worlds for game development, city planning or even training new machine learning models.”
Pop on over to the Nvidia website to read their extensive post on the new application.
Source: Nvidia
Dan Sarto is Publisher and Editor-in-Chief of Animation World Network.
