The tech giant will present 20 research papers advancing generative AI and neural graphics — including collaborations with over a dozen universities in the U.S., Europe and Israel – at the computer graphics conference coming to LA this August.
NVIDIA announced that 20 NVIDIA Research papers advancing generative AI and neural graphics — including collaborations with over a dozen universities in the U.S., Europe, and Israel — will be presented at SIGGRAPH 2023, this year running August 6-10 in Los Angeles.
NVIDIA research innovations are regularly shared with developers on GitHub and incorporated into products, including the NVIDIA Omniverse platform for building and operating metaverse applications, and NVIDIA Picasso, a recently announced foundry for custom generative AI models for visual design. Additionally, years of NVIDIA graphics research have brought film-style rendering to games, like the recently released Cyberpunk 2077 Ray Tracing: Overdrive Mode, the world’s first path-traced AAA title.
The research advancements presented this year at SIGGRAPH will help developers and enterprises generate synthetic data to populate virtual worlds for robotics and autonomous vehicle training. They’ll also enable creators in art, architecture, graphic design, game development, and film to produce high-quality visuals more quickly for storyboarding, previsualization, and production.
AI With a Personal Touch: Customized Text-to-Image Models
Generative AI models that transform text into images are powerful tools for creating concept art or storyboards for films, video games, and 3D virtual worlds. For example, text-to-image AI tools can turn a prompt like “children’s toys” into visuals a creator can use for inspiration — generating images of stuffed animals, blocks, or puzzles.
Designed to enable specificity in the output of a generative AI model, Tel Aviv University and NVIDIA researchers have two SIGGRAPH papers that enable users to provide image examples from which the model quickly learns.
One paper describes a technique that needs a single example image to customize its output, accelerating the personalization process from minutes to roughly 11 seconds on a single NVIDIA A100 Tensor Core GPU, more than 60x faster than previous personalization approaches.
A second paper introduces a highly compact model called Perfusion, which takes a handful of concept images to allow users to combine multiple personalized elements — such as a specific teddy bear and teapot — into a single AI-generated visual:
Serving in 3D: Advances in Inverse Rendering and Character Creation
Once a creator develops concept art for a virtual world, the next step is to render the environment and populate it with 3D objects and characters. NVIDIA Research is inventing AI techniques to accelerate this process by automatically transforming 2D images and videos into 3D representations that creators can import into graphics applications for further editing.
A third paper created with researchers at the University of California, San Diego, discusses tech that can generate and render a photorealistic 3D head-and-shoulders model based on a single 2D portrait. This breakthrough makes 3D avatar creation and 3D video conferencing accessible with AI. The method runs in real-time on a consumer desktop and can generate a photorealistic or stylized 3D telepresence using only conventional webcams or smartphone cameras.
A fourth project, a collaboration with Stanford University, brings lifelike motion to 3D characters. The researchers created an AI system that can learn various tennis skills from 2D video recordings of real tennis matches and apply this motion to 3D characters. Simulated tennis players can accurately hit the ball to target positions on a virtual court and even play extended rallies with other characters.
This paper also addresses the challenge of producing 3D characters that can perform diverse skills with realistic movement without using motion-capture data.
After generating a 3D character, artists can layer realistic details such as hair, a computationally expensive challenge for animators. Traditionally, creators used physics formulas to calculate hair movement, which is why virtual characters in a big-budget film sport much more detailed heads of hair than real-time video game avatars.
A fifth paper showcases a method that can simulate tens of thousands of hairs in high resolution and real-time using neural physics, an AI technique that teaches a neural network to predict how an object would move in the real world.
The team’s novel approach for accurate simulation of full-scale hair optimized for modern GPUs offers significant performance advancements compared to state-of-the-art, CPU-based solvers, reducing simulation times from multiple days to hours. It also enables quality hair simulations in real-time. This technique allows for both accurate and interactive physically-based hair grooming.
Neural Rendering Brings Film-Quality Detail to Real-Time Graphics
After filling an environment with animated 3D objects and characters, real-time rendering simulates the physics of light reflecting through the virtual scene. Recent NVIDIA research shows how AI models for textures, materials, and volumes can deliver film-quality, photorealistic visuals in real-time for video games and digital twins.
NVIDIA invented programmable shading over two decades ago, which allows developers to customize the graphics pipeline. In these latest neural rendering inventions, researchers extend programmable shading code with AI models that run deep inside NVIDIA’s real-time graphics pipelines.
In a sixth SIGGRAPH paper, NVIDIA will present neural texture compression that delivers up to 16x more texture detail without taking additional GPU memory. Neural texture compression can substantially increase the realism of 3D scenes, as seen in the image below, demonstrating how neural-compressed textures (right) capture sharper detail than previous formats, where the text remains blurry (center).
The seventh paper features NeuralVDB, an AI-enabled data compression technique that decreases by 100x the memory needed to represent volumetric data such as smoke, fire, clouds, and water.
Also announced today are additional details about neural materials research shown in the most recent NVIDIA GTC keynote. The paper describes an AI system that learns how light reflects from photoreal, many-layered materials, reducing the complexity of these assets down to small neural networks that run in real-time, enabling up to 10x faster shading.
The level of realism appears in this neural-rendered teapot, which accurately represents the ceramic, the imperfect clear-coat glaze, fingerprints, smudges, and even dust.
More Generative AI and Graphics Research
NVIDIA will also present six courses, four talks, and two Emerging Technology demos at the conference, with topics including path tracing, telepresence, and diffusion models for generative AI.
Source: NVIDIA
