In this month's edition of "The Digital Eye," Peter Plantec chats with Paul Debevec about his latest research projects involving realtime 3-D display and capturing and rendering human faces, including a new skin rendering technique.
Image courtesy of Deron Yamada. © 2004 DYA367.
Understanding light is key to making art and movies, in particular. The old Hollywood masters knew how to create mood with light, enhance beauty with it and create a sense of place. Poorly understood, light can ruin creative effort and confuse an image. Real light is wonderful to work with in the hands of a lighting master. When you think about it, light is our main interface with life, supplemented by sound and bounded by touch.
Today we work as much with virtual as real light… perhaps more. And understanding virtual light is more difficult than understanding the visual impact of real photons. But understanding virtual light brings enormous power over image. Look at what Photoshop tools can do to a snapshot, refining a drab, ugly image into a vibrant beautiful final -- and that's just the beginning. Light reveals information and that information can be recorded and used in new ways to create images that never existed. As we move deeper into our understanding of light, amazing things are becoming possible and one man, in particular, is blazing our way into that light. Dr. Paul Debevec.
Debevec is associate director of graphics research at the University of Southern California's Institute for Creative Technologies. More than that, he is a true pioneer in many technologies that, once discovered, have become part of the pallet of every 3D artist and digital filmmaker. The large studios pay big bucks to be the first to use Dr. Debevec's inventions to enhance their vfx heavy films like the Matrix and Spider-Man franchises and King Kong. Let's take a look at what he's been doing lately.
Dr. Paul Debevec's lastest research includes designing a working prototype of an easy to build, 3-D display system using a standard video projector. Unless noted, all images © 2004 University of Southern California.
A Realtime 3-D Display in the Round
Debevec's team at the ICT has designed a working prototype of a reasonably easy to build, 3-D display system using a standard video projector, which has been modified to project over 4,000 frames/second. Their original design has a number of advantages over typical 3-D display technologies:
omnidirectional -- It generates 3-D images in the round, so you can walk around it seeing it from all sides.
autostereoscopic -- which means you don't need to wear any kind of glasses to get the effect… a big advantage.
animated/interactive -- the image can be updated 200 times a second allowing for fine detail interaction and full animation.
The main disadvantage is that in its present state of development it is monochromatic. The modifications on the video projector necessary to output the tremendous amount of information needed to reproduce such an image requires that all the color channels be repurposed to the main mission, leaving color behind. However, this is not a permanent problem as solutions are on the horizon using three -chip video projectors.
Dr. Paul Debevec.
The 3-D images are created by projecting super high-speed video onto an almost invisible spinning mirror. The projected images are synchronized so that as the mirror spins, it reflects a different image towards each potential viewing angle, accurately portraying the object from each perspective.
According to Debevec, "Our rendering approach is capable of recreating real scenes as well as virtual ones, with correct occlusion, horizontal and vertical perspective and shading." I did notice that tilting one's head causes some minor distortion in the image.
Debevec expressed optimism about ever getting rid of the confining mirror. Actually, looking at the image, seemingly floating in space as you walk around it, gives one an uncanny but cool feeling. You want to reach out and touch it, which is probably why the image is created inside a glass enclosure. I imagine the spinning mirror could be a bit painful if one lost his head.
Debevec explained that in order to succeed, the next-generation of 3-D displays must begin to represent the physical world more accurately and in the round. He's currently working on a color display using a three-chip video projector, experimenting with tent shaped mirrors to create a more solid and satisfying image, reducing the uncanny effect. He added, "The Holy Grail in this arena is to find a good solid state approach to make light wag back and forth like our spinning mirror does today." That one is a ways off.
Debevec's rendering approach is capable of recreating real scenes, as well as virtual ones.
Capturing Faces and Hearts
Another project that Debevec and his Graphics Laboratory team are involved with is the acquisition and rendering of ultra-high resolution models of human faces, bodies and performances. Debevec has long been involved in the development of image based lighting and he's taking that many steps further with new work using Light Stage 5. Similar to Light Stage 3 but using new ultra bright white LEDs that can be driven at any gray levels, LS5 is being used to capture the human face in remarkable detail. By measuring normal maps with spherical gradient patterns the team has been able to capture human pores and fine skin wrinkles down to an accuracy of 1/10 mm. To date, acquiring such detail has required high-res face casting which is expensive, time consuming and grossly uncomfortable.
Debevec explained further: "The neat thing about it is that we can capture different facial expressions very fast and build up a library from which we can reconstruct a wider range of different lighting scenarios."
And how does he manage to create normal maps using LS5? "We get the high-resolution skin detail by estimating surface orientations from the skin's reflection of polarized spherical gradient illumination. We have full control of the brightness level of each LED in the sphere, and we put polarizers on the lights to create polarized light gradients. From the gradient patterns -- where we create brightness ramps of light from the top to bottom, left to right and front to back of the stage -- we can estimate the surface orientation of each photographed pixel of the face. More importantly, we can use a polarizer on the camera to isolate the skin's specular reflection, which is light that has not diffused within the skin tissue and thus reveals the detailed structure of the skin pores and fine wrinkles, which are otherwise nearly impossible to detect. The result is that we can apply this high-resolution detail to a traditional 3D face scan to capture high face models accurate to a few millimeters in just a few seconds. Since we capture aligned, lighting-independent texture maps to go along with the geometry, the resulting models have all of the irregular colorations and skin detail, which make people look human."
Simultaneously the group is developing a new skin rendering technique that cleverly uses these highly detailed surface normal measurements for the specular and diffuse reflection components of human skin. Resulting renders can be spectacularly realistic and it's possible to accurately create match lighting for live-action backgrounds.
In his latest work, Debevec and his team are having success with realtime capture of facial performances. Using realtime data generated in their Light Stage using high-speed gradient illumination and structured light, they calculate the parameters needed to precisely synthesize face geometry, and then animate it with near perfect accuracy. "The facial details, skin folds and pores all provide excellent and accurate tracking points for re-creating extremely accurate face performances," Debebec stressed. Why is this important?
Experts in virtual actors agree that if there are enough perfect details, viewers will accept a virtual performance as real.
At the recent fmx conference held in Stuttgart, Germany, a number of us interested in virtual human actors, including Debevec and Christophe Hery -- the guy who developed the render technology for the baby in Lemony Snicket's A Series of Unfortunate Events and Davy Jones at ILM -- got together and discussed face capture and rendering. It quickly became clear how important these are to crossing the Uncanny Valley. That is, where a virtual actor's performance approaches the look of reality, we become increasingly sensitive to flaws in look and performance. This gave many viewers an uncanny feeling watching The Polar Express. However, characters such as Davy Jones in the last two Pirates of the Caribbean films and the baby in Lemony Snicket demonstrate that if you get enough details down almost perfectly, people will accept a virtual performance as real. The baby, of course, becomes uncanny after a bit as his behavior becomes ever more bizarre, but that was the plan.
Debevec's work is leading us across that Uncanny Valley and I speculate that within two years his laboratory will have developed face and performance capture technology with associated rendering, which will give us virtual human performances that we will, in our hearts, accept as real.
Peter Plantec is a best-selling author, animator and virtual human designer. He wrote The Caligari trueSpace2 Bible, the first 3D animation book specifically written for artists. He lives in the high country near Aspen, Colorado. In addition to his work in vfx and journalism, Peter is also a clinical psychologist with more than a decade of clinical experience. He has spent several years researching the illusion of personality in animated characters. Peter's latest book, Virtual Humans, is a five star selection at Amazon after many reviews.
