Oscar winner Joe Letteri isn’t resting on his laurels. His work is cut out for him in the upcoming The Water Horse and Avatar.

After winning an Oscar for vfx on King Kong, Joe Letteri has remained at Weta Digital to supervise work on The Water Horse (Sony/Revolution, Dec. 7, 2007), including the CG sea creature, and Avatar, James Cameron’s long-awaited, first feature since Titanic. Letteri also discusses early work on the CG Silver Surfer from Fantastic Four: Rise of the Silver Surfer (Fox, June 16, 2007). Under the vfx supervision of Kevin Rafferty, Weta has reportedly enhanced its CG animation process that employs performance capture techniques to add further dimensionality to the liquid-metal hero performed by Doug Jones.

Bill Desowitz: Please fill us in on the state of the industry with regard to digital actors and 3D environments.

Joe Letteri: I think as far as digital actors go, in terms of characters rather than digital doubles, we’ve come pretty far. Especially with motion capture and facial motion capture, you can really work with an actor to develop the character. It’s an extension of what we did with Andy Serkis with Gollum and Kong. There is still a huge call for an animation team to work the actors and help develop the digital characters the rest of the way, because, typically, there are characters that are non-human and invariably there are things that a human actor cannot do. It’s too dangerous or physically not possible because of the configuration of the character. So you need this integration with the actors and the animators to pull everything together. I don’t see that going away anytime soon. I’m not sure you want it to go away because it’s really a great combination to have.

BD: In terms of techniques, how do you see it evolving between performance capture and keyframe?

JL: What we’ve always done here is to try to make the path as two-way as possible so we can start off with performance capture but then layer on the same set of tools that the animators are used to working with. So it’s always the call of the animators to make requested changes. Sometimes it sails straight through based on what the motion editors are doing. Other times, if you get a request to change the performance, you just have to decide at some point the data is too heavy and it’s going to be easier to reanimate it. The performance is a guide. But again, by having the data there, you’ve got a good starting point for the character and what you need to do next.

BD: What do you think of the various techniques that are available, including what Sony is doing with Imagemotion, what ILM did with Imocap on Dead Man’s Chest, Face Robot from Softimage and the new Contour from Mova that was introduced at SIGGRAPH?

On Kong Letteri had realtime body capture but not facial. He looks forward to a time when a director will see realtime physical and facial capture on the set. All King Kong images © 2005 Universal Studios.

JL: I think those are all great ways to go. They all bring a little bit of something to the toolkit that you have and each has its own strengths and weaknesses, so you can tune which technique you use to the situation. In the course of a large feature, you’re probably going to use several of those techniques. It’s great to see all of these things being developed in different ways to look at an important part of the problem for a particular task and try and solve that.

BD: I had a chance to ask Andy Serkis about performance capture recently and he thinks there will be a time in the future when a director can look through a viewfinder of a handheld camera and see in realtime physical and facial capture. Is this something you’re keen on?

JL: Yes, I think that would be a great thing to have to be able to work with the actor to get as much of his performance as possible. For example, on Kong and even with Gollum, we had a lot of the body stuff working in realtime. But looking at the facial, there was a problem of translating and learning the character, particularly with Kong. Yes, I would like to see that happen. One thing that it means, though, is that all that character development has to be done upfront if you’re going to do it all on stage with a director. We spent weeks and weeks with Andy on the motion capture stage for Kong and we got to digest all of that and turn that into his character. So it means shifting the way you’re doing things. It’s another one of those paradigms where what we used to call post-production pushes more and more into preproduction.

BD: How close are we to conquering “The Uncanny Valley?"

JL: You mean the real/not real, the human/not human? I think it depends on the application. Obviously we have seen digital doubles done really well using a lot of image-based techniques, for example, where you couldn’t tell the difference between the original performance of the actor and the digital performance. So from that point of view, it’s almost already been hit. But to take a human performance and have it actually be a human character? I don’t know. I’ve never been in that situation before. We usually just hire a real actor. We’re always looking at creating the other characters.

BD: And that includes Avatar? (The sci-fi film is about Jake, a paraplegic war veteran who is brought to another planet inhabited by a humanoid race at war with humans.)