Search form

Joe Letteri Talks ‘Avatar: The Way of Water’

Wētā FX’s 4-time Oscar-winning Senior Visual Effects Supervisor discusses his collaboration with director James Cameron on the highly anticipated ‘Avatar’ sequel, now in theaters.

In 2009, director James Cameron introduced moviegoers to a world unlike any they’d ever seen with his epic sci-fi adventure, Avatar. The film, the first ever to reach $2 billion at the global box office, remains one of the most popular movies of all time, single handedly ushering in the era of theatrical – and TV - stereoscopic 3D. A four-sequel production deal with 20th Century Fox ensued.

13 years later, fans of the film who were anxiously awaiting the first sequel can finally stop holding their collective breath. Set more than a decade after the events of the first film, Avatar: The Way of Water tells the story of the Sully family (Jake, Neytiri and their kids), the trouble that follows them, the lengths they go to keep each other safe, the battles they fight to stay alive, and the tragedies they endure. Starring Sam Worthington, Zoe Saldaña, Sigourney Weaver, Stephen Lang, Cliff Curtis, Joel David Moore, CCH Pounder, Edie Falco, Jemaine Clement, Giovanni Rabisi and Kate Winslet, with a screenplay by Cameron & Rick Jaffa & Amanda Silver, Avatar: The Way of Water, which opened in theaters on December 16, seems poised to repeat the success of its record-breaking precursor.

Fortunately, as he did for the original Avatar, Cameron again enjoyed the support of Oscar-winning visual effects powerhouse Wētā FX in New Zealand, led by 4-time Oscar-winning Senior Visual Effects Supervisor Joe Letteri, famed for his innovative work on, among other digital splendors, the CG characters Gollum and King Kong. Together with Lightstorm Animation Supervisor Richard Baneham, Letteri helped ensure that the emotional truth of the actors’ work was effectively transferred to their characters, and that the film raised the bar yet again for state-of-the-art cinematic visual wizardry.

We spoke with Letteri about what it was like to work with Cameron again on the much-anticipated sequel, how things have changed (or not) since the first film, and how Wētā keeps pushing the boundaries of visual effects.

AWN: The film was not only visually overwhelming – in a good way – but also great from a story standpoint. In terms of narrative and pacing, it was more than I expected. So congratulations on what I know was a Herculean effort.

Joe Letteri: Jim spent so much time on the stories for this one, and I read them all back to back, and they did not disappoint. That's where you want to start. Because if you don't have that, you're not going to get very far. And Jim's films have always been about story.

AWN: People who don’t understand how the performance-capture process works think that it's almost like cheating. But folks who have done this kind of work for years tell me, “Yes, you can get a great photoreal image. But once it's got to move, with emotion, it's just incredibly difficult.” It seems like you guys really nailed that on this film.

JL: That's been something I've loved to do for years, starting back with Gollum [in The Lord of the Rings]. And it is incredibly important to the film. You have to have that for it to work. And it's not an automatic process. It involves a lot of animators' time to get that right. We write a lot of software to try to infer and understand what's going on, like this new facial system that we wrote for this film. But we still have to look at it and ask, "Does it hit the mark?"

Because, for one thing, it's not one-to-one, you're translating the actor’s performance to another character. An AI system can’t do that. You have to go deeper, which is what we have done. We've taken it down to the muscle level, to what's happening under the skin. Once you translate it, it’s a very good starting point, but you have a team of animators who have to go in and make that character work and make that translation work. You look at the character performance side by side with the actor's performance, and, first off, technically you try to match the two. Then you see how close you are emotionally, and then you start adjusting until those two performances sit together. That's really the only way you can do it.

AWN: AI tools are getting more and more sophisticated. Will things like deep learning or neural networks make a difference?

JL: AI, deep learning, neural networks, they're all names for the same thing, basically. None of them are really about intelligence. What they are is a big matrix, a big mathematical equation. If you think about when you're keyframing something and you just do a spline in-out, that's a mathematical equation, but it's just working in one dimension. When you're working with a face, you have over 170 dimensions that all have to work together at the same time. That is really difficult for animators to maintain. And that's why you tend to get rubbery kind of motion, at least in early passes. And then, you spend a lot of time trying to get rid of that. You realize the key is that all of these muscles are connected, so if you pull on one muscle on one side of your face, the rest of your face moves sympathetically.

We don't create an expression by telling our muscles what to do, they simply do it when we express ourselves. We created a tool that utilizes a neural network to manage the interconnectivity, but that will never be 100 percent accurate. The animators still have to drive the process. We try to give the animators a good head start, but every shot has to go through the animators’ hands to make it work. There's just no way to do it otherwise.

AWN: How long did it take to develop the neural network tool, and what was the process like?

JL: We had the idea for it after we did Alita: Battle Angel. Alita used the same FACS-based system that we'd been using since Gollum, and I just realized that system had too many limitations. FACS is all about the emotional states – the happy, sad, angry – but it doesn't cover dialogue at all. Most of what we do is dialogue, so that was one big shortcoming that we always had to work around. That's when I started thinking about what would be a better way to do it, and came up with this idea for the neural network. We started hashing it out, and we had our R&D team start to write it, but as they were writing it, we had to start gathering data for the Avatar actors because we were getting ready to start shooting.

What we did was we had a big extensive FACS capture, because a lot of those same poses are still valid, and then we did dialogue on top of it, with standard line readings to use as a baseline. Then, we did what we called FACS2, which was Jim sitting with the actors and going through their lines and having them give a performance. We did all this in a chair with eight cameras, so we could really see what the face was doing and reconstruct a 3D moving model of it.

AWN: All the gear in the film is highly detailed and looks and feels very real. Did that present any problems in terms of movement?

JL: It has its difficulties. You have things like these crab suits that they jump into when they're crawling around with the tulkun. Or, on the flip side, there’s all the battle gear that these guys are wearing. When we're capturing them, they're not wearing that gear, so their arms are in a different place. Suddenly, you put them in that gear and it all changes. So then we have to redesign the gear to fit the motion. It was a constant back and forth with animation to get a performance that looked good. Because, again, it's all about motion. If you don't have good motion, if your animators can't give you good motion, why bother?

AWN: One of the things many designers talk about is how they'll start with 2D designs and then, once they move to 3D and folks start looking at how to model and rig things, designs need to change, sometimes significantly. Did you work with the designers and, if so, what was that process like?

JL: I was there for hardware, for creatures, for all of it. Now, the way that's changing is, with a lot of the design and motion tools these days, even the art department is moving into that and has started testing some of this early on. For Avatar, we also had an embedded team of animators from Wētā working at Lightstorm to do exactly that, to start producing motion studies and animations that could be used on the stage.

AWN: Years ago, when I interviewed [VFX Supervisor] Roger Guyett, he kept coming back to lighting. Everything revolved around lighting. The lighting in this film – especially in conjunction with water – was incredible. Is this just because the tools have gotten better, or did you take specific steps to achieve the results you got?

JL: When Jim is doing his virtual camera passes, he's looking at composition, because if you want something in silhouette or side lit, that changes the feeling of the shot. When we did the first film, we were working in MotionBuilder. We had eight lights, with no shadows. Jim could sketch out what we wanted, but he couldn't really give us a one-to-one.

For this film, we wanted parity between what Jim was seeing onstage and the final rendering. We wrote two renderers – Manuka, which is our full-blown path tracer to give us all the physically correct lighting, and Gazebo, which is our real-time playback and lighting engine. We basically embedded that inside of MotionBuilder. What Jim was seeing was actually coming through Gazebo, and so we had really good visual parity with Manuka. We could use shaders that would give us similar results. We could use lights that had the physical attributes that would play correctly in Manuka. As much as possible, we tried to bring parity between the two.

It gave us all something to start with, to the point where, if we're using HDRI maps for the environment, we gave all of our library to the team on the stage and said, “Here, don't make something up. Start with these. These are all valid.” And so when we got them back, we knew we had valid lighting environments to start with.

AWN: You’ve spoken about how it can be problematic to use the traditional tennis ball as the focal point for an actor, especially when you’re dealing with many different-sized characters, and about using monitors instead to provide actors with a better visual aid. Where do you see that going?

JL: You have to think about this as an integrated system. At the very low end, you still need a tennis ball from time to time, but it's not ideal. It's hard to position in the right place over time, and in the best case you still have an actor acting to a tennis ball. We wanted our blue people on stage acting. So if you can't do that, what's the next best thing? If we can get the face and the monitor in the right place moving at the right time, great. We had their performances, and we programmed that into this cable cam system with the monitor there. We decided there would be no ceilings on any of the sets because it's the only way you can get that in there.

We also had mocap cameras all around because we were capturing so much as we were going, so we always knew where everything was spatially. It worked out ideally for us; in other productions, it might be harder to rig something like that up. But we tried to use it wherever we possibly could because of the value we got. The character's in the right place, the actor’s in the right place. And with this new depth compositing tool that we wrote, you can actually see that they're in the right place in 3D in the video feed. It works for the actors, and it also works for the operators.

AWN: With everything you’ve accomplished in The Way of Water, will the next film in the series be easier?

JL: Absolutely. For example, we now have a good understanding of the characters. And that's the great thing about working on a sequel – you roll into it, especially when you're doing two films back to back. But there will be new things in the story that we won’t have touched before. I just always hesitate to say that anything is going to be easy because that will prove to be the jinx.

Dan Sarto's picture

Dan Sarto is Publisher and Editor-in-Chief of Animation World Network.