The Weta Digital visual effects supes discuss the intricacies of motion-capture and Andy Serkis’ outstanding performance in Matt Reeves ‘Planet of the Apes’ sequel.
Some of 2014’s most outstanding visual effects work comes courtesy of Weta Digital, who brought to life the Andy Serkis-led simian colony in Matt Reeves’ Dawn of the Planet of the Apes. The complexity and sheer volume of sophisticated animation required to tell the story of humanity’s clash with a growing society of genetically empowered apes was staggering enough. Factor in outstanding acting performances led by pioneering mo-capped actor Andy Serkis and you have one of the more compelling sci-fi films of the year.
Dan Lemmon and Keith Miller, two of Weta’s visual effects supervisors, had the daunting task of harnesses state-of-the-art motion-capture technology, often employed within the misty rainforests of British Columbia, to make audiences really believe they were watching a thousand-strong army of angry apes bent on annihilating one of earth’s last pockets of humanity. I recently had a chance to speak with Dan and Keith about their work, the intricacies of motion-capture technology used to capture live-action performances onset, and how regardless of the tools being used, digital characters are only as good as the acting performance they’re based on.
Dan Sarto: When do you guys normally first get involved in a film like this? How does it all start?
Dan Lemmon: In the case of this film, we got involved before the director was even involved. The first film [Rise of the Planet of the Apes or Rise] was directed by Rupert Wyatt. The plan was for him to do the second film as well. He’d written a treatment but felt he couldn’t make the movie he wanted based on the proposed schedule. So Matt Reeves came in after we had already begun looking at different concepts, rethinking some of the locations and making a plan for how we might shoot the film.
Matt came up with a different treatment, taking things in a different direction. We worked with him the whole way, talking about how we might get our equipment into some of the locations, how things might change from a lighting standpoint, how things might change from an animation standpoint based on where he wanted to take the movie. That was really important, especially as he started scouting locations and trying to figure out where and how he was going to shoot the film. We were figuring out how our tools would work in those environments and how we could get the best performances to make the best movie we could with the time and budget we had.
Keith Miller: Weta is constantly developing technology. So even before we were actively involved in the production of Dawn [Dawn of the Planet of the Apes or Dawn] we were constantly developing tools for shading and lighting. We’re continually improving our technology and it all directly applies to the films we work on.
DL: One of the things that changed for Dawn in particular was the way that we do our hair. We handled it a certain way on Rise, but between Rise and Dawn we did two Hobbit movies and a bunch of other work as well. In the course of doing that work we completely retooled our hair pipeline, mainly in an effort to get more realistic natural pelts of fur, get the detail and the clumping to be more controllable by the artists and therefore more bendable for a realistic result.
DS: You guys basically setup little performance-capture studios in the rainforests of British Columbia. The logistics must have been a nightmare. In such an environment, how do the practical issues of capturing the performance data you need impact the director’s ability to tell his story? How much does the technology impact what you can accomplish on that type of location shoot?
DL: I guess we try to look at the problem from the other side. We try to start with the goals of the story and figure out what tools we have now and what tools we have to invent in order to support the director’s vision. That’s what makes the work we do so much fun. You’re constantly trying to problem-solve and figure out how you can achieve something. But first and foremost, it’s done in service to the story.
DS: Do you find sometimes that you are able to serve the director’s creative vision despite the technology at hand?
DL: You are always constrained by what’s physically possible with the technology you have. But in terms of your toolset, part of the creative process is working with your constraints and figuring out how to solve the problems at hand. But having said that, the big thing for us is to really focus not so much on, “Hey, we’ve got this great tool that can do this and that” but on what kind of story we think we can tell with this tool. We’ve got this great story we want to tell. How can we tell the story in a way that’s going to be believable and engage our audience as much as possible? So early on, we start talking about that [with the director].
KM: The technology evolves over time. Like on Avatar. When Jim [Avatar director James Cameron] first wanted to make Avatar the tools weren’t ready. So he said we can’t make this film from a technological standpoint…
DL: …I was at Digital Domain when we finished Titanic. Just a few months later we were doing character animation tests for Avatar and he basically said, “We’re not ready.”
KM: Yah. So the toolset can limit things as well.
DL: After the success in particular of Gollum and The Lord of the Rings movies, it became clear that the technology as well as the artists had evolved to the point where we could tackle a digital character that could hold the attention and engage an audience for the length of a feature film.
DS: On a film like Dawn, what goes into the decision making regarding what you will shoot live and what you will do with full CG? What affects those decisions?
DL: It’s an ongoing conversation that we have with the production designer. His job is to create fantastic sets and amazing environments. But his job also is to figure out how to tell the story within a budget. How much he can build to shoot practically with physical sets and after talking with us, how much is practical to extend? The whole time you’re shooting you’re having constant conversations and little negotiations onset. OK, can we point the camera over here? Yes, but it can take us as much as an hour and a half to drag this greenscreen from one side of the set to the other. Each day you need to figure out where the different components need to be and what you can accomplish for the amount of time you’ve got. You’re always weighing how badly we need this component versus these other components. It’s not only a constant give and take with the visual effects department, but all the other departments as well: the grips, the production designers and set decorators. We are all trying to collaborate, to figure things out so it helps the director understand what’s practical and possible and helps everybody make decisions about where that energy is best put.
DS: I had a chance to talk performance-capture with Andy Serkis at FMX 2014. Were there any new tools or techniques developed since Rise that gave you better results or made things easier or more practical on Dawn?
DL: Like Keith mentioned, our tools are always going through an iterative process of improvement. Sometimes we make big leaps and other times we make smaller advances. One of the things that’s happened is our ability to process motion-captured data has improved. We’ve reached the point where we can get a lot more characters into a capture session at the same time when we’re working onset. So rather than being constrained to four or five actors at a time we can get up to 10 actors on the set performing. That’s really helpful when we’re creating a crowd that must be put all over the village and in the city. So that was helpful.
Our gear changed a little bit. I guess you could say it was a little less bleeding edge. A little more battle tested. I think for the actors that’s probably kind of transparent. Once they get the gear on it doesn’t really affect what they’re doing. Our job is to get in their way as little as possible so that they can craft their characters and interact with one another with as few distractions as possible.
One question we often get is, “What makes Andy Serkis such a genius at motion-capture acting?” And the answer is he’s not a genius at motion-capture acting. He’s just a genius at acting. He’s a really talented character actor who puts in a lot of work and crafts his characters in an engaging way that connects with the audience. If you look at each one of his characters [performance-capture-based], they are all very different. Kong is in no way similar to Caesar apart from they both happen to be apes. Gollum as well couldn’t be more different. He is just really, really good at making characters. And the performance-capture technology has nothing to do with his performance other than the fact that we then change his appearance.
DS: Do you think there is a misconception with audiences about what performance-capture really is? They see it in games and filmed entertainment all the time and probably just figure you throw someone into a suit and the computer turns them into some creature. Do you think people really understand how unbelievably difficult the process is and how much animation you have to create to really capture the performance?
KM: Maybe people are starting to understand. They’re certainly becoming much more familiar with the processes. It’s much more out in the open now that a number of films these days make use of motion-capture. They’re exposed to it in general quite a bit more.
DL: There’s a notion that there is a keyframe look and a motion capture look and people will argue the merits of using one approach or another for a particular kind of character. They’ll say, “We don’t want this to look like motion capture.” That kind of underscores what you were talking about. There is a perception that these are tools that do one thing and it’s always going to look like that one thing. There are a lot of different tools that go into crafting a performance. Performance-capture is a major component. But what makes those characters so compelling is the contribution of the actor. If you have a less talented actor in the suit then you’re going to get a less interesting performance. For us, it also depends upon everybody down the line after the capture happens that puts their heart and soul into taking all the beautiful work that Andy has done in front of a camera and making sure it gets up on the screen.
I’m in no way trying to disparage the work of game artists. There’s some really excellent work being done today. But with a capture-plugged-into-a-character type approach, you do get a certain kind of robotic artifact. It’s a big part of our job to go in and break down those artifacts and make sure we are honoring the spirit of what the actor did on the set. We are shepherding an actor’s performance through all these tools that have their own quirks and vices to get what they did that day up on the screen.
KM: Aside from the pure emotion and performance aspect there’s so much of the aesthetics of the character as well which drive the believability.
DS: The scope of this film is enormous. It must have been an arduous production. There are often hundreds of apes on the screen at the same time. How much is simulated? How much is pure CG as opposed to animated performance-capture?
DL: It was a really big job. We’ve got people that put hundreds of hours into some of those scenes in terms of getting all those crowds populated, making sure nothing stands out as false, that you’re not seeing repetitions of the same motion but you’re perceiving infinite variation when in fact there isn’t infinite variation. There are repetitions in there. If you stared at those images long enough you could point that out.
You have a handful of actors that are being captured onset, but then you have a lot of background actions captured on another day and another place. We do a lot of these background vignette captures back is in New Zealand where we get…
KM: The pure massive simulations are restricted just to those show of strength scenes and…
DL: …Yah. There’s was a little bit in the background on some of the raid on the colony scenes and that was pretty much it…
KM: All the background apes in the tower fight are following a schoolyard fight style and it needs to match specific emotional beats that are happening throughout the story. That’s difficult to portray through massive simulation.
DS: This film involved such a huge volume of complex animation and visual effects work. What were the biggest challenges for you as supervisors? What kept you up at night? And conversely, what part of the experience did you enjoy most?
DL: [Laughs] I guess for me as a supervisor, the most challenging thing is probably just making sure from the beginning to the end that every day is being used as efficiently as possible in terms of chewing away on the big problems. It’s all about arranging things so that when you get to delivery mode at the very end, after all the decisions have been made about which shots are going into the movie, that everything can just kind of flow. You don’t want to end up getting into the final composite and rendering of shots and realize, “Oh dear, his face wasn’t high enough resolution and these wrinkles just aren’t working.” There always are little things like that but you want to try to tackle as many problems upfront…
KM: …It’s really satisfying when all those pieces you setup and staged in a particular way now just fall into place and the shots just start rolling through.
DS: Things are actually all working.
KM: Yah. There’s usually an element of shock. “Oh wow, things are actually working at this stage and we are pushing the shots out!”
DL: We are making a movie.
DL: For me, that’s the best part of the whole process. You’ve done all this work onset and all this work back at post, you’ve been chewing away at these shots and they’re all in different stages of process. All this different work is going on with animation and roughing and things always look kind of crude as you make a lot of judgment calls. You’re saying, “OK, we’re going to tweak this lip shape here, we’re going to adjust this eye wrinkle here and here, change the way the hair looks here, we need more water droplets in the fur.” You’re adding all these things.
But one day you throw the shot up with the background plate and suddenly you see this character you’ve been working on for months, sometimes years, and it looks believable. You see him alive and breathing and suddenly you get sucked into that moment. That for me is where the magic happens. Suddenly, something that was never there just appears. There’s a moment where you suddenly believe in this character you feel, “Oh my god, this is amazing.”
That’s one of one of the most satisfying moments as a supervisor. You have problem shots that you’ve been working away on for months. Then somebody shows something in dailies and you have this moment where you’re just blown away by the skill of the artists you work with. Something suddenly just got transformed into a work of art. That’s pretty amazing when you realize you’re working with such talented people.
Dan Sarto is Publisher and Editor-in-Chief of Animation World Network.
Dan Sarto is Publisher and Editor-in-Chief of Animation World Network.