Getting <em>Wild</em> About Facial Animation

Check out the trailer, clips and featurette from Where the Wild Things Are at AWNtv! [1]

Rising Sun and Iloura came up with the technique of "sequence projection" often used to create talking animals. Further R&D was supplied by Animal Logic. This would help maintain the reality of the performances. All images © Warner Bros.

At first, Spike Jonze [2] wasn't sold on a CG approach to the facial animation for the creatures that populate his adaptation of Maurice Sendak's Where the Wild Things Are. He wanted to try animatronics, but his good friend David Fincher [3] convinced him otherwise by showing him some of the early tests they were doing on The Curious Case of Benjamin Button [4] (which, ironically, Jonze turned down). And that gave him the faith he needed to go down this path.

Not surprisingly, it was an unusual path -- even for CG.

The creatures were defined; the movie had been shot; and there was a rough cut, but the faces were static because of the masks. But the emotional impact was still apparent.

In steps Daniel Jeannette as animation/visual effects supervisor, who had previously worked on Happy Feet [5]at Animal Logic after a prestigious stint at ILM.

 "Early on, somebody said you should use CG creatures because it gives you the freedom to do what you want," Jeannette recounts. "It gives you a lot of control, but there's also a lot of limitation that comes with it. And, ultimately, you're talking about a lot of greenscreen work. This didn't work with Spike's vision of shooting on location or the core of the story, which is the relationship between Max and the creatures. It was better to have him interacting with actors in costumes. It was real, the performances were nuanced and emotionally it hits you in the gut."

However, Framestore wound up doing all of the animation and vfx. Jonze was very hands on: directing and helping supervise the delicate and crucial task of "making sure these creatures lived and had a soul." Photo by Sonny Geras. 

They tried a lot of techniques before realizing that the best approach would be limited facial animation enhancement whenever they needed lip-synch. In other words, strictly for the movement of the face, they only animated the eyes, lips, teeth and tongue.

And the faces in Wild Things were brought to life at London-based Framestore primarily using a new twist on an old technique known as "sequence projection" or "projection mapping," often used to create talking animals. For Wild Things, however, it had to be taken to a new technical and artistic level. Character suit performances were filmed with static faces. The faces or heads were tracked in 3D and a CG articulated head was animated on top to achieve the appropriate performance to match suit and dialogue.

The texture was extracted from each original frame and re-projected back onto the 3D model with the image "warped" to match the animation. CG eyes, mouth interiors and fur were all added, and 2D paint work helped complete a convincing final image.

"The beauty of it is that the lighting was essentially making sure that the eyes and the mouth had the proper light direction, but it wasn't so complex that you had to spend a lot of rendering time and a lot of time defining and refining where the light would come from to be accurate," Jeannette adds. "It's what we used. They also had to do a lot of compositing to reconnect all of the detail. Post-production lasted two years but they had eight months to do the animation."

They originally worked out of Australia, where the movie was shot. Animal Logic, Rising Sun and Iloura did early work. The technique was devised primarily by John Dietz at Rising Sun and Iloura in Melbourne. And then further R&D was done by Animal Logic.

"We did three or four months of development with them and as we were gearing up for post-production, they decided to re-tool a little bit and reshoot some elements to refine the story," Jeannette continues. "And that was in the middle of the strike. So we had to stop work for a few months and then did additional photography last year. But when we came back, we had to basically find a new home for the project for all kinds of reasons. Some because our vendors' slate might've grown too big and they couldn't take on as much work as we were hoping and also because at some point, I realized that, given the time we had left to do the work, piecemealing it was a very daunting template to follow, so we looked to find a single facility to do all of the work. And that's where Framestore came in.

Character suit performances were filmed with static faces. The faces or heads were tracked in 3D and a CG articulated head was animated on top to achieve the appropriate performance to match suit and dialogue. Photo by Sonny Geras. 

"As it goes, a lot of the work we did in Australia was very useful but not necessarily translated into Framestore's pipeline. So we had to rebuild all of the meshes and everything to fit their pipeline, which is essentially Maya [6] for animation, and rendering in RenderMan [7] for the eyes and inside of the mouth. Given the money and time, this was obviously the best approach.

"The challenge for Framestore was developing a pipeline in a very, very short time. An animator works on a low-textured look of a character and that goes to lighting and they apply the shaders and it goes to compositing. Here, compositing in animation happen at the same time. In fact, you had to do a lot of compositing ahead of animation so that the animators could work with the textures as they were photographed, as opposed to working with gray-shaded characters, as you usually do. What was great for us was they came right off of Despereaux [8] and had a crew of animators already there. And they had that experience of character animation on a large scale."

"I think it produces a really lovely visual aesthetic -- one that was certainly right for the feel of Spike's film," suggests Ben White, Framestore's CG supervisor, "It looks photographed and real in a very unfussy, non-CG way, so it's easy to concentrate on the performances."

Jonze wanted the Wild Things to be every bit as big and grotesque as in the book, and to this end he had appropriately enormous costumes designed. The necessarily light costumes contained no working parts, their faces static, awaiting post- production animation. In late 2008, Framestore convinced Jonze and his team that they were the best crew for the task, and a team that would grow to 250 strong began work on one of the toughest deadlines in the company's history, with more than 1,100 shots, or 1,400 "character appearances" to deliver in a little more than four months from production.

The texture was extracted from each original frame and re-projected back onto the 3D model with the image "warped" to match the animation.

According to Framestore, a key feature of their post work was the degree to which Jonze was personally involved. But partly due to the film's elongated production period, and partly because of the extraordinary challenges he was posing the Framestore animators, Jonze came to London and established an office within Framestore's production floor. From there he met, directed and helped supervise the delicate and crucial task of, as he put it, "making sure these creatures lived and had a soul." Going out of his way to motivate the London team, Jonze started them off by shooting a hilarious video pep-talk from Sendak, who exhorted the team to "make this the best thing you've ever done -- no futzing around!"

CG eyes, mouth interiors and fur were all added, and 2D paint work helped complete a convincing final image.

Heading the animation team was Animation Supervisor Michael Eames (Children of Men [9], Charlie and the Chocolate Factory [10]). "Spike has a disarming, disingenuous side to him, which is hugely effective when motivating people," he says, "But at the same time, I think he came to us knowing pretty much exactly what he wanted, and left us having achieved exactly that."

Eames admits to some initial misgivings about the sequence projection technique. "I'd worked with it before," he adds, "And its Achilles Heel has always been inadequate performance in, or capturing of, the original material -- in this case the suit work shot in Australia. I didn't think that it would be good enough to support what we could bring to it. But I cheerfully admit to being wrong about that. Spike recorded the voice actors visually as well as on audio before going to shoot, and then the onset suit actors were given specific references and beats to match their performance to these rehearsal takes. The result of this painstaking work was that, when it came to adding in the animation, the body motion usually made sense. Such problems as we had tended to be the result of later editorial changes rather than the performances themselves."

The technique was not without its difficulties at the Framestore end, though. "Manipulation of the 3D model was limited to how far we could stretch or compress the texture before it broke," Eames explains, "Sometimes it was just not possible to get the desired effect and we would have to work together with 2D compositing to effectively hand paint or animate an expression."

Further challenges were added when Jonze decided he wanted some fundamental redesigning of the creatures faces, in particular all the creatures' mouths; one character was redesigned by narrowing her face in the composite to give her a sweeter look -- to the tiniest of changes in mood -- raising the emotional temperature of an expression through additional 3D and 2D shading around the eyes, for example, counteracting an unsympathetic play of light.

Creating the Wild Things involved three distinct stages, but it was up to Framestore to tie the two initial interpretations of the characters' performances together and add the emotion that comes from facial expression.

Under the supervision of Tim Webber, Framestore also managed to deliver hundreds of regular vfx shots, including: flying clods of CG dusty earth in the dirt clod fight, matte paintings, scenery enhancement, rig fixing and removal, CG missiles, storm sequences, a day for night at sea sequence and greenscreen elements.

The process of Wild Things was a fascinating one-off experience for Framestore. As CEO William Sargent puts it, "Creating the Wild Things involved three distinct stages which all needed to work together in order to bring the characters to life: the suit performance, the vocal performance and the facial animation. Each stage was crucial, but it was down to Framestore to tie the two initial interpretations of the characters' performances together and add the emotion that comes from facial expression.

"Jonze is one of modern cinema's true originals, and it's improbable that anything quite like Where the Wild Things Are will come along again soon. Framestore is honored to be part of something as special as this project, and proud that it was able to deliver a rumpus every bit as wild as its creators hoped."

Bill Desowitz is senior editor of AWN & VFXWorld.