<em>The Polar Express</em> Diary: Part 3 -- The MoCap/Anim Process

[1]

The Polar Express animation supervisor David Schaub. [1]

This is the third of four installments in VFXWorld’s The Polar Express production diary. Read Part 1 [2] and Part 2 [3].

To recap, “MoCap/Anim” is the work that the animation department does on top of (or in addition to) the performance capture delivered to us from integration. A description of the animation rigs and workflow is covered in the Animation Rigs & Controls [4] discussion in Part 2 of this series. The shots are delivered to us from integration as Maya files with all of the assets embedded. That includes audio tracks and all the pertinent video references set up on image planes.

Using the performance capture as our foundation, the animation task is to fill in the blanks — but only to the extent indicated in the video references provided to us from production editorial. Often a shot will come with notes from Bob Zemeckis indicating a desired performance change. For example, a character should look left instead of right — or perhaps do a double-take at a specific moment. Other than that, the actor’s performances were sacred unless Bob indicated otherwise. The hardest thing to do in the beginning was avoid the temptation of “helping” a performance along by pushing an extreme or making an action broader, etc. Bob knew these performances inside and out, and his goal was to replicate those performances without any further interpretation by the animators. One exception to this rule is body dynamics that often had to be animated (or re-animated) if there was some external force acting on a character that was not captured on set. The simplest example of this would be upper body animation to support the illusion of a rocking train, especially when the characters are on the roof.

[5]

Hero Boy’s movements were performed by Tom Hanks, but the digital artists found that when retargeted to the proportions of a child, the actions look like a slow-motion run on the moon. All Polar Express images © 2004 by Warner Bros. Ent. In [5]

The Sequences
The obstacles we encountered in the first six sequences were among the most difficult challenges we faced on the show. This is where our processes were being established, and where most of what was broken got fixed. I am not suggesting that life was easy after that, but those six sequences taught us much about what we were up against and allowed us to establish a viable workflow based upon the lessons learned. What I will do now is walk you through some of those milestones and lessons that paved the way for the remainder of the film. The dates obviously overlap because there are multiples of shots in play at all times.

Boy Awakes Sequence (BA)
Number of Shots: 18
June-November 2003

June 11, 2003
Our first shot through the pipe is also the first shot in the film where the camera travels in through the window, moves over the bed and closes in on an extreme close-up of our hero (Chris). The reference video shows Tom Hanks lying very still, “breathing slowly and silently.” There is no detectable movement in the close-up video reference, other than a hint of movement in his chest as he breathes. However, the performance capture provided us with some lovely body tremors and twitches. Given the fact that the character is sleeping, it made no sense to use this noisy data — so we turned it off and animated by hand.

When Tom opened his eyes in this shot, his eyes were absolutely fixated as he stared at the ceiling. Although it could not be seen in the reference, lead animator Kenn McDonald added a hint of saccade (tiny intermittent eye movements) to keep the eyes alive. One of those movements was perceivable change in eye direction. When Bob saw this he instantly pointed out that Tom did not do that in the reference video. He was right. Tom’s eyes were frozen in silence as he listened intently for the sound of the bell. A change in eye direction would indicate a change of thought (a distraction), and that was not the intent of the moment. We had missed the point! This is an example of “reinterpreting” a performance, which is something we needed to avoid — a lesson driven home right out of the gate. So we eliminated the obvious directional change, but kept a small amount of saccade in keeping with the natural physiology of the eyes.

[6]

When the train arrives, Hero Boy’s room and props shake violently. Stand-alone props with animation curves applied could not make it through the pipeline as-is, so a procedure was scripted to drive a rigged component. [6]

Once the performance had been replicated, I still thought that something was wrong. Even though breathing animation had been added, our boy still looked dead. Tom was absolutely still in the reference, so we couldn’t add gratuitous movement — we had been down that road before! So what was the difference between a sleeping boy and a dead boy? With that question in mind, I strapped a laser pointer to my head, lay on the floor and watched the little red dot on the ceiling. Try as I might, I could not keep the little dot from chattering all over the place, even while holding my breath. Beneath the random noise was a very distinctive rhythmic bounce — the heart! Obviously, the dot would not have moved at all had the pointer been strapped to a cadaver. So now we were on to something. Noise curves were added to the body, neck and head to emulate the noise pattern that I saw projected on the ceiling. Animator John Matthews hunted down an audio waveform of a heartbeat, and used that waveform as another driver on the same joints. Believe it or not, these very small nuances make a huge difference in the viewers’ perception of that kid lying there — subliminal as it may be.

[7]

Acquiring Tom Hanks’ likeness in the face of the boy was a losing battle until it was decided to add a hint of Tom in the brow area in order to help sell expressions. [7]

July 17, 2003
In the next shot, our hero gets up out of bed and goes to the window. This was a very complex shot with lots of contact moments between our character and props in the room. There was very little movement coming through in the motion capture spine, so the offset spine and redistribution controls were used to create a more natural compression and flexing in the spine as the character crawled across the bed. Deformers are used to animate depressions in the mattress and pillow as the character moves through the scene. Representative geometry is deformed and animated to define the desired behavior of the blankets, which will ultimately be simulated. As lead animator Chad Stewart continues to animate the feet and hands, the snapping controls “explode” when put to the test.

When we see our character posed at the window for the first time, it strikes everyone that his arms are freaky-long! The character goes back to modeling to shorten the arms — and we squeeze in a few other fixes while we’re at it. Because of this, the character must be re-rigged, and the snapping problem ultimately gets resolved as well.

July 21, 2003
In our next close-up, Chris hears the bell and looks over his shoulder toward camera. There is a very specific look on Tom’s face in the video reference that Bob is intent on replicating. When he saw our work he kept saying that he “just wasn’t seeing Tom in our boy.” However, the boy is a very different character design with little or no resemblance to Tom. Acquiring Tom’s likeness in the face of the boy was a losing battle, unless we deformed his face to look more like Tom’s. The area that Bob was focusing on was the brow region, and the shape of the boy’s brow was very different from the distinctive “Nicholson-arch” that Tom has. It was decided to add a hint of the Tom Hanks likeness in the brow area in order to help sell expressions like this in the future.

August 19, 2003
New textures were painted on our boy to represent the Hanks brow shape more accurately. Once this was approved, the hair-patch layout for the eyebrow was modified to match. The end result is that the desired expression comes through. Since the brow design changed, all shots leading up to this had to be reworked and all works-in-progress (which there were many) were upgraded to incorporate the design change.

To avoid this problem moving forward, the character design was locked. Our work in the future would be an interpretive evaluation of one character transposed to the next. We don’t want to skew our characters’ face in the interest of caricaturing the facial topology seen in the video reference. Instead we are reading between the lines and transferring the intent of an expression from one character to the next. For instance, there is a series of shots when the boy is frustrated that he cannot hear the bell. There is no dialogue, and little physical likeness between Tom and the boy; however, Tom’s mannerisms and expressions clearly come through.

[8]

The Conductor’s props were a challenge. The watch chain was rigged and the dynamics were animated by hand. The Conductor’s glasses had to be constantly adjusted too. [8]

July 17-October 24, 2003
Our next milestone occurred when our boy sneaks down the stairs, and when he realizes that Santa hasn’t visited yet, rests his chin on the banister as he ponders that thought. The problem was that his chin was rigid as it rested on the rail, and not deforming as you would expect. All of our facial controls were designed to imitate muscles affecting the skin from underneath. It was never considered how we would deform the skin and underlying tissue with an external force. John Matthews did some investigation into how we might do this and came up with tweak clusters as the solution.

A tweak cluster is a selection of points on a surface grouped together under one controller. The controller allows you to push, pull, rotate or scale many predefined points on a surface. The weighting of each point (how much it is influenced by the movement of the controller) is accomplished by “painting” the amount of effect that the controller has on the selected points. In general, the weighting is painted such that the central region of control has the greatest affect, graduating to minimal control toward the outer edges. This allows you to essentially sculpt the model, and animate that sculpt on the fly.

John creates a single control on the tip of Chris’ chin that would realistically bend and give way as he put the weight of his head on the banister. It was easy to set up, easy to use and the end result was very effective.

Train Arrives Sequence (TA)
Number of Shots: 36
July 17, 2003-February 16, 2004

October 9, 2003
A tricky shot of the boy pulling a number of deforming props from a drawer has been underway for quite some time. Several elaborate rigs were built early on to manipulate the props — but the results were only moderately successful. Upon discovering the tweak cluster solution, the shot was retro-fitted with tweak clusters and animator Marco Marenghi was able to produce the desired results almost immediately.

June 19-December 8, 2003
When the train arrives, the boy’s room starts shaking violently, which means that all of the props in the room need to shake also. Animator Stephane Couture begins animating each prop by hand, which seemed to be the most logical approach at first. As the shaking gets progressively more violent he begins experimenting with dynamic simulation and is able to achieve excellent results in very little time. A good example is the jar of pencils that falls over, and the pencils spill onto the desk and some roll off the edge. The lamps and lampshades were also set up for dynamics, and some very natural results were achieved.

Unfortunately, stand-alone props with animation curves applied could not make it through our pipeline as-is. An animatable rig is required in order to survive the upgrade that occurs when a shot moves forward to the next department. To solve this problem, Carolyn Oros (pipeline TD) scripted a procedure that was able to take those rudimentary props and use them to drive a rigged component so that it would follow the pipeline of our component assembly system. This became yet one more of our standard tools.

Boards Train Sequence (BT)
Number of shots: 30
August 13, 2003-April 12, 2004

August 18, 2003
After the train arrives, the boy opens the front door and runs down the steps toward the train. This action was performed by Tom, but when retargeted to the proportions of a child, the action looks like a slow-motion run on the moon. The reason is that the stride-length for the child is much less than that of the adult who performed the action. However, the timing foot-to-foot (and all of the body language that goes with it) is still in the same time-space as the adult performance. To make the adult-to-child conversion believable, the entire action was sped up by about 12%.

September 26-December 17, 2003
When we first meet the conductor, many of the same problems we experienced with our hero boy are revisited. At this stage there is no pose library, so all of the facial work is done with muscles and tweak clusters. Poses will be added to the library as we move forward. An interesting development is that tweak clusters can now be saved as part of the pose and saved as such in the pose library. The conductor is also equipped with several animatable props, including his hat, glasses, pocket watch and chain. I make a plea to our producers that the watch-chain be simulated by the effects group. Unfortunately their workload is insane as well, so the chain is rigged and the dynamics are animated by hand.

In the beginning, the addition of other kids to the cast caused Maya to crash with great regularity, costing time and work losses to the production. Originally, Sony’s digital team budgeted the background kids for simple plug-and-play, but they stood out like sore thumbs next to the featured cast. In the end, many hours were put into animating faces away from camera.

Meets Kids Sequence (MK)
Number of Shots: 21
November 11, 2003-March 9, 2004

Up to this point in our story we’ve only dealt with two characters: Hero Boy (Chris) and the Conductor. When we load these two characters into a scene (along with the low resolution train), we still chug along in Maya with a workable amount of computing power and performance. Once we raise the resolution of a character (which is necessary for facial animation), we often bring Maya and our computers to their knees.

The situation became critical once we stepped onto the train-car full of kids.

The typical scene file with just the train and Chris was running upwards of 200 megabytes and when opened in Maya was taxing the computer’s RAM to the tune of 1 gigabyte or greater. Needless to say, our first endeavors to add additional kids caused Maya to crash with great regularity, costing us greatly in terms of time and work losses.

Bert Van Brande and his animation pipeline team came upon the idea of splitting the scenes into manageable chunks. Scene files are put together in integration where they are using significantly lighter rigs for the characters and the train. Before the facial capture is applied, the integrated body performances are submitted for approval (into animation), and the scenes split before transferring the files to Maya. If the characters are constrained to the train, each split file would carry with it the motion from whatever node the characters were constrained to. All of these splits had to be recomposed into a single movie file for review by Bob.

After struggling through the process of using multiple image planes of each split in our render, character setup TD Amy Hronek came up with the Render Compositing Tool. This used Z-depth to automatically render and composite all the splits at once. It was a breakthrough that solved a plethora of problems. Now several artists could work on splits independently, and a single artist could render and composite each split in a final render for Bob to review.

Background Kids
There was an ongoing effort throughout production to seek simplifications that would drive the costs down. Given the amount of attention and effort put into our hero kids, it was decided at bid-time that the background kids would be plug-and-play straight out of integration. There was certainly no budget for any animation work on the extra characters, but I knew they would stand out like a sore-thumb if we mixed them in with our heroes. I proposed that we set them up as “bobble-heads,” where we only see the tops of their heads moving above the seat-backs. We could animate and refine a library of cycles that could be plugged into any shot that required seeing them. Bob agreed to try this simplification, but after presenting a few tests he thought that the kids were simply too disengaged from the surrounding action. At this point, production editorial began specifying specific performance beats for each one of the background kids. This made them prominent in the shots, so a certain level of animation became necessary. If the kids were bouncing on the seats, or running across the aisles it was necessary to do the 12% time scale (to correct the adult-to-child time space) and apply the supplemental animation where necessary to fill in the gaps. There were also the contact issues with seats, props and other characters in the scene. Unfortunately, the background faces were only rigged for basic expression changes, partly due to the simplified face topology and patch layout. We could move the eyeballs, but none of the procedural shape changes described above were rigged into these characters. Therefore, animation effort was put into animating faces away from camera wherever possible. None of this work was accounted for in our bids, and each hour spent on background kids would need to be reclaimed on other sequences in the future. The numbers were being watched very carefully.

[9]

Director Robert Zemeckis changed his mind about the look of Lonely Boy during the production, making it necessary to rework the rigging for his character. [9]

Lonely Boy Sequence (LB)
Number of Shots: 41
November 20, 2003-May 12, 2004

The Lonely Boy sequence is where we pick up Billy from “the other side of the tracks.” The original “Lonely Boy” was a different character design rigged with the same arsenal of animation controls featured in our other hero characters.

When Bob saw the first motion tests with this character, he was concerned that this particular “Billy” was miscast. Instead, he cited a specific background kid that he believed would fit the bill perfectly. Our setup crew went to work immediately in an attempt to rig the alternate model to make him a full-featured character. It took several weeks to produce marginally-acceptable deformations in the face, but nothing like the results we were getting with the other heroes. It turns out that the face topology was very different, and our best efforts could not overcome the fact that the patch layout in the background models did not follow the natural contours of the face. They were never intended for high-level performance rigging, and optimized as background characters instead. This rig would need a lot more work before it was fully-functional. The first step was to redistribute the isopharm layout in modeling, and then back to rigging for another round.

Unfortunately we couldn’t wait. While this rework was underway, we continued to move shots though the department using the existing rig. Thankfully, Billy’s performance is muted during the first half of the film, and he doesn’t show much expression until we approach the North Pole. By the time he launched into song, we were ready to go with the new character. Lead animator Keith Kellog had been working with the rig and preparing expressions for the pose library so that a smooth transition could be made the first time Billy opened his mouth to speak/sing.

Hot Chocolate Sequence (HC)
Number of Shots: 32
December 10, 2003-May 7, 2004

The dance sequence was choreographed in production editorial, and characters placed in the scene to achieve the desired composition for each shot. Since shot composition was king, integration’s challenge was to make all the characters fit without interpenetrating. The hot chocolate cart was designed to travel straight down the center aisle, but there was very little clearance, especially when the waiters were hanging off the sides. When the cart spins, it simply didn’t fit so the aisle had to be widened. When the waiters ran up the walls and around the ceiling we discovered that they were penetrating into the ceiling so we had to expand the size of the car. It was a house-of-cards in that performances of the other characters would no longer fit the expanded environment and had to be adjusted to accordingly. Again, these tasks were not figured into the budget so we knew that the numbers would have to be reclaimed elsewhere.

Tightly choreographed performances were captured separately, so characters never responded to each other as they would if they were captured together. Actually, there was plenty of interaction between them in that they interpenetrated horribly. It was a giant puzzle with eight waiters, two chefs, the conductor and a hot chocolate cart all packed into the center aisle, and it was a different puzzle on every frame because of the broad dance movements. The offsets on all of those bodies and limbs had to be adjusted frame-by-frame to be sure they cleared each other (and the props) in a dance that was packed into a sardine can. It was a huge relief to the crew each time the waiters hopped onto the tables because that meant they were out of the aisle!

[10]

Designing the Hot Chocolate sequence was like a giant puzzle with eight waiters, two chefs, the conductor and a hot chocolate cart all packed into the center aisle, with dance movements thrown in for good measure. [10]

The waiters were rigged to handle broad actions, including arms that twist 360 degrees so that they could pour hot chocolate while performing back-flips. Like the background kids, their faces were only rigged for minimal control. The idea is that they are absorbed in their Bubsy Berkeley moment, and “Hot Chocolate” is their only mantra. Special rigs were provided to animation from the effects department to control the arcs of the squirting chocolate, which needed to be tightly choreographed to the camera as well. Cups, saucers and trays were flying throughout the scene, tossed by a character in one split, but caught by a character in another split.

Tight coordination between integration and animation was necessary to make all of the elements work together. Up to this point we had always moved shots forward out of MotionBuilder (integration) and into Maya, but John Matthews was now anxious for a tool that would allow us to go the other way as well. This is because some animation tasks are easier to execute in MotionBuilder because of its unique tools and ability to handle more than three characters at a time. Character setup supervisor JJ Blumenkranz and his crew developed a “motion-collapse” script that enabled us to bake all of our composite animation curves back onto the integration rig and bring it back into MotionBuilder as a fresh new file. The animation curves could now be manipulated in whichever package was best suited for the task at hand.

Conclusion
These were the sequences that defined our processes. As time went on the data became more solid out of integration, and shots started moving through the pipe much faster. Over time the pose-libraries were filled with more reliable expressions for each character and the facial performances, including dialogue gradually improved as a result. Many of the pose-libraries were transferable to other characters as well — especially the adult Tom characters (Conductor, Hobo and Santa). Even so, it seems that every new shot carried with it some whacky problem that had not been anticipated. The difference is that now those problems did not seem insurmountable. For the most part, we knew where we were going and it was more of the same routine to the finish line. The costs were being monitored very carefully, so we had to stay focused on efficiency and choose our creative battles wisely.

I would like to reiterate that it was neither the design nor the intention of this production to create “photoreal” humans as it is often suggested. It is a stylized film with a painterly quality on every level — including the human characters, which were not designed for micro-expression levels of detail that would be required for photorealism.

However, the level of detail is high enough that critics tend to evaluate the imagery and characters on that level. They are quick to point out the flaws, but dismiss the areas that are in fact working very well.

I believe that there are many moments that are near perfect (within the design scheme of the film). Given the production schedule and shear volume of work, we had to focus on the high-value shots that we believed would make the most significant difference in the context of the film. Because we were producing every pixel of every frame, we had to be smart about which areas would get the most attention — because 16 months goes by awfully fast!

The final installment will focus on the keyframe-animated elements, and will be published upon the release of The Polar Express DVD.

Animation director David Schaub joined Sony Pictures Imageworks in 1995. On The Polar Express, he worked with director Robert Zemeckis and noted senior visual effects supervisors Ken Ralston and Jerome Chen. He was previously a supervising animator on Stuart Little 2, for which he and his team received a VES Award (Visual Effects Society) for “Best Character Animation in an Animated Motion Picture.” His other film credits at Imageworks include the Academy Award nominated Stuart Little, Cast Away, Evolution, Patch Adams, the Academy Award nominated Hollow Man, Godzilla and The Craft.