David Schaub breaks down some of the key sequences and creative/technical challenges in the latest installment of his exclusive Polar Express Diary.
To recap, MoCap/Anim is the work that the animation department does on top of (or in addition to) the performance capture delivered to us from integration. A description of the animation rigs and workflow is covered in the Animation Rigs & Controls discussion in Part 2 of this series. The shots are delivered to us from integration as Maya files with all of the assets embedded. That includes audio tracks and all the pertinent video references set up on image planes.
Using the performance capture as our foundation, the animation task is to fill in the blanks but only to the extent indicated in the video references provided to us from production editorial. Often a shot will come with notes from Bob Zemeckis indicating a desired performance change. For example, a character should look left instead of right or perhaps do a double-take at a specific moment. Other than that, the actors performances were sacred unless Bob indicated otherwise. The hardest thing to do in the beginning was avoid the temptation of helping a performance along by pushing an extreme or making an action broader, etc. Bob knew these performances inside and out, and his goal was to replicate those performances without any further interpretation by the animators. One exception to this rule is body dynamics that often had to be animated (or re-animated) if there was some external force acting on a character that was not captured on set. The simplest example of this would be upper body animation to support the illusion of a rocking train, especially when the characters are on the roof.
The obstacles we encountered in the first six sequences were among the most difficult challenges we faced on the show. This is where our processes were being established, and where most of what was broken got fixed. I am not suggesting that life was easy after that, but those six sequences taught us much about what we were up against and allowed us to establish a viable workflow based upon the lessons learned. What I will do now is walk you through some of those milestones and lessons that paved the way for the remainder of the film. The dates obviously overlap because there are multiples of shots in play at all times.
Boy Awakes Sequence (BA)
Number of Shots: 18 June-November 2003
June 11, 2003
Our first shot through the pipe is also the first shot in the film where the camera travels in through the window, moves over the bed and closes in on an extreme close-up of our hero (Chris). The reference video shows Tom Hanks lying very still, breathing slowly and silently. There is no detectable movement in the close-up video reference, other than a hint of movement in his chest as he breathes. However, the performance capture provided us with some lovely body tremors and twitches. Given the fact that the character is sleeping, it made no sense to use this noisy data so we turned it off and animated by hand.
When Tom opened his eyes in this shot, his eyes were absolutely fixated as he stared at the ceiling. Although it could not be seen in the reference, lead animator Kenn McDonald added a hint of saccade (tiny intermittent eye movements) to keep the eyes alive. One of those movements was perceivable change in eye direction. When Bob saw this he instantly pointed out that Tom did not do that in the reference video. He was right. Toms eyes were frozen in silence as he listened intently for the sound of the bell. A change in eye direction would indicate a change of thought (a distraction), and that was not the intent of the moment. We had missed the point! This is an example of reinterpreting a performance, which is something we needed to avoid a lesson driven home right out of the gate. So we eliminated the obvious directional change, but kept a small amount of saccade in keeping with the natural physiology of the eyes.
Once the performance had been replicated, I still thought that something was wrong. Even though breathing animation had been added, our boy still looked dead. Tom was absolutely still in the reference, so we couldnt add gratuitous movement we had been down that road before! So what was the difference between a sleeping boy and a dead boy? With that question in mind, I strapped a laser pointer to my head, lay on the floor and watched the little red dot on the ceiling. Try as I might, I could not keep the little dot from chattering all over the place, even while holding my breath. Beneath the random noise was a very distinctive rhythmic bounce the heart! Obviously, the dot would not have moved at all had the pointer been strapped to a cadaver. So now we were on to something. Noise curves were added to the body, neck and head to emulate the noise pattern that I saw projected on the ceiling. Animator John Matthews hunted down an audio waveform of a heartbeat, and used that waveform as another driver on the same joints. Believe it or not, these very small nuances make a huge difference in the viewers perception of that kid lying there subliminal as it may be.
Note These curves were ultimately added to many characters in the quiet moments to help get the blood moving in their veins, so to speak.
July 17, 2003
In the next shot, our hero gets up out of bed and goes to the window. This was a very complex shot with lots of contact moments between our character and props in the room. There was very little movement coming through in the motion capture spine, so the offset spine and redistribution controls were used to create a more natural compression and flexing in the spine as the character crawled across the bed. Deformers are used to animate depressions in the mattress and pillow as the character moves through the scene. Representative geometry is deformed and animated to define the desired behavior of the blankets, which will ultimately be simulated. As lead animator Chad Stewart continues to animate the feet and hands, the snapping controls explode when put to the test.
When we see our character posed at the window for the first time, it strikes everyone that his arms are freaky-long! The character goes back to modeling to shorten the arms and we squeeze in a few other fixes while were at it. Because of this, the character must be re-rigged, and the snapping problem ultimately gets resolved as well.
July 21, 2003
In our next close-up, Chris hears the bell and looks over his shoulder toward camera. There is a very specific look on Toms face in the video reference that Bob is intent on replicating. When he saw our work he kept saying that he just wasnt seeing Tom in our boy. However, the boy is a very different character design with little or no resemblance to Tom. Acquiring Toms likeness in the face of the boy was a losing battle, unless we deformed his face to look more like Toms. The area that Bob was focusing on was the brow region, and the shape of the boys brow was very different from the distinctive Nicholson-arch that Tom has. It was decided to add a hint of the Tom Hanks likeness in the brow area in order to help sell expressions like this in the future.
August 19, 2003
New textures were painted on our boy to represent the Hanks brow shape more accurately. Once this was approved, the hair-patch layout for the eyebrow was modified to match. The end result is that the desired expression comes through. Since the brow design changed, all shots leading up to this had to be reworked and all works-in-progress (which there were many) were upgraded to incorporate the design change.
To avoid this problem moving forward, the character design was locked. Our work in the future would be an interpretive evaluation of one character transposed to the next. We dont want to skew our characters face in the interest of caricaturing the facial topology seen in the video reference. Instead we are reading between the lines and transferring the intent of an expression from one character to the next. For instance, there is a series of shots when the boy is frustrated that he cannot hear the bell. There is no dialogue, and little physical likeness between Tom and the boy; however, Toms mannerisms and expressions clearly come through.
July 17-October 24, 2003
Our next milestone occurred when our boy sneaks down the stairs, and when he realizes that Santa hasnt visited yet, rests his chin on the banister as he ponders that thought. The problem was that his chin was rigid as it rested on the rail, and not deforming as you would expect. All of our facial controls were designed to imitate muscles affecting the skin from underneath. It was never considered how we would deform the skin and underlying tissue with an external force. John Matthews did some investigation into how we might do this and came up with tweak clusters as the solution.
A tweak cluster is a selection of points on a surface grouped together under one controller. The controller allows you to push, pull, rotate or scale many predefined points on a surface. The weighting of each point (how much it is influenced by the movement of the controller) is accomplished by painting the amount of effect that the controller has on the selected points. In general, the weighting is painted such that the central region of control has the greatest affect, graduating to minimal control toward the outer edges. This allows you to essentially sculpt the model, and animate that sculpt on the fly.
John creates a single control on the tip of Chris chin that would realistically bend and give way as he put the weight of his head on the banister. It was easy to set up, easy to use and the end result was very effective.
Note We started using tweak clusters everywhere, especially on our main characters. We were able to sculpt facial expressions that were simply unattainable with the muscle system alone. We could sculpt cheek creases, brow furrows (skin bunching between the brows) and convincing swallowing animation with tweak clusters in the jaw and neck. This was a huge relief, because whatever was not possible with the animation rig could surely be handled with tweak clusters if it came down to it. However, this kind of control can easily get out of hand. We ended up using tweak clusters on the train, the ticket, and at one point all the facial animation on several elf shots with tweak clusters alone. Rather than devising new animation controls for a specific action, our setup group would often defer us to the tweak cluster solution because it was so simple and effective.
Train Arrives Sequence (TA)
Number of Shots: 36 July 17, 2003-February 16, 2004
October 9, 2003
A tricky shot of the boy pulling a number of deforming props from a drawer has been underway for quite some time. Several elaborate rigs were built early on to manipulate the props but the results were only moderately successful. Upon discovering the tweak cluster solution, the shot was retro-fitted with tweak clusters and animator Marco Marenghi was able to produce the desired results almost immediately.
June 19-December 8, 2003
When the train arrives, the boys room starts shaking violently, which means that all of the props in the room need to shake also. Animator Stephane Couture begins animating each prop by hand, which seemed to be the most logical approach at first. As the shaking gets progressively more violent he begins experimenting with dynamic simulation and is able to achieve excellent results in very little time. A good example is the jar of pencils that falls over, and the pencils spill onto the desk and some roll off the edge. The lamps and lampshades were also set up for dynamics, and some very natural results were achieved.
Unfortunately, stand-alone props with animation curves applied could not make it through our pipeline as-is. An animatable rig is required in order to survive the upgrade that occurs when a shot moves forward to the next department. To solve this problem, Carolyn Oros (pipeline TD) scripted a procedure that was able to take those rudimentary props and use them to drive a rigged component so that it would follow the pipeline of our component assembly system. This became yet one more of our standard tools.
Boards Train Sequence (BT)
Number of shots: 30 August 13, 2003-April 12, 2004
August 18, 2003
After the train arrives, the boy opens the front door and runs down the steps toward the train. This action was performed by Tom, but when retargeted to the proportions of a child, the action looks like a slow-motion run on the moon. The reason is that the stride-length for the child is much less than that of the adult who performed the action. However, the timing foot-to-foot (and all of the body language that goes with it) is still in the same time-space as the adult performance. To make the adult-to-child conversion believable, the entire action was sped up by about 12%.
Note When retargeting adults to children, time-scaling became standard practice for broad actions such as running, jumping and other stunts where gravity plays into the equation. Because shot lengths have been defined out of layout, speeding up the action also requires filling in the front and/or back-end of the shot with supplemental animation to make up for the shorter performance that results from the time-scale.
September 26-December 17, 2003
When we first meet the conductor, many of the same problems we experienced with our hero boy are revisited. At this stage there is no pose library, so all of the facial work is done with muscles and tweak clusters. Poses will be added to the library as we move forward. An interesting development is that tweak clusters can now be saved as part of the pose and saved as such in the pose library. The conductor is also equipped with several animatable props, including his hat, glasses, pocket watch and chain. I make a plea to our producers that the watch-chain be simulated by the effects group. Unfortunately their workload is insane as well, so the chain is rigged and the dynamics are animated by hand.
Note 1) A recurring problem with the conductor is that his glasses were constantly being adjusted to assure that the frames were not intersecting his eye-line from the cameras perspective. The glasses would slide up and down his nose throughout the shots to avoid obscuring the eyes at all costs.
2) Each new hero character (Holly, Know-it-All, Billy, Hobo, Santa, Hero Elves, etc.) went through the same developmental process. It was typical to spend as much as three months on additional controls, deformations and geometric modifications before the first shot of that character is ever finalized.
In the beginning, the addition of other kids to the cast caused Maya to crash with great regularity, costing time and work losses to the production. Originally, Sonys digital team budgeted the background kids for simple plug-and-play, but they stood out like sore thumbs next to the featured cast. In the end, many hours were put into animating faces away from camera.
Meets Kids Sequence (MK)
Number of Shots: 21 November 11, 2003-March 9, 2004
Up to this point in our story weve only dealt with two characters: Hero Boy (Chris) and the Conductor. When we load these two characters into a scene (along with the low resolution train), we still chug along in Maya with a workable amount of computing power and performance. Once we raise the resolution of a character (which is necessary for facial animation), we often bring Maya and our computers to their knees.
The situation became critical once we stepped onto the train-car full of kids.
The typical scene file with just the train and Chris was running upwards of 200 megabytes and when opened in Maya was taxing the computers RAM to the tune of 1 gigabyte or greater. Needless to say, our first endeavors to add additional kids caused Maya to crash with great regularity, costing us greatly in terms of time and work losses.
Bert Van Brande and his animation pipeline team came upon the idea of splitting the scenes into manageable chunks. Scene files are put together in integration where they are using significantly lighter rigs for the characters and the train. Before the facial capture is applied, the integrated body performances are submitted for approval (into animation), and the scenes split before transferring the files to Maya. If the characters are constrained to the train, each split file would carry with it the motion from whatever node the characters were constrained to. All of these splits had to be recomposed into a single movie file for review by Bob.
After struggling through the process of using multiple image planes of each split in our render, character setup TD Amy Hronek came up with the Render Compositing Tool. This used Z-depth to automatically render and composite all the splits at once. It was a breakthrough that solved a plethora of problems. Now several artists could work on splits independently, and a single artist could render and composite each split in a final render for Bob to review.
There was an ongoing effort throughout production to seek simplifications that would drive the costs down. Given the amount of attention and effort put into our hero kids, it was decided at bid-time that the background kids would be plug-and-play straight out of integration. There was certainly no budget for any animation work on the extra characters, but I knew they would stand out like a sore-thumb if we mixed them in with our heroes. I proposed that we set them up as bobble-heads, where we only see the tops of their heads moving above the seat-backs. We could animate and refine a library of cycles that could be plugged into any shot that required seeing them. Bob agreed to try this simplification, but after presenting a few tests he thought that the kids were simply too disengaged from the surrounding action. At this point, production editorial began specifying specific performance beats for each one of the background kids. This made them prominent in the shots, so a certain level of animation became necessary. If the kids were bouncing on the seats, or running across the aisles it was necessary to do the 12% time scale (to correct the adult-to-child time space) and apply the supplemental animation where necessary to fill in the gaps. There were also the contact issues with seats, props and other characters in the scene. Unfortunately, the background faces were only rigged for basic expression changes, partly due to the simplified face topology and patch layout. We could move the eyeballs, but none of the procedural shape changes described above were rigged into these characters. Therefore, animation effort was put into animating faces away from camera wherever possible. None of this work was accounted for in our bids, and each hour spent on background kids would need to be reclaimed on other sequences in the future. The numbers were being watched very carefully.
Lonely Boy Sequence (LB)
Number of Shots: 41 November 20, 2003-May 12, 2004
The Lonely Boy sequence is where we pick up Billy from the other side of the tracks.The original Lonely Boy was a different character design rigged with the same arsenal of animation controls featured in our other hero characters.
When Bob saw the first motion tests with this character, he was concerned that this particular Billy was miscast. Instead, he cited a specific background kid that he believed would fit the bill perfectly. Our setup crew went to work immediately in an attempt to rig the alternate model to make him a full-featured character. It took several weeks to produce marginally-acceptable deformations in the face, but nothing like the results we were getting with the other heroes. It turns out that the face topology was very different, and our best efforts could not overcome the fact that the patch layout in the background models did not follow the natural contours of the face. They were never intended for high-level performance rigging, and optimized as background characters instead. This rig would need a lot more work before it was fully-functional. The first step was to redistribute the isopharm layout in modeling, and then back to rigging for another round.
Unfortunately we couldnt wait. While this rework was underway, we continued to move shots though the department using the existing rig. Thankfully, Billys performance is muted during the first half of the film, and he doesnt show much expression until we approach the North Pole. By the time he launched into song, we were ready to go with the new character. Lead animator Keith Kellog had been working with the rig and preparing expressions for the pose library so that a smooth transition could be made the first time Billy opened his mouth to speak/sing.
Hot Chocolate Sequence (HC)
Number of Shots: 32 December 10, 2003-May 7, 2004
The dance sequence was choreographed in production editorial, and characters placed in the scene to achieve the desired composition for each shot. Since shot composition was king, integrations challenge was to make all the characters fit without interpenetrating. The hot chocolate cart was designed to travel straight down the center aisle, but there was very little clearance, especially when the waiters were hanging off the sides. When the cart spins, it simply didnt fit so the aisle had to be widened. When the waiters ran up the walls and around the ceiling we discovered that they were penetrating into the ceiling so we had to expand the size of the car. It was a house-of-cards in that performances of the other characters would no longer fit the expanded environment and had to be adjusted to accordingly. Again, these tasks were not figured into the budget so we knew that the numbers would have to be reclaimed elsewhere.
Tightly choreographed performances were captured separately, so characters never responded to each other as they would if they were captured together. Actually, there was plenty of interaction between them in that they interpenetrated horribly. It was a giant puzzle with eight waiters, two chefs, the conductor and a hot chocolate cart all packed into the center aisle, and it was a different puzzle on every frame because of the broad dance movements. The offsets on all of those bodies and limbs had to be adjusted frame-by-frame to be sure they cleared each other (and the props) in a dance that was packed into a sardine can. It was a huge relief to the crew each time the waiters hopped onto the tables because that meant they were out of the aisle!
The waiters were rigged to handle broad actions, including arms that twist 360 degrees so that they could pour hot chocolate while performing back-flips. Like the background kids, their faces were only rigged for minimal control. The idea is that they are absorbed in their Bubsy Berkeley moment, and Hot Chocolate is their only mantra. Special rigs were provided to animation from the effects department to control the arcs of the squirting chocolate, which needed to be tightly choreographed to the camera as well. Cups, saucers and trays were flying throughout the scene, tossed by a character in one split, but caught by a character in another split.
Tight coordination between integration and animation was necessary to make all of the elements work together. Up to this point we had always moved shots forward out of MotionBuilder (integration) and into Maya, but John Matthews was now anxious for a tool that would allow us to go the other way as well. This is because some animation tasks are easier to execute in MotionBuilder because of its unique tools and ability to handle more than three characters at a time. Character setup supervisor JJ Blumenkranz and his crew developed a motion-collapse script that enabled us to bake all of our composite animation curves back onto the integration rig and bring it back into MotionBuilder as a fresh new file. The animation curves could now be manipulated in whichever package was best suited for the task at hand.
These were the sequences that defined our processes. As time went on the data became more solid out of integration, and shots started moving through the pipe much faster. Over time the pose-libraries were filled with more reliable expressions for each character and the facial performances, including dialogue gradually improved as a result. Many of the pose-libraries were transferable to other characters as well especially the adult Tom characters (Conductor, Hobo and Santa). Even so, it seems that every new shot carried with it some whacky problem that had not been anticipated. The difference is that now those problems did not seem insurmountable. For the most part, we knew where we were going and it was more of the same routine to the finish line. The costs were being monitored very carefully, so we had to stay focused on efficiency and choose our creative battles wisely.
I would like to reiterate that it was neither the design nor the intention of this production to create photoreal humans as it is often suggested. It is a stylized film with a painterly quality on every level including the human characters, which were not designed for micro-expression levels of detail that would be required for photorealism.
However, the level of detail is high enough that critics tend to evaluate the imagery and characters on that level. They are quick to point out the flaws, but dismiss the areas that are in fact working very well.
I believe that there are many moments that are near perfect (within the design scheme of the film). Given the production schedule and shear volume of work, we had to focus on the high-value shots that we believed would make the most significant difference in the context of the film. Because we were producing every pixel of every frame, we had to be smart about which areas would get the most attention because 16 months goes by awfully fast!
The final installment will focus on the keyframe-animated elements, and will be published upon the release of The Polar Express DVD.
Animation director David Schaub joined Sony Pictures Imageworks in 1995. On The Polar Express, he worked with director Robert Zemeckis and noted senior visual effects supervisors Ken Ralston and Jerome Chen. He was previously a supervising animator on Stuart Little 2, for which he and his team received a VES Award (Visual Effects Society) for Best Character Animation in an Animated Motion Picture. His other film credits at Imageworks include the Academy Award nominated Stuart Little, Cast Away, Evolution, Patch Adams, the Academy Award nominated Hollow Man, Godzilla and The Craft.