The Polar Express animation supervisor David Schaub.

With The Polar Express, based on the popular children’s book by Chris Van Allsburg about a magical journey to the North Pole, director Bob Zemeckis, star Tom Hanks and the film crew achieved a new kind of live action/animation hybrid as a result of breakthrough performance capture technology from Sony Pictures Imageworks called ImageMotion.

VFXWorld asked me to document my account of this production from my perspective as the animation supervisor. Thus, my focus will be on the animated portions of the film, along with brief insights into other areas of production. Virtually every element of live action and animation applied. From an animator’s point of view, I was drawn to Polar Express not only by the prospect of exploring a new process, but also by the opportunity to create a wide range of animation performances using more traditional means as well. All of the animals, the train, flying reindeer, elves, puppets and more would be animated by hand.

In this first installment, I will walk you through the earliest days of production, and the series of events that lead to my involvement.

The Polar Express presented a rare opportunity to explore something completely different. At the heart of this equation was one of the most cinematically adept filmmakers of our generation, Bob Zemeckis. With a history of applying talent and technology to find new ways to bring stories to the screen, this project promised to be an adventure.

Part of the magic of the book was in its painterly illustrations. From a production design standpoint, Zemeckis was intent on maintaining the integrity of the book’s artwork. If he were to do this as a live-action film, he knew that he would lose the paintings — and with them, much of the magic. It was the magical spell of the book — the sense of mystery and wonder so artfully conveyed on the page — that he wanted to retain. But he had to find a way to bring the paintings in the book to life, yet had no preconceived notions about how that look would ultimately be achieved on film. At first he was striving to replicate the brush-stroke imagery of the Chris Van Allsburg paintings, but that look ultimately gravitated to the more refined interpretations of those paintings that production designer Doug Chiang was producing.

To explore his painting-in-motion idea, Bob turned to Ken Ralston, five-time Oscar winner and senior visual effects supervisor at Imageworks, who has been Bob’s vfx guru as far back as Who Framed Roger Rabbit. Ken made some early explorations applying 2D approaches with vfx supervisor Sheena Duggal. The first test used a combination of image processing tools and customized brushes that allowed an entire image to be treated as a pastel painting. Another approach used optical flow software to create motion maps for each pixel in a series of live-action images. This motion analysis was used as a base to create time-coherent painterly effects over live-action sources. Artists would paint directly onto selected single frames. These paintings were used as keyframes, and the optical flow software applied motion to the paintings over time. The areas of the image that did not work using optical flow out of the box were manipulated in the composite, using a combination of 2D painting tracking and warping. The results were very promising, and development efforts in this area continued to move forward for many months.