The Polar Express makes a technological breakthrough by using performance capture. All images © 2004 Warner Bros. Ent. All rights reserved.

When Robert Zemeckis launched his digital center at USC four years ago, he sat at a computer monitor and confided that his dream was to someday shoot an entire movie, virtually. Well, he soon got his chance with Warner Bros.’ The Polar Express, the all-CGI Christmas extravaganza with Tom Hanks, based on Chris Van Allsburg’s popular illustrated children’s book, which opens today [Nov. 10, 2004] in both standard 35mm and 70mm IMAX 3D. Along with Sky Captain and the World of Tomorrow and The Incredibles, The Polar Express represents a daring technological leap in CGI moviemaking. In wanting to capture the spirit of Van Allsburg’s painterly look, Zemeckis chose to experiment with a new form of performance capture designed by Sony Pictures Imageworks and Vicon called ImageMotion. VFXWorld recently spoke with senior visual effects supervisor Jerome Chen about the unique challenges of making The Polar Express and what its significance means to the 3D community, which has already begun debating its technical and artistic merits.

Senior visual effects supervisor Jerome Chen found that motion capture didn’t match the movie’s needs. Instead, performance capture allowed the actors to interact.

Bill Desowitz: How did this process start?

Jerome Chen: [Senior visual effects supervisor Ken Ralston and I] were the creative and technical supervisors who were charged with coming up with a way of achieving this movie after a preliminary meeting with Bob, who basically said he didn’t want keyframe animation. We was pretty sure it would have to be CG, since it would be hard to do this live action. And then he wanted to preserve the visual spirit of the book, the pastel drawings. So we even toyed with the notion of shooting live action and treating in post à la What Dreams May Come. You still have all of the effects plus on top of that an artistic process, which is really daunting.

BD: Were you already experimenting with performance capture?

JC: We had done motion capture for movies like Spider-Man and other movies at Imageworks. But let me make a distinction between motion capture and performance capture. We started calling it performance capture because we were grabbing the entire performance at once, meaning facial and body. The other stuff we had done for stunt sequences was for body performance. When I started looking into the state of motion capture at the beginning of the show in June 2002, it felt pretty primitive to us, meaning when they did motion capture for games and other action sequences, it was about the stunt, not about the facial performance, so you could either keyframe animate the face or grab a separate motion capture session where the actor sits still and mimes a facial performance, and a technical animator would glob the two pieces together. But in our movie we have these four children interacting with each other on this adventure, so it didn’t make sense to capture everyone separately. So conceptually, what we needed was to create a place where you can get four actors together — they can look in any direction at each other and you can record the performance.

That was the design spec. At that point, we contacted a number of motion capture equipment makers and talked to them about what he wanted to do. One of them told us it couldn’t be done because there was too much data to capture, because we were going to use a full marker set on the face at this point. Nobody had done what we were talking about, which was really odd. Also a little frightening. One of the main problems we had to overcome was how could the cameras take in so many facial markers. Our system alone has 152 facial markers. What we ended up doing was working with Vicon to develop their software so that it could take in the amount of data that we’re talking about at a quality you could reconstruct without a lot of noise in the markers.