Chen needed 72 cameras to provide enough coverage for four actors and their facial and body markers.

BD: So what was the breakthrough here?

JC: The breakthrough was coming up with the number of cameras and the configuration of the cameras and the pipeline after you’ve gathered the data to apply it to the characters. It turned out that that we needed 72 cameras to provide coverage in this capture zone so we could grab four actors and their facial and body markers together. So that’s 152+48 markers per person x 4. I think that’s 80 gigabytes per minute. There’s a lot of other technology that had to be created to manage it — bookkeeping things for the data to be processed and visualized.

BD: What about the performance challenge?

JC: Part of what you’re doing is capturing Tom playing an eight-year-old boy [along with four other adults]. So already you have differences in how much a muscle moves on Tom’s face in relation to what’s happening on a child’s face. We wanted to get a character that looked like Tom when he was younger. We started scanning his son, who actually looked more like Rita [Wilson, his mom]. But we also realized that we didn’t want to make him into an actor. So we came up with a design that Bob liked, and as we started to apply motion to it; we made a couple tweaks so that the kid’s eyebrows and mouth looked a little more like Tom’s because the performance actually translated a lot better, because Tom has these really arched eyebrows and does a lot of acting with his forehead — he doesn’t move his face that much. It’s interesting how we were able to analyze his acting in that way.

Chen created a new smoke and snow renderer called “Splat.”

BD: Talk about how the production process was split into different phases.

JC: You have the performance capture first, then the integration process. After a particular performance take is selected by Bob and his editor [Jeremiah O’ Driscoll], it is sent to Imageworks and is ordered up. We then go through the process of applying the performance data to the digital character in a medium resolution. The digital characters are then placed into the virtual set and the props are put in. And at that point it goes into layout, which is similar to a traditional keyframe movie. So this is where we begin to talk about the point of view of the movie. What does he want the camera to be showing us? And one of Bob’s trademarks is visual storytelling, so the camera is very important to him. And what was liberating about this digital process was he was able to concentrate totally on camerawork as a whole separate phase during performance capture. And you can’t even begin editing yet because all you have at this point is video reference. So we created this process called “Wheels” where we brought in a real cameraman to teach computer animators how to act like cinematographers and it would feel like operating a remote camera on a gearhead as if they were on a technocrane. So the wheels basically allow you to pan and tilt and all we’re doing is recording the input from those wheels that will drive this virtual camera later, so you get all the nuances of Bob’s camerawork played back in realtime. We didn’t want to keyframe the camera, which gives you a different look.

BD: What other new technology did you have to create?

JC: We had to create smoke and snow and water and all of the effects animation. I think this was one of the largest effects animation crews that we’ve had at Imageworks. Traditionally smoke and water effects take so long to look correct in the computer, but because we were going for a more stylized look, we decided to create a new renderer called “Splat.” This was our smoke and snow renderer, and it was very fast. We used old technology to create some tests of smoke to be composited in, and these passes took 16-20 hours a frame to render. The simulation to get the smoke movement was done pretty quickly, but to render it you really had to like the movement because you only had one chance, and we had hundreds of shots that required smoke. So this new renderer would take 20 seconds, which was huge. So that meant the effects artists could do a lot of iterations of movement and lighting until we really liked it. I thought the smoke and all those effects, those subtle interactions, turned out great — it was one of my favorite parts of the movie.