After its Oscar-winning performance on Pirates of the Caribbean: Dead Man's Chest, which entailed the groundbreaking Imocap remote performance capture system for Davy Jones and his crusty crew aboard The Flying Dutchman, Industrial Light & Magic had a whole new set of challenges with At World's End.
First, visual effects supervisor John Knoll had a smaller crew to contend with and again was tasked with only five months to complete the work, since this time out his project was the first to finish the season rather than the last. With ILM taking on 750 of the most difficult shots, Roger Guyett was brought aboard to help supervise. He oversaw the shooting of 200 miniature elements and also worked on Captain Jack Sparrow's bizarre hallucination as the wall encrusted member of The Flying Dutchman. "Like Wyvern in Dead Man's Chest, where Jack is inside the wall, we used a similar kind of approach," Guyett suggests, "but the hard part was getting all the texture and doing all the paint work. It is very much a CG character, but Johnny Depp didn't wear the funny gray suit, but he did wear the checkered headband and armbands. We wanted him to still be recognizable." Meanwhile, much more work was farmed out to other vendors this time -- more than 1,000 shots -- under the supervision of Charles Gibson, including Digital Domain, Asylum and The Orphanage. Digital Domain, in fact, worked on 300 shots, including the waterfall sequence, icebergs and the high-speed destruction of an opponent's ship.
This allowed ILM to do the necessary R&D and execution of the film's most daunting innovation: the Maelstrom that envelops both The Black Pearl and The Flying Dutchman during their battle. "The water is a huge character in the movie, taking the technology that we developed over the last few years with simulation but up to a whole new level," Guyett explains. "But to develop the Maelstrom whirlpool, it needed to be a total 3D environment."
In other words, this far exceeded even last year's exemplary work on Poseidon. Indeed, as Knoll concedes, they had to rethink their initial approach. "The assumption at the beginning was, given that the Maelstrom was this very large environment that takes on this whirlpool shape, I thought you could sculpt a funnel shape and then put an ocean surface shader for the sequence. But it was pretty apparent from the first test that we weren't going to get enough visual complexity. This really required full-on computational fluid dynamics. That's a function of resolution, computational power and memory. Even with the biggest 32-gig machines, you couldn't properly convey what was required. What we needed were a few optimization tricks to get maximum amount of detail."
As a result, former Stanford researcher, Frank Lossaso-Petterson, who helped build the university's fluid sim engine that has assisted ILM, was hired on staff to create the Maelstrom. "We needed to simulate a lot more water and for the Maelstrom sequence there were a lot more shots that had to be done," explains the fluid sim expert. "It became clear early on that the resolution needed was above and beyond anything we or anyone else had done before. With respect to R&D, we needed a way of getting all that visual detail on the surface that director Gore [Verbinski] wanted and at the same time be able to run in a short enough time to get a few iterations. The first couple of simulations that we ran in the traditional, Poseidon way, in parallel and on the newest machines, revealed that we would've needed another 10x the amount of time to attain the required resolution. So it became clear that we needed some technological changes.
"The Stanford engine allows us to run a basic photorealistic simulation. However, it's been extended at ILM to allow for greater control for art direction and rendering. So I've developed a lot of tools to make that work on top of the basic engine, as well as other tools in order to make the engine more accessible to a lot more people. One of the new tools focused on manipulating data, so that if the director wants it to go a little faster or spin faster in certain areas, we could handle this in a quick turnaround. That led to us having to deal with other challenges, such as manipulating the data of water underneath a ship that is interacting with this water. So we developed technology where we could add in the ship interaction after running the general overall water motion, which had a lot of advantages. Not only could we manipulate the overall water motion and then add the weight afterwards but we could also run a single simulation for half the shots and then another simulation for the rest of the shots. We could also run individual simulations for highly detailed shots with greater resolution.
"On the rendering side, one thing that was a little strange was that because the Maelstrom was bowl shaped, you end up looking at the water from a shallow angle up close or from an over the top view farther away. The rendering challenge was that we would have to deal with this level of detail change for water that is just a couple of feet from the camera to water that is several hundred yards away from the camera. You obviously have to handle the water differently if most of it is obscured by other waves when you see everything from a top down view.
"Distances were in the thousands of feet. Just a rough calculation that we came up with for the amount of water that we were simulating for this was the 15 billion gallon mark. And if you need resolution down to just a few feet per cell, or even less than that, you can't just brute force that computationally, even with all of the parallelism that we have and state-of-the-art computers. This is where the ability to supplement simulation with wake and detailed sims comes in handy.
"One technique that we used was... to simulate the Maelstrom as though it were flat and basically perturb gravity... because the surface is what matters. Instead of sloping down, the gravity was really pointing inward. And then part of the post process tools that we have would be to deform the surface into the shape that we wanted. All the art direction is being established while we run these shots. All these advancements made feasible what otherwise would've taken 40-45 weeks to render. And with parallelism across 40 processors, you take that down to a few weeks. And then running the simulation in a flat domain, as I've just described, brought that down to a few days."
As Knoll explains, the ability to composite two simulations at different resolutions and then apply deformations and overriding animations, enabled Verbinski to achieve important visual clarity. "It was important to Gore to really read that the water spins faster and faster as you go farther down into Maelstrom," Knoll adds. "The story points required you to see certain moments clearly: When The Flying Dutchman and The Black Pearl are on opposite sides of Maelstrom and The Flying Dutchman cuts down lower latitude into the faster water, it takes this faster track to close the gap and pull in right behind The Black Pearl. But when we actually ran a fluid simulation with all the correct mathematical calculations, it may have been physically correct but it wasn't dramatically correct. Because when you framed up a shot, Gore didn't think the water lower in frame was moving sufficiently faster than the water at the top of the frame to convey his story points."
Another unexpected moment occurred during the Maelstrom sequence when there wasn't a sufficient number of Flying Dutchman crew to fight. At a certain point in the sequence, the two ships connect and their masts become entangled, and you have crusties fighting on both ships. "Having built 17 members of Davy's crew for Pirates 2, we didn't think we'd need additional characters for Pirates 3," Knoll continues. "But as we began laying out the choreography of Maelstrom and figuring out who's where and when, we came up short when we split up the Dutchman crew. We didn't realize until we began shooting this in December. We added around 10 more Dutchmen. And they are very labor intensive to build and paint and set up."
According to animation supervisor Hal Hickel, the design process went much faster. "Fortunately, we had already been through the design process and figured out what looked good with barnacles and what didn't work. Plus we had some new stuff to explore: a jellyfish guy and a sea urchin and a morey eel. So that was fun. We did a whole big menu and once that got down to 10 ideas, we sent that directly to our modelers and they created them really fast and Gore was great in approving them and not overworking them."
Meanwhile, Hickel says there were a couple of new CG crabs to create as well. "One transformed from stone to crabs. Figuring out that metamorphosis was a fun and interesting challenge. Ordinarily, we'll design a character and then the modelers build it and then the creature tds chain it or put the controls in it and then the animators animate it. However, in this case, it had to work back and forth a lot. The crab was roughly built and then we started to animate tests on how it might unfold and then that was handed back to the modelers to figure out how to seam that all together when it's in its stone form. The stone crab sequence starts out with some nice character animation: there were no tough technological challenges. We could just get into the performance of these characters, which is just very weird.
"And then we had normal real world crabs when Tia Dalma is transformed into Calypso and disintegrates into millions of crabs. In both cases, one of the most challenging things was the odd group behaviors. Having thousands of them crawling around wasn't a problem, but the stone crabs had to pile up on top of one another and carry the ship along. There was no way to do that by hand--we had to figure out a procedural approach to generating that motion. And there was more of a particle approach to the other ones cascading down and sweeping away the characters on the deck of the ship."
And although the CG Davy Jones was far less complicated this time, he still posed some new challenges. "We used our tentacle solving tool for most of Pirates 2, but on Pirates 3 where's he's in all kinds of action scenes, including sword fighting, we had to look at the settings all over again to achieve those shots," Hickel adds. "Otherwise, his tentacles would just flop over his shoulder or do something else undesirable. So we had to find ways to make him look great but make the tentacles behave the way we wanted."
Davy Jones was actually a great pleasure for Knoll. "On so many productions you do, you spend such large portions just figuring out the technology and working out the aesthetics. And just when you get good at it, you don't get to reap the benefits of what was done. This happened on Pirates 2 with Davy Jones. And so it was really great to go into Pirates 3 already pretty well practiced with Davy. We got really proficient with how his tentacles moved. We even took the tracking marks off of Bill Nighy's face because we didn't need them anymore."
And what was Knoll touting, now that he's finished with the Pirates of the Caribbean franchise? ILM's record computing power on At World's End: "Every evening we used about 70,000 processor hours -- the equivalent of one processor running for about eight years. And so for the entire production, I'm estimating that we used one processor millennium and that's just at ILM. This represents the biggest disc space footprint of any show at ILM: 103 terabytes -- a new record for us. Up until February, the entire company was 75 terabytes, so we had to expand the storage pool considerably. We ended up using about 130% of ILM's previous disc space just for [this] Pirates."
Bill Desowitz is editor of VFXWorld.