For Pirates of the Caribbean: On Stranger Tides, Industrial Light & Magic got to play in the water with mermaids, the Fountain of Youth and shrunken ships in bottles for 300 vfx shots. It was certainly a departure from the crustacean-like creatures and apocalyptic mayhem of the previous Pirates  trilogy.
And for Ben Snow, ILM's visual effects supervisor, it was a nice change of pace from the hard surface challenges of the Iron Man  franchise. The mermaids were especially different, appearing beautiful and human outside the water to entice and entrap the pirate victims and then menacing underwater with deadly fangs. But rather than going completely CG, they decided to apply a hybrid approach, in keeping with director Rob Marshall's glam aesthetic and desire to retain as much of the live-action performance as possible, particularly when it came to the hero mermaid, Syrena, played by Astrid Berges-Frisbey.
"The look of the mermaids was important," Snow says. "We conceived them as having an inner body that had all the scale texture on them and then an outer membrane that made them look human when they got out of the water. They evolved from being a little more human to a little more creature-like with vestigial gills, but we pulled back on that.
According to Tim Harrington, ILM's animation supervisor, the facial capture was driven by two Mova  Contour sessions: one to create the facial animation rig and another performance session of the actress watching previs or the shot (ADR style) on a monitor.
"Astrid did about 80 different facial expressions and we have a new proprietary system at ILM where we can take a group of expressions and decompose them into all of the individual shapes that create our facial rig," Harrington explains. "We started by creating a 1:1 match of Astrid and it was one of the most accurate digital doubles ever done at ILM. On top of that, we wanted to either be able to do facial MoCap or to animate by hand the way we did Davy . There were some transformation shots where we were just going to copy the performance from the plate, so we needed to have both approaches and a system that could handle both [as a hybrid]. We had a new Imocap set up for her because she was going to be in water and basically nude. We applied markers using a tattoo stencil on her arm instead of the traditional bands of Velcro to capture her upper body. It would then go to animation and we would attach our mermaid to that and animate the tail [designed by Aaron McBride] using her legs as a basis. We came up with a big fin that was long and elegant that could be simulated."
ILM not only used its PhysBAM fluid sim engine, but it also applied the Plume  GPU-accelerated technique developed with NVIDIA for smoke and fire on The Last Airbender. This was used for mist and for overall fast turnaround. ILM also used Houdini  and Maya  for additional sim help. "It was handy to have automated tools to help the artists and Rob Marshall to read the animation," Snow adds. "If the mermaid was moving in space, looking almost flat shaded, you could more easily buy the hardware render. But if we didn't put the splashes in quickly, it didn't look fast or powerful enough. The water interaction was key to selling the performance, and we learned a lot about getting characters to move through water convincingly. You had to have accurate-looking simulations right out of the box to show that the animation was performing well."
ILM collaborated with Scanline VFX  for the triggering of the Fountain of Youth, beginning with a droplet of water running along a leaf, and then when water flows along the walls of the cave to make a portal. However, for a spectacular liquefying death resulting in a nasty skeleton, ILM scrambled to get some extra water sim by adding a whirling effect. There were some rendered surfaces, meta-surfaces for more of a glossy look, different layers of particle sim created with PhysBAM and the simulation tools and some practical water.
"Initially, we just thought that the ships would be affected by the light in the room that they're in," Snow explains. "But then this idea came to have the ships look like they're frozen in time at the moment of capture: in battle or in a snowstorm in the Arctic or in a storm at sea. The art department made several bottles with real model ships inside. For the wider shots, we replaced our key ones of those and essentially added movement to some of the others. And then as we got closer, they became fully CG. We even played with macro photography on some shots.
"There were a couple of happy accidents where the compositors went over the top with a big explosion. I initially had them hold it back. But I showed Rob Marshall the early take and he requested that we go back to the bigger explosion."
Bill Desowitz is senior editor of AWN & VFXWorld.