ILM’s Ben Snow Talks 'Pirates', Mermaids and 3D Pipeline

Watch the complete Ben Snow interview at AWNtv!

ILM visual effect supervisor Ben Snow has worked on many of the largest and most complicated vfx-driven films made, including Iron Man and Iron Man 2, Terminator Salvation and Star Wars: Episode II – Attack of the Clones.  He’s been working at ILM since 1994, with the exception of his sojourn down to Weta Digital as vfx supervisor on King Kong.  We spoke recently about his work on Pirates of the Caribbean: On Stranger Tides.

Dan Sarto:  Can you tell us about your work on the most recent Pirates film?

Ben Snow:  Pirates of the Caribbean: On Stranger Tides was a little different from the previous Pirates films.  We didn’t have as much of the film [to work on] as we had on previous films.  Moving Picture Company and Cinesite both did portions.  ILM also worked with a couple of other companies as partners doing some of the work.  The bulk of the work we were responsible for was the big mermaid attack sequence that happens moderately early in the film, and then some sequences where Blackbeard has captured some ships, and got them in these little bottles, but they are actually the full size ships that have been magically shrunken down.  [We also did] the end sequence where they find a way into the fountain of youth and the climax of the film takes place.  For the fountain of youth sequences, we worked with Scanline, which was very good, and we also worked with a couple of other companies on some of the other stuff [we did on the film].  But all of the mermaid stuff was done at ILM except for a wider shot of a ship going down, that MPC did.  The big challenges for us really were dealing with the mermaids, dealing with the fact the film was in 3D and dealing with the issues of all the water and the interactions.

D.S.:  What were some of the most challenging or difficult parts of the mermaid sequences?

B.S.:  One of the first challenges was the design process.  The filmmakers already had some art work they had done on the mermaids, but there wasn’t anything that had really grabbed them yet.  And so we got this great opportunity to actually contribute design. One of ILM’s art directors, Aaron McBride, and I went down to talk to the director [Rob Marshall] and find out what he was after.  He showed us quite a lot of different designs that they had done up till that point, and we talked about different images he found inspiring.  Then we went back up to ILM and came up with some designs that he really reacted well to.  That was a really fun part of it, a fun challenge.  We didn’t want to do the classic mermaid like you saw in Splash and other films.  We wanted to do something that’s a little bit more like saying these are part people but they are also part animal. 

So, the animators, for instance, rather than looking at this sort of classic mono fin swimmer where you have her knees bent and they are really obvious, we tried to work out how could we break up the joints, so it’s more like a fish tail.  Instead of the knee joint, although we still want a hint of that, we might have multiple joints.  We looked at the whole anatomy of how this would work.  One of the images that inspired Rob Marshall was an image which had a glamour model on a beach, in a towel that is almost like a sheer fabric.  He really liked the fact that you got a hint of her legs through the fabric of this towel, so that was a quality he wanted us to think about.  We actually took that and said, “You know what, why don’t we make it like the mermaid has a scaly inside, but then has this sort of almost jelly fish or slightly translucent, transparent membrane.”  That might be useful because it might help us with the transformations.  So coming up with that was definitely a challenge, but a really fun part of it.  But then once we came up with that, it was, how do we execute this sort of thing?

There was a lot of stuff that we had to deal with.  There were a lot of mermaids and they were fighting, so we had crowd scenes to deal with, we had to work out where to package up the animation, because the mermaid design had a lot of trailing, beautiful flowing tendrils, and kelpie like growths, that came out of their hair.  We did a lot of work to come up with ways to simulate that in a way that we could then transplant between different models and across a pack of fifty mermaids.

Ironically of course, one of the biggest curve balls that hit us during the production was that quite late on, the design changed.  It was really important to him [Rob Marshall] that out here in the mermaid’s arena, she was a beautiful woman. But the argument had always been that the other mermaids that were attacking the sailors, earlier in the film, could be a little bit more aggressive and creature-like and be a little bit more different, not so human.  But as the film jelled, the director decided, “You know what, I really want them all to be human. Particularly when they are out of the water they should pretty much just look like the actresses that we photographed. ” 

Now of course, because these creatures attacking the pirates weren’t going to be actress-like, we hadn’t filmed actresses attacking the pirates.  All of the films where we tried to make digital humans, it was a pretty big challenge, and particularly a big challenge to be faced with only a couple of months left in the production.  So, we collaborated with the director, tried to work out how we would do this, and with the studio-side effects supervisor Charlie Gibson, we decided to shoot some actual footage of people that we could use.  We had material for a couple of the key performances and then we had to step back and say, “How do we make these creatures look more real?”  Instead of them having long kelpie tendrils of seaweed, they will have to have hair, which is a whole different can of worms.  It’s not [like] simulating long cloth-like objects, you are now simulating individual hairs, and it’s a big complication.  So we said we won’t be able to do that, we are going to have to color them so that they look like [they have] hair.  One of the things that really worked in our favor was he [Rob Marshall] really wanted to play [the mermaids] quite mysterious and dark.  You don’t really see what’s going on and he felt it would be a lot scarier that way.

It actually did work.  We went with the more human look, since we had used real people, and we had already done a fairly thorough job of working out how to integrate the mermaids with the live actors, add scales to live actors, because we are doing that with the main mermaid.

It is still is a scary sequence, and even though they [the mermaids] are not as creaturely as we had intended originally, obviously people have reacted well to it and like it.

D.S.:  What have been the studio’s greatest achievements in lighting the last few years?

B.S.:  We have tried to develop and improve our lighting over the years.  ILM has always had a really rich history with creatures, you know.  Davy Jones is obviously a really beautiful example of what ILM has been able to do with creatures in the past.  What’s happening of course is, studios are all faced with the situation where we have ever shortening schedules, and we want to try and maintain the quality of the work.  But in the shorter time frame, we want artists to get down to the creative nitty-gritty faster rather than having to do with a lot of technical issues.  So one of the things we started talking about a few years ago was creating an energy conserving shader set that behaved more physically correct, more like in the real world.  We started playing with this a little bit, talking about it a little bit towards the end of Ironman, and then implemented a system on Terminator Salvation.  I was able to use that extensively on Ironman 2.

During Ironman 2, the system really became mature and it essentially became the standard that we use across all films at ILM right now.  But, we hadn’t really used it much in creature work.  We used it for things like robots and metallic surfaces, and it really gave us a big boost there.  It works really well with things like environments, where you are outdoors, it’s sunny, there’s a lot of interesting stuff reflected in the environment.  On Terminator, we had been able to exploit the fact that in an artificial environment like the factory, it gave you a lot of control to mirror what’s going on in the real world.  That was more like the situation we were faced with Pirates. 

What was interesting was we were faced with a magical creature, the mermaid that did not necessarily want to respond completely.  Physically, you wanted a certain sense of magic to it. Rob Marshall decided to really make them more and more realistic, keep the human aspects more and not have them transform so much.  He actually reined back a little bit on how flashy they were.  The physical basis of the materials was really useful there but, I have to say that I felt that on Pirates, we didn’t quite get the optimizations of lighting that we had experienced on Ironman 2.  The creatures were different.

We did find we had to tweak things more in the different environments but I think we realized that part of the reason for that was the way that we had to engineer the materials so that we got all those beautiful scaly and isotropic reflections that made them so pretty.  That was sort of a cheat in a way [laughs].  You put that into your physically-based lighting system and it doesn’t necessarily work as easily out of the box as would something that’s a really physically correct material.  So it’s a bit of a lesson.

D.S.:  What were some of the special preparations you had to make for the most recent Pirates film, in terms of workflow, pipeline and new production tools?

B.S.:  Well, Pirates of the Caribbean: On Stranger Tides was a challenge because we had a short schedule and the studio really wanted to do the film in 3D.  So, luckily we’d just had the experience of [doing] the Star Tours Ride, and on the chunk of shots that ILM did on Avatar, of setting up a stereo pipeline.  It was actually pretty robust when we got into it.  On parts of the Caribbean, we were using footage shot with RED cameras, which was the first time we’d used that at ILM.  We were able to benefit from a really good relationship with RED where they were very cooperative with us, sent us up the camera.  We did a bunch of color tests and evaluations, and looked at ways that we could make the RED footage look more like film.  So we had a lot of help and encouragement on the technical side, and our team at ILM came up with some interesting ways to convert their data that I think helped the project.

We were dealing with the digital pipeline, we were dealing with the 3D pipeline.  The issues for us were, you’ve got a bunch of mermaids splashing around in water.  It’s really quite challenging to track in their splashes.  So we ended up going with a lot more computer generated splashes.  In addition, we went and filmed stereo splashes as well, separate, so that we would have that material to composite in.  But, that was where the stereo actually got complicated.  I have to say, one thing that really benefited us on the film, in terms of stereo, was that the decision was made early on, mostly at the suggestion of the stereographer and with our complete enthusiastic support, to shoot the project in as near parallel as we could, more like an animated film.  Stereo films that have been made up till this point tend to use a rig which has a convergence where the camera actually angles in and focuses on something.  The disadvantage of that [method] is that means you really have to decide where you want the convergence while you are making the film.  Also, it has complicated things like match animation, match moving cameras and rotoscoping a little bit.  So even though it [shooting parallel] didn’t make it easy shooting stereo, what we found was compared to other films we were doing in 3D at that time [at ILM], we actually ran into a lot of less headaches because of taking this parallel approach.  What it meant was that you use the ability of the RED camera to shoot an oversized image.  We crop that image and that’s how we have the flexibility to adjust conversions.  We are throwing away a hundred pixels either side of the frame but it does give us the flexibility and it did seem to make it smoother in terms of the things that we thought would be hard on the stereo.

D.S.:  Even with a pretty robust stereo pipeline, it sounds like working in 3D makes things much more complex.

B.S.:  Absolutely, yes.  Working in stereo, you have to be a lot more careful in layout and animation.  People get used to cheating.  Of course we always try and beat that out of people, because if you are doing any sort of physically-based simulation, like a particle simulation or a water splash, the 3D space is important anyway because the way the character interacts with the physics of the simulation.  So we had situations on Pirates of the Caribbean where we were match animating a body to a mermaid, and it just wasn’t working.  We discovered that the head of the mermaid was actually two feet behind her shoulders somehow, and it was some cheat that had happened in the layout phase. The problem is that you really have to be on top of your quality control, because if it makes it into the pipeline, what sometimes happens is, the next artist comes along, and says, “Well, you know it looks like it’s working so I can just band-aide this on top of it,” and it actually goes quite far down the pipeline.  And [then you find] this just isn’t working.

Once you open up the scene, oh my God, it’s two feet off the body, you know, and so, then you have to go back and re-do the work.  We had to really be on top of that, and work with the artists and say, “OK, you have to quality control that, everything has to be proper, in proper space.”  Animators can’t animate to cameras as much, so it [stereo] does make a big difference with that [animation].  Generally a rule of thumb was that lighting wouldn’t change a whole lot. Animation, we didn’t think would change a whole lot although the issue that I mentioned does add the fact that they can’t cheat as much.  Layout, definitely it added and rotoscoping it added.  Compositing, where it really got more complicated was when you were doing things like, you’ve got a water surface which is uneven, and it is going back in stereo space and you are adding three mermaids splashing around, and each of them have splashes, and you somehow have to make it look like those splashes are sitting on this uneven water surface.  That created a lot of nightmares for people.

What oddly didn’t create nightmares was one of the issues that you get when shooting as we did with a mirror system.  When one camera is looking down at the mirror, and one is shooting through the mirror, even just the fact that the eyes are offset, you get different reflections in different eyes.  The polarizing filter, and the mirror, has an effect on things.  So there would be shots of the mermaid coming up in a little sea pool for example, and you get reflections in one eye but not in the other.  That is something we experience in real life and is certainly common in stereo photography.  The tendency is to want to fix some of that stuff.  But we found that when we started trying to paint out reflections that were in one eye and weren’t in the other, the water stopped looking wet.  So, you actually want to leave in some of these imperfections.

D.S.:  Were there any significant pipeline changes you needed to make to handle this film?

B.S.:  I think the big thing was definitely the color pipeline, getting the RED camera data through our pipeline, how we dealt with that, how even we dealt with the facility that was handling the overall DI and the color correction.  We did a fair bit of work with them to work out how to get the data from the camera set to us, what happened when we got the data.  What we would do is bring in the plate say, “OK, is there any error in the stereo between the different eyes?”  We’d try and correct and offset that.  “Is there any color difference between the two eyes?”  We’d try and correct that.  We built a different front end to our color pipeline for the way we brought data on.  Also, because we were dealing with an over size image that we were going to crop, [we] changed all of our display techniques and the way we would look at dailies and then the way we would give data back to the client.  So the whole image pipeline changed quite a lot.

Then for the mermaids themselves, it was really the same sort of challenge as whenever you get a new creature.  How do you make a rig for a tail that is animateable, but not so heavy that the person can’t use it and so that the animator can intuitively just pull some portions of the tail and the rest will do the right thing?  How do you write a program or a mini program to make that happen?  How do we deal with the fact that they have this membrane over the top of their bodies and how does that behave?  How does the material on the underside get affected?  How do you deal with these things being in a crowd?  First it looked like it would be seven, then it grew to fifteen and then it was fifty mermaids all swimming at you.  How are you going to have someone animate that?

We set up tools that allowed us to use baked cycles where the animator, just in Maya, could go in, select a cycle and apply it to their mermaid.  We had seven different mermaids in the same cycle and could apply [tools] to the different mermaids and work correctly.  Then if the animation wasn’t quite working on one of the mermaids, the animator had a tool to say, “OK, I am going to pull this out and animate it again, automatically and then I can just make the tweaks I need.”  So there are a lot of tools that we did for handling the mermaid crowds, automatic tools for placing splashes, generating CG splashes, and then replacing the whole water surface if that was the easiest way to deal with it.  Sometimes it was easy to forget the water that was in the plate.  We are going to generate the entire water surface because otherwise we will never get the splashes to work properly.

D.S.:  The name ILM historically has been synonymous with spectacular, cutting edge visuals.  While there are a lot more companies today doing excellent work, ILM is still known as the “gold standard.”  What drives you and the company to continue innovating and doing such excellent work?

B.S.:  I think we are very lucky at ILM to have a huge, rich legacy of visual effects work behind us for years, and a great example of visual effects projects that have made a big difference, that were really at the front end of motion control photography.  [ILM] did some of the best work in that [motion control], and then they did some of the really best blue screen composite type work in the optical world in the late 80’s, and then became one of the pioneers of the digital world with Jurassic Park, and Terminator 2 and The Abyss.  There is a rich legacy to build on at the company and an attitude that we don’t want to rest on our laurels, we want to improve things, we want to make lighting tools better, we want to make animation tools better, we want to make it faster to do things.  I mean the real imperative for every company out there these days is shorter production timeframe, smaller budgets, and the need to up the anti on each movie and up the wow factor for the audience.

So we are looking at tools both to improve the quality of how the shots look and also to make it faster for us to make the shots, to make shots that look just as great but in a shorter time frame, so that we can afford to do the projects.  I have to say in the business today there is terrific work coming out of a bunch of companies.  I have colleagues that I have worked with at ILM that work all over the world now at different companies and they are all doing terrific work.  It’s actually inspiring for anyone to look at the work being done not just at your own company, but at other companies and say, “Wow, this is exciting stuff, it’s cutting edge stuff, and I am envious of this stuff.”

I think that envy and that desire to make it better is great and keeps us trying to make better and more exciting images.  That desire, coupled with the fact that we have been able to keep the quality pretty consistently high across projects that even if people hate the film, they can usually rely on the effects being interesting and well executed and fun to look at.  Sometimes the effects are the best part of the film.  That’s not great, we don’t need to like that, but people really still respect us and they still respect the work and so I think that the company has a certain standard and a prestige.  That’s one of our challenges, to aggressively be able to make the films for smaller budgets and smaller time frames, to maintain that standard.  It is certainly never questioned that we have to do that.  The desire is to not compromise on quality, to recognize that Industrial Light Magic is a mark or a standard for high quality visual effects, and the desire for everyone at the company is to keep that standard. I think that is the key.

Dan Sarto is publisher of AWN.

Dan Sarto is Publisher and Editor-in-Chief of Animation World Network.