Ellen Wolff explores how ILMs Stefen Fangmeier mixed cutting edge technology with a bit of old cinema magic to help conjure the retro look of Lemony Snicket.
Stefen Fangmeier is a serious guy, a 14-year veteran of Industrial Light & Magic with such credits as Terminator 2, Twister, Jurassic Park, Casper, The Perfect Storm, Saving Private Ryan and Master and Commander. He also possesses CG expertise gained at software company mental images in Berlin, and the National Center for Supercomputing Applications at the University of Illinois. But when Lemony Snicket author Daniel Handler came to speak at ILM, Fangmeier completely dissolved. My cheeks were hurting at the end of his talk it because I laughed so hard. Daniel was absolutely hilarious. He really has a wicked sense of humor.
In collaboration with director Brad Silberling, it was Fangmeiers to make sure that that Handlers offbeat approach to childrens literature was translated faithfully to the screen in this big-budget production for Paramount/DreamWorks. Silberling, who had worked with Fangmeier a decade ago on Casper, specifically requested that he supervise the effects in Lemony Snickets A Series of Unfortunate Events.
While Fangmeier may be best known for visual effects that appear to be photographically real, the assignment on Lemony Snicket was anything but that. For me the key was to understand the level of reality necessary. It wasnt the ultimate goal to create an absolutely real look but something that had some flair. The whole film has a theatrical feel to it. It is sort of a `non-period period film thats got its own aesthetic. I call it `Fellini for the family.
A Theatrical World
That look was developed first by production designer Rick Heinrichs, an Oscar-winner for Tim Burtons Sleepy Hollow. Fangmeier describes the films ambience as a little bit of Eastern Europe mixed with 1950s England. Everything is a little bit drab and dreary, yet its beautifully photographed. Theres no exterior work at all in this film. All of these sets were forced perspective, which was wonderful. It gives the feeling of being on an elaborately done theater set. The aesthetic is wide angle lenses almost every shot was done with a 17 or 21 mm lens.
To find the necessary space to achieve such large-scale staging, the production took over an enormous space at Downey Studios near Los Angeles, the site of a former aerospace plant. They even dug out a huge tank, and on one side they built an artificial beach. They also built a village at Lake Lacrymose, which was kind of like a northern Italian town. Part of the journey that the films three orphans take puts them in the Dickensian lair of Count Olaf (Jim Carrey) and the towering cliff-side home of high-strung Aunt Josephine (Meryl Streep). Her house was partially built and the interior set was surrounded by backdrop paintings a lot of these were full-size painted backings that were like curtains draped around the set. Theyre very stylized, like theatrical scrims, and some of them were backlit. At times we replaced or altered them to give some shots more reality. We also built miniatures of Aunt Josephiness house, because on set the stage at Paramount was not high enough to build the whole tower.
Of the hundreds of visual effects shots that ILM wound up doing for Lemony Snicket, 40 of them involved creating the environments that the three orphans travel through. We told Brad that we could shoot real locations, but since everything else was shot on interior stages, if we suddenly went outside that would break the frame of the film. I said `Lets fabricate this and make it look like theyre driving through an elaborate painted world. It was in keeping with the films aesthetic, making you feel like youre in a very art-directed world kind of like What Dreams May Come. We created all the environments that they drove through both city and landscape environments. Some of them were simple matte paintings for the lock-off shots. Others were 3D environments that we built for them to drive through. It made it easier on us that often the depth of field was on the kids in the car so we had an out-of-focus background.
With a lot of the matte-painted exteriors, there is reality, although its more beautiful than reality. Its like when you see a sky with incredibly beautiful colors in it. If I did that in most movies, people might say it looks fake, but in this movie, that was never the problem. We could make something look so rich, or the cliffs below Aunt Josephines house look incredibly steep. We had the license to create things that were production-designed.
The ILM animation crew had several slithery critters to create as well, including a two-headed cobra and the worlds most deadly viper. They were fantasy characters to some extent, notes Fangmeier. Its not that we made things look cartoonish. We wanted to make them look real, and feel like theyre really there, but we werent copying an actual python or a known snake. Theres a fine line of giving `touches like scale patterns in the simulation of the snakeskin, and how it slides around to give it an interesting look. That doesnt make it look fake, but it might be something that you wouldnt get with a real snake.
One of the more fanciful characters in Lemony Snicket actually presents itself to one of the orphans while hes sitting at the waters edge. He reaches down and touches this little shell and out of it comes this impossible sea creature with all these tentacles. Its a beautifully animated creature that comes out of this shell magically. It had to look real and yet it doesnt represent anything in real life.
The creepiest critters of all appear in what Fangmeier calls the leech attack on the lake. We modeled them to be about eight to 10 inches long. In some shots, thousands of them attack the boat that the children are in. Its a really scary sequence.
Another Perfect Storm
Fangmeiers team had to tread carefully to create the invented reality of the storm sequence in Lemony Snicket. Almost every shot in that sequence is a visual effects shot. They shot it on a stage with the kids on a gimbal, which they couldnt really run because it was kind of dangerous with the kids up there. So we created a whole lake below them in CG.
Fangmeier is no stranger to creating water-worlds, having won a British Academy Award for the visual effects in The Perfect Storm. He also earned an Oscar nomination for his work on that film, and then earned another one for the sea adventure Master and Commander. However realistic CG water needs to be, he observes, Youre using a few million particles to simulate something thats made up of billions and billions of molecules. Were always doing a reduction of the complexity thats there in real life so we can manage it.
The films climatic hurricane causes the precarious cliff-side house of Aunt Josephine to come tumbling down as well. The house basically falls apart in the wind all the wood scaffolding breaks up. All we really had was an interior set that they could drop a little bit so that you could get the sense that the people inside were having some interaction with it. But with the kids in there, we couldnt really, turn on the wind very much or blow debris. All those things had to be done later in CG.
For such a physical effects-heavy sequence, there was an animatic done. That helped us figure out how the house falls apart. First the library gets torn off and then the main living room starts to drop. The refrigerator and then the stove break free and slide around. That was the only previs done for this film.
A Synthetic Star
While audiences will notice many stylized effects in Lemony Snicket, the most innovative work in the film was designed to go completely unnoticed. In several shots, ILM substituted a digital version of the youngest of the films three orphans: the 16-month-old character Sunny. The character was played in the film by identical twins Kara and Shelby Hoffman, and Fangmeier knew from the start that wed have to match these twins. We built Sunny, and her dress and hair and everything, to be intercut with the live-action girl. We even have one scene where her CG face is in complete closeup.
We really had to live up to making the CG pretty much invisible, even when Sunny did things that were outrageous. Theres one very funny scene where shes bitten the edge of the table and shes hanging by her mouth with her feet dangling. To pull that off and make it look believable was fun. We also did a shot where Sunny catches a spindle that her brother throws to her. He throws the spindle at camera and she comes into frame from under camera, jumping up and grabbing it in her mouth like a dog with a Frisbee. Then she turns to the camera, grinning, with that thing in her mouth. Thats probably our absolute hero CG shot of her. Obviously Im a little bit close to this, but it really is quite convincing!
Doing things with babies is tricky. The important thing, of course, was to have the real girls to photograph. We did have a ton of references from photographs that we took while filming, and we had the girls come up to ILM as well. We did elaborate sessions with them to get colors, textures and shapes. We had to see what was natural to them and what wasnt. We couldnt really scan them because babies dont hold still. So the traditional technique of taking a cyberware scan and all those things that you can do with adults because you can tell them to hold still for 20 seconds wouldnt work. We had to devise a technique of getting their surface data scanned. We devised something with multiple cameras simultaneously photographing the girls from different angles. We could then use that to later reconstruct Sunnys facial geometry.
It was certainly something that was pioneered for this film. Our software group, which is led by Steve Sullivan, was still putting this together as we were getting ready to do the setup. Constructing the geometry to the level of quality that we needed was definitely a new thing. We needed it to be just as good as a cyberware scan, and maybe even better. By taking one snapshot of her and then another as she smiled, we could capture some interesting expressions. The information that you get with a cyberware scan - because people have to hold expressions for several seconds isnt necessarily that natural.
Compiling instantaneous snapshots of the girls provided a range of expressions that ILMs animators could work with. It was not a straightforward interpolation, because when you interpolate you have to have the same number of points when you go from one shape to another. But we used these meshes that we got out of our sessions with the girls as guidelines to actually create our own topology it was a regularly structured grid placed over the surface of the face. Weve done that a lot even with cyberware scans, because that data doesnt have the mesh structure that we need for all of our enveloping and muscle animation. So we did end up with a lot of sculpted shapes based on a close study of her expressions. And of course the animators were able to blend between different shapes. It was typical facial animation shape blending that allowed them to get certain expressions created from two different shapes.
The Devilish Details
Modeling the digital Sunny was done using ILMs proprietary modeling software. Fangmeier credits modeling supervisor Martin Murphy with making sure that Sunny was on character all the time. That task required the modelers to completely obsess about that character. She had very smooth skin, so it was a fairly simple surface. But it could have looked like a doll a really good doll, but still a doll and thats what we had to overcome. There was that smoothness, yet there are little dimples here and there. Its all about subtleties. Capturing her cuteness was a main concern, to make sure that we didnt create something horrific. CG people can look creepy.
Colin Brady, our animation director, did do a lot of motion capture for the baby to get some natural body motion. But we knew wed have to do the hands and face by hand-animating them. There were two main criteria for the performances: it had to look like Sunny and it had to be cute. If we didnt hit those two, the performance wouldnt work.
One of the most difficult things were the eyes. There isnt really that much to play with. Were basically looking at spheres with texture maps on them. The real trick to getting this kind of character, once you have all the elements, is that you make the smallest adjustments. In CG, you might say something is 5% off, or that something should be made twice as dark. But with this, we were making changes of .05 %. Then wed go a little bit more. It took baby steps to get it to the point where every level was just right. It was definitely a resolution thing.
What helped tremendously in creating a convincing appearance were the subsurface scattering algorithms used for rendering Sunnys skin. Created by Christophe Hery (who earned technical achievement honors from the Motion Picture Academy for this work) these shaders were essential to get Sunnys skin to look real, adds Fangmeier.
The final touches needed to sell the reality of the digital Sunny were her virtual costume and her hair. Hair is still incredibly difficult to get to look right. This hair was fine, downy stuff, so it was very challenging. This is probably the best hair weve ever done. It was also incredibly difficult to make a CG version of Sunnys dress with its lace and ruffles. Weve gotten really good with cloth simulation, with surface qualities and the way that fabric stretches. But it takes great attention to detail I must say that in the end, the shots with the Sunny character went very smoothly.
The Final Analysis
At the outset of this project ILM had expected to do 200 shots for Lemony Snicket, but the finally tally was more than 500. Although we did less 3D work with Sunny than we originally expected, there were a lot of cases where we went in and fixed up things, creating an extension or a new backing. Theres a lot of different work in this film that will go unnoticed.
Fangmeier admits, Im curious about how the general public will approach this film. It is so beautifully crafted, and I hope audiences will appreciate that. Its more `old cinema magic than effects-driven. There isnt anything that screams `Heres the big effects sequence. The effects are just serving the style of the film which is something that I really enjoy.
Ellen Wolff is a Southern California-based writer whose articles have appeared in publications such as Daily Variety, Millimeter, Animation Magazine, Video Systems and the Website CreativePlanet.com. Her areas of special interest are computer animation and digital visual effects.