In the first of three articles on Spider-Man 3, Bill Desowitz provides an overview of how Sony Pictures Imageworks threw lots of sand, a little goo and some other new 3D tricks into the mix.
For more details of all the visual effects in this record-breaking film, check out VFXWorld's additional Spider-Man 3 coverage.
In keeping with director Sam Raimi's mandate to raise the obstacles and emotional stakes in Spider-Man 3, with the introduction of three new super villains -- Sandman, Venom and New Goblin -- Sony Pictures Imageworks was tasked with raising the vfx bar as well.
Although the film's overall vfx shot count of 930 was not extraordinary, the volume and complexity of the 3D work were. So there were a lot of new advancements at Imageworks to meet the challenges. First, there was the creation of Sandman, which required ambitious R&D to build a series of solvers to animate the character. Then there was the creation of the symbiotic black goo from outer space that required its own special rigs, which eventually transforms into the malevolent Venom. Plus improved virtual camera movements to better show off the acrobatics of Spider-Man and his new black-suited alter ego; a new proprietary package, Katana, for more efficient use of lighting rigs; improved digital face scanning and computer-controlled camera work to replace the faces of digital doubles with live-action footage of the actors; and, finally, improved virtual environments.
"The Spider-Man franchise has always been about solving a series of problems," contends visual effects supervisor Scott Stokdyk, who enlisted the aid of about a dozen other vendors. "In the crane disaster, there was a section where a building is somewhat destroyed by a crane arm and beam; there are about 20 different elements, including bluescreen and CG. In general, we tried to make a concerted effort to get as much live action in there as possible. I mean, the trend in visual effects that I perceive is more and more shots going completely synthetic, which I really understand because there are some things you just can't get in camera. Sometimes it's actually easier to go completely CG. But even in the most fully virtual shots, we try as often as possible to shoot a live-action component. I think that was really a good strategy: it made for every shot being different.
"For example, during the alley fight with Harry Osborn, if you look at every shot across that sequence, there's at least 15 ways we executed them. So there wasn't a simple rule about shooting where we either shot bluescreen people on CG backgrounds or CG people on CG backgrounds. It was an extreme combination of real plate photography shot on a Spider-Cam rig in Los Angeles, with an incredible amount of CG and intricate stunt work, either computer-controlled cables that used previs information where the stunt team were almost motion controlling their people with this cable system in combination with bluescreen and CG people and small pieces of sets with CG backgrounds. What made this challenging was that there was no easy way to get in a groove.
"We've all gotten pretty sophisticated in being able to figure out what's real and what's CG. But the constant challenge across all the Spider-Man movies for us has been: What real world components can we bring in there to trick the brain into accepting it? And what emotion in the character animation will help that too? One of the things that [vfx Plate leads] Nic Nicholson and John Schmidt did was a physics-based ballistics tool that gave animators another tool in their palette to ground something in the real world. It basically allowed the animator to take the center of mass of Spider-Man and, based on some inputs, to map how the trajectory of that would work in 3D space. Instead of having to just create an arc or a swing from scratch, the animators would figure out if you through a body through space, this is what it would do. I think that Sam has become more savvy over the last few years about what makes these stunts more interesting whether they're CG or real and what Spider-Man does."
Stokdyk says that although Raimi understand the limits of vfx, he still tries to push them. A case in point was Sandman. The sand had to look and act just right.
"Up until the end, there was an incredible amount of custom code created for sand," Stokdyk continues. "The sand team likes to say it's the equivalent of six years of labor done over the last two years. It was quite complex. And aside from the development, the shots themselves were complex involving a very intricate mix of character and effects animation. In many cases, it was more than just effects animation layered on top of character animation: it was that the effects animation was so integral to the character animation that it became a very iterative process where the character animator would block out movement, even of a character that wasn't completely formed, including volumes and shapes to give a hint of the effects to come, and then the effects animator would take over and use some of those Maya surfaces in the particle system to create volumes and flow and then go back to the character animator and say, 'This isn't working here: we need to speed this up, slow this down; we need more volume here.' It was the combination of interlaced effects and character animation that elevated the complexity of it.
"What the sand team did is break the sand particle system into a series of solvers. For instance, there would be a fluid simulation solver and there'd be other systems that have other behaviors that have to do with avalanching and cascading, blowing and collision, targeted-driven shape movement and then all these systems had to talk to each other and we needed to have particles transfer between these different systems, so there was an overhead in terms of just accounting for this. And a whole suite of tools was built at the very start of the movie based on a test shoot where John Frazier and his special effects team brought out all different kinds of sand and sand substitutes like ground corn cob that was used for some of the stunt work because it was light weight. And we filmed all of this for specific things we knew were in the script. And then the very first thing on the show we did was side by sides of CG matching micro behaviors of sand. It has different behaviors depending on whether it's wet or dry or what its density, layout, shape are. And so the sand team created little sub-tools to get these kinds of behaviors. And then the real challenge was: once you've got this suite of real world behaviors that sand exhibits, when you apply that to a shot with character animation and a story to tell, how do you art direct it and animation direct it the way Sam wants, and keep those underlying characteristics that people recognize as sand?"
Spencer Cook, who was elevated to animation supervisor on Spider-Man 3, concurs that the greatest achievement was the integration of character and effects animation that reached new heights at Imageworks.
"We were trying overall to continue the great work in the past and elevate it to a new level," Cook offers. " On this one, there were just a huge variety of animation styles because of all the characters and their different body languages. For example, continuing to make the flying and acrobatics of Spider-Man as realistic as possible. And Spidey's black-suited alter ego also needs to have very realistic kinds of acrobatics but he has a whole different body language.
"And then there's the birth of Sandman, which is a very low-key performance where he's not doing anything dramatic. He's very subtle and there's a lot going on in trying to convey what he's thinking about his emotional state, which concerns his sick daughter. Sam very much considers the animation as acting, so we talked a lot about the emotional state and getting across to the audience the pathos of Sandman and trying to get some empathy for him. The golem legend [about the clay statue that comes to life] was brought up in discussions. He's very stiff and not very expressive. The challenge was keeping things subdued because our instinct as animators is to go big and to make sure everything is read clearly
"There was a lot of back and forth between character animation and effects animation to integrate new techniques and to make it whole so that sand was part of his character. We put spheres and different kinds of shapes on the character to try to establish a general guideline of where we thought the sand would be mostly accumulating and where it would be falling off. And we used that as a quick tool to show Sam and to find out what he thought the sand should be doing, because it was quicker for us to keyframe some spheres falling off than to go through a whole particle simulation, and so that was incorporated into the character animation and then the sand team would use that as their guideline as to where to run the simulations. And since the Sandman grows to such huge proportions, it was fun to develop his style of motion at this size where he becomes less human."
Stokdyk says symbiotic goo/Venom posed a similar problem with tightly integrated effects and character animation. "But -- and I think this highlights the subtlety of the effects animation -- while you have to have extreme control over very discreet particles with sand, there's more of a cohesiveness to goo: it's a creature with a skeletal structure and a gooey-like surface. It was a whole other team and what we again found was that when you have to combine character animation and effects animation that have to work together, it throws all the rules out the window because it turns the pipeline from a linear serial process into an iterative loop. And we had an fx animation team co-led by Ryan Laney, who created a toolset in Maya so the character animators were able to create their own tendrils and appendages and pieces on their own. This allowed them to define on a per shot basis exactly what they needed. Those tools were usable by character animators who did very effects-like dynamics as well as effects animators who were able to do character-like effects too, so it blurred the lines."
Cook says Raimi was insistent that symbiotic goo look unfamiliar -- not an insect, an octopus or a spider because of its otherworldly nature. "This was a tricky balance because it does have multiple limbs and lend itself to those creatures. It was a very evolutionary process developing as we went and there was a challenge in each shot to keep this thing looking alien. And basically what we ended up doing was piecing it together on a shot-by-shot basis instead of using a central rig. And it was a collection of wind tubes that we could scale out so it could pull itself along and then other appendages came out of that appendage."
Stokdyk says the Bell Tower birth of Venom posed its own set of challenges. "In this sequence, we had to very tightly couple the CG effects of the symbiotic goo covering Peter Parker and Topher, with cloth ripping off of Peter that was sometimes synthetic. And something I did new on this show was use one or two HD witness cameras every time we had to rotomate an actor. It takes a special skill of a matchmover to replicate the body. We had at least one shot at the end of the sequence: a 360° circular dolly track around Topher as goo's covering him and pulling on his face, wrapping around his torso and arms. We had the makeup and wardrobe people almost puppeteering his skin and cloth to give that real world interaction. That made for an incredibly difficult 3D track but we were pleased with the result. They spent at least eight weeks on that shot. It involved per-frame soft object manipulation of skin and cloth: individual points that got pulled. That is very much an artistic and talent-based achievement rather than a CG one."
According to digital effects supervisor Peter Nofz, the final battle consisted of 350 vfx shots and represented the most difficult sequence. "The reason that the final battle is so big is because the location in New York didn't quite exist the way Sam wanted, so they needed a construction building in there and a sand pit in there, so we needed to recreate this final battle area completely in CG. And getting three or four characters in every shot wasn't easy. It raises the bar right there; it means there's much more animation, much more effects and much more destruction. And obviously when things change [as a result of last minute reshoots], it's never a good thing for us because it means we have to partially start over, so there are all these continuity repercussions. But this time there were so many more elements that we needed to keep track of."
As a result, Stokdyk says they were ready for the final battle and whatever last minute changes were necessary in resolving the outcome. "The symbiotic goo team took on that challenge, and the way that Ryan Laney and CG supervisor Dave Seager built those tools made it very efficient for them to respond to changes. That's not to say that changes late in the game aren't hard, but the team was prepared for that and the way for Sam has worked on the last two movies. It involved the convergence of almost all our pipelines: sand, goo, character and environment. So the final battle was almost a miniature show. Because of that, I designated from the start, one CG supervisor, Francisco DeJesus, to plot out the master plan for the sequence. When I was in New York shooting, getting as much generic plates as well as acquiring environment, Francisco analyzed what could be used and also drove a lot of the development of the giant Sandman, and then once really the shots started rolling in and being turned over, we started splitting to other CG supervisors.
"One thing, in particular, that we did that I've been wanting to do for this franchise is pick a real location and replicate it, because there's definitely an urban planning element to the placement of buildings. But in every environment that was fully synthetic in Spider-Man 2, it was an environment that didn't exist in the real world. Here we had the benefit of matching a real world location exactly so we could intermix real location photography with CG. And, for the final battle, although we put another CG building under construction there, we were able to go and take real photography of the location, shoot real people there and give Sam the flexibility to do what he wanted in post.
"What's really cool about the way Sam works is that he does put a lot consideration into visual effects. He conceptualizes the creation of the villains, and in Spider-Man 3, it was the birth of Sandman and the Bell Tower sequence where Venom is created. He began very early in those sequences and shot a lot of that footage early for us so we could at least do our development of those characters. Once you've done that initial work, it gives Sam a little bit longer to figure out how to end it and wrap up everything. Obviously it was important to leave the audience with a sense of closure."
Bill Desowitz is editor of VFXWorld.
Bill Desowitz, former editor of VFXWorld, is currently the Crafts Editor of IndieWire.
