ILM VFX supervisor Ben Snow discusses the enormity, complexity and challenges of the work his team wrestled with on Marvel’s latest superhero adventure.
‘Avengers: Age of Ultron.’All images © Marvel 2015.
Marvel’s latest action-adventure hit, Avengers: Age of Ultron, expands the studio’s ever-increasing appetite for elaborate and sophisticated feature film visual effects. Managing the work on 800 of the film’s 3000 visual effects shots was no small task for ILM VFX supervisor Ben Snow. “This by far was the biggest canvas we had with Marvel and we had a lot more shots [than on previous films],” states Snow. “Chris Townsend [Marvel’s overall VFX supervisor] is ex-ILM and we worked together quite closely when he was here. We had good shorthand and there was a strong degree of trust with him and Marvel. It gave us a bit of freedom and also meant that things went smoothly.”
With other projects happening concurrently, and the comic book adaptation featuring varied and complex shots, ILM sought assistance from their facilities in Vancouver, Singapore and London. “It was a little bit of a stretch spreading the work out that much internally but because we have direct satellite hook-ups and Vancouver joins us in dailies via hook-up, we were able to get good visual effects supervisors in each of the houses and put the work out,” says Snow. “That’s how we did it. We were able to do most of our asset development at the base here in San Francisco.”
Snow was initially concerned about potential production pipeline issues. He explains, “One of our CG supervisors went over to London for a few months to set things up. We decided that we would keep everyone on our basic pipeline so that we could transfer work back and forth. The different facilities, even though they have the same toolsets, do have the independence to try different things. We always encourage that. The pipeline was impressive.”
One of ILM’s traditional strengths is their talented art department, which was used extensively on the film. According to Snow, “Marvel has strong visual development and Charlie Wood’s [Guardians of the Galaxy] art department did fantastic work as well. But there are always details to work out when you’re trying to make something functional. You have to do concepts for all of the thrusters, weapons, force fields, and damage levels as the different battles go on. We did the bulk of the environmental concepts for our work. There was some initial work that Charlie’s team had done for Sokovia but we came up with the final look. Different buildings that get damaged during the Hulk Buster Fight, the paint job on the Quinjet had to be refined and giant engines that lift off Sokovia needed to be created. We ended up doing about 900 pieces of artwork beyond what we got from the client.”
ILM’s efforts integrating their work with the film’s previs and postvis was extensive. Snow continues, “Even with previs, though often they were able to take it to a certain degree, we had to get it into proper animation and layout to refine a lot of shots. We ended up doing a lot of blocking. But it was a great [previs] team and we had great communication with them. We setup new systems on the show to make it easier to ingest their work so for a shot that was fully CG we could use their work quickly as a starting point. Then we were able to go onto the postvis stage.”
Ultron, an artificial intelligence created by Tony Stark (Robert Downing Jr.) serves as the film’s chief protagonist. “Ultron has several incarnations,” remarks Snow. “He starts life as a simple Iron Man costume and becomes one of the most intricate robots ever built, known as Ultron Prime. We were studying beautiful photographs of engines, watches and intricate mechanisms which were our touchstone for him. Marvel’s visual development department had done the basic design but we had to work out the intricacies of the face. That further evolved as we started rigging the face. It’s 10 times more complicated than one of the Transformers”
Ultron Prime was the most complex hero character ILM has ever done, with 2000 controllers, 600 in his face alone. This complexity was driven by James Spader’s (who played Ultron) great performance. Says Snow, “Our animation supervisor Mark Chu, Chris, and I felt like that we wanted to pin it [Ultron] to that [Spader’s performance] and Joss agreed. Like with Davy Jones [Bill Nighy] in Pirates of the Caribbean: Dead Man’s Chest, we had the animators study that performance and reproduce it on the face of Ultron Prime.”
Snow describes how in the case of Ultron Prime, bringing the performance to a machine, rather than a living creature, was not easy. “In this case it wasn’t an organic flexible face but one made up of little sliding plates and technology. The rigging for this required the rigger to work out that when Ultron opens his mouth, a plate is going to slide under the next plate and that’s going to slide under the next plate. Sometimes we would allow him to open his mouth in an angry expression which made him more like the iconic Ultron from the comic books. We were able to use that similar intricacy on his body so as his pectorals would flex you would see plates sliding on the body as well.”
Creating Ultron also required development of new metals and surfaces. “With the Iron Man suits in the past we did a lot of development for the different metals,” notes Snow. “In fact Iron Man and Iron Man 2 represented some our biggest work on metal developments together with Terminator Salvation. However, with Ultron you’re not just dealing with the metal surfaces and their different qualities. We were trying to suggest that there were other qualities, particularly, when he ends up becoming a vibranium character.” Snow’s team went over to the nearby Academy of Science, where they took numerous photographs of various mineral specimens. After showing their favourite materials to the director, they incorporated that new metallic look in a rough version of Ultron Prime. As Snow describes, “Joss liked it so we went back and the museum allowed us to pull the specimen out and photograph it from different angles. Riffing off some of the photography of intricate watches, using a range of different materials as well as adding in this unusual mineral were key.” HDRI was critical for integrating the sophisticated robot into the live-action footage. Says Snow, “With a character like Ultron you have to think of it more as if you’re lighting a car for a car commercial. It’s not so much the direct light that’s on him but what he’s reflecting that makes a big difference. So you end up modifying lots of things. The beauty of computer graphics is that we can put bounce cards wherever we want and you don’t see them.”
ILM was also responsible for the opening minute-long shot starring each of the Avengers. According to Snow, “There was a lot of work done in the previs. I broke it down and did an analysis here at ILM, the Marvel team did an analysis and we compared notes. We were pretty much on the same page. Then the second unit began to plan the shoot. They came up with some different ideas as well. There was an elaborate shoot process and we broke it down into different stages. You do the best you can with the shoot and try to assemble it afterwards. We did a whole round of postvis which sorted out a bunch of the problems but not everything. At a certain point we had to say, ‘You have to give it to us and we’ll start doing the layouts to blend the plates together.’” The big challenge was blending all of the plates and getting all the materials to work together. Snow continues, “All of it was shot for real but then we had to extend it. For example, you’re shooting a sequence with jeeps running along a track in the forest but in fact we’re in a large open area in a forest because we needed our camera to run alongside the jeep and we’ve got cable cameras going. Getting the hook-ups was one issue. Another issue was there are a lot of synthetic characters involved. There’s Hulk and guys flying all around the place, like Iron Man. These things sometimes evolve because you don’t always get what you want because it’s impossible to shoot. You’re mixing a cable camera with a crane camera or a camera being driven along a car on rails. We ended up creating a lot of CG environments to blend everything together.”
The Leviathan Chamber was a big set that Charlie Wood’s team designed and built on stage that Snow’s team extended digitally. “The idea is that Baron von Strucker [played by Thomas Kretschmann] and his men have been working on analysing the Leviathan and stealing some of the alien technology that allowed this giant thing to fly. We extend the environment and added the beast.”
“The Vibranium Lab is like the Leviathan Chamber,” continues Snow. “That changed a fair bit too in post. It’s an industrial area that Strucker’s men were using and gets appropriated by Ultron to create his own army of Sub Ultrons. There are couple of big shots that are fully CG that establish this and then blend into Charlie’s beautiful set which is like this Gothic industrial Wonderland. That was the same area that we were using for the Leviathan Chamber. Ultron modifies it beyond what it was like when Tony Stark visits it.”
ILM also worked on Tony Stark’s nightmare about the end of the Avengers and the world. As Snow describes, “This is a fairly brief moment early on. It involved a lot of matte painting work. Joss wanted it to echo the idea of the portal that they had in the first Avengers. That was one of the things we did with our Singapore group with Philippe Rebours, one of my co-visual effects supervisors on the show. It’s a nice moment because it’s shocking.” Establishing shots of the Avengers Tower appear on the big screen. Says Snow, “That was the New York City photography from the first film where we sent a team out for a month to photograph the hell out of that area around the viaduct and Central Station. We were able to leverage a lot of that photography to create our environment. The towers were redesigned by the art department and fleshed out here as a generalist digital matte painting.”
A new Iron Man Mark 45 suit gets introduced in the film. As explained by Snow, “It was quite a different stylistic direction from some of the older suits and had its own challenges because it was more streamlined and things fit closer together. This meant when he was moving his joints there was a bigger chance of holes opening up in the suit. There were some intricate rigging problems to solve. The Mark 45 was a bit more colourful than some of the other suits.”
ILM also handed extensive fighting shots that take place above the ground as an aerial battle ensues between Sub Ultrons, Iron Man Mark 45, War Machine and Vision. ILM handled the scenes primarily with their London team. Snow explains, “ILM VFX supervisor Michael Mulholland and his animation team were terrific. The sequence has all of the different characters firing upon one another and there are some really dynamic shots. It’s in and around the Helicarrier and on the edge of this floating Sokovian city. We created the environments synthetically. Our linkup points were the restored Helicarrier from Avengers and the floating city of Sokovia which was a large generalist creation.”
Sokovia was an elaborate asset produced by ILM for the movie. Snow continues, “We had to break it apart and get it back into the generalist world so they could render it with textures. That’s different from the way we have done this sort of thing before because we haven’t had to destroy things on this scale.”
Snow explains the complexity of creating the fictional city. “The lift off and the floating of Sokovia were a big challenge but fun to do. Charlie’s art department had suggested a design for the city. When we were filming in Northern Italy, where we shot some of the street level material for the sequence, I was up there in a helicopter taking plates of different parts of the Valle d’Aosta Valley. One of the things we liked about it was that it had a medieval look mixed with a Soviet Block era look. Sokovia is a fictional Eastern European city and that sort of mix of architecture was perfect. There was a location in London, the old police training station in Hendon that we used also because that had more of the 1960s and 1970s Soviet Block type architecture.”
Snows extensive photographic reference shoots were a critical component of the final fully CG sequences. Says Snow, “The plate material was used as reference and as a style guide. It allowed me to get some lighting studies to help guide the artists and to get a bunch of photography and material that we could use to create a completely CG version. Even though what you see in the film is almost entirely computer graphics it is pinned to live-action photography. We got to study the atmospherics and sun angles, which provided a good foundation. It’s an elaborate model that we built in 3ds Max. It’s complicated further by the fact that the city lifts out of the ground on this plug underneath it. You have all of this destruction as it has to tear itself out of the earth.”
What goes up must come down, as was the case with the city of Sokovia. As Snow describes, “That was a big challenge. First it starts free-falling. There’s a scene with Vision and Wanda where things are floating around them. We did some basic photography onset and photographed our actors but then we had to blend that with the CG environment full of floating debris and people. We had a combination of rigid body simulations for the larger things like the cars and bicycles, and a synthetic background for the entire environment. There’s rubble and dust floating everywhere. The idea is that the city is dropping. So, you have to feel like things are lifting off of it but you also want the zero gravity feel. You end up with incredibly complex CG geometry. You’re taking every prop that you’ve made from all of the Avengers films and asking, ‘How do we plunk that in there?’ Literally you have the kitchen sink in there.”
The re-entry of a space capsule was used as a visual touchstone. Snow continues, “Joss wanted it to feel like a space capsule re-entry, or like a meteor. The idea is that the falling city is going to impact the Earth and if that happens that would cause a catastrophe like a large comet impact. We gave it a re-entry type glow, and a long vapour trail like a comet.”
The film also introduces us to a new character, Vision, which Ultron Prime creates as a perfected version of humanity. “We weren’t really sure how Vision would work,” states Snow. “At one point it looked too synthetic and started to lose some of the character of Paul Bettany’s performance. There was a good exploration process with Vision. We worked with Lola VFX. We would add the cape if it was going to be a Paul Bettany live shot and we did some fully digital Vision shots as well. We had to extend that idea of the cape being something that looked like a real cloak, but not quite real. His cape was supposed to be this intelligent material and they wanted it to look like cloth a lot of time but also wanted these hints that it wasn’t quite real. The desire was for it to subtly suggest some of the stuff that we usually run away from with computer graphics.”
According to Snow, digital capes are a better alternative to physical ones. “Thor [played by Chris Hemsworth] rarely wears the full cape in action scenes; there are always cape extensions with him. There are two main reasons. You don’t want a cape to get in the way and trip people up. There’s also an art direction quality that you can bring to it if it’s synthetic because you can have it float in a particular way. We could have it really flourish but sometimes we would go too far with it and Marvel, Chris and Joss would rein us back.”
Another major ILM sequence was set inside a church, where the Avengers fight off Sub Ultrons attempting to activate the key needed to launch Sokovia. “That’s getting towards the climax of the film,” says Snow. “It has a big shot where you see all of the Avengers in action. That was a terrific thing to shoot partly because one of the characters, Pietro [played by Aaron Taylor-Johnson], is really fast. Sometimes we would go into slow motion and you can enjoy Thor coming down with his hammer. It was an elaborate shot to shoot. We built a church set and stunt people worked out the choreography for what each of the Avengers was doing. We had to go around and take notes. Hawkeye [played by Jeremy Renner] is stabbing this guy with an arrow in the eye and Thor is coming down and smashing someone. We were taking notes of what the intent was because the actors are essentially miming to thin air.”
The physical logistics of the onset work were challenging. As Snow explains, “We had to find a rig that was fast enough to shoot the shot because it was being photographed on a Phantom which can do high frame rates. We had to find this fast camera car and put a special platform down so we could rocket around this church interior while the Avengers were doing their work. We then had to do the math to slow it down. We would reshoot Pietro running around with a slower camera and then in post-production work out how to blend all that together and add the crowds of rushing Sub Ultrons.” In the scene, Thor unleashes a lightning strike. Says Snow, “They were looking for something iconic so you could take one of the frames and it would look like a painting. We shot a plate of Chris Hemsworth [who played Thor] in the church set and put a green screen behind him because we wanted to wrap lightning and floating dead Sub Ultrons around him.”
Thor and Captain America [played by Chris Evans] fight Sub Ultrons on a bridge together while saving pedestrians in falling cars. “That was an incredible challenge to shoot because you end up doing a fair bit of rotoscoping with the elements that are impossible to shoot,” notes Snow. “You have Thor on wires and against greenscreen and Cap hanging off a bridge plus you’re trying to create this world of the floating city around them. We had a set in Hendon in London for the edge of the floating city and the bridge. The broken bridge was built over the soccer field that adjoined the police training centre. You look down 10 feet to green grass in real life and in the film you end up looking down thousands of feet to the crater and what’s left of the city that [this chunk of Sokovia] just pulled out of.” Vision destroys the final robot in the Last Ultron Sequence. Snow continues, “Joss asked for a last minute rethink to get more of the James Spader character into the Sub Ultron which doesn’t have any articulation. Mark Chu ended up coming in and doing a bunch of the animation himself. We were lucky to be able to pin it to such a great performance.”
The size, scope and production schedule of the film required ILM to share work and collaborate with a number of other VFX studios. Along with digital assets there were also sequences that had to be shared between vendors. “Because of the way these things happen in Marvel film productions, the endings often change,” observes Snow. “Like other filmmakers I’ve worked with they use the post-production process to hone and refine everything. Things can change a lot. The end sequence had to be broken up because of the timeframe. We got the Sub Ultrons from Double Negative and we sent them Ultron Prime. We did the Mark 45 suit and shared that. The Vision model was originally developed between Lola VFX and Framestore. We added some stuff to it like the cape. There was a lot of to and fro but luckily we all have contacts and friends in these different companies. Ken McGaugh [Double Negative VFX supervisor] is an ex-ILMer and a good friend.”
Looking back at the project, Snow says the two pivotal CG characters of the Hulk and Ultron were extremely rewarding. “Hulk was satisfying because we set out to try to improve him and we achieved that. For Ultron, it was how closely we were able to capture and extend James Spader’s great performance. We helped create a rich feast visually as well as in terms of story. I was lucky to have a great team here.”
--
Trevor Hogg is a freelance video editor and writer best known for composing in-depth filmmaker and movie profiles for sites such as the CGSociety, 3DTotal, Live for Films and Flickering Myth; he is a big fan of Raiders of the Lost Ark, Batman: The Animated Series, The Hobbit, Studio Ghibli, and Peter Weir.
Trevor Hogg is a freelance video editor and writer best known for composing in-depth filmmaker and movie profiles for VFX Voice, Animation Magazine, and British Cinematographer.
