Bringing 'Benjamin Button' to Life

The long and winding 40-year journey to bring The Curious Case of Benjamin Button to the screeen ultimately became a vfx struggle as much as a storytelling one. How fitting for this reverse aging saga about maturity and mortality adapted from the F. Scott Fitzgerald novella and starring Brad Pitt.

Not surprisingly, director David Fincher (who's been attached to Button on and off for six years) turned to Digital Domain, with whom he's worked for more than a decade on commercials and smaller films, to solve the crucial performance capture challenge. It was a process that began with research in 2002 but didn't get solved until much later.

For approximately the first 52 minutes of the movie, when Benjamin appears in his 80s, 70s and mid-60s, he's not Pitt in makeup. He is completely CG from the neck up, created by Digital Domain. The first shot is where Button sits at the table and drops the fork. The last is where he turns his head on the tugboat in the snow.

"I've been working with David since 2002 on various commercials," recalls Eric Barba, Digital Domain's visual effects supervisor on Button. "He talked to me about the various challenges of making this movie. The thing that I love about David is that he doesn't have comfort that it can be done. He's more about: 'I can put the pieces together in my head of what you've shown me and what you've done. This can be done with the right budget, time and schedule -- you guys can figure this out.' He is just fearless with connecting the dots. In working with him, he's constantly pushing the envelope, even on commercials. We knew there were amazing challenges that we had to figure out."

Indeed, in 2004, Fincher got serious about getting the project greenlit by Paramount and Warner Bros. He put together an art department to start production design and scout locations, switching from Baltimore to New Orleans. "Ultimately, he asked us to do a test," Barba continues. "We had a five-week post schedule with a team of artisans to break the process down."

In the test, which did not involve Pitt, they tracked a head onto a body in a seamless way and rendered skin and hair that looked believable and compelling. There was no speaking and the range of motion was limited, but the test built confidence at Digital Domain after receiving a positive response from the studios and producers Kathleen Kennedy and Frank Marshall.

According to Ed Ulbrich, one of Digital Domain's VPs, who exec-produced the vfx, the test achieved its purpose "but also scared the hell out of us because we had to figure out the missing pieces and try to find other technology and tools that had been used for other purposes and repurpose them for this, and figure out what code needed to be developed in the middle to make it all work."

They also needed to work out an all-digital workflow, so basically Digital Domain used commercials as funded R&D. Along the way, they additionally figured out how to deal with cloth and fabric as well as ideas about how to deal with the human face.

And, as Fincher worked on Zodiac, Digital Domain made the controversial Orville Redenbacher spot, which, though turning out more creepy than uncanny, provided another R&D opportunity. "David may have had confidence from the beginning but we didn't," admits Character Supervisor Steve Preeg. "Having worked in rigging in this industry for quite some time, it's definitely a scary thing to even think about [achieving this feat]. In fact, everyone we tried to hire to work on it, felt the same way. It was actually quite hard to staff up for the show, which was surprising. But during the Orville Redenbacher [experience], we had a certain number of constraints put on us to test certain aspects, most of which didn't work out, but there were some little [kernels] of optimism."

Barba adds, "With the bleeding edge of technology, you certainly get cut and hurt and that was the case with Redenbacher. Commercial budgets and schedules are tiny compared to features and we put together a fantastic team, but schedule and the technology maybe let us down a tiny bit, and maybe some of our own plans. But if we hadn't done that, I don't think we would've gotten the head start on Benjamin Button. For me, as much as I hate failing, it was absolutely necessary to go through this process. It left a scar but it didn't kill us."

Before they actually started on Button, there was another head replacement test for a feature that didn't get greenlit, which allowed more experimentation. "I was by myself in a dark corner but started thinking that maybe we could get there," recalls Preeg. "Finally, we did some capture sessions with Brad and quick [wire frame] animation of Brad and showed it to David, who got excited at that point.

Then came pre-production. The chief hurdle was still animation: How do they make this believable? And tracking: How do they make this head sit in this world? And lighting and compositing: How do they create a system that could realistically drop these heads into this amazing cinematography?

Instead of relying on traditional marker-based motion capture and hand animation techniques to bring a live actor's performance to a digital character, Digital Domain went in a completely different direction, and developed a system called "emotion" (led by Barba and Preeg). This proprietary process incorporated several different technologies -- using them in ways that they weren't intended -- and brand new techniques. Digital Domain wrote software code that created a relational database of Pitt's micro-expressions and matched them with keyframe animation and/or image analysis data. The biggest breakthrough was the development of the idea to use FACS (facial action coding system) shapes (inspired by the work of Paul Ekman) with volumetric capture and Digital Domain's proprietary volumetric deformation rig that brought Pitt's performance to Button. This approach enabled animators to use the micro-expressions to manipulate individual movements such as a dimple or eyebrow twitch. The highly customized deformation rig could be driven by either keyframe animation or image analysis, but because it was built from Pitt's own face, it kept Button's expressions true to his performance. Mova's Contour system was a key component in this part of the process, and it was used to volumetrically capture Pitt's expressions. Digital Domain has been working with Mova's Steve Perlman for years on adapting their technology for this application. Meanwhile, Image Metrics' technology was used to provide primary animation curves and timing data from Pitt's live-action performance captured from four angles using Viper HD cameras. This gave artists a head start for a number of shots that saved them production time during animation.

Overall, 155 Digital Domain artists handled 402 vfx shots; 325 of which were head replacements. They worked from life-casts of Pitt and body actors to create three photoreal maquettes; then aged them to Button in his 80s, 70s and 60s. They created 3D computer scans of each of the three maquettes. They shot scenes on set with body actors in blue hoods. They created computer-based lighting to match the on-set lighting for every frame where Button appears. Digital Domain also developed proprietary tracking and hair systems.

So what was the secret sauce on Button?

"There were two aspects," Preeg offers. "We had actual information from Brad's face and how it moved, and we put it into a system that allowed it to be animated easily in a realtime, minimalistic way that gave you a lot of believable motion. There are rigs out there that, to make an expression, you need a hundred controllers, or you try to drive something directly with performance capture and it becomes very difficult to edit or, with too many animation curves, there is too much information in the animation that can't be reconciled to work together. The directive from early was to use as few of the controls as possible, maybe five or 10. So the movement of one control could get you a lot more believable motion in the face.

"The eyes were a different story: those were a pain in the butt. Unfortunately, the system that we used to get the movement of Brad's skin ended at certain parts of the face because of the nature of the capturing, and that included the eyes. There was a lot of work on the eyes by hand and I think we put some stuff into the eyes on Benjamin that I haven't seen in characters before."

"So there's a lot of interpretation in our emotion system," adds Barba."We learned early on that a lot of it comes down to artistry because we were transferring across Brad's current age to this older Benjamin character."

Meanwhile, lighting, under the supervision of Jonathan Litt, posed its own challenges. "One of the questions that we kept asking was," Litt says, "'Is he supposed to look old or young'? It's kind of a funny question considering the premise. But there was a youthfulness that David wanted to impart on the character, and he's not supposed to just look like your average 80-year-old guy. He's supposed to live in an in-between land of a child trapped in an older body. So that was an artistic question that came up a lot. At first, we were going to make him look 80 years old, period. We wanted to stick with photorealism, which is the safest thing to rest on. And the thing that David let us turn to was that it's OK to let him look a little disturbing, a little weird. And that was also a huge thing with the head size. David wanted the head to feel a little too big to have that childlike quality."

"On the one hand, if you're dealing with an almost supernatural quality, the more messed up he looks, the harder it is for us to sell it as physical reality," Barba adds. "So we had to make him look weird but not fake. I think there was a sweet spot that we found with the head size. That affected other things too like age spots. Is he supposed to look like he's been standing out in the sun for 80 years? Where we ended up going is there are certainly age spots in the first Ben 80 and David responded to them the more he saw them, but there aren't the more medically revealing types of marks on the skin."

Likewise, the lighting of the head was linked to the set lighting. "That was the philosophy," Litt continues. "Brad drives the performance and the director and DP drive the look and we hold true to that. In other words, it's 100% set capture of the environment and the application of that environment in the renderer. This was a pretty unique job for the lighters. It's an evolutionary step for the industry in terms of HDR capture. Like animation, we built a system that was intuitive and controllable, but all physically-based. Our tracking team did an amazing job of capturing the minute detail of every single lighting setup. It was a full survey of every lighting setup: photographic and 3D positions. They surveyed around 80-100 lighting setups with a CG head. That data then fed into the new pipeline built for the show."

Tight collaboration, therefore, was crucial between compositing and lighting. "A lot of the setup was done in Nuke, which, as everyone knows, is very powerful in both 2D and 3D," Litt suggests. "Teams of compositors would be the first to touch the shots by ingesting this data and bring it up in Nuke. We started off by using straight HDR spheres but slowly came to realize that it wasn't getting us the subtleties we needed in terms of localization of lighting. With a head moving around in a room or gradations of light across the head itself, some of the light sources are very close. There's a candlelit scene and certain types of falloff that you don't get with an infinite sphere, so we developed a hybrid system of taking all the data, mapping some of it onto a sphere and then reconstructing all that geometry and all the bounce cards and light cards and lighting sources and taking high dynamic range textures, putting them on all of that data and then, using custom shaders all in mental ray, driving the lighting from this combination of the HDR sphere and HDR set geometry and the 3D position of the head. Even the HDR sphere went through a new process that was developed in Nuke for this show, which we call 'Repositioned HDR' or 'Repo HDR.' In Nuke, they could import the camera, the head position and all of that and generate animated HDRs as the head would move around. It was a new way of lighting that worked really well in terms of volume. The head is all over the map; it's 325 shots with different lighting setups because of his movement within a scene. Knowing the schedule of the show and the ability and time it takes to match lighting by hand, you're never going to get all the way there. This allowed us to push through a lot of shots once the lighting pipeline got rolling."

And there were tracking and compositing breakthroughs as well. For such intricate tracking, Digital Domain made sure it could synchronize and time-code every frame.

"In the post process, we track the main camera as well as the high-end HD cameras to re-triangulate the performer on set," explains Tracking Supervisor Marco Maldonado. "We reorient to the survey, which is the onset environment scaled 1:1 to real world numbers. And once we figure out where that's going to be within a 3D package, that's embedded to some degree and everything else has to fall within the survey. Once we have him in 3D space, we take that point cloud to drive the CG head and CG clavicles. So there was always one person to track at a time and different body types to track with the CG Brad head. Every actor had its own set of challenges because we have to remap the point cloud to each performer. A lot of the tools for our tracking software were re-written to help with the placement of the point cloud."

The main difference on Button is that compositing actually played the very first part in the lighting pipeline. "We would take these HDRs, make sure everything was correct and reposition them to where Brad was in a scene, run that out and that would be the first pass, which would go off to the lighters, who would then finesse the technical set up of the HDR," adds Compositing Supervisor Paul Lambert.

"But we had a pretty new drill at the beginning of the pipeline and part of it was getting the grades into place, like how David wanted the scenes to look," adds Compositing Supervisor Janelle Croshaw. "We were setting the color and the tone. Nuke is the backbone of everything here, so every single department ended up having a check, a script or a comp. An image sequence was basically an auto comp of the head and we had track check, anim check, roto check. We figured out on Zodiac that digital data can get really messy and there can be a lot of it. And so we worked on automating on Benjamin, so everyone could see it immediately. We were able to automate the process going out as well and reverse all the grading we did because David likes to do all the grading in sequences up in DI with the shots that don't have visual effects. It was important that tracking needed a good way to view what they were doing over the plates, so we set up track tracks where Marco and his team could hit a button out of Maya and it would run an automated Nuke script that would grab the graded plates and the roto and do mini-comps so they could see how their tracked head fit within the shot. So it helped all the departments a lot."

But that's just the first third of Button. The rest of the movie and the remainder of the character's curious life required the skills of several other vfx vendors, including:

• Asylum (supervised by Nathan McGuinness) handled the tugboat adventure that travels from the Mississippi, up the eastern sea-board to the mid Atlantic and across to Murmansk, Russia.

• Hydraulx (supervised by Erik Liles and Greg Strause) delivered 79 shots, including CG falling snow and breathe enhancement (Murmansk and other Russian locations); 2.5D Murmansk matte painting enhancement; 2D Paris matte painting enhancement (the accident scene); CG head replacements for Daisy (dance rehearsal and performance scenes); and CG Old Baby Button (replace animatronic/puppet).

• Lola VFX (supervised by Edson Williams) was responsible for the "youthening" effects on Button and Daisy (Cate Blanchett).

• Matte World Digital (supervised by Craig Barron) did an assortment of matte paintings.

Digital water, boats and environments were a perfect fit for Asylum. With the advancement in 3D technology and techniques, combined with the ongoing development of a pipeline for water, McGuinness (Oscar nominated for Master and Commander) was confident that Asylum had the knowledge and experience to deliver exactly the look Fincher was expecting.

"The challenge for [us] was that the tugboat was shot on a greenscreen stage and its journey had to be seamlessly integrated into full digital environments that traversed rivers, and oceans environments," McGuinness explains. "This photoreal digital journey took place through morning, daytime, dusk and night, and there was fog and snow to complete the full gamut of environmental diversity."

One of the more spectacular moments in the movie was the tugboat battle with a submarine: the submarine was full CG and this sequence accounted for more two thirds of the (189) shots that Asylum was commissioned to design. Additionally, some of the tugboat shots that were filmed on a gimbal were replaced with full CG renderings of the tugboat to allow for creative dexterity in the battle.

McGuinness and his team were cognizant that when designing full CG environments and indeed full CG shots that there must be a photographic aesthetic and continuity must be preserved as if filming this as a live-action sequence.

As part of the project methodology for the battle, the decision was made to approach the animation design as one big shot for the entire scene. The CG team built a persistent world and animated the tug and submarine maneuvering at believable speeds to arrive at the final collision location. This approach assured that Fincher could see exactly what would have been there had the scene been shot for real out in the ocean.

For accuracy and simulation purposes everything was moved in "tug space" by setting the tug as the center of the world. Often details such as tires and bumpers or the hull itself had to be matchmoved to give the water team objects to splash against for proper interaction.

To deliver complete photorealism to these watery environments, development of digital water assets for the film involved a group of Asylum CG technicians and artists led by VFX/CG Supervisor Jason Schugardt. Meticulously integrating these elements seamlessly into each shot was a team of compositors headed up by Senior Compositor Mark Renton.

Each sequence had a unique water type whether it was the muddy Mississippi, choppy open ocean, chilly Murmansk Harbor or eerily calm waters of the Atlantic during the tugboat battle. The water team generated surfaces custom to each shot using proprietary code and in-house software for the RealFlow and lighting teams based on their specific needs. The RealFlow team would then simulate boat splashes on a low resolution proxy mesh that would then line up to the high resolution mesh lighters rendered in the final scenes.

Additionally, in the Tugboat battle sequence, Asylum had to create a burning ship with floating debris and dead sailors. The water was supposed to be a "dead calm" but even the smallest variation in ocean surface height made it obvious when objects weren't floating on the surface. A system was created that not only kept floating objects matched to the height and angle of the water but also moved them along the surface as the tide ebbed and flowed generating its own ripple and displacement.

Specific to the characteristic of water, as it approaches the horizon, it is mostly constituted by reflections; therefore, high detail 2D sky matte paintings were very important part of rendering the water. These matte paintings were integrated into large dome reflection textures to make sure the water color matched the final composite.

Velocity and caliber for gun-fire as well as resulting glass damage was designed down to the exact frame. To match the gun flashes, bullets hits and blood splatters were composited into shots using a combination of practical and digitally painted elements. McGuinness and his team carefully researched a host of other supporting digital assets to ensure they were not only accurately represented but visually exciting.

Lola, meanwhile, created 315 shots, greatly refining its "youthening" technique since X-Men: The Last Stand. In fact, Lola broke it down to a science, using Flame for compositing, Maya for 3D and Bijou for 3D tracking.

The artists adjusted the features of Blanchett and Pitt based off ratios and tables derived from anatomically correct formulas. Because of the large volume of age enhancement shots, Lola had to create an efficient pipeline that was both accurate and efficient. Blanchett was difficult because her skin is nearly flawless; it is hard to remove wrinkles that do not exist. For Pitt, the challenge was to match how he looked in Thelma and Louise, his iconic breakthrough performance.

In the dance studio scene, Pitt dramatically emerges from the shadows, and is 20 years younger. This is the first time Daisy sees Button as a young man, and the effect required strong audience reaction. Senior Animator Casey Allen was tasked with creating new techniques. "In the end, after removing and/or repositioning every fold and crease of the eyelids, Casey was able to create new geometry of the eye though re-lighting and resetting the structure of the entire eyelid," Williams explains. "This created an eye socket structure that Brad possibly never had, but the overall effect, combined with the re-shaping of the rest of his face, was that of extreme age rejuvenation. Casey worked closely with David Fincher and me to fine tune the look of Brad's eyes, and Casey devoted close to 60 hours for this one shot.

"[Later], when 21-year-old Daisy first encounters Benjamin back from over-seas, we adjusted Blanchett's already porcelain-fine skin and sculpted features to suggest the clinging baby fat of early adulthood. Lola's technique preserves the skin's natural texture while adjusting the depth and lighting of the face to push the character forward or backward in age, all while emphasizing the actor's performance, so we filled in her defined cheek hollows and more evenly spread the light across the facial features. Much of this was established in our earliest tests done by Senior Artist Leong Wongsavun. [Fincher] made adjustments to the overall look and then to individual shots with Artist Chris Ingersoll over a few months. The tracking challenges presented by Blanchett's youthful performance were addressed with a combination of solutions in Flame and PF Track.

"One of the challenges we faced when working on older Pitt closeups was enhancing the makeup already present and ensuring the most realistic looking skin possible. Latex and other prosthetic pieces just don't react to light like skin does, or necessarily have the same texture as skin. In the scene where Benjamin discovers his eyesight is getting better, Artist Sean Wallitsch readjusted the breadth and intensity of highlights and sheen on the skin, and used virtual skin grafts to smooth out the edges of prosthetic pieces and get more realistic looking skin overall. In addition, mesh warps were also used to distort features (like eyes) for an older look that you just can't do with prosthetics. In other sequences, we 3D tracked facial features with PF Track to allow us to accurately place even more complex details onto an actor's performance.

"In addition to drastically altering the primary actors' ages, it was necessary to carry through distinctive facial characteristics such as scars and bone structure to all ages of the characters. For instance, Pitt's character Benjamin has a scar across his cheek that is integral to the story. When the character appears as an infant, our Flame Artist Trent Claus was able to track the featureless skin of the baby using PF Track and then, using Flame, he could adjust the height, texture and other dynamics of the skin to create a dramatic scar across the baby's cheek. Trying to apply such a drastic make-up effect to an infant on set would be impractical, whereas we were able to seamlessly alter the skin of the baby to suit the story using digital effects.

Matte World Digital helped to create the many environments and periods needed for the "time traveling" locations.

Beginning with the Armistice and ending with the rising flood waters as New Orleans's levees break from Hurricane Katrina, Matte World Digital's contribution to the film covers the 20th century and the beginning of the new millennium, with notable timeline stops along the way that include 3D environments and digital matte paintings of New York, Paris and period views of New Orleans itself.

The time traveling required the kind of photorealistic effects work that has to seamlessly fit into live-action photography and match realistic, not fantastic, environments. "Such so-called 'invisible effects' have long been a Matte World Digital specialty and our most demanding projects," Barron suggests. "Our Benjamin Button work ranged from digitally expanding a New Orleans train station set to creating entire city environments out of pixels."

Matte World artists created a World War I armistice celebration that takes place the same night that Benjamin is born. "As fireworks go off and jubilant crowds gather in New Orleans' French Quarter, MWD filled out the sequence with crowds, put in period fireworks and created elements with interactive lighting to enhance the location and extended the visible streets back further into the distance," Barron continues. "For the crowds, we shot our employees in our parking lot at night, running around and waving sparklers, creating a library of elements for compositing.

"For the train station interiors, Fincher shot only a minimal set on stage of only a few benches and a façade that MWD filled out with complex 3D environments. We added ornate ceilings, rows of windows with shards of light coming through them, benches and other set dressings, and additional crowds of people. Photographers within the scene fire flash bulbs, and additional lighting renders created momentary cast shadows and highlights. The interiors were re-created in different eras and at various stages of deterioration as a 3D station model rendered with global illumining techniques as 32-bit high dynamic range photorealistic set extensions of a given scene that could be altered and aged as needed. Our pipeline included using sliders to manipulate lights to match the on-set plate lighting. Environments that were wholly, or partially, computer-generated included a snowy street scene in northern Russia, shots of New York in the '30s and '40s and a dramatic flyover of Paris in the '50s required modeling most of the city, and was textured with aerial still camera pictures photographed and retouched by MWD artists.

"Audiences might not suspect that they are viewing matte shots when they see the moody dawns at the Button family lakefront home. The house becomes an emotional touchstone for the film and was also created in 3D by MWD artists. We aligned our sunrises to a practical light source mounted at the end of the Button pier to illuminate the actors. The water was modified in composite to reflect the added skies, and combined with additional backlit water elements.

"MWD provided the theme of time itself in the form of the New Orleans train station clock, which runs backwards as testament to the clockmaker's pain at having lost a son in the Great War and his wish that he could turn back the hands of time. The clock face was a digital creation, tracked into the frame or part of our 3D environment, rendered with the effect of bright windows seen reflected in the glass face. The CG clock is seen throughout the film, including the end of the movie, after it's been taken down and stowed away (replaced by a contemporary CG LED clock whose numbers move forward). The final scene sees the flood waters of Katrina rise as the forgotten clock is revealed, still moving... backwards, with added digital painting illustrating the weathering and aging of the clock face and the effect of rising water seeping in and causing it to stain."

However, it all comes back to the believable performance of Pitt as Button. And now that Digital Domain has arguably created the first completely photoreal human performance, what's next?

According to Ulbrich, who believes Digital Domain has crossed the "Uncanny Valley," the studio has already begun production on the next feature that incorporates the technology developed for Button in its next iteration.

"Digital Domain first saw a script [for Button] in the late '90s when Ron Howard was attached," Ulbrich recalls. "It was deemed impossible. Tools would have to be invented that didn't exist. This was still in the realm of science fiction. Benjamin Button has been a quantum leap for us... we achieved what had never been done before."

Bill Desowitz is senior editor of VFXWorld and AWN.

Bill Desowitz, former editor of VFXWorld, is currently the Crafts Editor of IndieWire.