The Digital Eye: The Shape of Blockbuster VFX to Come

I expected to do a brief survey of all the summer blockbusters here, but thats not how it turned out. As I started to investigate, I ran into a wall of security like never before. To give you anything meaningful was going to take an extraordinary effort. So I decided to dig deeper and started picking up useful info on a number of important films, including one that stretches beyond 2005.

Clearly new vfx magic is providing a launching pad for remakes. Old effects films like War of the Worlds and King Kong are being brought back with a vengeance. Steven Spielberg, who, in the past, has embraced nice aliens, this time not only takes Barre Lyndons original script to heart but especially the source novel by H.G. Wells. Expect truly nasty aliens this go-round. You may recall that WOW started out as a Wells book that became a notorious Orson Welles Mercury Theater radio broadcast in 1938 that people thought was real. It caused mass panic throughout the East coast. My family lived only a stones throw from the center of the attack in New Jersey. Oh, the stories they tell. Spielberg has filmed it as a personal journey as a metaphor for 9/11. Youre going to see a lot of POV shots where he normally might have used crane and copter shots.

Standing where I am on the hill, it seems like a possible rebirth of the vfx film this season a new synergy of vfx/story/acting/art direction /direction. Im hoping that contemporary moviemaking has matured to the point where producers and directors understand that story is king and all the rest, merely servants. Trust me, great vfx are easier to come up with than great story. Great story is rare and beautiful and we must bow before it.

Before you start throwing shoes, let me explain. Story starts with a blank piece of paper (virtual these days) and the writer must come up with a complex, perfectly developed vision that will entrance readers whole cloth. We inspired by that great story, great previs and serious art direction and ultimately the directors eye, dig deep inside our creative and technical selves, and come up with our fantastic visuals that keep people coming back time after time to be immersed. Its been said ad infinitum, but truly: film is the ultimate collaborative medium.

There appear to be many great films coming up, but these are the ones that intrigue me the most. You can tell by the amount of detailso enjoy:

Lets Begin With Batman Begins

The original script for Batman Begins was written back in 1991! It took some serious time to get here, but its going to be good. Vfx teams have gone to enormous lengths and used extremely advanced tools to bring you stuff that even you as vfx people might find unimaginable. Im telling you, Gotham City is so amazing in every way, and I didnt even expect that. I mean a digital city has been done, right? Not like this.

Director Chris Nolan was an interesting choice. Hes usually done dark sort of movies like Insomnia and Memento.Chris hasnt done a vfx heavy film before, and he definitely approached this one with an attitude: I do not want this to become a digital cartoon. As an aside, that cartoonish look originated with the TV series back in the 60s, where they even composited in the Offff and Zap and Arrrrgh text balloons. Those TV shows were written by Lorenzo Semple Jr. Youll see Lorenzos continuing role as you read on.

Right from the start Chris was worried about computers taking over. He has consistently required that all the vfx reflect observed reality. That is, if youre going to animate Batman falling off a roof, you do it using reference footage of a man falling off a roof. If youre going to build a CG city, you must base it on a real city. One vfx team went to Chicago and spent seven weeks there shooting nearly 1.5 million reference images in 10 stop HDR. During the process, they literally shut down the entire center of Chicago to light the buildings for reference. Those shots could be used later to create theatrical lighting for the massive, virtual Gotham.Paul Debevec, whose pioneering work in image based modeling and lighting made the city possible, could almost be the mayor of virtual Gotham. As an aside, Paul mentioned that hes going to be discussing his remarkable LSP or Light Stage Process at SIGGRAPH this year. Thats one presentation that is going to be packed.

Although the movie has more than 600 vfx shots, youll be hard pressed to see most of them. Of course, thats the beauty of great vfx work. It doesnt distract from the telling of story by standing out and saying: Look at me; rather, it creates the moods and settings that forward the directors vision.

Chris is also a consummate filmmaker, and that means he learns rapidly as he goes. He has an interesting personal style. Although he plans his shots, hes also adaptable to spontaneous situations. He can change gears quickly when he wants to, in order to capitalize on situations that arise during the filmmaking process. Thus it was, that Chris began to embrace the digital realm. By the end of the picture he appeared to be quite comfortable. His work style presented an interesting challenge to the vfx teams. They had to be flexible, adapting to Chris style. And they did.

Batman Begins is really not intended to be a vfx extravaganza. Its intended to be a great movie that uses a bit of very sophisticated vfx to achieve a quasi reality look that somehow remains true to the Batman mystique. In fact, the trailer shows almost no VFX except for Gotham City in the bg.

Gotham City

This city is different. Gotham contains a half million CG buildings and every one of them is based on a real Chicago building. There are varied repeats of course, but the designers had library of 2,000 or so, actual buildings to use. By rotating and scaling them they were able to create this massive digital set of Gotham.

Nolan and his producer wife Emma Thomas went to extraordinary lengths to maintain strict security on the set, but I did find out some very interesting info. First, Ive seen a trend in this years crop of vfx blockbusters that I like. Everybody seems to be going back to miniatures. The big films are now combining digitally enhanced miniature shots all over the place.I build miniatures as a hobby, so I can appreciate the beauty of that.But miniatures are sometimes not so small anymore. Building miniatures at 1/3 or 1/5 scale are a bit of an oxymoron.

The miniature buildings used in Gotham City are colossalsome as much as five stories high! But the big story is Gotham City itself. It is so huge that ordinary pipelines cant deal with it. Maya was used as the backbone, with RenderMan outputting the many passes needed by the compositing team to create the final look. It was a massive job. Normally this would have required a large work team over a long period of time. But in this case, the entire job was done by one person, with two TDs plus one compositor as support team.How? Thats the interesting part.

Three excellent new proprietary tools were developed by the Double Negative R&D team. As you know, R&D has become the lifes blood of cutting edge companies in vfx. Based in Londons Soho district, Double Negative has always been on the cutting edge. Its a relatively small company, so they have to be efficient to remain on top.Their internal development guys have come up with three major new tools to keep that edge.

Always pushing to make its pipeline invisible to its creative artists, Double Neg constantly works to simplify their interfaces. Its one reason artists like working there. As you know, RenderMan has long been known to be expert friendly. It can be a little like wrestling a bear to work with, if its not your full time responsibility. Traditionally the bridge between Maya and RenderMan is handled with MTOR, a Pixar utility. Though its always improving, Mtor has a steep learning curve. Thus, at Double Neg., REX was born. Its a proprietary Maya plug-in that is a snap to learn and use, but you have to work there to enjoy it. Completely replacing MTOR, it allowed that one man wearing several hats to handle all the lighting and rendering for Gotham.

Another tool set was needed as well. Theyre not entirely original but highly customized. Based on Debevecs work, Double Neg. developed their own, very sophisticated photogrammetry engine to pull geometry out of all those hundreds of thousands of HDR images taken back in Chicago. The other tool in the set is used to take those same images and project them onto the geometry. I believe this system also was used in the lighting of those buildings. When you have a high dynamic range texture, you can adjust it to bake in lighting, thus relieving your pipeline of an enormous amount of render time. The tool set was developed by Oliver James and called Photofit.

They actually put together 200 gigapixel panoramic reference images 50K across to work fromjust imagine how you could zoom into that! The thumbnails alone were 2K across! Till now I had no idea that kind of imaging was even possible. They stitched it all together with an in-house tool called STIG. And BTW, the entire pipeline at Double Neg. is HDR friendly.

To get the proper look on the buildings, Double Neg. developed their own proprietary BRDF shading models. You probably know this, but BRDF stands for Bidirectional Reflection Distribution Function. It's a mathematical model that describes how reflection behaves with different materials. That is, photons encounter a surface and several things happen. Some are absorbed and some are reflected. In addition, there can be all sorts of subsurface scattering to produce translucency. Its possible to describe real-world surfaces materials in terms of BRDF parameters.

To carry out Chris vision for Gotham City, Double Neg. went to their own R&D department and developed their own proprietary DRDF models that were then used to build Maya shaders. I understand they used some of these new shaders in the Wayne Building interiors, which have a modern Art Deco look. They also used them in the train station, an integral part of the Wayne Building. Oddly enough, this environment is modeled after the Sydney Opera house, but they used metallic surfaces to give it the right look and it all works. I can also tell you that the CG Wayne Building is about the only non-Chicago, non-real building in virtual Gotham. Its design is influenced by the Chrysler Building and the Empire State, with that cool old Art Deco feeling that I love so much.

I have to tell you about the digital stunt double for Batman. Its a very high res, very carefully textured, exact copy of Christian Bale (Bruce Wayne/Batman.) Its mostly used in long and medium shots, but I understand it could be used up close. They did some test shots of the virtual stunt double next to Bale, and you couldnt tell them apart.

Building a stunt double in CG is virtually useless without bringing them to life via animation.Oh, how to do that believably. The first thing that comes to mind is MoCap, right? Unfortunately, the only places the digital stunt double is used is for shots that could not be done in practical live action. If they cant shoot it with a live actorthey cant capture it in MoCap. So how did they do that awesomely realistic animation of Batman? They did it frame by frame by hand. Yeah, you read that right. Remember that Chris wants everything done according to reality referenceso we take a guy, put him up on a nine-story building and wire drop him into an air bag. We shoot the fall from 10 different angles and we use all the footage as direct reference for animating the digital stunt double. Labor? Yes. Results? Wait till you see them.

The geometry for the batman stunt double was built in SOFTIMAGE|XSI and piped over to Maya for texturing and animation. Everything was done in polys with sub division smoothing. I think youll agree its pure digital artistry.

Rumor has it that the very last shot of Batman Begins is entirely CG.Im told that was never part of the original plan, but the vfx teams did such a wonderful job of creating reality, that Chris was to some degree won over. I was most impressed with the great lengths people went to for perfection. The results remain to be seen, on June 15. I cant wait.

And Im betting his next film will incorporate more of the subtle digital approach he developed on this one.

King Kong

Getting information on Kong has been more of a nightmare than on any other film for me ever. Weta is as tight lipped an outfit as I have ever seen.So Ive had to extrapolate, intelligently from all the bits and pieces Ive been able to gather, to figure out whats really going on down there in Kiwi land. I was able to get some verification on much of it.

So, Im going to tell you everything Ive been able to piece together. And, OK, its not a summer release, but its an important vfx film that is definitely worth covering here.

In addition to noticing on the Kong website that director Peter Jackson is looking great (if tired), after losing a lot of weight, a grid of white dots on the Kong set: bluescreen backdrops that I suspect are for tracking; and I also noticed that Peter is using realtime compositing for his bluescreen shots. Word is: his system uses the NVIDIA QuadraFX 4000 GPU engine for realtime compositing on the directors screen. This allows Jackson to see the actors within his virtual set. As for Kong finals compositing: Discreet Toxik is a not-yet-available, very advanced collaborative compositing environment. It allows the compositor, DP and director to all get involved in the process together.I have it on good authority that Jackson was given a pre-release Toxik suite, that he likes it and is using it on Kong. Discreet will be showcasing Toxik at this years SIGGRAPH.

As I understand it, Oscar-winning Joe Letteri is the key vfx supervisor on Kong. Joe was at ILM for a long time and played a big roll in bringing Jurassic Park to life. He specializes in lighting design and computer graphics and animation.

Several months ago, Jackson showed a select 200, NDA-strapped individuals, a healthy 18-minute preview. Afterwards, he also showed them around Weta and was particularly proud of his animatronic-like miniature forest. Apparently he has it set up to show the impact of Kongs enormous strides through it, with the branches and leaves shifting and rippling in the resulting shock waves. He assures us that nothing like this miniature effect has ever been seen before. Although none of the 200 will talk about specifics, there was a uniformly positive response to the preview.

At least one sophisticated viewer indicated that its vfx are so stunning, they remind her of how impressed she was the first time she saw Jurassic Park. So Joes participation is not so strange and very fitting. Expect to see more than 30 very cool Maya-animated dinosaurs in the Skull Island sequences. Kong himself doesnt show up until nearly halfway into the film.

OK, Im told the lead animation package on the film is Maya. But again, not Maya but Maya fully plugged with all sorts proprietary shaders and animation plug-ins. Not surprisingly, it has also been suggested that Jackson is once again extensively employing Discreets Lustre digital color grading to maintain the proper look throughout the movie. As with Toxik, the advantage here is that the color correction team can work collaboratively with both the DP and the director.

I understand that Zbrush2 was used for both sculpting and texture development for people milling about in scenes such as in the crowded Pelembang Docks in Sumatra, and possibly for Kongs face as well. Youre also going to see more than 30 CG dinosaurs roaming the integrated CG/miniature jungle, along with the biggest, fiercest, mother chomping gorilla seen in any previous version. This scary Kong is being animated both animatronically and in CG. The CG part will be the most important with never-before seen fur effects and some very special facial work by Mark Sagar. Naturally, Mark kindly declined to talk to me about it, but if you look at some of his SIGGRAPH papers you can extrapolate.

From what I can gather, Kongs CG facial structure is extremely complex with very subtle controls and a new kind of semi-mechanical face animation system that is proprietary to Weta. All this was designed to help Kong act believably.

Local Wellington rumor has it that things were made particularly difficult for everyone involved with Kong when Jackson had another one of his epiphanies and, after more than two years of development, he decided to change Kongs overall design. I can see Peters mind churning up an initial vision that sits, evolving. At some point this new vision surfaces and we get Kong just isnt working for me any more. Then he disappears and returns after significant time with a new Kong. Fortunately, most of the hardware and CG elements are fairly adaptable and all was not lost. As I recall, Peter did something like this with Gollum, changed the design late in the gameevery genius has his own waysGollum certainly worked as revised.

OK, Kong has to do some serious acting, and like Mighty Joe Young, his face is ever so important. Since he doesnt speak, much of his personality and mood has to be expressed through body language. As you recall, Andy Serkis did an amazing job with the Gollum. Andy has again been drafted to become the body and soul, and probably the face of Kong. Some say hes been walking around like an ape on the set. Weta is using some of the most sophisticated body and face MoCap seen to date. Ive heard the number of capture points on the face is in the many thousands. As I understand it, Kongs very expressive face will be animated both by this new face animation capture and augmented with artistic frame by frame tweaking.

NYC-33

Now we come to New York City. The city is almost like a character in this film. Weta has gone to great lengths to create NYC, circa 1933 in a way that will both make you believe and fascinate you at the same time. I understand the art direction on the city will create a dark mood. The city was built under the direct supervision of Joe Letteri. Joe is much more than a technical expert; hes a brilliant artist as well. He knows how to get the look Jackson sees in his mind. NYC-33 required his touch to get that noir mood going. HDRI was used in creating texture lighting, after the fact. This lighting was baked into building textures to reduce the need for resource-expensive lighting on such massive geometry as this city entails. The pipeline had to be enhanced to handle the massive data requirements of such large-scale geometry. I also have reason to believe that Joe and his team have incorporated some photogrammetry to extract building models, and a lot of original Art Deco building design, much of it, based on early photographs of NYC.

One last thing: Weta used Massive to create the virtual actors for battle scenes in Lord of the Rings. They have maintained an in-house Massive development team. Im absolutely certain that Massive is being used in creative ways on Kong. Massive is the Academy Award-winning like the collective brain and animation engine for virtual actors developed by Stephen Regelous. These virtual actors can now see and hear; they can react according to a library of behaviors for given situations.So I expect the hoards of horrified New Yorkers, animals on Skull Island, etc., to behave in believable ways as they actively mill and flow and battle about. Oh, and one last thingIts been mentioned that a fair amount of rendering and simulation is being done using GPU engines, both for previs and for some of the final shots.

I hope this whets your appetite until Dec. 14, when we can finally see Peter Jacksons reimagined Kong.

Superman Returns

Director Bryan Singer was the third choice of Warner Bros after Brett Ratner and McG. for Superman Returns. Normally that looks a little like disaster. But it looks like Singer, who did such an awesome job on the X-Men movies, and who knows vfx from the ground up, is the right man for the job.

Previs has been assigned to Pixel Liberation Front and a large portion of the vfx to Sony Pictures Imageworks (sharing with Rhythm & Hues, The Orphanage and Rising Pictures) with Maya plus plug-ins being the main 3D suite. Good choices.

Panavision sent over its new working prototype Genesis digital camera for Bryan to play with. There was some discussion about the shots taking too long to get to editingwhich makes no sense to me.Im hoping they work out the bugs. I saw the first test shots of the farmhouse up in Tamworth set at sunrise and they looked great to me. With this camera you start out with the DI ready for vfxcool.

Genesis is the future. It was designed as a hand held direct to DI camera, incorporating proprietary gamma and colorimetry, which enables seamless intercutting with standard 35mm emulsions. It should be great from a DPs POV with a nominal ISO sensitivity of 400 that can be easily extended to ISO 1600 for low light situations, with relatively low noisepretty amazing. It has a whapping 35mm, 12.5 megapixel sensor. Talk about HD! The chip was designed that size so that the Genesis could use all Panavisions regular PRIMO primes, and zoom lenses. Shutter angles from 0.8 to 360° and frame rates from 1 to 50fps were designed-in to enable compatibility with Panavisions regular family of film cameras. The Genesis is built by Sony exclusively for Panavision. Im pretty sure that Bryan is going to shoot most of the film with the Genesis.

Metropolis

Bryan realizes that Metropolis is one of the key characters and we dont want to mess it up with the reality of New York. BTW, does this city is really one of the characters idea strike you as a trend here? Indeed. Creating any of these famous cities has more to do with art direction and digital vfx than anything real. It could as easily be done in Prague. These are escapist moviessmart directors create a believable city that isnt too real even Nolan. Otherwise, theyd forego the vfx and just shot in Vancouver. With vfx we have control over Metropolis, everything from the weather to the citys style and mood.

From a practical point of view Bryan had the sfx team, headed by Neil Courbould, build the most massive computer-controlled multi-axis hydraulic Gimbal platform that Ive ever seen. It took 11 weeks to build and is controlled via a game-like joystick, and all the motion control data is sent back and forth between the computer and the Gimbal through a very tiny cable that moves with the gambol via pulleys, and through a series of Teflon roller guides. If the wire breaks, the entire Gimbal with its 20-ton steel platform, will hopefully glide to its default horizontal position, but as I understand it, this has not been fully tested.

Because the platform must make some pretty radical shakes, it appears to me to be turbocharged by some sort of high-pressure gas. You can see a large bank of red gas cylinders under the rig. Neil says the crew takes great pleasure in watching the faces of extras as they take their first ride on this monstrosity. Youll see those honestly frightened faces in the movie.

The huge and remarkably detailed Daily Planet set with more that 35 miles of cables, is beautifully art directed to look both modern and still retain that 1950s Superman feeling.And if you look very closely youll see the Bryan Singer Award as one of the plaques on the wall above one of the desks. I dont know if youll actually be able to make it out in the final cut. The same building is, of course, available in both CG and large-scale miniature. Mostly, the practical set is for the hero building. For the crowd scenes, theyve put a false Art Deco front on a modern Sydney building where they can garbage matt out the modern building and composite in the CG version.

I am old enough to remember Superman on TV when they simulated flying by having the actor stand in front of a fake cloth sky that was hung sideways and had a fan blow down on him from above to flutter his cape. It was a lock off shot, and you could see the backdrop rustling along with his cape. But I liked the flying stuff in Superman with Christopher Reeve, another great vfx film co-written by my mentor, Lorenzo Semple Jr.

As for the flying sequences, Courbould says: If they can do it physically, its the best way, but if we cant, the vfx team can help us out. Its a blend of all these elements and what we really want, is for the audience to sayhow did they do that? So the sfx crew has built the most advanced greenscreen flying set ever conceived in a huge Quonset-like studio, hijacking ideas from decades of film flying experience plus a few new ones, as well, including some impressive flight control rigs.

Everybody seems to agree that this is going to be an extraordinary blend of practical and digital vfx. That remains to be seen next year, but there has been a very high level of synergy among the various teams.

Meanwhile, theres a lot more promising vfx films coming this year, including Charlie and the Chocolate Factory, Harry Potter and the Goblet of Fire and The Chronicles of Narnia: The Lion, the Witch & the Wardrobe. So perhaps Ill do another installment down the road.

Peter Plantec is a best-selling author, animator and virtual human designer. He wrote The Caligari trueSpace2 Bible, the first 3D animation book specifically written for artists. He lives in the high country near Aspen, Colorado. Peters latest book, Virtual Humans, is a five star selection at Amazon after many reviews. You can visit his personal website.