Compositing 3D elements into live-action footage is a very important facet of post-production. Bill Fleming instructs us on how to get it just right.
Today, stop-motion effects in classic films like The Seventh Voyage of Sinbad have been replaced with 3D composites. Merging 3D and reality is one of the most challenging aspects of digital effects. I'm sure you've seen this art at its perfection in films such as Dragon Heart, Men in Black and Starship Troopers. In these films the compositing was exceptional. The creatures seemed to be one with the environment. Of course, these are the few films where the composites were a success. The sad fact is that only 1 out of 25 films will do the composites correctly. More often than not the composited characters are over-lit and with the wrong light color--making them stand out like a beacon. One painful example of this problem can be seen in the film Alien: Resurrection where the 3D aliens are self-luminous when crawling through the dark corridors of the spaceship. They are brightly lit, yet the environment around them is dark. As a result, it completely destroys the believability of the shot.
Mastering Composite Lighting
OK, so what's the secret to doing a perfect composite? Well, it's actually not terribly difficult as long as you know the main points of concern when compositing. Let's take a look at the most critical points to consider when compositing 3D effects with live-action. Light Color It would seem that a bright white light is most frequently used when doing composite work. The problem is that light is never white! In fact, it's anything but white. You see, the color of light varies depending on the heat of the light, which is rated in Kelvins. For example: sunlight during midday has a blue hue to it because it's a very hot light, about 16,000 Kelvins. On the other hand, a candle flame is cool with a Kelvin rating of 3,500, so it has a red hue to it. The closest thing we have to white light would be florescent light, which is slightly green in color. Now you're wondering why we don't see these colors in the world around us. Well, we actually do but our eyes compensate for the color shifting of light through a process known as chromatic adaptation. Basically, our eyes attempt to convert all light to white. They are usually very accurate, though there is a 5% margin of error, meaning we will see some evidence of the actual light color in the environment around us. To test this, just look at any object in the far distance, such as mountains, during a sunny day and you'll see a slight blue hue to the colors. Keeping this in mind, you'll need to determine the actual color of light in the live-action scene so you can properly light your characters so they become part of the scene. Of course, direct light color is only one part of the formula. There is another light to consider: indirect (reflected) light. Indirect Light Color In reality, light is reflected off the objects in the environment. This is known as radiosity. This light, as it is reflected off the surface of an object, will take on the color of the surface. Basically, when light is reflected off a surface it will assume the color traits of the surface and the light source combined. The strength in color will lean toward the surface colors. If you have a red wall in a room, the light reflecting off it will be slightly red-not much, but about 5% red. Therefore, if you are compositing a 3D character into that room, you'll need to add the reddish hue to the reflected light sources for the character to blend into the environment properly. Light color was one of the major faults in the composite work in Alien Resurrection. The aliens were in a gray metal passageway, yet the lights illuminating them were white. The actual color of the light should have been gray. Of course, there was another problem with the light: the intensity.
Light Intensity Light intensity plays a major role in compositing 3D effects into live-action. As 3D artists we have the tendency to light all of our images directly. The problem is that in reality there are very few direct light sources. For example: outside there is only one direct light source, the sun. All other light is typically reflected, which means it's diffused. Reflected light has far less saturation than direct light. In the case of the alien creature, the model should have been lit entirely with indirect light so it blended into the passageway. There were no direct lights in the passage. All of the lights were reflected off the metal walls or filtered through the protective covers of the lights. The direct lights heated up the character far too much to blend with the environment around it. Indirect light is easy to simulate through the use of point lights in the scene. A point light is omni-directional so it simulates the volume of indirect light very well. Simply place point lights around the character to simulate the indirect light. Be sure to apply some of the reflected object color to the lights to ensure the effect is accurate. That does it for the most critical points to consider when compositing 3D and live-action. Now, let's take a look at how the actual compositing process is handled. Compositing 3D Effects with Live-Action For the purpose of this discussion, we'll take a look at a single cel of an animation in production at Komodo Studio. The goal of the project is to composite a goblin into video of an actual forest. In addition to the character composite, there are several items added to the scene to turn the forest into a goblin forest. Let's take a look at the steps in creating 3D composites with live-action. Step 1: Creating the Foreground Models When doing composite work you are often required to place a character behind an existing object in the scene. This sounds complicated but it's actually very simple. The first thing you need to do is import a frame from the live-action into your modeling program as shown in Figure 1.1
Figure 1.1 Importing the live-action shot into the 3D modeler. © Bill Fleming. Figure 1.2 Creating the foreground objects. © Bill Fleming. Figure 1.3 Creating the ground plane. © Bill Fleming.
Then you create the objects in the scene that will be in the foreground. In the case of this animation it's the two trees in the middle of the image. Using the image as a template, build a basic mesh that takes on the form of the trees. When completed, you'd have something like the image in Figure 1.2. Of course, for our character to blend into the scene we need a ground plane so it appears as if he's standing on the leaves. Since the leaves are rather chaotic, we need a chaotic ground plane or the surface will be too flat underneath the goblin's feet. To do this, I created an uncomplicated plane with about 100 polygons. Then I jittered the polygons and subdivided them to make a bumpy surface. This process was repeated several times to produce the jagged, yet organic, ground plane shown in Figure 1.3. Now, when the goblin model is placed on the ground, it will actually sink into the leaves. You'll also notice the ground plane was rotated to match the perspective of the image. We could have kept the plane parallel but then we would have needed to rotate the camera in the render and that would cause the trees to be out of proportion with the image. All right, now we're ready to look at Step #2.
Step 2: Staging and Surfacing the Scene
The key to an effective composite is to match the surfacing of the source image. This is done through the use of a projection map. A projection map is basically nothing more than the source image projected onto the models from the front. The difference between a projection map and a normal image map is that a projection map covers the whole scene. Basically, the image is projected over the entire scene, so anything with the surface using the projection map will reflect the scene. This is a great tool for compositing. You need to load the source image into the background of the scene as a template. Then load the models and position them over the elements in the background image as shown in Figure 1.4.
Figure 1.4 Positioning the models for the composite image. © Bill Fleming. Figure 1.5 Proper rotation of the models for shadow casting. © Bill Fleming.
You can see that I've also added the goblin model and the mushrooms and rock, which are part of the scene's transformation into a goblin world. Figure 1.5 shows how the models are all rotated to be parallel to the ground plane so the shadows drop properly. Once the models are in place, you need to apply the projection map of the background image to the tree and ground models. Now the last step in the process is the lighting, which we now know to be the most crucial. Step 3: Lighting the Elements Like we discussed earlier, lighting is the most important aspect of compositing. In the case of this scene, we need to place a light above and behind the objects to simulate the sun in the source image. It will take a bit of testing but you want to position the light so the shadows cast by the new elements match the source image. The next step is to apply the indirect light to the scene so the new elements blend properly. Once again, this takes a bit of experimentation but it's well worth the results. What I did was place five lights in the scene. The first was a direct light to re-create the sunlight behind the models. The second was a direct light in the upper left to add volume light. The shadows were turned off on the light so there wouldn't be conflicting shadows in the scene. The third was a direct light in the upper right for the same purpose. An additional point light, with no shadows, was added near the middle of the trees to light them up. Finally, a light was added to the character to pull him off the background. You will almost always need character lights so they don't overshadow themselves--particularly when they are back-lit. The results of the compositing effort are shown in Figure 1.6.
Figure 1.6 The final composite. © Bill Fleming. The original source image before the composite effort. © Bill Fleming.
As you can see, both the goblin and the new elements are convincingly blended with the live-action shot. We can now see that our little goblin friend Grumpy is tracking a snail who is destined to become soup for dinner. Pulling It Together While compositing 3D effects with live-action can be a bit of work, it's really not terribly difficult as long as you pay close attention to the lighting in the shot and take the time to add the elements that make the composite really convincing. The ground plane that simulated the leaves was very effective at drawing the 3D objects into the scene. Nothing is more artificial than a straight seam between objects. The jagged ground plane literally wrapped the leaves around the composited objects. Grumpy's feet are nearly buried in them. Put a little attention to detail into your composites and you'll soon be showing those Hollywood "big guys" how it is done! Bill Fleming is president of Komodo Studio, a 3D studio specializing in photorealism. He is the author of many 3D books, including the 3D Photorealism Toolkit, published by John Wiley & Sons. He also serves as editor in chief of Serious 3D magazine, a 3D magazine featuring nothing but intermediate/advanced tutorials for artists interested in taking their 3D graphics to the next level.