Search form

ILM's Seth Rosenthal on Motion Capture

Seth Rosenthal, the motion capture department supervisorat Industrial Light & Magic, took time out of his "menacing"schedule to discuss a gamut of topics relating to motion capture with HeatherKenyon.

Industrial Light & Magic began their motion capture department almost three years ago as a research effort. Now with the hiring of department supervisor Seth Rosenthal eight months ago, the team has come of age. Jeff Light, a veteran of ILM's computer graphics department, first began the motion capture department and spent years investigating technology, developing tools, methods, techniques and finally hiring a staff. Seth's background at Microsoft working with motion capture for games and with the Graphics Group of Microsoft Research working on human figure animation technology, combined with his experience on a project integrating motion capture capabilities with Softimage software, made him the perfect candidate to take the reins, while Jeff has moved over to the Star Wars production as the Motion Capture Supervisor. While we won't be able to see the department's handiwork until the most anticipated film in history, Star Wars Episode I: The Phantom Menace, is released, ILM plans for their motion capture department to become another cutting-edge, state-of-the-art tool in their mighty arsenal of special effects gadgets. Seth took time out of his busy schedule to run through a gamut of topics relating to motion capture.

Heather Kenyon: Let's start simple. What is the difference between magnetic and optical motion capture technology? How is the digital data recorded in both methods? Seth Rosenthal: Optical motion capture technology is a subset of computer vision. It uses multiple cameras to identify features by the use of markers or other image processing techniques and generates three dimensional information about those features by triangulating between the different cameras. Magnetic systems use a magnetic field and small receivers that can detect their position relative to the magnetic transmitter. Magnetic receivers can also, and this is extremely useful, detect their orientation in space. HK: Were the methods developed simultaneously? SR: No, they weren't. To the best of my knowledge, optical motion capture was developed primarily for biomedical applications and research, like sports related injuries and for the analysis of athletic performances. It was used, for instance, to study runners' gaits and later it became extremely popular for tracking golf swings! HK: Really! What year was this technology first developed? SR: Well, that is difficult to say. What technically constitutes optical motion capture? Animators still look back at the Eadweard Muybridge photographs. In a way those are optical motion capture because you can go back and reconstruct motion by tracking joint locations. Contemporary motion capture adds two things to that. It automates a large part of the process, and second of all, it adds the third dimension by triangulating from multiple cameras. Fundamentally though it is the same idea. More specifically, however, by the early `80s optical methods were being used in research. I don't have first hand knowledge of this, but one of the leading providers of magnetic technology developed the sensors for use in military applications, particularly for tracking the head movements of pilots. HK: It is very interesting the amount of technology that is used by the entertainment industry or for entertainment, like the Internet, that was not developed specifically with entertainment in mind. SR: Until fairly recently the majority of optical equipment was designed for the bio-mechanics research community. Only very recently have vendors begun redesigning their systems with entertainment applications as a priority.

A shot of ILM's motion capture studio. Photo © Industrial Light & Magic.

HK: What is magnetic motion capture most used for in the entertainment industries? SR: It is used for a whole range of motion capture. For games, research, but its real strength is that it is capable of doing real-time tracking. On productions where you see a live digital character for broadcast, they are probably using a magnetic system of some sort, often layering on top of that animation from other input devices. Any kind of live or puppeteering application, until recently, has been solely the province of magnetic technology. Its second advantage is that it can measure orientation for each sensor. There are some drawbacks having to do with data quality, but there is a definite ease of use that comes from the ability to get both the position and rotation data. That is not the case with optical. Optical has a number of other advantages, primarily accuracy and the ability to capture multiple actors in larger volumes. HK: What do you mean by volumes? SR: Oh, we are just talking about the area in which we are capturing, whether it is ten feet on a side or thirty feet on a side. By the way, optical vs. magnetic is not necessarily the most useful way to categorize motion capture technology. There are a number of other techniques available including body suits and electro- or opto-mechanical exoskeletons. Another way to think about the tools is active systems vs. passive systems. Magnetic systems are active, they require hardware on each performer. Most optical systems are passive in that they use only reflective markers on the performers. However some optical systems have synchronised lights on the perfomer instead of markers; these are active systems. When you look at it this way, you begin to consider issues like reliability, complexity and cost per performer. In a production environment equipment really gets used hard and equipment failure can be very expensive; the relative simplicity of passive systems is an advantage here. HK: What is optical most used for in entertainment? SR: It is also used in the games industry, particularly for sports games or fighting games. As feature film productions have turned slowly toward using motion capture, they have pretty much selected optical for quality reasons. HK: So magnetic seems easier to use and has real-time capabilities, while optical is more exact. SR: That is fair to say. I first started in motion capture working with a magnetic system. It was much less expensive, and very accessible and powerful. I have moved on to using whatever technology is available. Right now, we are using an optical system made by Vicon, the Vicon 8, and it is extremely powerful. In the end, although there are some complexities to using it well, it is not really a whole order of magnitude more difficult than using a magnetic system. It is all fairly accessible technology. HK: So, you wouldn't say this is difficult technology to learn? SR: No. Anyone who is inclined to learn the technology probably wouldn't have a problem. HK: Is your path typical? Do most people start with magnetic and then move into optical? SR: You know, I don't think so. First of all, I don't know "most people." It is not a very established field. Everyone who is working in it has come to it from different backgrounds. Yet, in the entertainment applications, people who work with magnetic systems tend to be focused on the real-time aspect of it and often have a puppeteering bent. They are working out ways to layer various sources of performance data onto a single real-time creature. So magnetic is not really a phase on the way to optical. They overlap but they also have unique applications. HK: We have a lot of student readers who write in and ask how they can learn this technology. Unfortunately right now, I don't have an answer for them. How do people normally enter this field? It doesn't seem like there is a University of Motion Capture yet. SR: I doubt that traditional animation schools will soon be adding motion capture departments either. If someone is extremely interested in this technology their best bet would be to look at universities or colleges that have computer graphics departments that are doing animation, or biomedical research. There may be no specific programs or courses pertaining to motion capture, but if they talk to the researchers and professors, they may find that there would be ways for them to gain access to the technology. There are an awful lot of schools that have a magnetic capture system in the closet, or an optical capture system in the athletic department, that was used a few years ago and now no one knows what to do with it.

HK: Are there any trade magazines, web sites or other resources that professionals in the biz turn to? SR: None that I know of that specifically pertain to motion capture. You can poke around the web for an article here or there. Some periodicals that discuss technology and special effects sometimes cover motion capture. HK: Right now motion capture involves a small number of professionals and a small number of vendors. However, with some of the breakthrough effects we saw in Titanic for example, it seems like we are going to be seeing motion capture use expanded. SR: I obviously hope that is the case and I am doing everything I can to ensure that, but I would say in the short term by no means is it guaranteed. There have historically been a couple of problems with motion capture at the level of feature work. The biggest one is that the community of vendors have exaggerated the capabilities of the technology and are given to making claims that are almost guaranteed to offend anyone working in animation; usually regarding how much cheaper it is, how much superior it is to traditional animation, etc. That has created an obstacle to get around. Motion capture is irreplaceable for some applications and inappropriate for others. When used well, it can be incredibly cost effective, but, in practice, some of those savings are displaced to other parts of the production. Second, the tools for acquiring data are good at this point. One can with a relatively modest investment get equipment and expertise that will allow one to capture high quality motion capture data. But incorporating that data successfully into sophisticated animation pipelines is still extremely difficult. The software tools are not adequate at this point, so it requires a high level of expertise to use the data effectively. HK: Some projects though are completed by using traditional computer animation techniques on top of the motion capture data. Doesn't this seem to be the "best of both worlds" solution? SR: That is a viable solution, but with the proper direction and the understanding of how you are going to apply that motion in a scene, it is possible to bring in an actor and get a complete performance. Even for a performance with complex interaction with characters say, in a background plate, and difficult timing issues, it is possible to capture data and, this is of course without secondary hands and face animation, put it directly into a shot. There are a variety of techniques and tools available and none of them replace the other. They are all just more colors on your palette. We are working with the entire spectrum. We have motions that we use directly. We have motions that are heavily modified or enhanced by animators to great effect. We also have motions that we use purely as reference. They are all great ways to use motion capture. However, to incorporate all of these techniques into the same pipeline is challenging. This might be one of the things that holds people back from using two different techniques on one piece of animation. ILM has a remarkable R&D group and very deep expertise in technical animation techniques; so animators here have access to good tools and techniques. The strength of the work that we have done to date -- you can't see it yet! -- is largely a result of having some extremely bright and talented people attack the problem of applying and using data effectively. HK: You need a very strong support staff then to make sure you are optimizing all of your tools. SR: Motion capture can provide animation for a character. But animation is just a starting point and doesn't buy you much if you don't have good modeling, lighting, compositing, texturing, and everything else that makes a shot work. That's one reason there is so much cheesy-looking motion capture out there. It seems so easy! You get motion that looks very real, but if you don't have a way to incorporate it into a pipeline to make good CG images, then you are not accomplishing anything. No matter how good the motion capture is, if you don't have a way to incorporate it into images that look good, then all you really have is great movement. HK: And if you don't have a great story then you can have great CG images and nothing else. SR: That's right. We are a tiny part of the big picture at ILM. We are just one little tool that ILM can use. Moviemaking is a collaborative effort. HK: Absolutely! That and government; both of which are just as entertaining these days. I can imagine that you are selecting different tools for each scenario in order to get the best effect. Sometimes optical. Sometimes magnetic... SR: That is true in general, although it is a bit simpler than that. We have been using magnetic for animatic work because it is fast and inexpensive, and we have been using optical for feature work because it is accurate.

HK: What are the next breakthrough innovations we can expect to see in motion capture technology? SR: Well, that is an interesting question, and what I have here in response is our classified R&D document from ILM Research. I thought I'd just read to you what we are working on...(jokingly) HK: Oh, okay. This is my lucky day. SR: Actually the common question which you started with is, `What is the difference between optical and magnetic?' It makes a lot of sense because those are the technologies that are available. From a production point of view, however, the real questions are: `What do we want to do with motion capture?' and, `What kind of technology is implied by our existing production techniques?' Therefore, further questions are: `Can we take it on a live set?' `Can we use it without interfering with live-action photography?' `Is it flexible enough to be immune to the heat, noise and light that are present on a set?' `Can we take it underwater?' We are looking at the technology under the lens of, `What kinds of things do you do in feature film production?' Now, are our existing technologies going to get to this point? Are they going to become fully integrated with time code support and various other synchronization solutions for audio? It isn't clear what the step is between existing technology to this kind of ideal set of tools that we would like to have. In a general sense, optical motion is, as I mentioned earlier, a subset of computer vision techniques. As a field, computer vision has developed a lot of good techniques for doing very circumscribed things. I have a friend for instance, who works with vision systems to identify fat on a piece of chicken going down a conveyor belt, and then to aim a water jet to trim the fat off. Or to sort burned tortillas out of the mix. These are very controlled vision tasks and can be done well. More general things, like deciphering a random image, determining what is moving in it, where it is and why, is much more difficult. This kind of research though is going on, and advances in computer vision will eventually lead to better tools for taking images and extracting motion from them. We will then have tools that will be much less susceptible to needing specific lighting environments, a certain number of cameras, etc. HK: How can we expect to see these innovations impact the entertainment we see? SR: In terms of movies, if we do our job well, you'll never know. It will open up, aside from movies, new art forms. I am particularly interested in seeing how live performance of various kinds, dance, multimedia performance, makes use of the ability to abstract one performer's motion onto other characters or computer generated characters. Something we are going to see a lot of unfortunately, are bad Saturday morning cartoons. This gets back to one of the problems motion capture has to overcome. When it is used naively or primarily for simple economic motives, you end up with characters whose motion does not match their design. You get cartoony looking characters that have this hyper-real motion that is disconcerting. On the whole though, it is another tool that will add to our ability to create compelling media, to put more, and more varied, characters in films, television or games. HK: Computer animation companies, besides Pixar, are beginning to produce shorts that are more artistic and story based, and they are touring the international festival circuit. Do you think this will happen with motion capture? I think it would be very helpful in promoting performance animation's place in the animation world -- so it would be seen not just as a technique, but as a viable form of art. SR: I think you will definitely see artistic shorts done. They are going to come from artists who have a more general interest in performance rather than traditional animation. I also think the technology will become more accessible, but it is going to be an issue of the right artist finding the right technology before we see really compelling short work. Obviously, this is something I would love to see. HK: Another one of our articles in this issue mentions two stop motion directors, <the bolexbrothers>, who are exploring in their stop motion work `working with human actors without conventional live-action methods.' In a way, you too are working with live actors but your final product does not technically include live actors. SR: Yes, but one of the limitations is that the audience is seeing real motion, and if your story or images don't make sense with hyper-realistic motion, then there is going to be a clash. This draws a circle around the whole application of motion capture. As soon as you don't want this realistic motion or the high resolution that motion capture provides, then you very quickly get into territory where you should have skilled animators and artists creating the motion. In the big picture, of all possible things to animate, those suitable for motion capture are a small group. There is a huge advantage to using the tools appropriately, but there is a penalty for using them for unsuitable applications. HK: Short films are a very exciting thought though because people who come to animation through motion capture come from a completely different background. SR: I definitely empathize with animators who are frustrated with exaggerated claims or expectations that are placed on motion capture. I can see why some would object to it being called animation, but that does not mean that it is not an art. It is just a very different type of art. It is much more about performing, directing talent and being able to benefit from the spontenaity of live perfomance. Performance arts always have the potential to create incredible momements that are completely unplanned. That is part of what makes movies and theatre magical. Motion capture has that dimension to it and I think that will lead to some great work. Heather Kenyon is Editor-in-Chief of Animation World Magazine.