Sebastian Sylwan and Joe Letteri give Bill Desowitz the scoop on the new R&D unit at Weta Digital, and what they hope to accomplish at SIGGRAPH 2009.
Weta Digital recently formed a dedicated R&D group headed by former Autodesk senior manager Sebastian Sylwan. He will be going to SIGGRAPH 2009 in New Orleans in two weeks to recruit talent for the group. The aim is to bridge the gap between R&D and production. Sylwan and Joe Letteri, a Weta Digital partner and visual effects supervisor on Avatar, spoke to VFXWorld exclusively about the new R&D unit.
Bill Desowitz: So let's start off by discussing the new R& D group.
Sebastian Sylwan: At the moment, it is pretty much me and we are pretty much collaborating with all of the departments here. And I'm starting to talk to a few people to lead some of the activities. But at the moment, it's very early stages. We are identifying some projects that we are working with, very much the core of CG activities, but we are heavily starting the recruiting process right now. We just finished a little bit of set up internally.
BD: Is this the first formal R&D initiative at Weta?
Joe Letteri: Yes and no. We always think long-term as much as we can. It more comes down to the actual research rather than development. If you want to dig behind the scenes a little bit and actually looking at collecting some data to justify some position or figure out what's really going on. That's the bit we really haven't had. We haven't gone out and actually built any tools and gather some samples to figure out what we're doing here. We always do that as needed. But we decided that it might be better to just establish this capability because it would be easier… We're seeing something here that we really don't understand with this particular material or this type of fabric. We can set up a lab to test these kinds of things and keep the results and go back and study them. And collaborate with other people who are dealing with similar problems.
BD: So what are the initial areas that you're targeting?
SS: We have pretty much three areas that we will be focusing on that are core to all of the activities here: we're planning to do some research on rendering and appearance, capture, model and transfer, etc., and then high performance rendering; of course, we're planning on looking at simulation in all of its various facets and forms; and we're planning to also look at one of the areas of great strength for Weta, which is the onset virtual studio. So one of the things that's important is that Weta continues to have a very significant code group, which is dedicated to building day-by-day tools. And that will be a part of this larger group: we don't want to make the research a separate thing. A lot of the research that we will be doing in these three areas will be applied research. We want to be solving day to day problems that have a direct impact on the connection to production.
And the other part that is important is that we are trying to do a little bit of outreach. We have very good connections with universities [including UC San Diego and Columbia University] and researchers all over the place that were more generally treated on a case by case basis. We want to try and give that a more deliberate and problematic approach: keep relationships going so that their research objectives are tied to very specific production problems or very forward-looking things that we're examining.
BD: So what are your goals for recruiting at SIGGRAPH?
SS: Of course, we will have a number of people there. We will be mostly focusing our efforts on the recruiting side, mostly in the R&D area. The idea is to get more profiles that have possibly some experience with 3D tools but more on the researcher side, so people that are doing a PhD or a post doc in computer graphics or math labs but are approaching problems that are in the three areas of interest. We are going to be actively looking for people there. We are not going to be having a booth, but we will be scheduling meetings ahead of time. And we are setting up an email address for resumes to be sent ahead of time so that we can schedule interviews with specific people.
We will also be following the paper sessions, and there are some that we have eyed and we will be getting in touch with the researchers.
JL: We want to get as much collaboration going as possible. In the past, you've seen these interesting papers and people were trying to work out these techniques and you'd spend a lot of time banging your head against the wall trying to figure out what was going on. What we've been trying to do is invite some of these people to spend time with us and see, from our end, what it's like to actually translate some of this research into practice. And then just asking students working in these areas if they'd be interested in spending a few months with us to see the work from a practical level as well. It gives you some real world perspective.
We are trying to apply the approach we are taking as real world as possible and the good thing about that does allow for common platform for this collaboration.
SS: In the program, we have a specific section that is for internships and collaboration. And we think that if you can get a real sense of the data set or the technique to solve a particular problem not only in isolation but also by fitting it in a production pipeline, it is a win-win situation when you get back to the lab.
BD: What are some of the problems you are looking to solve more directly?
JL: Generally, the problems are the ones we've all been looking at. On the simulation side, it's still water, fluid, cloth. It's all those sorts of things that people have been cracking and been getting better and better at, but every time you solve one problem, you realize there's a whole class you haven't looked at yet: different types of materials that you might want to get into. The things that are hard continue to be hard, so we're trying to make them easier and get into these others things we haven't touched yet. Similarly, with the rendering side, speaking of big data sets, you go from putting one character in a plate to having to do the whole world. That has its own set of challenges so you're looking at the things that work on a small scale to now being able to scale that up with larger groups of characters and creatures. Big lighting environments; global illumination types of solutions. Things that will actually work for creating an integrated world.
The realtime side of it is interesting to us because it's always been kind of out there but nothing you could bring to bear fully in production… so we're wondering where we can take advantage of those engines and try to figure out a hybrid approach: how to take the most appropriate data and loop it through graphics hardware and bring it back into the pipeline. That has both a research and engineering side to make it work.
BD: What has been the impact of Avatar and now Tintin on where you're headed technologically?
JL: That's what's laid the foundation for all of this. It was really just a question of saying, if we're really going to be doing all of this and there's so much we're going to be learning, we need some way to manage it: there needs to be some group responsible for looking at all of this information, making sure that things we learn in one area apply to another area and just generally pull it all together and what we do learn continues to evolve.
But it's built on a pretty strong foundation: the last couple of years have been pretty good for us in terms of research and just going back to basics, trying to evaluate all the assumptions we've been dealing with the past 10 or 15 years and in a sense just starting all over again.
BD: You're really on the cutting of edge of a new paradigm of moviemaking with performance capture and stereoscopic 3-D.
JL: There are two things that really change: One is character, which I alluded to before. Rather than just having a few hero characters in a scene, now you've got to populate the whole world. And it has to be just as believable. It's not good enough to have characters in the background who aren't speaking and aren't articulating. It all has to look as good as the people in front. How do you take what you're learning about facial animation for your hero characters and apply that to background characters without having to do the steps of building each one to the level of hand-crafted detail. The fact that you have to [digitally] build all of these sets now [is another challenge]. Doing a whole sandy floor that people have to walk on, for example, is still really hard to do in computer graphics; getting all the character interaction with all those particles is a huge problem. But where you can still get this nice, organic, dirty kind of feeling easily in the real world is still hard to do. But it's background, so it can't be a big deal.
BD: And stereoscopic must be posing a lot of detail-oriented challenges because you can't really cheat anymore.
JL: That was a big part of the impetus. You're right: you can't cheat a lot anymore. Not that using elements in miniatures, for instance, is completely out of the question, they can still be made to work. But your last minute cheats no longer work. You really have to think hard about it to make it all integrate properly by the time you hit the final render button.
For those interested in a research position at Weta Digital, you can contact them at: http://www.wetafx.co.nz/jobs/.
Bill Desowitz is senior editor of AWN and VFXWorld.