How Do We Judge the Best?

It’s the time of year when the animation and visual effects communities, along with the entertainment community as a whole, take stock of their work, celebrate outstanding achievement and bestow awards on a select group of films, studios and creative artists. 

When you have awards, you have voting.  When you have voting, you have subjective judgment.  When you have subjective judgment, hopefully you have responsible voters employing responsible criteria. Then, at the conclusion of the designated voting process, undertaken by a designated group of voters, winners are declared and by default, losers as well, however biased, mysterious or seemingly inane the final choices or process may seem.

That leads to a basic but complex question: What makes certain films “better” than others?  The web has more “best” film lists than Optimus Prime has polygons.  Or cans of WD-40. Everyone has a right to their opinion, so says the popular maxim.  But spend any time in the blog and Twitter-sphere and you’re more likely to conclude, “Everyone has a right to their own uninformed, ill-conceived and ridiculous opinion.”  But I digress.

From the Annie Awards, to the VES Awards and ultimately, to the Academy Awards®, groups of professionals are tasked with voting for the “best” in a host of creative and production categories.  And while there is no shortage of snarky arguments tearing apart every aspect of the awards process, these shows continue to thrive, driven at one level by the welcome and appropriate desire for each community to celebrate its own work and at another level by our society’s thirst to crown “winners” and demonize “losers” (or more often than not, the opposite).

The point of this article is not to question the very notion of making judgments about the relative merits of wholly subjective artistic efforts, or the value of big awards shows from an entertainment standpoint.  The point of the article is to get a sense of the perceived value of such awards within our industry and more importantly, gain insight into how creative professionals, who themselves make the films being considered, tackle the issue of determining what makes a film “good” and why they consider one film “better” than any other.

Winning is a Good Thing.  A Very Good Thing.

Much is made about the “politics” of awards voting, how ultimately, creative awards are nothing more than popularity contests. The absence of consensus on whom and what should be nominated, let along win, further illustrates the inherent problems that will always plague such awards. Advertising and promotional blitzes during awards season dominate every conceivable medium.  Despite the inherent flaws in every judging process, the importance of winning the main entertainment awards on artist careers and studio coffers cannot be denied.

Consider, for example, that when you walk into the main building at Pixar Animation Studios in Emeryville, a glass case in the atrium prominently displays not only the studio’s Oscars®, but their Annies as well.  The same goes for DreamWorks Animation in Glendale.  The value of a major award win cannot be overstated.  According to Chris Edwards, CEO and Creative Director of The Third Floor, the Academy Awards and VES Awards are extremely important.  “Winning an Oscar helps everyone associated with the production, from a lucky production assistant all the way up to the studio that financed its release.  Professionally, there is no doubt that the nod can be a game changer for VFX supervisors.  They are forever seen as one of the key architects of the production’s success.  VES Awards are a very big deal for the visual effects community.  The collective work of visual effects artists around the world has a huge impact on the global box office, and there is no better place to focus on this accomplishment than at the annual VES Awards.”  That sentiment was echoed by Jenny Fulle, Founder and VFX Producer, The Creative-Cartel. “The Oscar is the holy grail of awards for the visual effects community, but the Visual Effects Society has done a great job over the years with its annual VES Awards…a VES award is definitely a badge of honor among our peers.”

As Henrik Fett, Co-Founder and VFX Supervisor at Look Effects pointed out, it’s important for the industry to recognize excellence and great work.  “If it were not for institutions like the Academy, our industry could easily fall victim to being viewed as a commodity business that we all know it is not. It is wonderful to focus - at least once a year - on the artistry and creativity that is needed to create stunning visuals like the ones that are nominated this year.” Fett went on to say, “It is an incredible achievement to be amongst the nominees, let alone to win this prestigious award. Because it is so highly recognized, it is of great benefit to the recipients and definitely increases their ability to be invited to be part of interesting future projects. This seems to hold true for studios as well as for individuals.”

This year’s Annie Awards were bigger in every way than ever before.  One factor that has positively impacted the growing perception of the Annie’s significance is change in the very mechanism by which the awards are judged. The hope is that the voting better reflects excellence in the work rather than just popularity. According to ASIFA-Hollywood president Frank Gladstone, “[In the past] There has been criticism that the voting could be gamed. That’s a criticism you find in almost all the awards ceremonies. Somehow, there’s a way to game it.  You spend more money on ads, this and that, you influence the voting.  In our case, the criticism was that a studio could manage to control the election. Historically that actually wasn’t true. But, be that as it may, the perception was that it could be done and in fact, it probably could have been if anybody really wanted to do it.  So we had to come up with a way to change that perception.”

What the Annies did was two-fold.  First, taking a lesson from some of the other creative professional academies, they changed voting eligibility so that only professionals who contributed materially to the work could vote.  Second, they implemented an Electoral College-style system where they have a group of noted individuals, with extensive credits, that looks at the top contenders and finalizes the vote. As Gladstone noted, “What that does is not allow any studio to control the vote. This random selection of judges represents all the studios and is weighted absolutely fairly through the industry. There are individuals, studio people and representatives of each of the nomination committees, who all determine the actual winners.”

What I Look For in a Great Film

So, what makes a film great? What makes it worthy of award? What do industry professionals look for when they judge a film? 

David Silverman, who directed The Simpsons Movie, scores of Simpsons TV episodes as well as this year’s Best Animated Short nominee Maggie Simpson in “The Longest Daycare,” described what he looks for in a great film.  “A lot of things come to mind.  Being surprised, I guess, is a big one.  I want to have a feeling about the characters. I want to feel. I want a story that not only has great characters but ones that I care about. In a non-narrative film they’re going to do something visual that I hadn’t expected.  That would mean some sort of surprising thing.  I think good non-narrative films have a sense of story to them.  There is a rhythm to them that’s sort of the story.  The best non-narrative films are done by people who are still storytellers.  They don’t tell [stories] in the traditional sense but they tell in terms of art and rhythm.”

Travis Knight, Laika CEO, Producer and Lead Animator on the Oscar-nominated animated feature ParaNorman, summed up how he judges a film.  “There are a number of aspects to look at. There’s the craft aspect. Lens choices, camera choices, acting choices.  Is the world unified or is it just strapped together? There are a lot of elements that come into play when you’re evaluating the ‘quality’ of something like that. Ultimately though, a film is a story. Just like any storytelling experience, it needs to have an emotional component. If you’re not feeling anything, then it’s not working. If you walked out of a film feeling like you consumed empty calories, like it’s a little pop culture concession, then it didn’t work. Any story is supposed to connect with you emotionally. Art is supposed to allow us to look beyond our own experiences. It’s for those aspects of our shared humanity that we haven’t thought of or that we get in touch with.  If a film can do that, it’s a powerful piece of filmmaking. It deserves recognition.”

Travis went on about how rare and ultimately powerful a great film can be.  “It’s incredibly difficult to do any film, even a bad film. It requires a lot of work and a lot of people pour a lot of their hearts into it. But to craft something that can connect with an audience emotionally is extraordinarily difficult. If you can do that in a beautiful and evocative way visually, it’s very rare. You don’t see it very often. To have an experience like that in the theatre, those are things that you live for.  That’s the reason I’m in this business. When I saw movies like that as a kid, it just changed the way I looked at the world. Those are things that still live with me to this day.”

Mike Gabriel, Art Director on Wreck-It Ralph, also talked about the emotional aspects of a film. “A great film shows us old truths in new ways.  A great animated film requires we show that truth through powerful visual invention. The more you buy in emotionally that the cartoon characters on screen are real and in real peril, the greater the animated film.”                     

With regards to judging a film’s visual effects, Weta Digital’s Oscar-winning VFX Supervisor Joe Letteri talked about how a great film pulls you in.  As he explained, “Are you seeing something that can't possibly be real, but your brain is telling you that it is? Do you want to keep seeing more of it? I think this applies in any category, not just Visual Effects, and a great example of that is Lincoln. From the first frames, the illusion is so convincing that I felt that I was watching Abraham Lincoln and not Daniel Day-Lewis. If you break it down, you realize that it is not just the performance that stays with you.  All of the elements surrounding that - costume, makeup, cinematography, production design, directing - have to come together in those moments to draw you into the film.”

Fellow Weta artist, Animation Supervisor David Clayton, echoed Joe’s sentiments.  “Does the work go beyond good execution and creativity, and make you engage with the story playing out on the screen? Do you forget that you're watching visual effects?"

Aardman Animation’s Oscar-winning Co-Founder, Producer and Director Peter Lord described how honesty is of paramount importance to him in how he judges a film. “I want to be moved by a film. The thing I’m always looking out for in that respect is very personal of course.  The word I always think about and often use in such conversations is ‘honesty.’  Do I believe that the filmmaker is being honest with me?  We all know it’s not easy, but is possible, to play on your audience’s emotions for effect. That’s what we do as storytellers.  We manipulate it in our choice of shots, our choice of music and everything that happens.  So I try to judge, ‘Is that real? Do I believe the filmmaker believes that? Or, are they just going through the motions?’  Of course, that judgment is completely subjective.  I don’t expect anyone else to share it on any given film.  But that’s a very important thing for me. What I hate is being manipulated”

On the subject of Oscar judging, Peter was quite adamant in his assessments. “An Oscar is a huge award.  It is of massive importance.  That accolade should be given with a great deal of wisdom.  I’ve judged other competitions in animation.  Short films in particular work in lots of different ways. When you see a program of animated films, sometimes a very short thing comes up and you laugh with delight. They’ve caught a moment, they’ve landed a joke, they’ve charmed the audience. Maybe you’ve just finished seeing something 20 minutes long, rambling and incoherent and now you see something which is tiny and perfect and probably funny.  You laugh with delight at seeing it. But I don’t think you should give an Oscar to a film like that.  I’m not referring to anything in competition this year. Delight is a great thing in filmmaking. There’s not enough of that in filmmaking. But surely something more substantial, like an idea I’ve never seen before, combined with technique which is in some sense a virtuoso, that’s a reasonable thing to look out for. The film you give it to doesn’t have to be 20 minutes long.  It doesn’t have to be big in that sense. It doesn’t have to be expensive. But it needs to be exceptional. I don’t mean pomposity.  I don’t mean heaviness. It must have substance.”

And the Winner Is…

How do you judge artistic and creative production achievement in general when so much of the work is so spectacular? 

As Chris Edwards explained, it’s his impression that Motion Picture Academy members employ a wide range of criteria when voting in the animation and visual effects categories.  “I’ve heard that the VFX Branch of the Academy tends to vote for the most well-rounded use of special effects and visual effects techniques.  Many of the follow-up questions at the Bakeoff often focus around the clever use of practical special effects and integration with model miniatures.”  He went on to say, “However, my sense is that the larger Academy membership will not be as swayed by technique, and therefore judge the film at face value.  Were the effects incredible, and were they integral to the story the director was trying to tell?  These are the questions I suspect many voters will be wrestling with.”

Chris also noted that he thinks voters are looking for great stories with great characters.  “For the best Animated Feature Film category, I believe the Academy membership will also heavily weigh story and characters.  These days, there are so many well-animated films that a certain level of artistic proficiency is a prerequisite for making the short list.  Beyond that, it will come down to how well the animators were able to speak through their characters and connect with the audience.”

Jay Redd, a VFX Supervisor at Sony Pictures Imageworks, spoke about how over the years, his focus has shifted when he watches and judges films. “When I was new on the scene in the late 80s and early 90s, I was obsessed with learning how everything was done.  I’d get into a film and go into analyst mode. ‘How do they do that, how do they do that, how do they do that?’ instead of getting involved in the story. So now, I’ve made a number of films since then, and have worked from the ground up, from being an artist to a supervisor. Now that I know how it was or could have been done, I’m not worrying about it when I’m seeing a film. If something sticks out to me I might say, ‘Wow, that just pulled me out of the story.’  But really, it’s about letting go of the process and being immersed in the story. I care more about being immersed in the world created by the team that made that film. I just want to escape.  When certain visuals are presented in a certain way, that’s what the visual poetry of the storytelling is for me. I’m looking to forget about how it’s done.”

When it comes to judging the work for awards, however, care must be taken when breaking down and assessing the work itself, regardless of whether or not the film, the story or characters are deemed “good.” Jay described his process of evaluation and what criteria he uses for his assessments.  “The work today is so stellar across the board.  As visual effects artists, we have big challenges.  At the judging stage, getting caught up in the details of how something was made is exactly what we need to do. For example, voting for the Annie Awards, or the VES Awards, when I judge at a film festival, it’s the same kind of thing.”

According to Jay, he employs a three-part part process to judge films.  “The criteria for me, which I hope is fair, are kind of a three part thing. It’s a little triangle I keep in my head.  First of all, is there an emotional impact to the image that I’ve just been presented?  What is the emotional content? Does it elicit that ‘Wow!’ response?  That’s a big point for me. ‘Whoa, I’ve never seen that before.’ That leads into the creative aspect.  Some group of people has put their minds together to create this idea. So there’s the emotional impact, the creative impact and then the technical impact, which is absolutely key to our process.  Without the technology, without the inventions, without the engineering, without the science, none of this emotional impact could present itself to us. That’s my three pronged approach and I hope other people are looking at that as well.”

Inherent in the process of judging film is an obvious but important issue.  Not every film is good, by any criteria. But the work of an individual or group of artists on pieces of that film may be worthy of award. Conversely, we’ve all enjoyed films which, upon further review, may not contain any work or discernible creative element we deem worthy of an award. Ultimately, as professionals, the hope is that at award time, judging will be done with thought, care and fairness.  As Jay concluded, “Let’s say a film wasn’t really well reviewed, or it had pacing issues, or let’s say the movie ‘wasn’t great’ but the work inside it was amazing. We as visual effects artists have to look at the visual effects as it is and try not to analyze the whole film. An important aspect of this is that we’re talking about a visual effects award. Not a screenwriting award. Not an acting award.  Not an award for sound and editing. All those things contribute. But we’re talking about visual effects and that’s what we should be looking at. It’s a challenge.  You don’t want to be biased by thinking, ‘God, I really hate that certain actor.  I just couldn’t take the movie.’ We should be focusing on the task at hand.”

--

Dan Sarto is editor-in-chief and publisher of Animation World Network.