Transcript of my VR storytelling principles presentation just given at the 7th Int’l Conference & Exhibition on Visual Entertainment in Beijing.
The following is a transcript of a presentation on VR storytelling principles that I gave on December 1st, 2016 at the 7th International Conference & Exhibition on Visual Entertainment in Beijing. I like to speak extemporaneously, so please bear with any verbal clumsiness. :-)
I'm here to talk about the development portion of the entertainment workflow, specifically related to virtual reality. My own background focused on production during the first half of my career, the 12 years I spent with Walt Disney Feature Animation. After moving to China in 2008, I shifted my focus to development. This development work began in traditional areas of film and television - which I have taught here at the Beijing Film Academy - and shifted to virtual reality over the past year.
Virtual reality requires a different way of thinking. I believe you've heard this already. There have been great comments made today on this point, not restricted to virtual reality, but related to any new means of storytelling. When Demetri Portelli talked about shooting at 120 frames per second in 4k, he said something obvious, but also easily overlooked: the director needs to think differently about how to direct; the actors need to think differently about how to act; everybody involved in the production chain needs to review their assumptions, adapt and expand upon what's possible in the new media environment. This applies to VR as well. It's easy to bring your preconceptions and old ways of working into play. In this respect - and I'm not the first person to make this observation - the current state of virtual reality is very much like the early days of silent film.
This is the Lumiere Brothers' ARRIVAL OF A TRAIN. It was initially shown at carnivals. You'd go to circus sideshows and watch this train rushing at you. Audiences would flee screaming from the tent because they thought they were going to be hit by the train. The experience was so real and frightening that the spectacle overwhelmed them. I think VR likewise produces this type of spectacle. We talk about how cool it is to be underwater surrounded by sharks, or in outer space blasting aliens. These experiences are valid entertainment, but they still fall within the realm of spectacle. A mature language of virtual reality has yet to evolve. We still speak of VR in terms of what came before - games, film and television - while trying to figure out what VR can be.
It's analogous to the early development of the car. Most of you either have cars or have been in a car. There was a time when the car was referred to as the "horseless carriage." When cars were first invented, they were defined by what they were not: the carriage now has an engine and no longer needs a horse, so it's a "horseless carriage." It took decades for the horseless carriage to become the car. Think about the differences between a Model T and a Tesla. Aside from the fact that both have a "T" in their name, they're at opposite ends of the spectrum. The Tesla is almost like a space ship versus the Model T, which was in many respects still a horseless carriage. VR likewise has not yet come into its own.
This is a scene from one of Georges Melies' early film productions. Melies was an innovative filmmaker who was working at a time when folks were just starting to figure out what film could be. You see, however - and this is not a knock against Melies - that at the same time during those early days he was exploring film and introducing visual effects, he still was constrained to a theatrical space very much like this stage. We see the action through a planar frame of view; we're not moving around with the characters. This is understandable in context, and it's reflective of what we're seeing now with virtual reality. In much the same way that we had the look and feel of live theater influencing film, we now have the look and feel of film, the look and feel of television, the look and feel of games influencing VR.
Here's another example: a brilliant filmmaker and performer, Charlie Chaplin. Note the exaggeration. When you're performing onstage, you need to over-exaggerate your facial expressions to project your emotions to the people sitting in the back of the room. You see Charlie doing that here, perhaps for comic effect. This is during one of the more emotional scenes of City Lights, when he's expressing his feelings to the girl. To our eyes, there's a level of over-exaggeration. He has not yet adapted his performance to the new realities of the medium. As with Melies, this is simply an observation, not a knock against Chaplin.
My previous experience in virtual reality is relatively limited. Most people now working in VR are coming from other fields, and I'm no exception. I had some virtual reality experience during graduate school in the early 90's, with VR caves and the like. But as Chris Edwards observed earlier, the attempt in the 1990's to create an industry and a market for VR didn't pan out - for a number of reasons that I won't address here, but we can talk about on the panel. I followed the boom in animated feature films. The exciting thing is that as we're coming to VR from these other fields and mediums, it's incumbent upon us to adapt and evolve. Indeed, it's our responsibility.
As storytellers, we must consider the point of view of what we're trying to say and through whose eyes we're trying to say it. This is something that any storyteller ideally does - in a book, in a play, in a film. Virtual reality simply expands the options. It's important to remember that things which are now considered to be natural in film - the storytelling conventions, editing, montage, etc. - are essentially unnatural. There's no such thing as a natural experience of cinema. We have learned to accept the conventions of film over time. And so it will be with VR.
The great thing about VR is it allows us to look at things in different ways, to flip the space. We can use VR to augment the experience of what we're trying to say in standard film. We can also mold the space. We can scale down to a micro level or up to a macro level. There are many ways to think about how we are going to move through the space, what we are trying to convey as we manipulate the space, and ultimately what experience we want the viewer to have. I shouldn't even say the "viewer" - I should say the "experiencer," who is immersed in the VR scenario.
This is a piece called Birdly. This is not a story. It's what you would describe as a VR experience. I've tried this myself - it's terrifying and exhilarating at the same time. You're basically a bird, flying high over a simulated city. You're on a gimbal and you have control over your movements: you can flap your arms to fly through space; you can tilt your hands to dive and to pull out of the dive. The thing I love about Birdly is that there's a fan blowing wind in your face. This is a low tech, effective way to enhance the illusion of being a bird. It's also a great example of how sometimes a simple cheat is just what we need to complete the illusion. There's also a panic button. You can't see it in this photo, but there's a little red button that you can press to cut the simulation, because it's so visceral that it gets your heart going. I saw one person nearly faint because it was too intense for them.
This is a fascinating piece called The Doghouse. It's a Danish short film, twenty minutes long and shot from the point of view of five different people in a family having dinner. They're having a rather awkward family dinner, as families often do. There's a father, a mother, a young son, a teenaged daughter and the daughter's boyfriend. Each viewer can watch the film through a different person's point of view. You can watch it as the father, as the daughter, as the mother, etc. I watched it first as the father and then as the teenaged daughter. The fascinating thing is not only do you get to experience the same twenty minutes through different points of view, but your feelings for these characters change depending on whose eyes you're seeing through.
The first time I experienced this piece, I was in New York at the Future of Storytelling Summit. Our group watched the film and were then asked a series of questions. The first question was, "Who is the main character in the film?" We all raised our hands. The second question was, "Who is the most sympathetic character in the film?" We all raised our hands again. We each felt that it was us. This was a powerful reminder of how VR can be what Chris Milk calls an "empathy machine." Traditional film does this, but VR takes it to the next level. It allows us to identify with and relate to experiences through the eyes of another person. This can be used in fiction, as it is here, but it can also be applied to documentary situations. It's used by charitable organizations to help people understand what it's like to be impaired or different or discriminated against in some way. How does that feel emotionally? In this way, VR is being used not only for storytelling, but also for therapy. And it's this aspect, the ability to see through the eyes of another, that is the most revolutionary, ground-breaking aspect of VR technology from a story standpoint.
Many people look at these things and say, "Ah, it's a gimmick." Flying around like a bird, that's a game. Shooting a short film through the eyes of four or five people, that's an experiment. It's not real storytelling. But everything that we currently accept as a convention of film or television storytelling was at one point an experiment. And the experiments which mature, the experiments which stay with us - not only in our heads but in our hearts - become the new conventions, the new normal.
OK, next point: provide immersive direction. That sounds easy enough, doesn't it? As a traditional director, you really are a sort of dictator, with the audience following along in a fixed direction much like these beer bottles.
As the viewer, you might not realize that you are in many ways compelled to see and feel what the director wants you to. If we cut to a close-up of this light, you must look at this light. Now we're back to me on the stage. Now we're out to a wide view of the room. You have no choice but to watch what the director insists upon. This isn't necessarily a bad thing, but that's how the form works. When Steven Spielberg expressed his misgivings about VR, he said that he thinks it's a problem if people can look around anywhere they want to. How do you keep them focused? Does the story become diffuse? Does it fall apart?
As a VR director, you're more like this diver. He's generally got the school of fish in front of him, but you see some are drifting away. You need to influence the audience in this situation. The director needs to become an "indirector." You can't insist that people look in one particular direction in VR, because they can wander any place they choose. You need to use sound cues to direct the eye before something comes into frame, use visual elements to focus attention, and program the VR environment to respond intelligently to the viewer.
Many virtual reality storytellers are not only incorporating architects for the navigation of space, but also psychologists to understand how the human mind works, how human perception works. We must take more care in directing a person's attention through what they hear, through what they see, through what they feel. This school of fish serves not only as a metaphor for the VR audience, but also as a metaphor for the VR content itself. The first thing many VR creators do is start throwing story points in front of you, behind you and to the side of you. You're in a VR environment, and you're looking here, and then you have to look back there, and now you're looking over here... it just gets exhausting.
It reminds me of the early days of 3D stereoscopic filmmaking, where stuff was constantly flying out of the screen at you. Of course, 3D goes back to the middle of the previous century, when the spectacle of stereoscopic filmmaking was important. It wasn't until recently that people settled down and started to think about the stereo space not only from the perspective of what was coming at you, but what you were looking into. For an emotional moment in a 3D stereo film, this interior space may be more appropriate to the story beat than the in-your-face 3D effects, which are more suitable for action scenes.
Related to VR, my point is that everything doesn't have to be happening all around you. It's enough to be immersed in a 360-degree space with the focus of what you're doing more or less in front of you. If I'm walking to the subway station, my attention is in front of me. I'm surrounded by sound and motion, but I'm not constantly spinning around to see what's coming at me from another direction. Beijing may be an exception to this, where indeed you're wise to look out in all directions. In VR, you can have a 180-degree shell of primary information in front of you, with another 180-degree shell of secondary information behind you. If you don't look behind you, you're not necessarily missing anything. But if you do look behind you, the environment is confirmed. Even through your peripheral vision you have the sense - much like in a widescreen theater - that you are immersed in a particular setting. And immersion is one of the core values of virtual reality.
VR storytelling must be an organic experience. I mentioned Spielberg's concerns about virtual reality. People often think if they can wander around as they please, they won't have a cohesive story experience - that the environment will fragment and the story will go nowhere.
I would suggest that we can look to gaming for solutions. If you're playing a narrative game, there are certain key points you must go through.
The black and white dots in this diagram represent key story points that we must absorb in order have the complete narrative experience. But at the same time, we know that people may wander over to look at a bug, or enjoy the light coming through the leaves on the trees. We want to allow people to organically explore the space while not missing key story points. We want to structure an experience in which any given person - the person in red, the person in green, the person in yellow - can navigate as they please while still being steered through these anchor points. Much of this development work can be done on paper, using Post-It notes and index cards to plan out the structure and logic of the story.
Sleep No More is a fantastic piece of theater breaking new ground in New York. I believe it just opened in Shanghai. This is an interactive theatrical experience in which a warehouse has been converted into an old hotel where various tableaus take place with different characters and different story cues. The audience, wearing masks, wanders through this space freely as part of the experience. You're not restricted in how you move through the building, how long you stay in one place, what you see and what you don't. The performances cycle, and if you experience it more than once you may catch something that you missed the previous time through.
When people are done, they generally have a common understanding of what happened, but with a different take on the details. Friends of mine who have been through Sleep No More have spent hours afterwards comparing notes, reconstructing the story and discussing and how they felt about it. And while it may sound like a contradiction, at the same time that I've been saying we should not be beholden to traditional forms of storytelling, there's a lot to be learned by adapting forms such as Sleep No More to the medium of VR in an intelligent, evolved way.
Proximity should be used appropriately. When we refer to VR as an "empathy machine," a great example is Clouds Over Sidra - a VR documentary short produced by the United Nations on the Syrian refugee crisis.
There's a wonderful scene in which you're surrounded by a group of kids: you feel like you're standing right there amid this circle of children. It's a powerful, touching moment, to have that sense of presence. People come out of the headset crying, which they wouldn't necessarily do if they were watching on a TV screen and looking at it as opposed to being in it.
But proximity cuts both ways. A horrible image such as this is disconcerting to have in your face, because virtual reality replicates the environmental logistics we're accustomed to. If I have a close-up shot, that close-up - which might be perfectly acceptable in a traditional film - suddenly comes across as somebody sitting in my lap.
This can be uncomfortable, but it can also be acceptable. To have a baby that close to us is something we find warm and cuddly. So, the application of proximity is crucial: how close something is to you, and why something is that close to you, are important considerations.
Here is an illustration of some common proximities we experience in daily life. The person on the left is you. Within a half meter or so you have people who are emotionally close to you: children, family members and loved ones. Within about a half meter to a meter is the comfort zone for your friends: your pals - people who are not necessarily related to you but who are an important part of your life and who you have a certain level of intimacy with. Between about a meter and four meters are the people you work with: colleagues, etc. These are people who you're friendly with, but are not close to you - people who will smile at you in the workplace, but won't come over and help move your furniture. Outside of the four-meter range are strangers - people who pass by us. They might be nice people, they might not - we don't know.
Of course, proximity scales up or down depending on what country you're in and what culture you're in. For example, here in China, proximity can be compressed quite a bit. I mentioned the subway earlier. If I'm on Line One, I may be mashed against someone I've never met before, and that's an experience many people in the West don't have. However, you still try maintain proper emotional distance by averting your gaze. You don't stare in that person's eyes, but you look to the side - even if you're pressed up against them. So, proximity is something that should be scaled according to the circumstances of the culture. We can't just say, "Here's the way to do it," and approach it the same way in the U.S., in India and in China. We must to consider the cultural norms of what's acceptable in the territory that we're making the content for. Some things are going to be universal and some things are not. Keeping family nearer to us and strangers further away from us may be universal, but the parameters of what constitutes "near" or "far" are much different in a country like the USA than in a country like China.
Last but not least: support presence with agency. When people talk about virtual reality, they use the word presence a lot. The feeling of being there is great, but presence can be frustrating if it's not coupled with agency - the sense of being engaged. Any story form can provide agency. A book can provide a sense of agency to the reader. A film can provide a sense of agency to the viewer. But virtual reality provides the opportunity - the expectation quite frankly - for true agency wherein you can engage and receive a response.
I remember when I visited Noitom here in Beijing a few weeks ago and experienced Project Alice for the first time. They handed me a barstool in VR that was tracked to an actual barstool. I could see the digital barstool in my virtual hand while feeling the physical barstool in my actual hand. Even though I was aware that I was in VR, there was a powerful synthesis of feeling a physical object while seeing a virtual representation. What happened next is that I got "hungry" - I wanted to interact with everything in the Project Alice demo.
But of course not everything was set up for this. The next demo was a Buick showroom. A virtual car appeared, and I opened the passenger door. As I leaned into the car to take a closer look, I lost my balance and tried to grab the seat. But there was no actual seat there, so I nearly fell over. My point is that although I quickly accepted the virtual environment and wanted to do everything - to turn the light off, to kick this box over, to climb that rock - I could only do what the environment was programmed to permit.
There's a VR phenomenon you've probably heard of called the "Swayze Effect." I always age myself by explaining this. There was a movie called Ghost that came out 20 or 30 years ago - seems like only yesterday. A lawyer played by Patrick Swayze is murdered. I won't go into the details, but he comes back as a ghost to try and protect his girlfriend from being murdered by the same people. However, he can't have any physical effect on the world. He's there - he can see things and talk. But nobody hears him, feels him or has any sense that he's around, and this is extremely frustrating. The cat notices him, because cats have this sixth sense - they don't care about the world, they just care about the next one apparently - but nobody else can see him except for Whoopi Goldberg, which isn't worth explaining.
In virtual reality, the "Swayze Effect" is commonplace: you're there, but you're not there. The characters and environment may react to you, but not to the extent you'd like. Addressing the interaction is a huge amount of work for VR creators and programmers to consider, but we expect it. People won't be satisfied with an unresponsive VR story or environment. However, there are ways to deal cleverly with the Swayze Effect. One company plans to provide VR access to exclusive events: Oscar after-parties, fashion shows, etc. I smiled at this and thought, "Ah, and you can have the same feeling of being ignored as you would in real life." I share this half-jokingly, but there are indeed ways in which you can design the VR experience so that you accept the fact that not much reacts to you. This calls back point of view: how are you viewing the story, through what perspective and for what reason.
We're ultimately going to need this: artificial intelligence. Recent advances in AI have been touted in the media. My first thought was, "Hmm... AI is coming and the robots are going to kill us all, so why bother doing anything?" But though AI seems to be progressing rapidly, as I researched further I found that true AI - the sort of machine self-awareness that will enslave humanity - is way, way off. Most of the so-called advances in artificial intelligence are not due to improvements to the science or the algorithms, but simply due to the massive amounts of computing power that are now being applied to the same algorithms and processes that we've had for 50 years. And emulating the human intellect is not merely a scalability issue.
That said, the degree of AI we need to apply to a virtual reality story or a virtual reality environment is not on the order of HAL, but a limited amount to provide the illusion of agency, to provide the illusion of interaction for the purposes of that particular content. And this is where the intelligent planning of the virtual reality experience - the story that's being told, the way in which it's being conveyed, and the technology that's being applied - is crucial.
So I'd like to leave you with that thought. I know that this has been a mad dash over the surface, and there's a lot more to say on this topic. I lecture on it here at the school and in other forums. If anybody wants to talk about this further, I'm easy to find, online or on earth. Thank you.