Cooking Up a Chinese VR Short

360 view: http://vr8.tv/88/3F1840

Earlier this month, immersive content development took an experimental step forward with China's first improvisational VR "table read" on the interactive cinematic VR short film, Four Dishes and a Soup.

360 view: http://vr8.tv/88/3F187D

Four Dishes and a Soup is being developed by Magic Dumpling Entertainment and VR filmmaker Nikk Mitchell as part of the iAVRrc PROJECTS program at the Beijing Film Academy's International Animation & Virtual Reality Research Center. The iAVRrc PROJECTS program facilitates the development & production of compelling content in AR, VR and MR. The first iAVRrc PROJECTS slate represents a variety of approaches to immersive media, with Chinese culture as a common thread, and edutainment as a common goal.

Four Dishes and a Soup is a scripted, interactive cinematic VR short experienced from the point of view of a foreigner who is invited to dinner with his Chinese girlfriend's family. The viewer / participant is in the hot seat as the foreign boyfriend: influencing his behavior and reaping rewards or consequences. The VR experience aims to humorously highlight East-West cultural differences while showcasing delicious Chinese cuisine.

To achieve naturalism and authenticity in the development process, we recently conducted what can best be described as an "improvisational VR table read." In a twist on a traditional table read (where actors work from a prepared screenplay), we assembled a group of participants and supplied them only with character descriptions, scenario outlines and interactive direction from filmmaker Nikk Mitchell. When selecting people for the VR table read, we focused less upon physical appearance (grandma and grandpa were played by folks in their 30's) and more upon flexibility and playfulness.

360 view: http://vr8.tv/88/3F187C

In order to make our participants comfortable, the Four Dishes and a Soup improv VR table read was held in my Beijing apartment at a round table covered with drinks and snacks. After a few improv warm-up exercises, we began exploring scenes filmed with a custom-built 360 camera from the foreigner's POV. The learnings were immediate. We adjusted on the fly when our preconceptions fell apart, and came away from the session with a wealth of observations, notes and 360 footage that we are now reviewing in advance of drafting the complete screenplay and VR interaction model.

To mark this phase of the project, I interviewed our director Nikk Mitchell the day after the shoot to get his take on the experience...

KEVIN: Nikk, you're directing this VR short entitled Four Dishes and a Soup. Tell us about the purpose of yesterday's improv VR table read.

NIKK: Sure. We had the VR table read for two reasons. The first is for dialogue: to see what we could discover through improv, what sort of natural interactions would come out. Getting everybody into their characters was great: establishing their fictional backgrounds, education levels, occupations, previous exposure to foreigners, attitudes towards the boyfriend, etc. Some new scenes and other things came up by playing around, which was very cool. The second reason for the VR table read is that I wanted to test the interactive aspect of the project, because that will have many unique challenges, as we've learned already.

KEVIN: Tell us more about that.

NIKK: Well, in Four Dishes and a Soup, the viewer has agency: you can control the outcome of individual scenes - and on a broader scale, the final ending of the film - by nodding your head "yes" or shaking your head "no" in the headset. Figuring out how to film that is a huge challenge. Does your story keep branching, or is it modular? How do you organize the filming of the scenes? Yesterday, when a scene came to a decision point we'd have everybody freeze, remember their positions and then proceed through one possible outcome. We'd then have everyone return to their freeze positions and continue the same scene with another possible outcome, and so on. Now we'll take that footage into post and see how well the theory worked in practice. How do the various choices cut together with the setup? If people aren't in the exact same place, is that jarring or can we apply fades and other effects so it's not a big deal? We'll see.

360 view: http://vr8.tv/88/3F1876

KEVIN: There were things you learned during the shoot and there are also things you anticipate learning after you assemble the footage.

NIKK: Definitely. Who knows, maybe we did it completely the wrong way.

KEVIN: [laughing]

NIKK: Which won't make the shoot a waste, but a valuable lesson. [laughing]

KEVIN: Hopefully. Even before reviewing the footage, I feel that we learned a lot already from the improv. I'm sure there's stuff that won't be usable in the way we thought, but may spark new ideas and new ways of approaching the script and the final shoot.

NIKK: Yep.

KEVIN: You've done many VR projects. How does the experience of doing this improv VR table read compare?

NIKK: Well, I've shot VR projects that were improvised to some extent in the final result, but never this sort of pre-production improvised table read working from character descriptions and scene outlines. I think that doing this in advance of the final shoot is so valuable. Now we can review the footage, identify the good stuff, and work that into the script. And even if the final shoot is still improvised to some extent, we now have very real information to build on. I'd like to use this approach on most of my VR projects in the future.

KEVIN: I think it's the first time this has been done here in China.

NIKK: I haven't heard of anyone doing this anywhere.

KEVIN: What surprised you most yesterday during the VR table read?

NIKK: There's just so much more stuff you need to anticipate in VR than when you're making a flat film. VR brings up all these things that you're not used to considering, even when you're trying to expect the unexpected.

KEVIN: Indeed. Although we weren't dressing the apartment for a final shoot, I remember standing where the 360 camera would be, turning around to see what was visible, and then pulling down conspicuous background items - things you don't normally need to worry about when you're shooting in one direction instead of in all directions.

NIKK: Yeah, I shot a Kickstarter video for a friend last week with a normal DSLR camera and it was wonderful! [laughing] I love VR, so I don't want to talk shit about it, but you're always worrying about... everything. And then still missing things. But shooting with the standard camera was like... not having to worry about anything. I can focus on one direction, the viewfinder shows exactly what it's going to look like, and I just run around filming stuff. It's so refreshing! [laughing] I might be oversimplifying a bit.

KEVIN: [laughing] As a VR director, what's the biggest thing you need to concern yourself with? Aside from "everything".

NIKK: OK... "everything", plus delivering a good film. It's so easy for a VR film to be either boring or nauseating.

KEVIN: [laughing]

NIKK: There are all the things you "can't do" in VR: you can't have certain moves, you can't have certain cuts... all of these rules. And once you follow all the rules, you end up with this static 360 footage of the camera just sitting there. Nothing dynamic, nothing really happening.

KEVIN: Yes, once the "wow" factor of the VR environment has worn off, you're like, "Ok, now what?"

NIKK: Yeah, and it seems like so many VR videos are these passive experiences where you're underwater or on a mountain, but there are no interesting stories. So, that's my big challenge: how do I make a VR film that someone would want to watch, or that even I myself would want to watch? I don't like watching most VR films, and I'm a VR filmmaker.

KEVIN: That's a great segue to my main question. What excites you the most about the the potential of VR storytelling?

NIKK: I'm glad you asked that, because I feel like I just deeply insulted my lady VR. [laughing]

KEVIN: [laughing]

NIKK: The potential of VR as a medium is that you're not "telling someone a story," you're creating an environment and guiding them through a story that they "co-author." And that really excites me.

KEVIN: In a way, that's what happened with the VR table read participants yesterday. We had them visualize and inhabit their characters through improv and then placed them in scenarios where they were essentially writing their own story. They had guidance, but there was room for them to explore, literally and figuratively.

NIKK: Yeah, and the extension of that exploration in the final film will be the viewer, who will be placed into the scenes and will help create a story.

KEVIN: This is an interactive short film where the viewer has the agency to make choices and face the consequences.

NIKK: Which is so different than a normal film. You watch a normal film from start to finish, and everybody sees pretty much the same thing. You might focus on one character more than another, but you're seeing everything. It all happens in your field of view. An interactive VR film, such as Four Dishes and a Soup, takes things to the next level. Even a non-interactive VR film changes the game. People see different things from different points of view, and come away with different conclusions. For example, we could be watching a VR film about a young couple: you think the guy is a jerk who ignores his girlfriend, but I see that his girlfriend is cheating on him, and our friend thinks it's about a bird that they've been watching through a window. A lot of directors I know hate this. It's sacrilege to them.

KEVIN: Yes. People with strong traditional film backgrounds - Steven Spielberg comes to mind - are leery about VR as a cinematic storytelling medium. They see the "potential" as a problem.

NIKK: Yep.

KEVIN: And then there are others - people such as yourself with non-traditional filmmaking backgrounds - who just jump in and see what they can do, embracing the possibilities as they go.

NIKK: Well, it's that issue of the director needing to control everything. They have a story they want to tell, and they want you to understand that story their way. It's like when an artist creates a painting and then gets upset because you admire the shapes and colors instead of appreciating that it's about oppression. When I create something, maybe I have a clear idea in my head and maybe I don't - it could be an improvised bit of expression. Either way, another person can take anything from it, and I love that. It's more meaningful to me when someone puts together their own story, because it's theirs. The meaning is created by the combination of what I made and how they experience it.

KEVIN: What do you want the experience to be with Four Dishes and a Soup? You just said that individuals may have different experiences, but do you hope for anything in general?

NIKK: I hope that people have fun with it, and that it surprises them. And I hope that it feels immersive and natural. I don't want people coming out of it going, "Hmm. That was interesting," but coming out of it going, "Woo! That was intense!"

KEVIN: Where do you see all of this taking us in ten years? How do you imagine people will experience VR entertainment ten years from now, maybe even five years from now?

NIKK: That's a great question. Hopefully, there's going to be content that is enjoyable to watch. [laughing] I'm sure there will be. There are so many smart minds working on this, so many creative people involved, that we'll figure it out.

KEVIN: And what do you see yourself doing five years from now?

NIKK: [pause] I want to make blockbuster interactive films that go deep into choice. This project has given me so much inspiration on what could be done with interactive film. Five years from now I can see having a little... "empire" is the wrong word, but a new kind of film... empire. [laughing]

KEVIN: [laughing] Let's go with "empire", why not? A virtual empire.

NIKK: [laughing] But there's so much to be done to get there. I just want to keep creating and keep pushing. Hopefully five years from now, I'll enjoy the experiences I make and other people will enjoy them as well. And the experiences will hopefully be amazing - not because they're VR, but because you're immersed in something wonderful and meaningful... and the "VR" part is forgotten about.

KEVIN: I'm glad to hear you say that. Many VR evangelists seem intoxicated by their own sizzle, but the average person doesn't care. Most folks could give a shit about the technology itself.

NIKK: Yeah, it's like, "Hey, do you want to come over and see my TV screen?"

KEVIN: [laughing]

NIKK: Nobody want to see your TV! They want to see a good show or a good movie. Nobody cares about the device.

KEVIN: And if they do care, it's short-lived. "Ok... got it."

NIKK: Yeah, "got it"... now what? The focus in VR is still too much on the tools and not enough on the content.

KEVIN: So, having just dissed VR's fixation with technology, I want to ask a technology question.

NIKK: [laughing] Nice!

KEVIN: Because technology is an enabler after all. The 360 camera that you used yesterday is unlike any other I've seen. It seems like an interesting hybrid of a professional 360 camera and a DIY rig.

NIKK: Yeah, I call it "Frankenstein." [laughing] The guts of the rig are eight Go Pros that have been completely torn apart. The sensors and motherboards are stuffed in the bottom of this cylinder along with the SD card slots and the chargers, and wiring runs up through the cylindrical camera housing to this ring of wide-angle lenses around the top. It shoots awesome stereoscopic 360 video.

KEVIN: You're getting high-end results from a relatively low-cost rig.

NIKK: Yep. This camera was built by my partner, Wilson Lee. He's been working on VR cameras for the past five years, tweaking and tweaking, working out the ideal distances, angles, and so forth. Now he's got this great custom hardware system combined with his own optical flow post-processing software that results in great stereoscopic 360 video with virtually no seams.

KEVIN: So, he's engineered the camera and written the software?

NIKK: Yeah, and what he's working on now - which is really amazing - is positional tracking. He can already get some positional tracking out of this camera.

KEVIN: Cool. So, we can start to get 3D immersive video...

NIKK: ... that you can move in. With a relatively normal 360 camera, you're going to be able to...

KEVIN: ...step forward.

NIKK: Well, currently you can't step forward. It's only like 5 centimeters of positional movement at the moment. [laughing]

KEVIN: [laughing] A small step.

NIKK: But the next version of the camera will be a larger rig that can handle more data, so you'll at least be able to lean around in a chair.

KEVIN: Fortunately, everyone in Four Dishes and a Soup is sitting.

NIKK: [laughing]

KEVIN: Is there anything else you wanted to say about this improv VR table read or about VR in general?

NIKK: Well, I'm really happy to be part of this project. It's been awesome. I think the VR medium is in such an interesting place. Lots of stuff is going on, but there's also not a lot of great content being made. I'm not even happy with the stuff I'm making myself.

KEVIN: That's usually a good sign.

NIKK: [laughing] True.

KEVIN: Well, it means you're pushing for more.

NIKK: Yeah, if anybody in VR now was like, "I think I've got it all figured out..." [laughing]

KEVIN: [laughing] They probably haven't.

NIKK: I love that VR is in this amazing, awkward place where we know that something incredible is out there, but we just haven't found it yet. We're digging around in the mud. I know I'm going to find gold, and that's exciting. It's also sometimes discouraging, but it's a cool place to be.

KEVIN: Any time you're pushing the boundaries, you're in that uncomfortable creative space which exciting and gut-wrenching at the same time.

NIKK: Yep.

KEVIN: Well, it's been great to have you pushing the boundaries on Four Dishes and a Soup.

NIKK: It's been my pleasure.

KEVIN: And we're not out of the woods yet.

NIKK: [laughing] No, we're not. We're just now stepping into the woods.

Kevin is the author of AWN's Reality Bites blog, his musings on the art, technology and business of immersive media (AR, VR, MR) and AI. You can find Kevin's website at www.kevingeiger.com and he can be reached at holler@kevingeiger.com.