We’re in Phase 4 of VR and Things are Only Getting Better
By Andy Marken
“People come to the Oasis for all the things they can do, but they stay for all the things they can be.” – Parzival, “Ready Player One,” Amblin Entertainment, 2018
Sometimes timing is everything.
And in our latest research to “prove” that VR filmmaking is real, SIGGRAPH couldn’t have come at a better time.
For adult kids, SIGGRAPH is great any time because you get to see some awesome special effects and film work as well as get an idea of what tomorrow will look like.
Of course, for folks like our grumpy friend – we’ll call him Ted – it’s all about the game stuff.
When it comes to VR, he knows why the stuff hasn’t taken off, “People just can’t get comfortable wearing a headset for extended periods of time and no one likes how solitary you are when you’re viewing a VR film. The key to a good movie or TV show is sharing.”
And he made that comment about the commonly voiced issue while staring at his iPhone, oblivious to anything around us on the show floor!
Go figure!
Even before Facebook bought Oculus for about $3B in 2014, folks like Digi-Capital were forecasting heady growth and sales numbers.
Ever since then, they’ve been backing off the stratospheric market numbers, conjuring up all sorts of rationale for the market failing.
Not that Ted played many VR games, but we reminded him that the early games sucked!
They were just good to decent video games that developers slapped VR technology into using one of the two leading VR game engines – Unity or Unreal.
People got queasy.
The same was true of some of the earliest VR film efforts, because it was a new filmmaking medium.
Sci-Fi stuff like Brainstorm, The Lawnmower Man, Matrix and Avatar whetted the market’s appetite for great VR and VCs/market analysts drank their own Kool-Aid.
Consumer industry folks rushed out expensive (yes, Ted, clunky) HMDs (head-mounted displays) and waited for stuff to be delivered.
But it was an idea still in its infancy and all realities (virtual, augmented, mixed, cross) were thrown in the same pot.
Indie filmmakers have been testing, refining, producing results for years.
Hollywood has been selectively using it for years.
Want to draw a crowd to the theater? Do a VR trailer, put it on social media and people will click in – 180, 360, immersive – doesn’t matter.
They’re paving the way.
We saw The Martian VR Experience a few years ago at the SMPTE conference and it was freakin’ awesome.
We worked, struggled, sweat right along Mark Walberg…had to see the movie!
With a lot riding on each release, Lionsgate released posted their Shatter Reality trailer to accompany Shatter Reality.
Paramount’s publicity folks hyped the action of several Mission Impossible films (especially MI: Fallout) and the fact that Cruise performed his own stunts – no doubles.
What kind of action?
They let viewers be a part of every cringe-worthy piece of action you wanted to experience with him.
Of course, since it was “only a movie,” you didn’t get to experience his broken ankle, which set the production schedule back … a little.
It’s OK Tom, we’ll pass.
Granted, VR hasn’t taken off like “those folks” predicted.
There weren’t a lot of economic, off-the-shelf VR cameras in the early days; so smart, adventurous filmmakers developed their own solutions.
Rush to Capture – With the excitement and popularity of VR, firms large and small rushed to bring together 360-degree camera rigs like the Head Camera (center). Jaunt (l) quickly established itself as a major producer of high-performance VR cameras. To keep costs under control, many filmmakers creatively developed their own VR rigs using multiple economic GoPro cameras.
Stitching, image rendering and post production solutions were still works in progress, people who could “see” the potential and were anxious to get their content produced for the anticipated audience.
Most computers were woefully underpowered to handle the brutal processing load–even with the best GPU cards available at the time.
Power GPU – At SIGGRAPH this year, Nvidia’s Jen-Hsun “Jensen” Huang introduced the new Turing-based GPUs that deliver ray tracing and real-time graphics to make VR content production faster, easier and more realistic.
The VR filmmakers at SIGGRAPH couldn’t wait to get their hands on the new 8th generation Nvidia Turing-based GPU. Enabling light ray tracing, it’s going to help shooters produce killer film realism.
Yes 4K, 8K showrunners, filmmakers and post people drooled when it was announced; but their mouths went a little dry when they heard the cost and that they might need a new computer.
But imagine the time Indies will save in production–and time is … money!
And with some of the SIGGRAPH demos, it was hard to see if you were in the virtual or real world.
Immersive Production – To produce the best VR entertainment possible, post personnel had to work between proven production software/techniques and HMD reviews to see what worked and what didn’t. It was often an “uncomfortable” task.
Most of the early projects combined the narrative forward momentum of things people knew worked with films with an immersive exploration of VR.
The results were often the worst of both mediums.
Immersive pioneer Andrew Shulkind has been there since the beginning.
Besides doing much of the seminal experimentation with wide angle lenses, stabilized camera movement, storage for high frame rate capture and stereo conversion at the highest level, he was one of the first to diligently bring the best of “worked before” in this burgeoning new medium.
But it took that kind of experimentation with camera arrays, steadicams and a million different versions of framing, composition, and editing to get broadcast quality.
Andy Cochrane, an equally experienced interactive and immersive creator, added that 360-degree VR content is obviously more immersive than standard 2D films.
“We followed many of the same rules,” Andrew MacDonald, creative director at CREAM, added. “Some of the work reflects the fact that people often integrate interactive VR elements into their cinema experience rather than expand on it.”
“But the most recent work that Shulkind, friends at CREAM, 30 Ninjas and other VR pro teams are releasing are true VR films with a beginning, middle and end,” Cochrane added. “It’s a creative challenge when the viewer can direct the content flow; but by using the right audio and visual guides at the right times, you can help them through the story.”
“Game producers had all of the same problems,” Lewis Smithingham, 30Ninjas, observed. “Their development engines were great, but they were having trouble helping the viewer through the game. The result was a disappointing gaming experience. Many of them came to VR filmmakers to help develop way points for the game player to get to the next level or go off on their own to experience immersive options.
“We helped them deliver the scripted and unscripted game play that gamers really wanted, and it also helped improve film work viewers are experiencing now,” he added.
Developing improved and enhanced films required hours of stitching the content, removing rendering glitches and refining VR content. Helping viewers navigate through the new virtual space was time-consuming, hard on the eyes and hard on the stomach.
The stuff VR shooters are delivering today is darn good and the more folks experience it, the faster it will grow.
360 is Real, Good
Nick Bicanic, RVLVR, was one of the first filmmakers to visit Cuba when the travel ban was lifted and wanted to do more than just give viewers a visual tour; he wanted to help people experience the country, the people, the culture.
He spent three weeks in Cuba capturing the essence and excitement of the cities, countryside, people and their pastimes.
Music Rules – During his VR documentary covering the color and vibrance of Cuba, Bicanic had the good fortune of being there at the same time the Rolling Stones who did a concert to reinforce Cubans’ love music of all types.
“We gave them a solid taste of Cuba that wasn’t possible with a flat 4K film,” he emphasized. “Viewers shared Cubans love of baseball, quiet checkers matches, color and music of all types, including a special Rolling Stones concert.”
Sometimes, timing is everything and he was able to capture the excitement of the legendary group and the interaction of Cuba’s love of music.
“It was a great way to mix the old and new of Cuba,” he noted.
Curious VR
MacDonald and the rest of the Toronto-based Cream VR team have been instrumental in helping Hulu produce a solid library of VR shows including their most recent, A Curious Mind featuring Dominic Monaghan.
Your World – For HULU, Toronto-based CREAM developed a VR series featuring Dominic Monaghan called ‘A Curious Mind’ to enable viewers to ‘participate’ and learn more about the world around them.
A Curious Mind evolved over several years, when 360 and VR were just in their infancy and was similar to the style of Carl Sagan’s Cosmos: A Space Odyssey; but gives viewers a closer, more personal experience.
“The strength and weakness of the immersive content is that the viewer is ‘involved,’ MacDonald said. “Creatively, we needed to control the viewing to tell the complete story.
“The user can physically participate in the story telling and shape the experience simply by looking away from the POI (point of interest),” he explained. “Our challenge was to guide him or her so Dominic and the viewer both arrive at the conclusion at the same time.”
In A Curious Mind, Dominic and the viewer make the journey together; manipulating time and space, pulling apart atoms and exploring/experiencing science as never before.
Next Phase
Now that 360 video is becoming established and “understood,” all three Andrews are working on next-phase projects – volumetric image capture technology.
Total View – One of the best views of Microsoft’s HoloLens technology was in the appearance of Princess Leia’s message to Obi-Wan in ‘Star Wars.’
Originally developed for Microsoft’s platform HoloLens, it uses multiple cameras to map a target location from every angle which is then combined with 3D imaging techniques to create accurate representations of depth.
Using volumetric or 4D video (4D = 3D space + x,y,z + use time as the 4th dimension) technologies, the filmmakers capture the shape and form of human performances and place them in immersive experiences.
“This marks a new paradigm in how an audience engages with content,” Shulkind explained. “With 360˚ video you are locked to a fixed position. Capturing assets volumetrically restores the filmmaker’s ability to creatively block action around the viewer to make their own decisions. It marks a paradigm shift in how audiences will engage with content in the future.”
Dancing Volume – A ballerina performs her moves in Microsoft’s MR studio which will ultimately be a true volumetric video creating an interactive hologram of the artist that can be enjoyed from any angle.
Instead of capturing reality based on a specific perspective, storytellers capture the entire scene with infinite camera positions and infinite camera angles.
Volumetric is captured from the outside in, so people really have a true, no-holds-barred three-dimensional viewing experience.
Each of the VR filmmakers we talked with are working on volumetric projects scheduled for release late this year.
Cochrane noted, “We’re building ways to let you go anywhere you want and do anything you want within our stories.
He added, “There’s a lot of work still to be done, but it’s already getting quite magical.”
Those who had a chance to try it at SIGGRAPH said it is well worth the immersive isolation.
Of course, they also knew they would need Nvidia’s next-generation GPU to process the huge volumes of data that are generated.
Standards Emerge
To ensure that the industry will reach the analysts forecasts of – pick your favorite number, clusters of companies intent on making big money off filmmakers and viewers have been busy holding consortium meetings to develop “open” standards for VR/AR/MR.
What we seldom see in these groups though is direct involvement of filmmakers who must take the gear and software to really memorable stuff.
Still, we’re looking forward to getting an update on the Consumer Technology Association (CTA) standards at IBC as well as the Video Electronics Standards Association (VESA) standards recommendations, The Khronos Group Open XR update and the Global Virtual Reality Association’s (GVRA) best practices.
There are probably other “standards groups” at work we don’t even know about – yet.
Then, we want to see what comes out of October’s Society of Motion Picture and Television Engineers (SMPTE) conference.
Maybe they’re working on things like file formats, compression, distribution and IP delivery.
Trying to calm the hungry horde, Cochrane recently told a concerned and frustrated pro-VR audience, “Just (expletive) calm down. This is a marathon, not a sprint.
“We’re going to change humanity in uncountable ways,” he continued. “We’ve accomplished a lot in a few short years and we’ve still got a lot of work to do.
“It’s going to be a heckova’ run but we can do it,” he said.
Halliday in Ready Player One agreed with him when he said,” I realized that… as terrifying and painful as reality can be, it’s also… the only place that… you can get a decent meal.”
