How Star Wars is blazing a trail in VR storytelling

Star Wars has always served as a catalyst for innovation. George Lucas’s 1977 film redefined the summer blockbuster, Industrial Light & Magic transformed the visual effects industry, and even the much-maligned Star Wars prequels pioneered performance capture technology with Jar Jar Binks. That’s not to mention Pixar (hatched from Lucasfilm’s The Graphics Group), Skywalker Sound, THX and LucasArts’s contribution to the video game industry.

Star Wars’s ability to take audiences to a galaxy far, far away has always been one of it’s great strengths, so it’s perhaps no surprise to see the franchise explore the exciting terrain of virtual reality. That’s being driven by ILM’s nascent “immersive entertainment” division, ILMxLAB. Founded in 2015, ILMxLAB’s executive in charge is Vicki Dobbs Beck, whose career with Lucasfilm stretches back to Willow and Indiana Jones And The Last Crusade.

Speaking at GTC 2019, Beck said that ILMxLAB was born out of the question “what’s next?” for cinema, TV and video games. “We started seeing disruptive technologies being announced: virtual reality, mixed reality, new application for mobile,” she explained. “We believed that these devices would allow us new ways to convey emotion and establish deep and intimate connections to story and place. We felt these were opportunities worth pursuing and so we launched ILMxLAB. We wanted to be pioneers, and our goal was to allow people to step inside our stories.”

Though still in its infancy, the division has been quick to experiment with VR and MR (mixed reality) storytelling. VR game Jakku Spy arrived on the Star Wars app in 2015 in support of The Force Awakens, while 2016 saw the arrival of Trials On Tatooine and Rogue One prequel tie-in Recon A Star Wars 360 Experience. ILMxLAB backed ventures outside the Star Wars universe, too, notably Alejandro G. Iñárritu’s Carne y Arena, which sees participants put into the shoes of migrants crossing the US-Mexico border.

For Beck, ILMxLAB’s philosophy is simple: creativity leads and technology enables. “Why is [immersive storytelling] different to other mediums?” Beck asked. “For VR in particular it’s really two things. First is the power of being there, sometimes called the power of presence – it’s really about worldbuilding and placemaking. Second is the power of connection, an immersion provides a unique opportunity to interact with characters. If they respond to you in compelling and meaningful ways, it opens up entirely new possibilities.”

More recently the “hyper reality” experience Star Wars: Secrets Of The Empire, seen in the UK at The Void in Westfield, has taken guests to the molten planet of Mustafar, where a briefing from Rogue One’s Cassian Andor (Diego Luna) and real sets combine with VR headsets for a fully immersive experience. There, floors rumble, fans whir and heaters and scent generators work on overdrive to help marry the physical and virtual world. When your skiff glides across Mustafar’s lava surface, that’s real heat and wind you’re feeling.

Testimonials from those taking part in Secrets Of The Empire have proved encouraging. Beck recalled one moving account of a guest in a wheelchair who, after emerging from the experience, felt elated because it had given her the chance to walk alongside her friends in the guise of a Stormtrooper. It may have been in virtual reality, but to her the experience was a real one she’ll treasure.

“[Secrets Of The Empire is] not a film, not a game, not a theme park attraction but we could learn from all of those,” Beck said. “This is the power of VR, it creates a memory of a place as if you’ve really been there.”

Powering Secrets Of The Empire requires some serious technological force. It has to render out 90 frames per second with only 11 milliseconds to crunch through each of those frames – any noticable latency and the entire ‘reality’ falls apart. For reference, a photorealistic ILM frame from a 24fps movie like Star Wars: The Last Jedi can take hours, sometimes overnight, to render.

With VR striving for as close to cinematic quality as possible, ILMxLAB’s Star Wars Reflections short, developed in collaboration with graphics card giant Nvidia, provides an intriguing glimpse into the future. The short clip, showing an encounter between Captain Phasma and two Stormtroopers, was made via the Unreal Engine using real-time ray tracing, dubbed the holy grail of high-end cinematic imagery.

“When we talk about authenticity through photo real-time imagery these advances are key,” said Beck. “We’re able to create soft shadows, lifelike reflections… all of these contribute to the sense of immersion and the power of storytelling.”

This autumn ILMxLAB will release its most ambitious project to date, the three-part episodic series Vader Immortal on Oculus Quest. A brief trailer shows the Sith villain’s Monolith castle (first glimpsed in Rogue One) and teases a VR lightsaber battle. The Dark Knight Trilogy’s David S. Goyer brings some Hollywood clout to the project as writer and executive producer, while input from the Lucasfilm Story Groups means it connects to the wider Star Wars universe.

“The series takes place between Episode III: Revenge Of The Sith and Episode IV: A New Hope. We intentionally chose the iconic character of Darth Vader because we knew it would be powerful to connect with him in VR,” Beck explained of the series.

“This is also our foray into connected stories. So Star Wars: Secrets Of The Empire essentially sets up Vader Immortal. Whereas in Secrets Of The Empire you were on an outpost on Mustafar and you could see Vader’s Monolith in the distance, now you’ll have the opportunity to go into the Monolith and engage with Vader on his home turf, not to mention have the opportunity to wield a lightsaber.” 

With ILMxLAB striving to blur the lines between cinema and video games, soon we won’t just be watching Star Wars stories, we’ll be active participants in them. “We can move from the notion of storytelling, a one-way experience,” says Beck, “to the idea of storyliving.”