Future Vibes With Spatial Sound Design
By Anne McKinnon
@scifiannemarie
The Future of Sound Design in Immersive Technology
If VR is to become the next wave of entertainment, it needs to have a solid value proposition. What does virtual reality add to the current model of experiential narrative? Making virtual reality feel real and sensational in an effective way is crucial at these early stages where hardware and software is going through massive overhauls and consumers are constantly adapting to disruptive changes in a non-standardized industry.
Co- founders Helena McGill and Anna Wozniewicz of Noctvrnal, a spatial sound design studio, realized early on that the world of spatial audio would be an integral part of this next wave of entertainment.
In VR and 360 experiences, “You want the sound to mimic real life,” said McGill. “So if you’re sitting in front of me and I’m turning my head, I still want to hear you as if you are sitting in the same spot, not as if you are moving with me,” she said.
It’s this ability of spatial sound design to add plausibility to VR that makes it extremely valuable because the brain interprets the experience as real. While there are many dynamics to spatial sound design, simply put, it’s a technique that uses audio to mimic the way we hear in real life.
Spatial Sound Design for VR/AR and 360
There are a few ways to work with spatial sound design. One, object localization where sound is tied to an object in the virtual environment. Then, there’s HRTF or head related transfer functions that characterizes how sound is perceived by the listener. This depends on several factors such as size and shape of head, ear canal and general structure, and can be used to stimulate 360 audio through a pair of headphones when the audio is rendered in a binaural format, so that sound is perceived as coming from a particular point in space.
In 360 there are x, y and z axis of sound for localization. For incredible accuracy along these axis of space where sound can travel, McGill and Wozniewicz have been experimenting with parametric speakers which, “are very pinpointed and if you move a couple feet in either direction, you’ll know, because you’ll hear it,” said Wozniewicz. Parametric technology used to be a military technology, but now it is often used in entertainment, and specifically in LBE (location based entertainment) for immersive experiences.
Then, AR adds an additional component in sound design of superimposing into a space, dealing with the actual physical location an experience is played in, while VR is a closed of world so there are less uncontrollable variables to account for.
Industry Standards and Workflow
Working with spatial sound design comes with new and more complex workflows. For 360 video the audio production process is more linear like sound design for film, but for VR and AR there are additional components.
One of the programs used for 360 is Facebook’s 360 spatial workstation. “They are really good about posting updates for the community,” and it’s free, said Wozniewicz.
“We’ve also been doing a bit of Reaper and have played around with G-audio’s spatial tools as well but they have a proprietary format which makes it a little harder to integrate across platforms,” she said.
On the game engine software side of spatial sound design, they use Fabric, but note that Unity has a pretty powerful built in audio spatializer. Combining this with Google Resonance, which is Google’s fairly new SDK that is really compatible cross platforms, is a good way to increase flexibility across platforms.
Overall, workflow is still a clunky process for spatial sound design, but this is a part of streamlining with new tools, programs and technologies. Noctvrnal work closely with developers through access to beta programs where they develop feedback for improvements and standardization.
Applications of Spatial Sound Design
McGill says the market has shifted away from solely 360 content and is branching out to more interactive VR and LBE experiences. Some demand comes from VR games, but it’s not as common as demand for spatial sound design in LBE, “We’re seeing a huge market for that,” said McGill.
McGill and Wozniewicz also work with installations that include live animations and biometrics, and they were a part of the sound team for VRLA’s multi-sensory Mezo installation that incorporated a spatial music mix across a massive pyramid with over 20 speakers.
“With this added dimension of 3D audio, this has challenged us to push the boundaries of where sound design can go. Before we were limited to linear structure and now that we are operating in this limitless interactive 3D space that’s either closed off VR or an open world AR system. It has challenged us to approach it from a different standpoint and it’s also given us the flexibility to implement a little more abstract or experimental techniques that we wouldn’t have been able to do in a traditional 2D setting,” said Wozniewicz.
Scripting for Spatial Sound Design
Even before spatial sound design became a reality, sound design in general has often been viewed as a finishing touch to the production process rather than as a part of the original narrative. Working closely with scriptwriters, Noctvrnal is able to maximize the value sound can contribute to the experience by adding sound cues into the story rather than trying to fit them in retroactively.
From a software standpoint, continuing to build solutions to map out interactivity is a work in progress on the developer side. As technology for workflow and object/sound relationship mapping progresses, this will enable a greater involvement in the actual narrative that is being soundtracked.
This also ties back to current applications in VR/AR. For LBE you have to think “how can we use this unique location to enhance the sound,” said McGill. They have to think about the constraints of the space, and how to enhance the experience with sound design. Catching this at the earlier stages of the project is better, so it’s more like creating and less like fixing, said Helena.
In essence, the space is 360 and the sound should be too. Working it into the narrative enhances the power of storytelling in an organic and effective way.
Pushing the Boundaries of Spatial Sound Design
In architecture, sound is an incredibly important part of the design. Certain structures are designed to amplify sound and to have it carry such as in a theater, and other buildings are designed to limit the transfer of sound such as in galleries or places of work.
Beyond architecture, there are many more applications of how spatial sound design can enhance an experience, or product. For one, there’s an incredible drive to design new and futuristic cars. One of these futuristic dynamics is sound and designs are done with more in mind about the positioning of speakers and how this can enhance the way sound is heard, or how it can cancel out sound in the surrounding environment.
“Entertainment is always the first application [of sound design], but there are so many more,” says McGill.
In addition to design, there are also applications of AI and machine learning emerging to assist with the creation of spatial audio.
Next Steps for Noctvrnal
One of their goals is to create an installation that is almost entirely auditory. It may have some visual assets to make it more accessible, but as the supporting act to a sound based experience.
“We are using that as our own R&D project to really integrate parametric speakers and other sensors into an experience that is engaging, immersive and pushes the envelope when it comes to integrating audio into an environment,” said Wozniewicz.
They want people to notice how there are very little visuals, and mostly sound, and to find that they are still engaged.
“We are excited to tell a story that is completely through sound. That’s often something that is overlooked,” said Helena.
VR/AR and 360 is gaining traction against all odds because of its incredible ability to immerse a user in a new world, space or reality. It’s the most effective form of experience in modern media. However, it’s not just software and hardware that make it worthwhile, but the content creators and how they engage users within current limitations.
