Learning VR Audio: First Steps and DesigningSound.org VR Audio Resources

As many of the writers that contribute to DesigningSound have explored, there is a huge benefit gained by the community when tutorials and learning materials are freely shared.  Well established tools and pipelines have a multitude of learning resources available, including video and written tutorials, making it relatively easy to learn new techniques outside of a school or workplace setting.   You can usually learn how to use industry standard tools, and find industry standard techniques for learning fairly typical sound design tasks with a quick Google, or even searching Youtube, or any of the social circles.

For elements of the industry that are well established, it makes learning a breeze, but for newer technologies such a VR Audio, a lack of learning resources can be a huge pain point.  How do you learn when there is a very small amount of resource available to do so, very few people writing about their experiences, and new tools available every Quarter.  This article will explore several resources that are available, and how both myself and others have cobbled together a pipeline from snatches of information scattered across the internet.

It’s no secret that people may have been working in the Game Development space of VR Audio for, at the very least 3-6 months, or upwards of 2 years + for those who have been embedded into Hardware production companies (like Oculus or Valve, it is inferred).  This does bestow a certain amount of time for learning and development, and it gives these people a bit of a head start in figuring out how things work in this complicated new technology.

Personally, I’ve been working in VR Game Development for about 6 months now with Zero Latency VR, but have developed a number of binaural radio plays, and have researched psychoacoustics at a Masters level.  At the moment, I’m working in an FMOD Studio to Unity 3D pipeline, using 3DCeption as my spatialisation plugin.  I also use the Facebook 360 workstation for “canned” spatialisation of some of my audio.

 

In VR Audio for Game Development, we are fortunate enough to be able to continue usage of our Middleware and Engine tools with the addition of a Spatialisation plugin to take care of the rendering of the 3D position of the audio.  This article will focus mostly on this pipeline, but it is worth mentioning that the point of difference between VR Games and VR Film/Linear VR is that there are a few new tools coming out for linear VR that are slowly surfacing in the social groups.  These tools will be explored in a future post.

Learning where to Learn

As a first port of call, video and written tutorials and brand maintained wiki-style information is usually a fairly sensible place to start.  For “standard” Game Audio, this can be as easy as dialling up the brand website or YouTube channel and picking the topics that you need clarification on.  This style of content is usually created by Tools Developers themselves, or “Identified Expert Users” – users that either self-publish their processes (which are self identified as superior to other users in some way), or users selected by the brand to show their usage of the tool in a tutorial format.

Other designingsound.org writers will explore the relationship between Brand, Tutorial and Tutorial Maker this month, and hence this is not a critical element of this article, however it is useful to note that the “Brand seal of approval” is a useful way to know the information shared in the tutorial is a useful and efficient method for using that tool.  Unfortunately for people looking for information on VR Audio, a lot of this brand maintained information is actually quite basic and is usually focussed on getting the tools working, or delivering a foundation of knowledge for using the tool.

Please note: Throughout this piece, many of the resources are listed in the DesigningSound.org VR Audio Resource page, and hence we refer to those entries in the database by their sheet name and line number, e.g., Sound Design Line 15 is SD15.

 

Let’s look at the Branded content and information to determine what can be learned to help you get your pipeline running and get you operating in the creative process and understanding the technology.  Content like this is shared from the Vision Summit (SD15, SD16) and Oculus Connect Conferences (SD14, SD12), from VRDC (SD17, SD18) and in the format of video tutorials on branded websites and YouTube channels.  With the rapid development of tools, there is also a rapid update of resources for these Middleware solutions (and other tools such as Koloreyes (L03), such as the AudioKinetic website’s quick introduction to using Ambisonics in Wwise (SD22).  The Oculus Audio SDK website also has a solid introduction to thinking about and working with audio in VR when using the Oculus Spatialiser (SD23).  These resources are very good starting points, and will help you get started in setting up your pipeline, but it’s important to note that branded content gives perspective of using one set of tools to achieve the potential of that set of tools.  It may be necessary to assess many tools to find the right solution for your project.

So if you’ve dialled up the Brand curated information and got the pipeline functional, you’re certainly a good proportion of the way there.  You can start diving in and continue with designing sound the way you have done previously – experimentation is an important part of discovery and learning.  Indeed, when speaking with Stephan Schutze on the topic, he admitted that many of the techniques he had learned were from experimentation.  This resonates with what I’ve discovered as well – experimenting with designs and concepts are one of the few ways whether you’ll work out if they have the desired effect.  It can be useful to have a launchpad for this experimentation, though, and this is where the “technical” and “social” knowledge bases come in handy.

[perfectpullquote align=”left” cite=”” link=”” color=”” class=”” size=””]Experimenting with designs and concepts are one of the few ways whether you’ll work out if they have the desired effect[/perfectpullquote]

Breaking down the knowledge bases into these major formats is important because there are both technical and creative skill sets that contribute very different types of knowledge to our professional practise.  We’ll explore the various places (on the internet) that you can find these types of resources, and how you can use these to start learning more about the state and skills relevant to VR Audio.

 

 

Learning from Technical Resources

Technical Knowledge Bases are the type of resources that you might have used when you were studying, or when you need to scrub up on your solid Psychoacoustics research content.  These can be heavy reading and require a significant amount of mental energy to process, but understanding the background to the technology is useful.  Understanding psychoacoustics (how humans interpret sound) empowers exploitation of psychoacoustic principals in your design work.  Knowing what frequency ranges correlate with certain psychoacoustic effects like localisation will mean that you can enhance frequency effects, design and mix in the “spatial frequency range”.

A great place to start if you want to learn the foundations of Psychoacoustics, and understand the basics of what the HRTF technology that drives a lot of the spatial audio tools is the UC Davis Website for 3D Audio for Human-Computer Interaction (SD2) This series of pages is great because it’s concise, well illustrated and covers the foundations very well.  It’s written by one of the Academic greats (Richard O Duda) and includes references for further reading.

This leads us to the deeper research and development style publications – the Journal of the Acoustical Society of America has a sub-site for their Express Letter format.  These are mid-length research papers that are freely available to read from the Scitation website (SD3) .A recommendation from my Masters research is that finding follow up studies that explore Covert Peak Areas or Directional Bands will give you interesting perspective on the inherent directionality of ‘sound’.

You can, of course, search on Google Scholar for a more general survey of the research that is available.  JASA is not the only academic publication available that explores acoustics and psychoacoustics – the AES also publishes papers on these topic areas, as well as others.

Learning from your Community

Social Knowledge Bases are the more creative resources that allow you to reflect on the techniques and results of other practitioners.  Write ups or presentations from individual professionals about specific projects, and hopefully as we approach the AES and GameSoundCon Conference week in September, there will be more generic and widely applicable information available as well.  We are going to discuss a number of different resources that have published pieces about VR Audio, but it must be noted that this type of content is sometimes published by Branded Blog websites or other Aggregating websites, and you must consider the objectivity of Branded content when reflecting on such content.  At the moment, Reflective articles are scattered across the internet in a somewhat sporadic manner.  Although there is the AudioVR.com community and website active now, the aggregation of these articles into one resource isn’t something that has happened yet.

Interesting perspectives are shared from major projects like Crytek’s The Climb (SD1) and Oculus Story Studio’s reflection on Henry and Dear Angelica (L01), which discuss the creative approaches of the teams, and rationalise the techniques deployed based on the pipeline and requirements of the projects.

[perfectpullquote align=”full” cite=”” link=”” color=”” class=”” size=””]“It’s that level of detail that makes the immersion work. Even though in real life a sound so small would probably be masked by other environmental sounds, in the intimacy of VR, those sounds engage the player.” – Jennifer Walden (Post Perspective on Crytek’s The Climb)[/perfectpullquote]

There are more sporadic articles that are scattered across the internet, and many of them are documented in the designingsound.org official VR Audio Resources page (here).  This includes our own articles about VR Audio, articles from The Sound Architect, DesigningMusicNow, Block Interval, Winifred Philips, and many more sources.  Development on VR titles can be slow (in this initial phase of the consumer facing industry), so the rate of release of articles covering detail are currently slow, and level of detail can be fairly low.  Taking the widest possible sample of these articles is important, as every practitioner has had to make judgements about the needs of their project and act on opportunities and hurdles as they present themselves during the course of development.

Learning as much as you can about the tools and technologies available to you is also possible through these more social knowledge bases as well – Vision Summit (here), VRDC (here) and Oculus Connect (here) all publish their content online for free.  Vision Summit and VRDC have featured several technology partners presentations, where you can learn the background and how to use technologies such as Visisonics’ Realspace 3D, Crytek Engine and Two Big Ears’ 3DCeption.  Oculus Connect features iterative presentations from their in-house content team, where their approach to sound design for VR is updated on a yearly basis.

The Relevance of Linear VR

At this stage, you can even learn by practising with VR Film content and using VR film tools.  The H2n is capable of A-format Ambisonics recording, which can then be layered with sound effects and included on to 360 video in a DAW with tools such as SpookFM (L03), Facebook 360 Workstation (here) or Sound Particles (here).  There are many lessons about sound staging, sound design and the treatment of space in VR that can be learned from working with VR film tools which are immediately transferrable to working with VR games, so reading up on the materials offered in the Linear VR page of the Designingsound VR Audio Database will help get you on that pathway.

[perfectpullquote align=”full” cite=”” link=”” color=”” class=”” size=””]There are many lessons about sound staging, sound design and the treatment of space in VR that can be learned from working with VR film tools which are immediately transferrable to working with VR games[/perfectpullquote]

Learning and Sharing Socially

Finally, one of the most engaging ways to stay afloat of developments in the industry and what other people are doing and learning is by engaging with social media.  This can be a tricky thing to always have time for, but even if you can’t keep a close eye on the following groups, you’ll still be able to search for and ask questions into the groups for tips and techniques that you need a hand with.  On Facebook, the most active and helpful groups I’ve found are the Spatial Audio in VR and AudioVR Groups.  The AudioVR (here) and Virtual Reality Creators (here) Slack groups are also useful places for conversation and questions, which you can engage with as you please.

[perfectpullquote align=”left” cite=”” link=”” color=”” class=”” size=””]Keep reading and experimenting, as these actions will continue the development of your knowledge in this field[/perfectpullquote]

Good Luck and Happy Learning

Make sure that you check in with the VR Resources page from time to time, as we update this resource with new content that we discover.  If you publish any content that teaches VR content, reach out to us on the contact page to get it listed in the resource as well.  Keep reading and experimenting, as these actions will continue the development of your knowledge in this field – the most you will discover will be from your own processing of these many resources and conversations you have with other professionals in the field.

 

Thanks to Stephan Schutze for his perspective on learning (he’s on twitter @stephanschutze).

Sally Kellaway has a clone that constantly lives on Twitter @soundsbysal (jokes! She wishes but she has to do alllllllll the twittering herself).