VR Sound Design for Touch Controllers

This is a guest contribution by Earl Scioneaux, III. Earl is a New Orleans-based game sound designer. He also does sound for film and produces music under the monicker The Madd Wikkid.

---

 

When I first used touch controllers, I was amazed at how they enable the player to instinctively “grab” things in the game world. This can allow for some amazingly intuitive and realistic interactions, some of which require special consideration when it comes to sound.  While working on The Mage’s Tale, I was tasked with crafting a number of sounds in this arena, from seemingly simple things like a hand crank and a lever, to supernatural things like telekinesis and a spell that blasts electricity out from the player’s hand.

Reach Out and Touch

In a non-VR game, turning a hand crank might typically be handled by pressing a button that would trigger a specific animation for the cranking action – something for which a complete predetermined sound could be easily added.  With touch controllers, though, the player would be able to wind a crank with the actual motion of their arm. There’s no way to know if the player will move quickly or slowly, change speeds, or reverse direction. A simple sound or loop wouldn’t pair well with this, and such a natural mode of interaction deserves sound that is believable.

To solve this, I imagined the crank as kind of a ratchet. At regular intervals, as it rotates, there’s a singular click, and the overall “cranking” sound is a composite of these clicks. Designing the sound of the VR crank accordingly, it seemed, would allow it to respond believably despite changing speeds and directions of the cranking action.

To do this, I made a handful of varied click sounds that would play randomly one at a time. Scripting was implemented to trigger the sound based on the change in the angle of the crank, with a variable we could set for how far apart the clicks would fire (e.g. every 30 degrees). Later, I added a second layer of deeper clicks, triggered half as frequently as the main ones, to give some impression of large internal gears clunking along in tandem with the crank.  It worked so well, we ended up using the same core system for ratcheting door handles, the creaks of treasure chests opening, and a handful of other things.

Next was a pullable lever.  This was similar to the crank in that the player could move it at various speeds in two directions, but the sound needed to be scraping, not clicking. This required a different approach.

For this, I made a few loops of similar, randomly selected scraping sounds as the base, with an added layer of occasional, short, high-pitched squeaks.  That composite sound had its overall volume modulated by the velocity of the lever’s movement.  When the lever was pulled, the sound became louder the faster it moved (until it reached full volume), and quieter as it moved slower.  When the lever stopped moving, the sound would stop.  This created a result that matched the feel of the interaction well, and we were able to reuse the same logic for several other objects that rotated on an axis.

 

Long Distance

The Mage’s Tale, as the name might suggest, is a game that endows the player with magical abilities.  This sorcery sometimes allows the player to interact with objects at a distance.  After a bit of experimenting, I found it possible to achieve rather satisfying results for some such situations simply by having related sounds in 2 places – one on the player’s magic-wielding hand, and the other on the object being manipulated.  Triggered in sync, this duplicity of sound sources served to create a sense of connectedness between the player and the distant object. To illustrate, let’s look at 2 examples: telekinesis and the lightning spell.

For telekinesis, something akin to using “the force” to grab things, I placed a darker, constant version of an airy magical sound on the player’s hand.  On the object being grabbed, a more intense version of that sound would play and crescendo (I used the game engine’s attenuation settings to have the volume increase and a low pass filter sweep upward with proximity to the player). The combination of those sounds created convincing results that felt linked even when the player moves their hand around while the object is being drawn to it.

For the lightning spell, a chaotic beam of electric arcs shooting out from the player’s hand, I realized it wouldn’t be practical to have a string of sounds along every point of a wildly-moving electrical arc.  As such, the dual-sound method came in handy again.  I chopped up electric spark sounds into very short bits (about 0.25 – 0.3 seconds) as the basic building blocks, with a darker set to play at the hand and a brighter set to play at each point of contact as the flailing arcs strike objects.  Whenever a beam made contact, a random instance of each would trigger simultaneously.  With this arrangement, as the lightning beams jumped around, the cacophony of tethered near/far spark sounds actually seemed to fill the air, creating a very responsive and believable effect.

These are a few of the more interesting challenges I’ve tackled while devising sound for touch controller interactions. As sound designers plunge ever deeper into the VR frontier, developing great techniques will be an ongoing process of brainstorming, experimenting, refining, and sharing ideas for years to come. As such, I hope that some of these methods I’ve shared will prove useful to others in their sonic pursuits.

Enjoy!

---

Thanks to Earl for contributing his piece. You can read more from him on his VR sound design blog or checkout his Facebook music page.