It may be premature for me to turn the focus of the series towards the future, as we find ourselves deep in the throes of the current generation console development, but I think by now those of us submerged in creating ever-expanding soundscapes for games at times suffer under the burden of our limitations. Of course, it isn’t all bad, given a set of constraints and creatively overcoming them can be as satisfying as coloring outside the lines.

I can’t help but feel a little Sci-Fi on occasion when I see some of the interesting work being done academically or within the DIY community. The explosion of information and accessibility to resources seems to enable those with a mind, and the time, to do so with a bottomless well of potential that, when focused, can provide the maker with something to substantiate their creative vision. Whether it’s the current craze for Kinect hacking, a modular code bending instrument, or simple pleasures of circuit bending, there are communities of people working together to unlock the inherent ability of our modern lifestyle devices. That’s not to say that every hack comes with a purpose, for some the joy is in the deconstruction, destruction, or the creation of something new.

One technique that keeps showing up in game audio is the pairing of an available game engine with a alternative audio engine not generally associated with game audio. Whether it’s the work of Leonard J. Paul using OSC (Open Sound Control) as a bridge between HL2 Source or more recently with Unity, Arjen Schut and his experiments with HL2 and Max/MSP, or this months featured Audio Implementation Greats spotlight: Graham Gatheral, I can’t help but see the resourcefulness of a few brave hearts boldly moving forward to fill a gap in the current functionality of today’s game audio engines.

I had a chance to talk with Graham and discuss his work using the Unreal Development Kit and OSC in conjunction with SuperCollider, an environment and programming language for real-time audio synthesis.

DK: Hey, nice to meet you!

GG: Yeah, you too!

DK: You’re in Vancouver, is that what I see?

GG: That’s right yeah, I moved here last summer with my partner who got a job over here, so I’m over here waiting for my Visa to come through, and I’ve got a lot of time on my hands which I’ve been devoting to game audio.

DK: Fantastic, and doing some really cool things working on good mods, did you do some university and graduate?

GG: That’s right, I did a Masters degree in sound design for the screen at Bournemouth University in the UK. That was about 6 years ago, and after that I worked in post-production sound for film and TV. I’ve been doing bits of freelance work since then, been making music doing live improv electronic gigs, working as a webmaster, and teetering on the fringes of game audio for a bit of that time. So yeah, that’s what I’ve been up to.

DK: Yeah, when you say teetering on the edge of game audio, it’s like, the stuff you’re doing is very edgy. The work with integrating SuperCollider with UDK, it’s cool work, I dig it a ton because it brings together two technologies that are not usually tied together.

GG: Yeah, I’ve been using SuperCollider for live performance since 2004, so I’m reasonably OK coding in SuperCollider. When I started in game audio seriously there was one [UDK] project in particular where I thought “surely I could do this more easily and more effectively in SuperCollider – wouldn’t it be great to hook up UDK and SuperCollider somehow?” So that’s kind of what I’m doing.

DK: It’s a little bit sci-fi for people who are not used to that, again you kind of came from linear and film, but your education then bred you a bit on the SuperCollider side of things and marrying that with UDK did you find that there were already things in place that would let you do that?

GG: Absolutely yeah. When I had an idea to do this I started looking around and obviously did a bit of Googling, and I found a project called UDKOSC, which is a project run by Rob Hamilton at Stanford CCRMA department. Basically he’s been doing a fair bit of work over the years with game environments and interactive music; in this instance he’s using UDK as a tool for music performance, sound installations, networked music performances – this kind of thing. So I got in touch with him and fortunately he was able to send me a few scripts and give me some advice get me going. It seemed fairly close to what I was trying to pursue.

DK: Right, and you mentioned having a project in mind that seemed well suited to the SuperCollider workflow. What was it about SuperCollider, or even the idea of procedural or synthesis that you felt like you could accomplish better by marrying those two toolsets that you couldn’t do inside of UDK or instead of standard game audio middleware.

GG: Well my experience is really only with UDK at the moment, I don’t have any real world experience with middleware like FMOD or Wwise, although I’ve obviously had a good look at them. I enjoy working with UDK, I think Kismet is great, you can get a lot done, it’s quite a creative tool, I like the modular approach. It’s obviously limited, so with SuperCollider or any sound synthesis language, I think the main accomplishment is going to be better immersion. I think that if you’re looking at almost infinite varieties of sound for things like weapons, wind, or collision impacts, this kind of thing, and also a closer matching of sound to the visual, it seems like you can get a better hookup between the mechanics of what’s going on in the game engine to what’s happening in sound – a closer marriage there.

DK: Right so, with all of the parameters going on within a game that can be fed over to into the synthesis tool or programming language, you have all of that data that you can use to affect things in real-time, and something like SuperCollider seems like it’s really well tooled for taking in those parameters taking in that data and tying it to aspects of the sound and then manipulating it and working with it.

GG: Yeah exactly, I think it’s super flexible; you can do so much in SuperCollider. You could have, say, thirty lines of code for a metal impact and have a bunch of parameters in there which you could adjust to provide sound for anything from, say, a small solid metal object right up to a big hollow metal object. So in thirty lines of code you’ve wiped out hundreds of assets that you normally have to deal with. So there’s clearly a space saving advantage there and also a time saving advantage for the sound designer I guess.

DK: Definitely, and it just seems like the pipeline between those two is a way clearer delineation, in that they can talk to each other using the same language of parameters and values and it’s already set up to do that without having to be highly modified, SuperCollider just comes out of the box in expectation of receiving all of this parametric data to manipulate sound.

GG: Yeah that’s it, SuperCollider has obviously been around for a while and it can take in OSC, so anything that can feed out OSC can control SuperCollider. This is essentially what Rob Hamilton’s project is, a way of getting OSC out of UDK.

DK: On that note, can you explain to us the pipeline of how that communication happens between game and audio engine, what kind of pieces did you have to put in place and what is the channel of communication between the two?

GG: So effectively UDK OSC is using the .DLL bind function feature in UDK – it’s calling in a function in the .DLL which is compiled from a bunch of OSC C++ classes which Ross Bencina wrote that can send OSC. I guess what you’re doing really, is creating the OSC message in UnrealScript (or getting the bits you want to send via OSC to your client into UnrealScript) and calling the function in the .DLL to get it out as OSC. It’s UnrealScript coding.

DK: Sure, so you’re specifying that in Unreal, and you have to explicitly specify with an Unreal script to be communicated to the .DLL and then converted to OSC that SuperCollider can then listen for and pickup.

GG: That’s pretty much it yeah. You set up a OSC Responder node in SuperCollider and make sure it’s listening on the right port, and you’re away. I’ve made a Kismet node for doing this so I can feed in any values, floats, integers or what have you, via Kismet using this link to the .DLL.

DK: Right, and as OSC…is OSC open source, or a standard of some sort?

GG: Yeah, Open Sound Control it’s an open source (library).

DK: …and this is a protocol (able to be used by) a ton of different applications not just SuperCollider but also PureData, Audiomulch, all kinds of applications that (can) use OSC to communicate information interoperably.

GG: Absolutely yeah, there’s tons of clients out there that can take in OSC that could be used for this application as well I guess.

DK: That’s beautiful to me because now you can get information out of the game, bring it over to an application where you can use that to manipulate sound or manipulate the audio. You’ve got some great examples up on your website at http://www.gatheral.co.uk and I’m seeing things for procedural weapon fires, synthesized creaking doors, the metal impacts that you mentioned…a lot of really interesting stuff using SuperCollider so you’re well on your way to some of that.

GG: Yeah I think so, it’s dependent on how good my synthesis skills get really. I’m still learning all of this stuff and getting better at SuperCollider, but I can see a time when I can get a lot more sounds in for a level in this way. And sounds that don’t synthesize very well can also be played back through SuperCollider, using the buffer. So even linear samples that would normally be played through the audio engine could be controlled in interesting ways. You could affect the rate of playback in a non-linear way, you could chop the sample up and playback only parts of it, or maybe combine tiny parts of the sample to make something that’s always new and interesting. I wouldn’t say that I’m aiming to get every sound in the game to play through synthesis in SuperCollider, but there’s certainly ways to get samples to play back in interesting ways too I think.

DK: Absolutely, you have a lot more tools in the toolbox than a modern game audio engine for manipulating sounds. I guess I wonder, when you are outputting this OSC data, you’re not turning off the audio engine for Unreal, for UDK…you still have that audio engine available to do things with is that correct?

GG: That’s right yeah, so you could use a combination of both.

DK: We’ve seen a few attempts at tying different game engines to different external (audio) implementations, whether it’s Leonard Paul and PureData tied to the Source Engine or Arjen Schut doing something similar. There’ve been a few different applications of this: getting information for a game and using it for an external application to do audio with. I think it’s trailblazing stuff, because as people involved in interactive audio we want the ability to use all of this real time data and harness what’s happening in the game and, kind of like you said in the beginning, to react dynamically to what’s going on in the game.

GG: Yeah exactly, you know it’s just a closer match between the audio and the visual for me. A fairly good example is the creaky door test I did; where you’ve got very small movements of the door it will only make a few creaks here and there – larger movements increase the rate of the creak playback. So rather than playing a sample every time or a bunch of different samples every time the door moves, it’s a much closer marriage between the audio and the visual, and I think that aids the immersion… for me anyway.

DK: Absolutely, and the idea of dynamic…we’re not talking about canned one shot samplers in representation of a dynamic event like this door you’re talking about, for which there’s a great example on your website. So being able to react dynamically to it with sound, it definitely takes you closer to being there with it and reacting with it.

GG: Yeah, it just seems more real. I mean, like I said it’s dependent on how good you are at synthesizing sound; it’s possibly not as real sounding as a sample of a creaking door but I think it’s a trade-off between that and how close the match is between what’s happening on the screen and what you’re hearing. I think that’s a worthy trade off.

DK: Definitely, and I think our models are increasing in their complexity as far as what sounds good as far as synthesis goes. I mean it’s not too far a stretch to take wind, for example, and synthesis the aspects of that successfully or believably. I think that as we move forward with this technology if we focus our attention on trying to make things sound better, they will get better.

GG: Yeah definitely! I was listening to the Game Audio Podcast #4 (Procedural Audio) and someone was talking about how this approach to procedural audio for games would free up a lot more time for Sound Designers to become sound synthesists in a way, and you know, why not? Having more time could certainly lead to much better synthesis and better sound using procedural audio.

DK: Absolutely, the specialization of a sound synthesist. And then also I think, presets with synthesis is nothing new, and the idea that you would open up your game audio toolbox and it would come with a preset for “Creaky wood door” and “Creaky Metal door”.

GG: Yeah, and half a dozen parameters that you can tweak.

DK: Exactly, feed in this data, get out this sound. I think that it would be a good place to start, and if that’s not selling it for your game, if your game is all about creaky wood doors, maybe you need to go with sample content. But if there’s three creaky doors in your game and you can get by with the “Creaky Doors” preset then bingo, it wouldn’t be a bad thing. We certainly have enough to focus on in other areas of sound for games, it couldn’t be bad. It’s cool talking to you about it though because, again, I feel like there’s a big future fort this synthesis side of things with the ability to just turn on a game and have some things out of the box that just work. Whether you want to call them presets or things that come embedded I just think that there’s a lot to worry about when it comes to game sound, and we could use a few freebies.

GG: Yeah yeah, I think it would free up a lot of time to get much more detail into the game as well. I’m fairly new to all this so I don’t know what it’s like, but I’m wondering if there’s a lot of collisions or other aspects to the game that don’t get the kind of time devoted to them that they need for decent sound; I don’t know if freeing up time for the sound designers would allow more time to pursue the intricacies of the game.

DK: One would hope so for sure. I mean, there’s always things that go unscored, at the same time not every door is about the creaky door hinge for example, you have to choose what’s important and put your time and effort into making sure that sings and sells. Whether that’s the Foley in a free running platformer like Mirror’s Edge, or the weapons sounds in FPS of the year, you just have to focus on what’s important for the gameplay, I think that comes above everything else and then if you can add detail to the other aspects of the world that make sound, you kind of just give it the attention that it deserves, or at least hope to.

GG: Yeah

DK: Yeah, dig. I like the direction you’re taking with the project I can’t wait to see more. Every video you release is an awesome thing to behold, I can see the experimentation, I can see the direction you’re going and it’s really cool stuff.

GG: That’s really cool, thanks very much. I should say that Rob Hamilton is going to host the code for UDKOSC on Git fairly soon so if anyone wants to get their hands on it, that’s where it will be. [UPDATE: UDKOSC is HERE!]

DK: That’s gonna be great. What’s that going to include?

GG: It’s going to include some Unreal scripts, the .DLL, and basically everything you need to get going.

DK: Awesome, I’ll be sure to point people to that, and look forward to the experimentation that comes out of it.

GG: Cool.

That’s it for this time, thanks for peering into the crystal ball of future forward interactive audio with Audio Implementation Greats.

Be sure to check out Graham’s excellent Blog as well.

Here’s a round up of related links:

Graham Gatheral: UDK+Supercollider: real-time synthesis for sound effects

Open Sound Control (OSC)

Unreal Development Kit combined with Open Sound Control

Unreal Development Kit

q3osc: an introduction

Q3OSC – Demo1

Q3OSC – Demo2

Stanford Laptop Orchestra

RedFly/Frank Teran

If you talk to anyone in game audio today about successful tempo synced synergy between music and sound effects it wont take long for your discussion to end up at REZ and the work of  Tetsuya Mizuguchi. The quintessential poster boy for synesthesia in video games and a stunning example of overt spontaneous interactive music creation.

But imagine for a moment stripping away the throbbing electronica pulse and replacing it with an organic instrument-based soundtrack created by one of the foremost prodigy of curiously inspired noise making bass thumpers, with sound effects locked to a groove oriented metronome, and you’ve got makings of a monster.

“Whereas many games today occupy free-formed soundtracks that respond entirely to the player, Mushroom Men is recorded to a beat. “You have sparks sparking in time to the music, and there are moments when the background music backs out and you hear the cricket cricking on beat,” says Jimi Barker, another sound designer with Gl33k.

“Piersall continues, “You want to make it seem like the world plays to a beat.””

-Austin Chronical Article: Making Mushrooms Dance

Click here to view the embedded video.

This brief overview sparked a fascination in me regarding the potential for experimentation in the realm of reactive and interactive music. I was lucky enough to end up down in Austin, TX for the 2009 Game Developers Conference and had a chance to connect to the fine fellow’s at Gl33k, the audio house behind the sound for Mushroom Men, The Maw, Comic Jumper, Ghostbusters (Wii), and more recently Epic Mickey. Not only did they unlock the secret door to the Austin underground, they also proved to be unmatchable hosts and a great group of guys with their sights firmly set on revolutionizing outsourced audio.

I had a chance to chat recently with Matt Piersall, Pres and New School Beast Handler at Gl33k, about everything from Drinks at the Death Star to syncopated bee buzzing. The full audio proceedings of this meeting of the minds can be found at GameAudioPodcast.com and a transcription has been provided by esteemed game audio professional Michael Taylor.

[DK]: Damian Kastbauer
[MP]: Matt Piersall

DK: Mushroom Men, man, that was a beautiful thing with regards to the way that the music came together, and that was the first title for Red Fly. Can you talk a little about how you guys got involved with that project, and what the first steps ended up being?

MP: Yeah, basically the way that we got involved was, my very first game job was at Terminal Reality as a Contractor. A guy there that (has since) left, his name was Jeff Mills, he was a Level Designer. He left Terminal Reality. They did Ghostbusters, and have tech they built called the Infernal Engine,  which we still use on some projects, and is pretty cool. What ended up happening was that I ran into him, like four or five years later at GDC. I had a drink with him at, you know that Death Star bar.

DK: Yeah! At the top of the Marriott?

MP: Yeah, we had a drink there, and I gave him a USB key demo, and was like, I really wanna…I saw the project, and I thought it was gonna be really cool. It just seemed like something I always wanted to do, in terms of establishing a new IP. Everything I had worked on up to that had had been a sequel;  a two, or a three; something was wasn’t as completely creative. That’s how we initially got involved.

I love sound design, and it had always been a goal of mine to work really closely with a developer. Basically what Gl33k does now is support developers with outsourced implementation, sound design, and audio direction. I sent in a bid to Red Fly, and they were like “Oh, can we get some music for this amount of money, because that is basically all of our budget for sound” and I said: “Yeah, that sounds really cool. I’d love to make some music” They seemed like a young enough studio, open to new ideas so I was like “ Yeah, lets make some music”

So, we ended up doing a couple of tracks, and there wasn’t a lot of movement at first, and then they came down to Dallas, where we were originally located. When they came down to Dallas, we got margaritas, and we started discussing ideas. I started talking about how I had always wanted to do a musical video game. Not as in playing music, but as in, literally a musical, people are singing and stuff like that.

DK: Like ‘Westside Story: The Videogame’?

MP: Exactly! [laughs] So, we had this long talk, and everybody got kinda tanked, and they agreed at that meeting the we should have this Metronome system. After that meeting, months went by, things were very slow moving. They didn’t have a way to get us builds in Perforce, so they had to package them up (via FTP). This is really early y’know! We were downloading 4-5Gb of data, and not understanding how things worked at all, and really having very, very little communication with the team.

We came to (their studio up in) Austin one time, and I did a mock-up: I captured some footage from one of the builds that I had, and I made some music, and I made an environment noise in Ableton, and the environment noise was rhythmic. So I did all this all this rhythmic stuff, and then I comped a video together, and I show them and they loved it. They were like “Alright, cool. We’re definitely gonna move on this, and make this metronome thing a reality.”

In that same meeting, they said; “You guys should move to Austin, what are you doing in Dallas?” I was like; “You know what? That’s a really good question!” At the time, all of our clients except one were either in Austin, or LA . We had five months of work, and I was like “Let’s move to Austin!”. In retrospect, that was really crazy. I’m trying to be more, I guess careful, so I’m trying to plan six moths out,  if I don’t know what’s going on in six months, I start to freak out!

DK: Awesome. And that kind of positioned you closer to Red Fly, your studio is right next to their development house now, and kinda that’s how you stepped into that pond up in Austin?

MP: Yes, actually Austin has a lot of developers, maybe around fifty or sixty? And its helped us out a lot. We always work with Retro Studios, we work with Disney… they’re all in town. I think where we are there are four developers on the same street.

DK: Not to jump the shark on this, but as an outsourced audio company, you have this autonomy and location (theoretically) becomes less of an issue. But I think that, maybe where your driving with the Mushroom Men story, the move to Austin, and your proximity to Red Fly, it must have really helped with development.

MP: Well, at the time, just to be completely real about our experience level at that time, we had mainly delivered assets. We delivered them kinda over the fence, like a lot of sound houses do it. The proximity thing was completely necessary. I believe that you can be remote and do a great job, because we’ve had to do that before, but what you don’t have at a distance is that you don’t get the same camaraderie with the team. You have to have a lot of face-time, even if you live thousands of miles away. In this case, I loved Austin, and always wanted to live here, it just has a cool life to it, and so many developers. The game community and the development communities here are really active. I don’t think we could have accomplished what we did had we not moved.

DK: Sure. It sounds like you could pass builds over the wall, you could step next door and talk with the guy who was doing the wiring under the hood, because you were not using any middleware for Mushroom Men, it was all proprietary Red Fly developed action.

MP: Yeah, it was the old Infernal Engine. It’s been updated a lot since and is really great. The old engine, from a level design standpoint was pretty cool, but from an audio standpoint, we didn’t even have things like pitch randomisation. It was very, very bare bones. But when we got here, it was great. We had Perforce access, and everything happened so much faster. We could attend meetings, and meet some of the guys doing animations, camera, effects.. We just sort of became a part of their team.

DK: Right on. From that point, once you had moved across the state, how did things come together technically? Over drinks you had decided this was going to be a metronome based system, that had repercussions that cascaded throughout the entire game design, because now you are not only syncing audio, but you are also syncing visuals, animation and VFX.

MP: The reality is that I wished that we had used it in a more extreme way. We got the functionality, and from the audio side we used it all over the place, and we did a lot of experimentation with that. From the design side, a lot of people used it, but it was so subtle that you barely notice it. In retrospect, I would have said we should really play it up with camera cuts, and UI elements coming in on beat, which those do actually, but its super subtle. Animation on beat was a little more difficult, because they had to build the animations to a beat. Particle effects were fairly easy, because we had emitters that would fire them off on beat. Essentially the whole system was kind of like a sequencer. Kind of like an 808, it would listen and fire off events.

I look at Mushroom Men as a great learning experience. We worked on it for about a year, but we did most of the work after beta. We had nine months of preproduction, then we worked for three months! [laughs]

DK: That sounds about right! Was there quantisation used as those events were fired off? I mean, did they have to do a lot of juggling of those events in order to line them up? That’s got to be part of the system, right?

MP: Yeah, its totally part of the system. The way the system worked was that you would have a stream. The stream was your backing track, that was the main music that made everything fall on beat. So we started that paused, and then we waited and we kicked out an event to everything. Everything would update every second on a tempo of 120 BPM to recheck that is was still on beat. It was really cool, and I wished we had used it even more. I ended up getting in a little trouble, I took too much liberty and snuck in and synced some things that weren’t originally synced. You just kind of had to do that. It ended up making the end product cooler for sure.

DK: You bet! Can you give us some examples of things for people who may be picking up the game for the first time after reading this article? What kinds of things should they be listening for that you think exemplify the system and its capabilities?

MP: Well, actually there are about three or four things that I’m into, and I hate everything else. [laughter] Ok, so there’s these bees in one of the levels, and we made a track that was the theme for the bees that was like [sings/buzzes theme]. We tied this into the bees particle effect so that its in sync with the…[buzzzes like a raver bee] So yeah, the bees are buzzing on beat, which I thought was really fun. Actually, every ambience in the whole game is rhythmic. I took everything into Ableton and made all looping, streamed ambience rhythmic somehow, even if its not real obvious. You know how in Ableton you can do these ‘stutter’ edits? It’s really subtle, but we did a lot of that with volume ducking for the streamed ambience., and we kicked off wood creaks and crickets and all the insects you hear which are making a beat, and every single localised and spatialized emitter based ambient sounds are on beat too. The only sounds in the game that aren’t on beat are the enemy fighting sounds, because they didn’t want to change their game design too much to support this Metronome thing, because that was not what the game was originally pitched as.

DK: Totally. So is this a good point to segway into talking about how you roped in Les Claypool to lay down some magic on this? This is maybe a side topic to the actual Metronome system, but it plays a huge role, and I feel like its one of the few crossover examples of popular music contributing to games in a very unique way.

MP: Yeah, totally. The way Les Claypool got involved was through the guys at Red Fly, who are huge Claypool fans. They got some cash together and approached Les and said “Hey, do you want to do music for this video game?” I don’t think he was that interested at first, as I don’t think he digs games that much, but his son loves games, and convinced him to do it. Actually, his son, who is like seven or eight, actually plays on the Mushroom Men soundtrack. I think his daughter does too, I think they play percussion on it, which is really cool. But basically he ended up getting involved, and I met him, his wife and his kids and we talked about what we wanted. I didn’t really feel like I could give him much direction, but I said “Hey you’re gonna do what you do, but if you could just write everything at 120BPM that would really help me out.” and he said “Sure, no problem”. In addition to that, I also requested stems, because you have to have that. We had written a bunch of themes at Gl33k, and we had done this one thing that was really, really fun, and I would love to do this on a bigger scale. We went to this small college town in Texas called Denton, and we recorded with this band called Midlake. So we just holed up in their studio for three days, and just recorded a ton of material. And so we went and remixed some stuff and got some tracks out of that material. Then we got Les’ stuff, and we remixed that and mixed it with what we already had and oddly enough it all kind of worked.

DK: That’s such a great story because I think what you’re talking about is angling toward the way a lot of game music is coming together these days, with regard to having composers composing music, and then having to rip it apart for the various systems and remixing it to achieve the different intensity levels for instance.

MP: Absolutely. Its cool too, because I could never do what Les Claypool does, and I couldn’t do what the Midlake guys have great guitar players, a great drummer and a great keyboard player, and they added some little bits in there. Basically what I ended up being was a remix artist, which is more my background; electronic music stuff. Basically Mushroom Men is a huge remix album, with a bunch of random sources.

DK: Remix and Orchestrator, while your designing and interactive orchestrator of sorts, which the Metronome system is, you have to be the maestro of that system and build content to support that. I think that is a great concept and a great analogy for the way a lot of game music comes together these days

MP: Yeah, it’s really cool, and is kind of exciting to me. I’ve been guilty of it in some of the games I’ve been involved in – the wallpaper music. Sometimes you can’t get away from it, because time is always an issue. And if you don’t plan for it,   or set out to say you’re gonna do this crazy interactive music thing, then you’re not going to accomplish it. It’s not one of those things where during alpha you’re like “Oh by the way, we need a intensive dynamic music system.” I love Wwise so much, because you can kind of fake some of that if you get in late in the game. But you still need to sit down with your programmers and the tech people and get a cool system in there earlier or it’s not going to happen.

DK: Right. It’s the two-way street of communication. In order to be interactive music, it has to interact with the game. If the game isn’t sending it information to react to or interact with, then you can’t truly have interactive music.

MP: Exactly

DK: I hear you. You have to plan for it. It has to be part of the design, and I think that when you do approach it holistically like that as part of the design, then you can better communicate that to the composer and take it to the next level with through remixing to make sure it hits all the right spots.

MP: Definitely.  One of the biggest things that I found remixing did for us was that it just gave us so much content, and that’s one of the things that makes it work. It was easy to make new content, a new track would take only a couple of hours rather than produce a minute every day, or two days.

DK: Do you ever see re-visiting the Metronome, and the idea of beat-synced sound? Is there a place for it?

MP: There’s absolutely a place for it. I really wanted to explore this for Epic Mickey, but I got involved a little to late to really push forward with that. Hopefully, maybe on a future project we could maybe do something along those lines.

I think the place for it is that it’s not just for cartoon stuff, or weird, psychedelic worlds. I think you can use it in a lot of ways. You could use to enhance any moment. The gamer doesn’t necessarily always want everything on beat, but even having something on rhythm in a super realistic, really intense first-person-shooter could be really effective if it’s used right.

Exploring new systems and concepts like these is one of my passions, and keeps me interested in this business. The reason that I got into game audio was that I felt that it was a way to let people experience this creative outlet that you have, but you don’t have to be there. And so, the more creative and interesting the system, and I guess I’d call it ‘audio-driven design’  where its not necessarily that the audio drives the entire design, it’s just that the audio provides a bigger role than just a supporting one, like supporting player feedback. There should be bi-directional communication; the audio should communicate the the game, and the game should communicate to the music. Right now, a lot of games are starting to do it, and the quality bar of that has risen so much.

DK: Right, and at the end of the day, you still have to serve the game, and the style of game. But also having this two-way street of communication to allow for greater interactivity. I think we’re taking slow steps towards that.

MP: To be completely open about the Mushroom Men stuff, we just scratched the surface. We had the chance to do something kind of interesting, and it turned out to be pretty cool. I really loved working on it, and working on a system like that, which ended up being kind of a feature. We got to do some cool stuff with ‘Splosion Man too, so we’ve had a few chances to do some cool things. It’s not always an easier sell though, because sometimes they call audio teams only when things are on fire [laughter].

DK: [laughter] I saw that coming a million miles away! That’s when they call all right!

MP: I can understand though, you don’t want to waste a bunch of money on an audio team if you aren’t ready for them yet. But at the same time, I’d rather have a small team the whole way through, and then go from two people to twelve people at the end of a project.

DK: Beautiful thing! This has been a great talk about Mushroom Men. I feel like we covered a ton of space on it, and it’s great to hear about it all. It’s such a unique moment in time that you captured through both the implementation, the remix and the composition of Les Claypool’s work and that synergy that came together for Mushroom Men that was extremely unique in my opinion.

MP: We did some cool stuff, but if there’s anything I’d take from it, it’s that it’s a way that I like to work. It proves that you can outsource and bring in specialists to do certain things for a project. It basically proved that the way I like to work is viable from a critical standpoint.

DK: For sure. And creatively satisfying?

MP: Oh my gosh, yes! I miss that part of it. It was the most fun I’ve ever had on a project. It was a great time of life, we had just moved here, I lived across the street from the studio and there are 90 bars within a square mile! Great times!

Gameplay footage focusing on synced ambient sound @ 6:20 and with music @ 8:00:

Click here to view the embedded video.

Gameplay footage focusing on Bee combat:

Click here to view the embedded video.

Stick around after the interview for a discussion on the late Austin GDC audio track, AES Game Audio, Footsteps experiences, and general banter between friends.

So this is the New Year…

The turning of the new year marks the one year anniversary of both the Audio Implementation Greats Series here at Designing Sound, as well as the Game Audio Podcast which I co-founded with Anton Woldhek around the same time. Over the course of the year we’ve covered topics that are relevant to professionals working in the industry such as: Footstep Design, Interactive Music, and Procedural Audio, and will be pressing into the new year with exciting new topics.

Beginning in 2011 we’ll be kicking out a fantastic discussion about Interactive and Dynamic Mixing with David Mollerstedt (DICE), Rob Bridgett (Radical Entertainment), Kristoffer Mellroth (Microsoft Game Studio). It’s out hope that these will continue to fill in the relevant gaps of information between other articles, dev diary’s, and interviews out there and bring some new insights into the art of interactive sound.

If you have any examples of outstanding audio in games, or interesting techniques being used, please drop a line!

Special thanks to Michael Taylor for transcription of the interview which can be found over at the Game Audio Podcast. “Michael Taylor is an audio designer, composer and proficient tea-boy and can be found at www.stomp224.co.uk “

ShroomMan & BossShroom Artwork © Aaron Armstrong

When embarking on a sequel to one of the premier tactical shooters of the current generation, the Audio team at Red Storm Entertainment, A division of Ubisoft Entertainment, knew that they needed to continue shaping the player experience by further conveying the impression of a reactive and tangible world in Ghost Recon Advanced Warfighter 2 Multiplayer. With a constant focus on creating dynamic, real-time, multi-channel systems to communicate information using sound, they successfully helped to create a world that visually reacts to sound and further anchors the player in the world. Their hard work ended up winning them the 2008 GANG “Innovation in Audio” award for the GRAW2 Multiplayer implementation of an audio system that turns sound waves into a physical force used to drive graphics and physics simulations at runtime.

The Audio Implementation Greats series continues this month with an in-depth look at the technology and sound design methodologies that went in to bringing across their creative vision. We’re lucky to have original Audio Lead/Senior Engineer Jeff Wesevich and Audio Lead / Senior Sound Designer Justin Drust laying out the detailed history of what promises to be the most extensive overview of the widely discussed feat of implementation greatness: the Ghost Recon Advanced Warfighter 2 Multiplayer: Wind System.

Hang on to your hats, and catch the breeze!

The Set Up

JW: “I think the thing I like the most about the wind effects modeling system we came up with for GRAW2 MP is that the idea behind it is so simple–everyone gets it on the first take. To wit: the amplitude of the currently playing wind wav files are converted into a force that can push particle systems around or into a force that a Havok physics simulation can work with. We add a semi-random direction to the force to keep things looking natural, but that’s the whole shootin’ match in just two sentences. It is astonishingly simply to create complex, difficult solutions, but the simple ones–man, that takes work.

The genesis of the system came from a specific multiplayer map. In this scenario, a C5A Galaxy (one of the largest airplanes ever made) has crashed in the middle of a huge thunderstorm and burning chunks of the thing are splattered the length of the map. Great idea, but with that much intense fire and smoke spread out across 1600 sq. m., we quickly exhausted our basic flame and “burning stuff” audio sources. And truth is told, the particle guys were having their own problems with the fairly strict limits on the number of different particle systems they could deploy. Things were looking and sounding pretty monotonous, instead of dangerous and exciting.

We knew we had a basic output meter provided as part of the X360 XDK, and we knew that the sound of the flame would have to be influenced by the howling wind, so our first cut was to simply attach a Matt-McCallus-optimized version of the RMS meter to the wind mix bins, and then use that output to change the volume of our flame samples. Not bad, but not ready for prime time either. We then created a type of emitter that would allow us to layer specific fire effects at a given place; triggering their play and level based on the wind’s RMS sum. This allowed us to mate a big whoosh with an accompanying wind gust. Once the sound designers started playing around with this, things got interesting.

We were making great progress, but now we had audio that didn’t sound at all like the static, looping particle systems sprouting all over the map. Well…could we change the flames and smoke the same way we were the accompanying sounds? Sure enough, there was a parameter in our generic particle system that allowed for wind to deformation, but of course this hadn’t been used before, and even the Art Director couldn’t be sure if the systems had been constructed with the parameter in mind. Fortunately, the first couple we tried were done that way, and it proved very simple to take the wind amplitude and update the particle systems in real time. At that point, we added the direction component, tweaked it, and suddenly we had huge gusts blowing flames all over the place.

We were really excited at this point, but even I was concerned about how this would go over with the artists–what do you mean everything will play back differently in-game than it does in our tools? How do we debug our work, etc., etc? Surprisingly enough, they loved it. Unbeknownst to us in our little corner of the world, the particle guys had been struggling big-time with issues like short loops and repetition; and their only real tool to combat this was to add more particles or systems to a given effect to make it appear moving and somewhat random–which of course hurt performance. (This should sound familiar to the audio guys out there). And then suddenly someone shows up at their doorstep with a gizmo that dynamically perturbs their systems–in a naturalistic way–for free. We weren’t a threat–we were problem solvers.”

GRAW 2 MP Dynamic Wind System

Click here to view the embedded video.

The Execution

JD: “Building on what Jeff has already explained, our environmental audio design contained three base wind layers, Breezy (being the lowest), Windy, and Gusty (being the highest). These wind levels were setup as zones within the mission space and could be cross faded between each other and respond to elevation; climb a mountain and transition to a higher wind intensity. Each base wind layer also had three overlay layers that would randomly playback to enhance the bases. The base layers were designed to be constant 2D looping stereo files while the overlays (also 2D looping and stereo) contained gusts and swells, providing a realistic alternating wind experience.

These varied wind intensities gave us the ability to calculate different RMS levels via playback which became the fuel for the dynamic wind system. While in pre-production we knew specific maps were going to call for higher wind intensities (such as the Crash Site level which mimics a intense wind storm and the crashed C5-A airplane mentioned above) so we knew in advance that specific levels would call for specific wind intensities.

As Jeff explained, we wanted to harness the wind to allow us to do more with actual in-game audio emitters and help bring the once stagnant multiplayer audio environments to life. For this we created a special emitter class that could be triggered to play or not play based on the RMS output of the wind. To achieve this, each “wind emitter” had mapped RMS values that would trigger specific playback time intervals based on specific RMS amplitudes. For example, increased wind RMS = increased playback frequency. These settings would be set by the sound designer on a per emitter basis within the mission space:

Example: -6 RMS = playback time interval of every 2 – 6 seconds (the stronger the wind = increased playback frequency of the emitter)
Example: -40 RMS = playback time interval of every 20 – 30 seconds (the weaker the wind = decreased playback frequency of the emitter)

Each wind emitter placed in the world had properties like those shown above that could be set based on the structure, location, and type of sound asset. This way we could simulate a low creaking sound of a building settling after a large wind gust by setting the properties to trigger playback at a decreased frequency when the wind was at its lowest amplitude. When the wind picked up, an entirely different sound emitter of a shutter or door rattling could be triggered to play during the higher wind amplitude and more frequently, thus creating the effect of the wind physically moving the shutter or door and responding to the gusting wind.

So now we were not only adjusting the playback frequency to match the wind intensity, but also alternating between sound assets to provide even more immersion.

These audio emitters would then be placed in specific areas of the maps or structures to enhance the environmental audio of specific features. For example, a building could have anywhere from 2 to 3 separate wind emitters (with typically 4 to 8 assets in each) placed on or around it that contained different audio content based on three levels of intensity, High, Medium, or Low.

High emitter content contained more action within the sound assets, heavy creaks, rattles, etc. Low emitter content contained low end groans and settling sound assets, while Medium was the glue that helped bridge the gap between the extreme amplitude sounds being triggered and the lower amplitude sounds. This system resulted in a very immersive and dynamic atmosphere that was all based around the RMS output of the wind assets. When the wind system gained or decreased intensity, so would the effected audio emitters. (We also had a special case “after the fact” emitter that could be triggered to play after a lull in the wind and was typically used on very large structures).”

The Sound Design

JD: “As you can imagine there were a large number of audio assets created in order for this system to work properly. Each emitter group alone would typically contain anywhere from 4 to sometimes 16 assets in order to add more randomization to the system. We spent a lot of time recording and editing sounds that would work within these levels, everything from the obvious fences, shutters, doors, dirt, to even large church bells. Once all of the assets were recorded and implemented into our audio engine, it was then time to create the emitters and set the properties. A great deal of time was spent auditioning the levels during development to make sure the playback intervals were set properly in order to maximize the results of the system, and of course fit within the overall mix.

In turn this spawned creating wind emitters not only for specific structures but graphical effects as well. Fire for example, also had multiple audio emitters that would randomly playback pending the wind settings and RMS. For this we triggered specific fire sound assets to play when the wind intensity was at its greatest, creating a bellows sound when the wind would gust. When a player walked past a burning jet engine or building on fire, the gusty wind overlay would trigger the fire to react to the increased amplitude. Tie this in with the actual fire particle effects growing along with the wind intensity, and voila, it all came to life.”

GRAW2 MP In Game Sound Design Montage

Click here to view the embedded video.

The Physics Bonus Round

JW: “It was a short jump from there to swapping out the random tree sway to work with the wind input, and then from there to playing with physics. Since this was 2006, we already had a pretty fair soda can effect to handle the can or two the artists put in the maps, but you only heard it after you actually shot the can. It was pretty simple to attach a sound listener to the Havok object code, which would translate our wind force into the Havok system, and have it act on the attached object. While not at all useful from a game-play or immersion standpoint, it was still great fun to watch a gust of wind knock a can over and then roll it underneath a car.

And as it all turned out, there wasn’t a lot of effort required, and it cost almost nothing at run time to get perfectly synched effects and sound. While pushing coke cans around is pretty trivial, it did show us that the concept was very practical for things like monster’s voices, or other effects that could look very cool if the sound produced could be directly translated into a force that affects the appearance of a game’s environment.

For example, Matt was able to quickly create an effect for helicopters…These were an interesting challenge, because the designers had just added a new game mode called “Helo Hunt,” which was proving to be very popular among the dev team members. Suddenly the bad guys had more ‘copters than Col. Kilgore and the 1st Air Cav… The initial piece we hooked in was a downward force from the blades that would affect particle systems. There tended to be a lot of burning buildings in the MP levels, so watching a helicopter come through a screen full of smoke, pushing the columns around in a fairly realistic way was fun to watch, as was watching the smoke gradually reconstitute over the fire below.

Our levels also had trees throughout and they proved to be good places to find cover. Since we didn’t want you to feel completely secure, we added a force to the blades that would push the foliage around underneath the rotors. Hovering helicopters in a search pattern got a lot more exciting throwing the tree limbs and leaves above your position around while they hovered nearby. And it gave you a lot more information on the helicopter’s position than just the audio alone. Overall great fun to do, but there was a lot of tweaking—our initial cut at the feature made the place look more like a hurricane than anything else…

The Wrap Up

JD: “In conclusion, this seemingly all came about by searching for ways to enhance the Multiplayer environmental audio experience within our levels. Having various wind layers and overlays helped to tie the entire system together opposed to just playing a static 2D wind background for each level. Keeping the system as robust and dynamic as possible was the key to its success. By creating such a dynamic system, it was easy to keep file counts at a minimum and still achieve outstanding results due to all the various dynamic elements involved. The wind had three distinct bases, with an additional three overlays, which would trigger 3D emitters that contained multiple variants, with three additional levels for various wind intensities (High, Medium & Low), which could all be set by the sound designer. It was a robust, but intelligent system, which helped bring life to our audio environments.

Many kudos to the team at Red Storm / Ubisoft for allowing us to experiment with this system and putting up with us when we’d break the build…I promise that only happened a handful of times.  ”

JW: “One final note – while we did have quite a few things fall into place for us to be able to implement these systems in the time available, the fact that this all happened at Red Storm was no accident. RSE’s engineers and artists were always ready to try new approaches, and more importantly, were always ready to give audio a chance to shine. I will always be grateful for the strong collaborative environment that allowed us to play with the possibilities. Thanks guys!”

While it has become common to affect audio using parameters coming from various parts of the game engine, like passing materials for footsteps or velocity for physics impacts, it’s not often that we see the audio engine used to manipulate game side data. By harnessing wind volume to drive visual parameters in the game, the GRAW2 MP team not only added character and realism to the different systems, but they also succeeded in bridging the gap between the audio and game engines. With most areas within the game industry reaching for greater and more realistic interactions which utilize every aspect of a games underlying simulations, we could do with a bit more cross engine communication in order to make audio an integral part of the equation. With shining examples like the GRAW2 MP wind system to stand as a testament to what can be done, hopefully the road ahead will be a 2 way street with audio helping to drive us forward into the next generation!

This months Audio Implementation Greats series returns with the overarching goal of broadening the understanding of Procedural Audio and how it can be used to further interactivity in game audio. While not specifically directed at any one game or technology, the potential of procedural technology in games has been gaining traction lately with high profile uses such as the Spore Procedural Music System. After reading this article it should be obvious that the idea of real-time synthesis and procedural audio in games is something I have a great interest in, and this article should be taken more as a call to arms than as a critique of our current state of affairs. In the current generation of consoles we are deeply indebted to the trailblazers who have gone before us, and I feel that in acknowledgment of the history of game audio we must do what we can to build on past accomplishments and use everything at our disposal to increase the level of interactivity within our medium.

I can’t wait to hear what’s in store for us in the future!

Today’s article is also being released in conjunction with the Game Audio Podcast Episode 4: “Procedural Game Audio” which brings to the table Andy Farnell, Francois Thibault, and David Thall, who all work in different areas of the gaming and audio industry. What starts out as a definition of procedural audio eventually ends up as a speculation of sci-fi proportions. We discuss, among other things, the role of interactive audio and how it can be used to support immersion in games, how to employ already existing systems in order to more accurately reflect underlying simulations, along with suggestions for moving forward with procedural in the future. It is an episode that has been along time in the making, and Anton and I both hope it will ignite a spark of inspiration for those of you who are interested in what procedural has to offer.

With that in mind I encourage you to explore all of the different materials presented: this article, GAP#4, and the collection of procedural related links at the LostChocolateLab Blog.

I look forward to the continuing discussion!

I pains me to have to be the one to break it to you all but, from where I’m standing, the year is already over. As the past months spiral behind us in an unending blur of tiny victories and insurmountable setbacks it’s impossible not to feel like time is rapidly accelerating towards the inevitable end of the year. Having just completed and rolled off an epic product cycle has forced me to immediately downshift and reflect on what we were able to accomplish and, when it comes to bigger picture questions in game audio, what is left to accomplish.

Call it “taking stock” if you like but from where I’m standing, interactive sound still has a long way to go towards delivering on the promises of it’s past successes, and even more work catching up to the current state of affairs elsewhere within the game development pipeline. I’m not prone to complaining, and would rather just get on with the task at hand, but there are times when I feel like audio has resigned itself to sit quietly while the visual aspects of our industry struggle to bridge the uncanny valley.

That’s not to say there haven’t been advancements in game audio in the current generation. As each successive sequel or variation on a theme rushes out the door to greet players, we have quickly made the move towards unifying the quality of sound between: cinematic, fully rendered cutscenes, and gameplay. Where once we had to accept the discrepancies between full frequency linear audio and it’s slowly maturing (and highly downsampled) interactive counterpart, we are now fooling even the best of them into thinking that there is no difference. But in our quest for this unification I feel like we, as an industry, have neglected some of the strengths of our medium, namely: interactivity.

It’s become common practice during “Quick Time Events” (QTE) or linear “In-Game Cinematics” to stream in linear audio to support or replace the scripted action on screen, thus guaranteeing that sound be communicated appropriately every time. While this is in direct support of the Film vs. Game convergence concept, it also effectively removes randomness and variation from the equation resulting in a more consistent but less surprising experience. Every time the sequence is repeated it will sound exactly the same. Every footfall, sword impact, explosion, and dialog line reproduced in stunning 5.1 meticulously as designed each and every time, over and over. If the other big concept constantly bandied about in the industry is “Immersion”, then there is nothing that will break that faster than repetition. I’d rather opt for a more consistently surprising experience than risk taking the player out of the moment.

The point I am trying to make is that, during this race for convergence, it seems that we have left to atrophy some of the fundamental tools and techniques utilized during previous generations of interactive audio. Things that could potentially bring our current quality level in line with our ability to manipulate sound in a way that is directly reactive to the drama unfolding on screen.

A brief divergence: Once upon a time all was synthesized in the world of game audio, and everything was made up of sounds – skronking, bleeping, & blarping – out of arcade cabinets and home game consoles everywhere (the origins and development of which have been lovingly detailed by Karen Collins in her book Game Sound). From those synthetic beginnings, and through the transition to PC’s and MIDI based sound and music, we arrived at CD-ROM and then consoles that supported the playback of CD-quality sound. For the first time we weren’t limited to the textures created in code and, in the rush to better approximate reality with higher quality sample based sounds, we never looked back.

While no one would argue that this first phase of sound synthesis in video games resembled the sound of the real world, at that time there was a greater ability on the part of the programmer/ sound designer/ composer to leverage gameplay and the game engine in order to dynamically affect the sound being output. Each generation begets it’s own tools built from scratch to support newly unlocked functionality and to support hardware that has never before existed. As each new platform enters the market developers are stuck in an endless loop of constantly evaluating new technology and re-evaluating the old. The road between then and now is littered with implementation, toolsets, and processes that were previously used to connect audio to gameplay. In the beginning, synthesis was the foremost methodology for sound in games but advancements in that area fell by the wayside as sample based technology took over.

However, work on synthesis outside of game audio has sped forward in other areas where resources are more readily available and computational restrictions are less of an issue. There has a much ground covered in the convening years in all areas of synthesis, for example: music production. We have arrived at a point where a quantity of music written for games (and otherwise) is created with a combination of samples, synthesis, procedural techniques, and sound modeling. To a degree, the difference between a real orchestra and a synthetic representation of an orchestra are starting to become indistinguishable. In short, we are no longer bound by the limitations of synthesis outside of games, but in game audio we’ve for the most part abandoned synthesis as a tool to be utilized at runtime.

I’m wondering, as I sit in front of a display crammed with magical audio tool functionality, why I feel like this is the first volume slider (or fader) I feel like I’ve ever seen in game audio that even remotely resembles the bling-d out interfaces of our fellow sound compatriots over in the world of Pro Audio? You know what else is great about it? It measures volume in decibels (dB), not some programmer friendly 0 -1 or 0 -100 scale that has nothing to do with the reality of sound; and I think to myself “Have we finally arrived at the 1970′s of Game Audio?” Which should go along way towards framing our current situation in relation to that of our audio associates outside of games.

While direct comparisons to other disciplines may seem unfair, it’s hard not to look to other disciplines and wish for some semblance of standardization to hang our hat’s on. It would seem that due to the speed at which things change in the games industry, and the constant need for evolution, we have instead opted to lock ourselves away in isolation each working to find the best way to playback a sound. In the rush to move the ball forward individually we have only succeeded in bringing it halfway towards what has been realized for sound outside of games.

During my time with Audio Middleware I have been constantly relieved that we no longer have to tackle the problem of how to play a WAV file. If any kind of standards have emerged it has been through the publicly available game audio toolsets of the current and past generations. The coalescing of all the different implementation techniques and methodologies are represented across the different manufacturers. Several best practices and lessons learned across the industry have been adopted; from abstracted Event systems, bank loading, basic synthesis, interactive music, and in some cases the beginnings of Procedural audio. Middleware is leading by example by providing a perfect storm of features, functionality, accessibility, and support that has helped to define exactly what implementation and technical sound design is right now. This is not to say that proprietary toolsets shouldn’t exist when necessary, but that our drive to stay proprietary is holding us back from working together to raise the bar.

Aside from whatever whizz-bang sound gadgets we might wish for any given project, at the end of the day, the single most important role that audio must play is to support the game being made. In addition to that, If we as an industry are truly attempting to simulate (and manipulate?) parameters based in reality – physics, weather, AI, etc – in our medium, it would make sense that our advances in audio should mirror the randomness and variability of the natural world. Increasing CPU power and exploding RAM spec’s are all steps in the right direction but unleashing that functionality calls for a drastic rethinking of how to keep things dynamic and in constant action/ reaction to the variables happening on the gameplay and visual side of things. One of the ways to further this relationship is by embracing the strengths of our history with synthesis and taking a page from the exciting work going on in procedural audio outside of games.

Let’s pause here for a definition of Procedural Audio:

“Procedural audio is non-linear, often synthetic sound, created in real time according to a set of programmatic rules and live input.” – “An introduction to procedural audio and its application in computer games.” by Andy Farnell

And a more specific definition regarding it’s application in games:

“Procedural audio is the algorithmic approach to the creation of audio content, and is opposed to the current game audio production where sound designers have to create and
package static audio content. Benefits of procedural audio can be numerous : interactivity, scalability, variety, cost, storage and bandwidth. But procedural audio must not be seen as a superior way for creating game audio content. Procedural audio also has drawbacks indeed, like high computational cost or unpredictable behaviors.” – from paper entitled “Procedural Audio for Games using GAF”

In an attempt to shed some light on what technologies exist, I have included answers to some basic questions I solicited during my research phase for this article. Three audio specialists who were gracious enough to bring me up to speed on several emerging technologies offer opinions on the current state of Procedural Sound and Synthesis.

Interviewed via email were:

- David Thall (DT)
- Mads Lykke (ML)
- Andy Farnell (AF)

Damian Kastbauer: Where do you see the greatest potential for procedural audio in games today?

DT: Physical Modeling.

The current video game sound design paradigm suggests that sound designers add ‘effects’ to a scene in some post-processing stage. For example, they might be given the job to sonify an animated simulation of a character or environment. However, there is no reason why sound designers should need to mix ‘canned’ recordings of sounds into an environment that is already in partial to full simulation.

For example, if we are ‘already’ looking at a mechanical simulation of an electricity-producing water mill, why not simulate the sound production using the same model? A sound designer could provide the model with ‘grains’ of sound, each of which would be procedurally combined by the underlying model to generate the sound scene in real-time.

ML: In sound effects based on object information from the 3D-objects (size, shape, surface, motion, interaction with other objects etc.)

That said, I do not think that the resources available for procedural audio in today’s games allow for advanced and detailed synthesis models – like the models used in computer games graphics. I like to compare the current state of procedural audio with the situation of 3D graphics around 1990. There is a lot of work being done in the labs and papers concluding on the promising potential of procedural audio but no one has really proved the benefit of procedural audio in commercial games yet.

In my opinion this has to do with the focus of commercial game developers today. For years the goal of many game developers (and programmers) has been to achieve the most realistic games possible with the present hardware. This focus has caused the more experimental and simplistic audio designs (including synthetic audio designs) to lose terrain for the benefit of sample-based audio. This focus has led to today’s paradox where sample-based audio is the auditive counterpart to the highly detailed modeled computer graphics.”

AF: In terms of sound classes? Engines for racing games, some animal models maybe (especially in lightweight children’s games), vehicle dynamics, environmental and ecological sounds. That’s the stuff for the next couple of years. Generative music systems of course. There are many aspects to procedural audio and many interpretations of the use of that technology. At one end of the scale you might consider offline automatic asset generation a la James Hahn (Audio Rendering), at the other end (of the real-time scale) you might consider behaviouralist interpretations of control (when applied to ordinary samples) as the next step. In fact interfacing problems in the gap between method and physics engine are very interesting as transitional technology right now.

But procedural audio is about much more than interesting synthetic sounds, it’s a whole change of approach to the audio computing model and the real potential is related to:

- LOAD (level of audio detail)
- Sounding object polymorphism and inheritance
- Automatic asset generation
- Perceptual data reduction

DK: What is the biggest obstacle to bringing procedural audio to consoles?

DT: Weak Sound Engine Integration.

Sound engines must be integrated into the larger graphics pipeline, so that tools and runtime systems can be developed that work audio into the ‘specific’ (as opposed to ‘general’) case.

For example, the rhythmic variations in a piece of music should be accessible by the animation and environmental systems. This would allow animation to better synchronize with the dynamics provided by a game’s sound track.

Another example might allow programmers or sound designers to modulate the volume of one sound with the scale of some visual effect, such as an explosion. Something as simple as tool-driven inter-system parametric modulation would increase the procedural variation factor by orders of magnitude.

In a galaxy far far away I think procedural audio will be able to produce extremely detailed and realistic synthetic audio that to some extent can match the present 3D graphics with regards to “realism”.

ML: The biggest obstacle to bringing procedural audio to consoles is the mismatch between the realism of the procedural audio we are capable of producing with the present hardware, compared to the computer graphics of our consoles.

In order to create a game where the player will not miss the usual sample-based audio we need to consider creating games that support a more experimental and simplistic auditive expression.

DK: Do libraries exist that would scale to a console environment? (pd, max/msp, csound, etc.)? Are the CPU / memory requirements efficient?

DT: Max/MSP could easily be modd’ed to work within a game console development environment. The graph node interface is a well-established paradigm for building and executing ‘systems’ such as those that might be architected to solve a procedural audio algorithm. With some work on the GUI-side, both pd and CSound could be extended to do the same. Arguably, all of these environments would be useful for ‘fast prototyping’, which is always a plus.

On the other hand, none of these systems are well-suited to handle the static memory allocation requirements of most game console memory management systems. Software would ‘definitely’ need to be written to handle these cases from within the respective environment.

ML: My experience with consoles is limited although I have some experience with max/MSP and pd. But I have never tried porting some of my patches for consoles etc.

AF: Chuck remains very promising. So does Lua Vessel since Lua already has a bit of traction in the gaming arena. Work on the Zen garden at RjDj is the closest project I know with the stated aims of producing a axiomatically operator complete library aimed at embedded sounding objects for games and mobile platforms.

DK: What examples of procedural audio are you aware of? (current or past)

DT: Insomniac’s sound engine has a walla-generator that builds ‘crowds’ of dialogue from ‘grains’ (or snippets) of dialogue. Various parameters are modulated by gameplay to control different aspects of the walla, such as density over time and volumetric dispersion over space. These are used extensively to increase the perceived perception of scene dialogue and action.

ML: Through my research I have not come across any contemporary examples of procedural audio in commercial games. In stead I have found a lot of inspiration in the good ol’ console games that rely on sound synthesis – like eg. Magnavox, Atari, Nintendo did in the 70s and 80s – before the entire game community went MIDI-berserk in the 90s, which subsequently spawned the sample-based mentality that dominates today.

AF: Best examples, on the musical side, are still those that go back to the demo scene. A demo title like .produkt (dot product) from Krieger 1997 shows the most promising understanding of real musical PA I have seen because it integrates synthesis code and generative code at an intimate level. Disney, EA, and Eidos all have programmers that venture into proc audio.

DK: What work on procedural audio outside of games would you say is currently the most exciting?

DT: Granular Synthesis of Environmental Audio

Granular Synthesis allows sound designers to easily sample, synthesize and parametrically-generate environmental audio effects, such as wind, water, fire and gas effects, among others, with a level of detail and morphology only hinted at in game-specific procedural audio systems. The fact that sound designers and composers have been successfully using these systems for decades to create easily modifiable ambience shows its greatest potential.

I think the development within virtual musical instruments (like vst instruments) is interesting in relation to procedural audio in computer games. Many virtual instruments run native (without external DSP) so they need to have a relatively small CPU usage, since they are a part of a recording environment, where other plug-ins need the CPU as well.

ML: I am very interested in virtual instruments capable of producing complex but controllable sounds, rather than virtual instruments trying to emulate single instruments like an electric piano to perfection.

We need to consider developing models for procedural audio that are able to produce complex and yet controllable sounds, in order to minimize the CPU cost. If we manage to create such models we can lower the instance count of the models needed to create more complex sounds – and thereby also move away from a procedural sound design that sounds simplistic.

I think that the developers of virtual instruments are 5-10 years ahead of procedural audio designers which is why this field can serve as an inspiration to many of us.

AF: So much happening outside games (it) is amazing. Casual apps on iPhone and Android are moving into the grey area between gaming and toy apps, things like Smule, RjDj and so forth. Combined with social and ad-hoc local networking these can threaten the established ‘musical’ titles, ‘guitar hero’ etc in short order. (An) Italian group around the original Sounding Object research sprint are always on my mind. We need more of that kind of thing. Mark Grimshaw from Bolton is editing a book for IGI available later this year and has some good contributors on the subject of emerging game audio technologies, Karen Collins, Stefan Bilbao, Ulrich Reiter. My own book (Designing Sound) is now with MIT Press. There is some new focus on modal methods with Dylan Menzies work on parameter estimation from shape being a promising transitional technology (taken a long way by den Doel and Pai of course with their Foley-matic)

DK: What area of procedural audio are you most interested in?

DT: Geometric Propagation Path-based Rendering

Geometric sound propagation modeling, though not a new concept, would benefit game audio immensely if it were procedurally generated in real-time. This would include reflection, absorption and transmission of indirect sound emissions into geometric surfaces, diffraction and refraction of sound along edges, and integration over the listener’s head and ears. Done correctly next-gen, this could increase perceived immersion immensely. Nuff said!

ML: At the moment I am mostly interested in virtual musical instruments. But I am also very interested in procedural audio in computer games because it is on “the breaking point”.

AF: The bits I haven’t discovered yet. :)

To get another earful of Procedural Audio for Games, head over to the Game Audio Podcast Episode #4 and listen to experts working in the field weigh in with their experiences and opinions. If you’re still itching for more, I’ve posted a round-up of related links over at the LostChocolateLab Blog which should help fill in the gaps. Please drop by to explore additional Procedural Audio articles, video’s, and white papers that I’ve collected during my research.

I was inspired recently by a recent keynote made by Warren Spector where he called this moment in game development a “sort of renaissance“. I feel the same way about game audio, in that there is so much change happening, and so many breakthroughs being made, that we really are in a golden age. By taking a moment to reflect on the past, widening the scope of audio for games, and staying true to the strengths of interactive I think we can travel the next section of road and ride off into the sunset. Saddle up partners!

UNLEASH THE KRAKEN

In Star Wars: The Force Unleashed we were working with two physics middleware packages: Havok Physics, and Pixelux’s Digital Molecular Matter (DMM). In addition to the simulation data that each provided, we also needed to manage the relationship between both. While Havok has become a popular choice for runtime physics simulations, the use of DMM spoke to the core of materials and provided each object physical properties enabling – in addition to collision’s – physically modeled dynamic fractures and bending. In some ways tackling the sound for both systems was a monumental undertaking, but there was enough overlap to make the process more pleasure than pain.

Before Jumping into the fray, I just wanted to take a moment to echo a couple of things that were touched on in the companion this article; specifically, that collaboration and iteration are the cornerstones of a quality production when it comes to systems design. Collaboration, because the stakeholders involved usually include people across all disciplines; from programmers to sound designers, modelers to texture artists, build engineer’s to game designers. Iteration, because the initial vision is always a approximation at best and until things get moving, it’s difficult to know what the eventual shape things will take.

While simultaneously reigning in and letting loose the flow of creativity ebbing and flowing across the development team, there is nothing more important than the support of your colleges. Leveraging the specialties of different people helps to bring new idea’s to situations in need of a solution. Your greatest asset as a team member is to recognize and respect the uniqueness of your co-workers and stay open to the constantly shifting requirements of the game. Good listening and better communication will improve the productivity of meetings, and reinforce the fundamental desire of everyone – to craft the best player experience possible.

DIGITAL MOLECULAR MAGIC

Starting with Digital Molecular Matter, Audio Lead David Collins worked closely with Pixelux to identify the core components that could be utilized in bringing sound to the simulations. Prototypes were created offline in pre-production driving toward the best way to score the sounds of the dynamic physically modeled objects being created by the art team. With a list of over 300 types of DMM materials, we chose to abstract a group of about 30 that would cover all of the sound types and object sizes. These DMM Sound Materials were added as a “Sound Material” property to the meta data for each DMM Material type. This was the first step in defining the sound an object would make when calculations regarding collisions, fractures, and bending where concerned.

Behind each of these Sound Materials were a set of parameters for speed (fast/slow), size (small/medium/large), and quantity (one/few/many) – in addition to specifications of sound interaction between surface types – that enabled us to specify different thresholds for each and scale sample content across the different values. The content itself – abstracted Sound Cues (or Events) – were defined for use by the DMM Sound System using “Sound Buckets” which essentially specified the sound content that would be used for a given parameter’s action when triggered.

In this way we were able to appropriately employ the sound of different sized collisions and fractures based on the number and type of actions requested by the system. Behind the Sound Cue referenced in the Bucket for each sound type we had the usual control over file, pitch, and volume randomization in addition to 3D propagation min/max distances and priority – which became crucial to reigning in the number of instances of a Sound Cue during a given request from the system.

We also had bending information to deal with; specifically for metal, wood, cables, and organic vines. When the system determined that bending of a DMM object had begun, it would start a loop that would continue as long as a minimum threshold of force was being applied to the object. While looping, the system also played several single element bend “sweeteners” when spikes in the amount of bending occurred. The best example of this can be heard when wrestling one of the giant doors between area’s in a level.

For an addition summary of the DMM audio system, check out Jesse Harlin’s fantastic overview in Game Developer Magazine from September 2008.

BRING IT TO THE TABLE

We adopted a different approach to handle data coming from the Havok side of the physics simulation – where we had a greater level of detail between objects that were throw-able and caused impacts, across the different environmental material types.

One of the often used audio techniques of Physics integration in the current and previous generations is the look-up table or matrix that is used to define material actions and their interactions. Using a spreadsheet format as the starting point for the system, surface materials can be arranged along the top and far left side of the sheet. At the point where a row and column intersect the Source Material to Destination Material sound interaction can be specified, usually as an audio file or an abstracted reference to a group of files with additional properties for randomizing pitch and volume values – what we were calling a Sound Cue.

We took this methodology one step further by enabling the additional layering of Sound Cues for the Source and Destination objects. This allowed us to not only specify a Sound Cue for the specific interaction between materials, but also a default sound for the inherent object or material type. In this way, a single collision between a metal barrel and the dirt of a forest floor could incur the following impacts: 1. Metal Generic (Source Layer) 2. Dirt Generic (Destination Layer) 3. Metal Barrel on Dirt Explicit (Source + Desination Layer)

Let’s take a step back and look at how each of those things are handled within the lookup table.

MATERIAL 1 & MATERIAL 2

“Material 1” (Column A) is used to define the material type of the actor being used. (ex. A metal object would be tagged with the “metal” material) The material name is defined at the top of each material section. The size of the material selection can be adjusted using the modifier adjustment in the top left corner cell (A1). “Material 2” (Row1) is used to define any other material types used within the game environment.

SOURCE LAYER

The “Source_Layer” (Column B) is used to define a set of sound content that will play every time an object – with “Material 1” defined as it’s material – impacts a surface with any “Material 2” in the game. The “Source_Layer” has a multifunction ability: If there is an entry in the first row of a material type (ex. phy_imp_dirt) then all levels of impact will register as the same “size” and “weight”; otherwise. If the first entry in a row is left blank, you can then slot 3 sounds that will react to the size and weight of an impact as specified in the Threshold tab (sm/md/lg).

DESTINATION LAYER

The “Dest_Layer” (Row 2) is used to define a set of sound content that will play every time an object whose material is defined in the “Material 2” (Row 1) is impacted by an actor with any “Material 1” (Column 1) in the game. The “Dest_Layer” has a multifunction ability: If there is an entry in the first row of the “Dest_Layer” then all levels of impact will register as the same “size” and “weight”, If the first entry in a row is left blank, you can then slot 3 sounds that will react to the size and weight of an impact as specified in the Threshold tab. (sm, md, lg)

SOURCE + DESTINATION LAYER

The source_layer + dest_layer provides a look-up table where a sound is played specifically between a “material 1” and “material 2” impact. In the following example, when an actor with a material of dirt impacts a concrete surface the phy_imp_dirt_concrete content will play.

COLLISION’S COMBINED

In this example we are playing a combination of the 3 sounds when an actor with a material of dirt impacts a concrete surface.

MODIFIER AND EXPORT

The modifier defines the number of rows between each material as a way to prepare the values to be exported into game ready data. The export button is used to convert the spreadsheet to an efficient XML file that will be used by the game engine at runtime.

THRESHOLD

Threshold is used to define the “size” and “weight” values that are used to transition between the 3 slots defined for an object with sm, md, lg.

BODYFALL

We were able to extend the use of our matrix system to incorporate our player and non-player character (NPC) bodyfall collision’s which were handled using a combination of Havok Physics and Natural Motion’s Euphoria.

INTO THE FUTURE

While there has been some laboratory work done in the area of Synthesizing Contact Sounds Between Textured Objects by the GAMMA research group at the University of North Carolina at Chapel Hill, this approach has yet to cross over to games at runtime. In place of true synthesis, the industry is currently invested in a sample playback methodology which requires a multitude of discreet sound files that are used as a representation of a given visual. Whereas once upon a time the game industry was embroiled in the hardcore synthesis detailed at length in Karen Collin’s excellent “Games Sound”, the change to sample playback has caused the synthetic muscle of game audio to atrophy. On the horizon is mounting a recombination of the power and flexibility of synthesis and procedural audio techniques, and the fidelity of linear sound content. Beginning in 2008 with the release of Sound Seed Impact and their Sound Seed Air suite of tools, Audiokinetic is leading the charge in audio middleware towards a return to synthesis that aims to add creative options that leverage the increased CPU and reduces the dependency on predetermined sound content stored in RAM.

With everyone in game audio engaged in battle for the resources needed to achieve an exponential level of quality in the current generation, we need all of the creative tools and tricks at our disposal to accomplish this goal. I’m a fan of anything that expands upon the growing possibilities of interactive audio in a way that puts control in the hands of people who are actively looking to push the boundaries of what is possible. Where it goes from here is up to the people making choices about how we move forward as an industry and where the focus continues to be.

Until next time!

Art © Aaron Armstrong

SYMPHONY OF DESTRUCTION

Physics, the simple pleasure of “matter and its motion through spacetime”.

In games we’ve reached the point where the granularity of our physics simulations are inching closer and closer towards a virtual model of reality. As we move away from the key-frame animated models of objects breaking, and the content swap of yesteryear, towards full scale visual destruction throughout our virtual worlds, we continue to increase the dynamic ability of objects to break, bend, and collide in relation to our experiences of the physical world around us.

“It is just inherently fun break things, and the bigger the thing is the more fun it is to break. It can be a stress relief or just give a feeling of power and control. We worked extremely hard to create a virtual sand box for the player to create and destroy as they see fit, we just hope it gives them the same pure joy they had as a small child kicking over a tower of blocks. “ Eric Arnold, Senior Developer at Volition (CBS news)

Because of the joy involved in seeing the reaction of objects when force is applied, there is developing the potential to derive great satisfaction from the realism of these simulations in games. Piggybacking on the work of people doing the hard thinking about how to manage the visual response and “feel” of this technology, audio has the ability to use information from these systems in order to attempt a similarly pleasing sound accompaniment to the visual display of physical interaction. Call it orchestrating the symphony of destruction if you will, there’s nothing finer than the sound emanating from the wreckage of a virtual building.

Hooking into these systems is no small task, in part due to the tremendous amount of data being constantly output; object to object collisions, velocity, mass, material type all being calculated at runtime to a level of detail necessary to display an unfolding level of realism on-screen. Sorting and sifting through this data becomes one of the main focuses early on in production in order to gain an understanding of how sound can be designed and played back in reaction and relation to these variables.

CONSTRUCTING CONTENT

When it comes to creation of these sounds – in which the smallest impact to the largest fracture must be represented with enough variance to discourage repeatability within a short amount of time – the asset count for any given material type can easily spiral out of control.

Some considerations when defining the assets include scaling based on: number of collisions, object size, object weight (mass), the speed at which an object may be traveling (velocity), and material type. It’s with these parameters that we can begin to build an abstract audio system in order to switch and transition between the different sampled content sets and gain the ability to apply parametric changes to the content based on information coming from the simulation. These changes could include: applying a volume or pitch reduction based on the declining value for velocity, or changing between samples based on the number of collisions at a given time.

“As all of this is going on we also play audio and video cues to let the player know which areas are getting close to breaking. Beyond making the world more believable they serve as a warning system that the structure is unstable and could collapse on the player’s head if they aren’t careful and hang around too long. This small addition took the system from a neat tech demo to pulling the player in to the game world and generating very real chills as they flee from a creaking, groaning building while tendrils of dust and debris rain down around them. “- Eric Arnold (Eurogamer)

RED FACTION: GUERRILLA – CASE STUDY

In an expose’ on the physics of Red Faction: Guerrilla, Senior Sound Designer Kate Nelson lays out the fundamental systems design and decisions that went into orchestrating the sounds of destruction:

“’Players love to blow stuff up’ was a popular phrase echoing in Volition’s halls throughout Red Faction: Guerilla’s development. GeoMod 2.0 technology made real-time destruction an exciting focus of the game design team. For that reason, one of the primary goals for the audio team was to ensure that when players ‘blew stuff up’ they experienced satisfying and immersive audio feedback. Overall, designing the destruction audio system was a rewarding challenge.”

THE ROAD TO RUINATION

Everything to do with audio for the destruction system is piggybacked on top of the underlying physics simulations being run by engineering. Because there are so many stakeholders involved with bringing this to life on screen and through the speakers, it’s usually an exercise of collaboration to the highest degree in order to find commonality between disciplines, and pave the way for elegant design.

“Once art/programming/design had a good prototype of what behaviors were expected, our audio team spent a good amount of time picking the destruction system apart and narrowing it down to the basic core components that affected audio:

– Every building and object in the game was made up of different materials, and those materials each referenced a core “destruction” material

• All things artists made out of metal, regardless of texture, could be thought of as “steel”
• All things artists made that were rocks and concrete could be combined into “concrete”

– Each material was engineered by artists and programmers to break in a specific way:

• Concrete would break into smaller pieces of concrete which would range in size but had consistent shape
• Steel would break into smaller pieces of steel which would range in size and vary in shape (sheets, poles, and solid chunks)

– Each material and shape combination would react differently in the environment when tossed around:

• Concrete blocks impacted the ground, rolled down hills, slid across flat surfaces, etc.
• Steel sheets impacted the ground, slid across flat surfaces, but would not roll, etc.

Leveraging the the core commonalities in order to appropriately simplify things from a sound perspective led to a greater understanding of the content needs and allowed a greater focus and level of detail on a few types of materials. When every game has to temper the fidelity of it’s systems – be it ambient, footsteps, or physics – with the availability of CPU resources and RAM, choosing where to allocate resources can be a constantly shifting management game that relies on constant gardening to make sure every sound type get’s what it needs to survive and be heard.

“From these observations and with keeping in mind that we had to keep our asset count down, we were able to extract a basic idea of the kind of audio assets we would need to create and trigger:

[Material type] + [shape] + [size] + [destruction event]

Examples:

• Steel + Sheet + Large + Impact
• Concrete + Solid + Small + Roll

“Once we identified the kind of sounds we wanted to produce with the destruction engine, we found the best way to be able to communicate our vision was by capturing footage of the game and adding sound design that illustrated the kind of support we wanted. This video was key in selling our concept to the project producer and explaining to our programmers what we needed to hear and why – nothing says “this is why it’s fun” better than witnessing the moment itself.”

FUNDAMENTAL FACTORS

Working closely with Eric Arnold and senior audio programmer Steve DeFrisco, the audio team (including Kate Nelson, Jake Kaufman, Raison Varner, and Dan Wentz) drove towards a system that was able to identify the four destruction factors they had identified for content.

– Material type (destruction sound material) was added to objects, environments and materials by artists or members of the sound team.
– Shape was determined after destruction – in real-time – by code that measured each material piece after destruction.
– Size was determined at the moment the destruction event triggered, based on the mass and velocity of the object:

• A X kg piece of concrete traveling at Y velocity impacted with Z energy value
• Audio designers specified which Z energy value triggered a large, medium, or small sound event for each destruction material.

– Destruction Events were calculated in real-time determined by the action of the material piece:

• Upon colliding with something, an impact would trigger
• if the piece continued to move in a horizontal fashion and was turning on its axis, a roll would trigger

The combination of systems and content design forms a mighty Voltron of destructive power and sound mayhem which can be heard resonating through every destructible element in Red Faction: Guerrilla. Once everything was up and running with sounds being triggered appropriately, the complicated task of applying sound voice limiting and additional sound finesse helped to reign in the multitude of sounds escaping from every impact.

Examples of other key features:

• Volume attenuation was applied based on the velocity of destruction objects.
• Playback of destruction events was controlled in order to not overwhelm the player.
• An additional material layer could be specified if needed to differentiate between surfaces that were struck.

Additional thoughts:

• Adjustments to Mars gravity during development made tweaking size energy values interesting!
• It was necessary to strike a balance between having enough destruction assets to provide necessary playback variety, and having so many assets that we blew our memory allotment.
• It took time to determine how many destruction sounds should be allowed to play at once to provide the necessary destruction feedback for the player without swamping the overall soundscape.

It can be said, that with the escalating demands across departments in game development, the single greatest asset on the side of quality continues to be the ability to quickly develop and implement systems that can then be iterated upon as we all work towards simulating models that meet and exceed expectations. The development of any complex system absolutely depends on this iteration in order to dial in the important aspects which will help to sell the desired effects.

UNTIL NEXT TIME

Tune in for Part Two when we look at the systems behind the physics in Star Wars: The Force Unleashed and look towards the future beyond our current reliance on sample playback technology.

Art © Aaron Armstrong

In a continued attempt to shed light on some of the best examples of Technical Sound Design in the current generation, I’d like to call attention to several titles that have pushed the envelope when it comes to the art of ambience. The all encompassing experience of “being there” in a game, where the sense of place is encapsulated in the sound of the environment. Stepping beyond the background din of a given location, we’re moving forward towards the players ability to affect the sound of a space by their interaction with it. This can be as simple as turning off a machine that had been emitting a constant loop of activity, or as complex as scaling the dynamics of a crowd dependent on the current artificial intelligence activity in an area.

THE SHACKLES

Despite leaving behind the memory restrictions of previous generation consoles, hearing a single looping ambience throughout a level or area within a game continues to be common – making any recurring distinct elements of the background clearly identifiable when repeated. While these backgrounds, well designed and teaming with character, still contain the potential to keep the player immersed in the game world, anyone who chooses this approach runs the risk of exposing the limitation this technique to the player. Several best practices have evolved and taken root to combat repetition and further lend a sense of randomness to the sound aspect of the game world.

In an article by Nick Peck back in 2004 entitled “Tips for Game Sound Designers”, a case for highlighting ambient elements which vary in time, duration, and position in order to “Generate 5.1 content without full bandwidth sources” was made. This included the idea of a subtly shifting background ambience with randomly placed elements as a solution to static looping soundscape, and presented a way out of the confinements of the locked loop. While likely that this presentation was not the first time a solution was defined, the practice of ambient creation using these methodologies perpetuates today in step with the advancements in available resources and the increased creativity of audio toolsets.

THE LAND OF THE LIVING

The world of Oblivion can be bustling with movement and life or devoid of presence, depending on the circumstances. The feeling of “aliveness” is in no small part shaped by the rich dynamic ambient textures that have been carefully orchestrated by the Bethesda Softworks sound team. Audio Designer Marc Lambert provided some background on their ambient system in a developer diary shortly before launch:

“The team has put together a truly stunning landscape, complete with day/night cycles and dynamic weather. Covering so much ground — literally, in this case — with full audio detail would require a systematic approach, and this is where I really got a lot of help from our programmers and the Elder Scrolls Construction Set [in order to] specify a set of sounds for a defined geographic region of the game, give them time restrictions as well as weather parameters.” – Marc Lambert

In a game where you can spend countless hours collecting herbs and mixing potions in the forest or dungeon crawling while leveling up your character, one of the keys to extending the experience is the idea of non-repetitive activity. If we can help to offset that from a sound perspective by introducing dynamic ambiance it can help offset some of the grind the player experiences when tackling some of the more repetitive and unavoidable tasks.

“[The ambient sound] emphasizes what I think is another strong point in the audio of the game — contrast. The creepy quiet, distant moans and rumbles are a claustrophobic experience compared to the feeling of space and fresh air upon emerging from the dungeon’s entrance into a clear, sunny day. The game’s innumerable subterranean spaces got their sound treatment by hand as opposed to a system-wide method.” – Marc Lambert

Also on the topic of injecting randomness into the soundscape, from a Game Informer interview with Don Veca regarding the ambient sound design of Dead Space:

“Veca told his audio team to make their ambient tracks a little shorter, more vanilla, in order to create audio with fewer effects that would catch a player’s attention when looped. The team then wrote some software Veca calls “the creepy ambi-patch.” This little piece of code would play separate small sounds, gradually changing their pitch and volume as they panned around Dead Space’s 3D environments. The effect worked beautifully, ensuring that every moment of Dead Space’s ambient backgounds were different no matter how long a person explored, or how many times they replayed a section.” – Game Informer

It should come as no surprise that ambience can be used to great effect in communicating the idea of place, either with ties to reality or to the abstract extreme. When you combine the use of soundscapes and level-based tools to apply these types of systems appropriately, the strengths of dynamics and interactivity can be leveraged to create a constantly changing tapestry that maintains a sense of immersion, and creates a personal experience for every player.

THE TOOLS OF THE TRADE

When it came time to design the creative tools used to implement ambiences in Fable II, the sound designers were able to “paint ambient layers” directly onto their maps. In a video development diary, Lionhead audio director Russel Shaw explains:

“I designed a system whereby we could paint ambient layers onto the actual Fable II maps. So that as you’re running through a forest for instance, we painted down a forest theme, and the blending from one ambiance to another is quite important, so the technology was lain down first of all.” – Russel Shaw

In what could be seen as another trend in the current console cycle, enabling the sound designers to handle every aspect of sound and the way it is used by the game is just now becoming common. The ability to implement with little to no programmer involvement outside of the initial system design, setup, and toolset creation is directly in contrast to what had gone before.

In the past, it was not uncommon to create sound assets and deliver them with a set of instructions – how they should be played back – to a programmer. A step removed from the original content creator, the sounds would then need to be programmed into the level – including any parametric or transition information – where the ability to adjust values would be out of reach for the sound designer. It is clearly a benefit to the scope of any discipline to be able to create, implement, and execute a clear vision without a handoff between departments to accomplish the task.

Many of the audio middleware toolsets currently available enable the Sound Designer a high level of control over the way sounds are reproduced. Some include the ability to randomize a sounds 3D position, even going as far to enable the ability to map positional “paths” using a custom interface. This opens up a further level of variation beyond sound files or pitch and volume randomization, and distributes the action across the soundscape in a way that better resembles our experience’s in real life.

As familiarity with these techniques and functionality of available toolsets increases, we can hope for a level of randomness that keeps the player firmly rooted in diversity and appropriateness of the game world. In this way I feel like we are gaining in the art of audio implementation and sound integration – by putting creative tools in the hands of the interactive-minded sound designers and implementation specialists we are paving the way for the ability to simulate living breathing worlds of sound.

THE NAKED CITY

“There are a thousand tales in the naked city” and if you listen closely enough you might be able to hear most of them during the gameplay of Prototype. That is, if you can keep yourself from wreaking havoc among the citizens of New York…which, let’s face it is nearly impossible. While you probably won’t hear all of the stories that the city has to tell, you will be able to hear the changing voice of that city during your progression from introspective lost soul to amped up superman thanks to the attention to detail by Radical’s Sound Department. In a detailed article, Sound Director Scott Morgan details the implementation that gave Protoype’s cityscape it’s voice.

“So we decided to develop a dynamic system for ambient sound so that New York could speak, through its inhabitants, of its current “emotional” state. If the city was in a relatively “normal” state, we would hear the traffic, the pedestrians and the busy sounds of New York that we all know. As panic ensues, so does the voice of the city, with screaming pedestrians and honking horns. If the player guides his character up to a quiet rooftop or the middle of Central Park, the sounds of New York adapt accordingly. As the infected hordes take over, the sounds voice the pain and suffering of the city and its inhabitants. As the story progresses and the city heads towards its darkest hour, its voice dynamically follows – expressing its state as a character in the story, revealing its suffering.” – Scott Morgan

This touches on a diverging point in the creation of ambience: that the sound of the ambient should react to the gameplay and change dynamically based on what’s happening in the environment as a result of player interaction. As we continue to move closer towards realistically representing a model of reality in games, so should our worlds react and be influenced by sound and its effect on these worlds. This was foreshadowed by Crytek’s Christian Shilling during the production of the original Crysis:

“Ambient sound effects were created by marking areas across the map for ambient sounds, with certain areas overlapping or being inside each other, with levels of priority based on the player’s location. ‘Nature should react to the player,’ said Schilling, and so the ambiance also required dynamic behavior, with bird sounds ending when gunshots are fired.” – Christian Schilling

Schilling went on to explain the basic concept and provide additional background when contacted via email:

“Sneaking through nature means you hear birds, insects, animals, wind, water, materials. So everything — the close and the distant sounds of the ambiance. Firing your gun means you hear birds flapping away, and silence. Silence of course means, here, wind, water, materials, but also – and this was the key I believe – distant sounds (distant animals and other noises)…So, after firing your gun, you do hear close noises like soft wind through the leaves or some random crumbling bark of some tree next to you (the close environment), all rather close and crispy, but also the distant layer of the ambiance, warm in the middle frequencies, which may be distant wind, the ocean, distant animals”
In addition to the triggering of various one-shot sounds, various mix decisions are being made behind the scenes in order to further focus on the appropriate sounds – Insuring that the important aspects of sound are communicated to the player.

THE WAY FORWARD

Thankfully, the continuation of work started years ago to diversify the ambient landscape in games and bring variation and randomness to environmental sound is starting to make headway. With standout examples leading the way, and dedicated individuals sharing their processes and reasoning, we can hope to expand on the creative possibilities enabled by toolsets and best practices to create rich sound worlds for players to inhabit. Focusing on these ambient techniques during game development means the player won’t feel inclined to focus the negative aspect of hearing loop point’s while playing – instead, they can marvel at the interaction between their character and the game world, and let it set the tone for their experience.

Concept Art © Aaron Armstrong: http://aaron-armstrong.blogspot.com/

As here I sit four years later having been lucky enough to participate in the undercover debug of several titles, mouth still agape at the possibility to visualize sound and sound properties as a way to understand what is “going on” sound-wise at a given moment, I continue to be fascinated by these environments created by hand for the sake of debug. While this aspect of game audio may continue to be a closely held secret of developers leveraging internal pipelines and processes, a few screens have escaped which show off various functionality.

There may be more out there in the wild, but in my cursory scrubbing of conference presentations and recalled articles of the past – alas – this is all I’ve been able to turn up. (feel free to add links to other in game audio debug screen grabs in the comments section.)

In addition to presenting – what was then – state of the art examples of sound design and under the hood implementation considerations such as: speed of sound, distance propagation of sound, and reverb interpolation; he also went on to highlight an example of creative realtime DSP in Bizarre creations XBLA title Geometry Wars.

In a brilliant execution of a simple idea, Geometry Wars took a Low Pass Filter and applied it to the entire mix when dropping a bomb, submerging the high frequencies of both sound & music, and then releases the applied filter over the course of the next few seconds.

Listen for it at 2:27 in the following video:

Elegant in it’s simplicity and beautiful in it’s effectiveness, this creative use of DSP illustrated one of the strengths of the current generation to be able to effect sound dynamically, reactivity, and interactively as a way to further meld sound to gameplay. At the end of the day, finding ways to nurture and support the complex relationship of interactive audio in games is something we should all be reaching for.

One area that has been gaining ground since the early days of EAX on the PC platform, and more recently it’s omnipresence in audio middleware toolsets, is Reverb. With the ability to enhance the sounds playing back in the game with reverberant information from the surrounding space, you can effectively communicate to the player a truer approximation of “being there” and help to further immerse them in the game world. While we often take Reverb for granted in our everyday life as something that helps us position ourselves in a space (the cavernous echo of an airport, the openness of a forest), it is something that is continually giving us feedback on our surroundings, and thus a critical part of the way we experience the world.

While It has become standard practice to enable Reverb within a single game level and apply a single preset algorithm to a subset of the sound mix. Many developers have taken this a step further and created Reverb regions that will call different Reverb presets based on the area the player is currently located. This allows the Reverb to change based on predetermined locations using predefined Reverb settings. Furthermore, these presets have been extended to area’s outside of the player region so that sounds coming from a different region can use the region and settings of the sounds origin in order to get their Reverberant information. Each of these scenarios is valid in an industry where you must carefully balance all of your resources, and where features must play to the strengths of your game design.

While preset Reverb and Reverb Regions have become a standard, and are a welcome addition to a Sound Designers toolbox, these techniques ignore the inherent physical characteristics of a space and are unable to dynamically react to reflections from these surfaces relative to the player. In order to bring Reverb closer to realtime expectations, level geometry could be referenced based on the originating position of a sound within a space, and that data could then be used to apply appropriate reflections.

One way of accomplishing this in the current generation of consoles is through the use of Ray Traced Convolution Reverb, a technique which snuck in under our noses in the Xbox 360 launch title Crackdown, from Realtime Worlds. 

”When we heard the results of our complex Reverb/Reflections/Convolution or “Audio-Shader” system in Crackdown, we knew that we could make our gunfights sound like that, only in real-time! Because we are simulating true reflections on every 3D voice in the game, with the right content we could immerse the player in a way never before heard.”-Raymond Usher (Team Xbox)

So, what is realtime Reverb using ray tracing and convolution on a per-voice implementation?
Simply put, it is the idea that every sound that happens within the game world has spatially correct reverberation reflections applied to it. Let’s dig in a bit more…

Here’s a quick definition of Ray Tracing as it applies to physics calculation:

“In physics, ray tracing is a method for calculating the path of waves or particles through a system with regions of varying propagation velocity, absorption characteristics, and reflecting surfaces. Under these circumstances, wavefronts may bend, change direction, or reflect off surfaces, complicating analysis. Ray tracing solves the problem by repeatedly advancing idealized narrow beams called rays through the medium by discrete amounts. Simple problems can be analyzed by propagating a few rays using simple mathematics. More detailed analysis can be performed by using a computer to propagate many rays.” -Wikipedia

On the other side of the coin you have the concept of convolution:

“In audio signal processing, convolution Reverb is a process for digitally simulating the reverberation of a physical or virtual space. It is based on the mathematical convolution operation, and uses a pre-recorded audio sample of the impulse response of the space being modeled. To apply the reverberation effect, the impulse-response recording is first stored in a digital signal-processing system. This is then convolved with the incoming audio signal to be processed.” -Wikipedia

So what you end up with is a pre-recorded impulse response of a space (room, cave, outdoor) being modified (or convoluted) by the Ray Traced calculations of the surrounding physical spaces. The distance of a originating sound from the level geometry defined in the Ray Traced calculation gives the reflections their parameters which effect the Impulse Response sample of the space type.

You can hear the results of their effort in every gunshot, explosion, physics object, and vehicle as you travel through the concrete jungle of Pacific City. As the player walks around the city, a passing car is with it’s radio blaring can be heard positionally from the open window in addition to it’s reflection off of a nearby wall; meanwhile footsteps & gunshots are continuously being reverberated realistically depending on the changing environmental characteristics. What this allows the sound to communicate is a greater sense of location and dynamics of the sound at the time it is triggered.

While the result may be less impressive than the complexity of the implementation, the additive effect that it has on the multitude of sounds happening throughout a game can bear a significant effect on adding realism to the environment. It’s also worth noting that Crackdown2 will be hitting shelves soon from Ruffian Games, along with the Realtime Worlds’ new MMO All Points Bulletin. No word yet on whether either of these will continue to push realtime Reverb but all ears will be on them for advancing the potential of this technique.

With a future for convolution Reverb implied in a recent press release for Audiokinetic’s Wwise toolset, and the brief outline of it’s use in Radical’s recent open world game Prototype, let’s hope the idea of Realtime Reverb in some way, shape, or form continues to play an integral part in the next steps towards runtime spatialization.

Article: The Audio of Crackdown
Podcast Interview with MGS Kristofer Mellroth (Starts at 12:00 in)

See video example of Realtime Reverb Debug @ 4:15

Crackdown – Sound Study Footsteps

Developer created proprietary toolsets continue to provide a vital form of audio integration in the industry, especially in cases where the gameplay specific features need to be exposed and interacted with at a deeper level than what comes “out of the box” with most middleware. In the days before robust audio middleware toolsets this may have been the only way to get past the basic “Play Sound” functionality of an audio engine and pull off amazing feats of groundbreaking interactive audio.

Because of the nature of game development and custom technology, there is not a lot of information made publicly available regarding these tools, and what has been exposed is usually little more than a screen shot or casual mention in a larger article regarding a specific titles sound. In most cases additional links and information have been provided in an attempt to gain a clearer picture of their uses, and what is available may be of value to those who might be involved with tool creation and especially those who are attempting to understand some of the additional esoteric aspects of audio implementation that have developed over time. While this is not a comprehensive look at proprietary toolsets through the years, it is an attempt to survey the landscape of what is known.

Sony SCREAM

Veiled in secrecy, the SONY SCREAM tool provides direct access to the Sony hardware specific audio library MulitStream. Public information has been scarce, unless you are a developer for the PS2/PS3 and the only screens around are found embedded in Presentations or articles. One interesting thing to note is Sony’s adoption of the iXMF interactive standard established by the IASIG in 2009 for their future toolset iterations including AWESOME (audio scripting solution) , SULPHA (a Multi-Stream analyzer/ debugger), and FUSION (modular based synth engine) as noted in an article with SCEE’s Jason Page and subsequent Delop article with Oliver Hume.

Sony’s Page: ‘Next Gen Audio – Is That It?’
Develop Article: SULPHA, so good
SCEE to implement iXMF
Next-Gen Audio Square-Off: PlayStation 3 vs. Xbox 360
IASIG – iXMF
Better Tools for Sound Designers on PlayStation_3

Bungie: Guerilla Toolset

The Guerrilla Toolset used by Bungie for the Halo games has been discussed in several articles, but aside from some general information and a tiny picture in the wild, there aren’t many supporting details. I was able to cobble together some details from notes I recovered from an Audio Boot Camp presentation Jay Weinland gave back at GDC 2006. Among the features over viewed in the session were the ability to subscribe and mix sounds based on their sound_class, visually represent distance rolloff in-game, update values in the tool and update in-game with keystroke, and Reverb interpolation between zones. I’m sure much has changed since 2006 as Bungie continue to ratchet up the quality of audio across their titles. One thing that stands out in Halo 3 is their newly implemented use of Waves EQ and Surround Limiter at runtime to modify the output of the game’s audio. This move toward the use of high end DSP taken from the pro audio realm and converted to work in the engine is a trend that we should be seeing more of in the future.

Halo 3 Audio: Locked and Loaded
Halo 3: Bugie.netAudio Presentation
GDC 2005 Report: Audio Production for Halo 2
Halo 3: Waves DSP at Runtime

Bethesda Software: Oblivion

Some of the audio functionality of the Elder Scrolls Construction Set is over viewed in this insightful development journal, along with what are becoming some of the best practices for Ambient sound integration. It’s great to see a visual representation of the ambient being used across an area, along with the parameters for playback. This kindof specific use tool is a great example of bridging the gap between game engines and audio functionality.

Article: Elder Scrolls Construction Set – Audio

Day 1 Studios: Ducking

In a technical article the overviews the theory of Ducking, Zach Quarles exposes the tool he used at Day 1 Studio’s to accomplish the interactive effect of reducing the volume of non-priority sounds in order to better differentiate mission critical sound or dialog..

Article: Game Audio Theory Ducking
Internal Ruminations of an Audio Monkey 

Day 1 Ducking Tool

Radical Games: Prototype

Thanks to the fine folks over at Radical Entertainment, much has been revealed regarding the various tools and techniques utilized by the sound team to push audio quality in games further towards a Hollywood model of sound. While the idea of interactive mixing is a broader topic for another day, some of the recent work that was done with scalable crowd ambience has gone along way towards pushing the dynamics background sound. In a series of articles Scott Morton details of their use of Max/MSP style procedural, runtime Reverb, and the proprietary toolset “Audio Builder” to accomplish the feat of implementing a 18 channel sound file in order to orchestrate the ambient and limit the impact to the disk at runtime. Some of the tools used for this process have been outlined including a peek behind the curtain on some of the creative tools used at Radical.

Article: Dynamic Game Audio Ambience
Article: The Sounds of Prototype

Battlefield Bad Company

In a presentation given at the Game Developers Conference in 2009, DICE Audio Programmer Anders Clerwall gave an overview of their “High Dynamic Range” audio solution spearheaded by David Mollerstedt (Head of Audio, DICE). Simply put, High Dynamic Range Audio allows for the realtime interactive mixing of sound based on prioritization, culling, and loudness measurement during gameplay. The result is an overall sound mix which adapts to the action going on in the game. Furthermore, menu options are made available to adjust the playback device type as explained by Ben Minto (Audio Director, DICE):

“TV/HiFi/Cinema – these (Options) change the way the game is mixed at runtime through Frostbite’s HDR system. The settings are self explanatory – HiFi is default, TV is if you need to hear everything at a quiet volume through a small speaker, and Cinema is if you have a nice posh setup with full range speakers.“

While the technique of providing different playback profiles is not unheard of in other titles, the HDR system easily allows for the dynamic balancing of the listener experience in order to provide the best possible soundscape.

Download: Automatic_Audio_in_Frostbite
Article: Heard About Battlefield Bad Company
How HDR Audio Makes Battlefield: Bad Company Go BOOM


In addition, here are some screens from the Audio Panel within the Frostbite Editor where you can see the “Loudness” attribute for a sound source exposed.

EA: The Simpsons

From a presentation given by the audio team at EA Redwood shores for The Simpsons Game several screens for proprietary tools are exposed including a view of the EA proprietary procedural toolset AEMS or Audio Event Management System, and several custom tools used for modifying parameters related to their physics implementation. AEMS is a general tool at EA made by EA Tech, and it’s been in use since before 2002. Its been used by almost every game during those earlier days, and even today is used in many.

Presentation Slides: Sound Design for The Simpsons Game

EA Tool used on "The Simpsons"

EA Tool used on "The Simpsons"

EA Tool used on "The Simpsons"

Impact Tuning

Roll Tuning

Slide Tuning

Despite the rise of audio middleware in today’s market, it is clear that is not a one size fits all mentality when it comes to creating a toolset and workflow to support the strengths of a particular games design. Hopefully by seeing, and understanding the choices studio’s and game audio professionals are employing to help them creatively add sound to games, we can continue to drive the innovation of game audio through the current console generation and beyond.

Tune in next time when we pontificate further on some of the creative techniques used to bring interactivity to audio in games.