Since embarking on my informal game sound study of footsteps and movement sound, things have continued to develop in a remarkable way. What started casually, quickly spiraled into a lengthy debate involving many people and uncovering  some interesting patterns.

I recently reprised a presentation I gave at GDC this year at my local IGDA chapter in an attempt to share the findings of the initial study and continue the discussion with some of the new information gleaned in the meantime. If you missed it at GDC or are interested in some of the additional insights that came out, this presentation breaks down the fundamentals and unique considerations that emerge across a diverse cross section of game genre’s and uncovers some curiosities and aesthetic choices .

It may be not so surprising to have found people who feel passionate about the role footsteps play, but it’s no less fascinating to hear how deep people’s experiences go, and how willing they are to share their perspectives. What once seemed like a small part of game audio, has taken on a whole new light after being placed under the microscope. I’m thankful to everyone who has contributed to the conversation during the last year, and hope that by sharing these insights we can all move forward and give appropriate attention to movement sound in games.

Hit the IGDATC link for some additional related articles: IGDATC Video – Footstep and Movement Presentation

Damian Kastbauer and Anton Woldhek have published a new episode of Game Audio Podcast discussing stuff from the recent GDC 2011.

In which Damian & Anton discuss highlights from the 2011 Game Developers Conference with a focus on super sessions, off the hook parties, and the impression of a procedural audio groundswell.

Listen – GAP Episode #9

Aaron Marks in the driver seat of a Marine Corps Amphibious Assault Vehicle (AAV) during field recording for Operation Flashpoint: Dragon Rising

This month has been awesome! Aaron Marks made a terrific special, which sadly ends today. Thanks to all the readers who sent his questions. Below are the answers to them.

Designing Sound Reader: How often do you use your musical background in sound designing a film or video game? Do you pitch certain hard FX or drones to musical notes to strike emotional chords in audiences?

Aaron Marks: I think I’m pretty much always in the ‘musical’ mindset when doing sound design, whether it’s recording, mixing and mastering or on a creative level. Music is definitely in my blood and I think, subconsciously at least, it’s always there when I’m doing any audio work. I definitely have some techniques where I ‘tune’ sound elements to be either pleasant or something a little more harsh. I also tend to pay attention to the music that is accompanying a scene or game level and often tune the sound effect to a note which blends well or I’ll go the opposite direction and detune it, especially for evil characters and their weapons, for example. You definitely want the audience to ‘feel’ the evilness and this is a great way to do it without being in their face about it.

In games especially, part of the goal is for all of the sound effects to feel natural to the game and not bring any undue attention to themselves. By doing this, the ones you want to really stand out will definitely be noticeable and have more impact. I’m always very careful for sounds like tractor beams, engines, humming machinery or ambience to tune them so they all work together and with the music.

DSR: How loud do you monitor when working 8 hour days (if you do that)?

AM: 8 hour days…what are those? It’s more like 12 to 14 hour days sometimes when the clock is ticking. Ear fatigue is definitely something on my mind when sitting in front of monitors all day. I do listen during the creation phase at a relatively low level, the volume control never really goes over 2 or 3, all to protect my ears and my sanity. But, that’s not to say I don’t turn it up and feel it a bit more at times. After I’ve created all of the sounds and before I deliver them to a client, I do a final mastering step at full volume to make sure they’re doing what I wanted them to do. And even then, I don’t call them ‘done’ until I return the next morning and give everything a fresh listen.

DSR: What are your staple pieces of equipment? Go-to EQ, mic, compressors? Does each different sound require a different piece of equipment?

AM: For field recording, my trusty Fostex FR2 is the first thing I grab and then whatever mic suits the situation. My ‘go to’ mic is the Neumann RSM191but I also drag along a Sennheiser 416, Røde NTG2, NTG3 and the NT4 and a couple Countryman B3 lavalieres if I anticipate the need. As far as using compressors and EQ, I don’t use them much and don’t use anything special, I’m afraid.

I’ve gone entire weeklong recording sessions with the 191, recording vehicle Foley and interior sounds while underway and never thought twice about using another mic for any of it. But, we also had other recordists and gear to capture engine, exhaust, suspension and track/wheel sounds since the 191 wasn’t practical in those situations. The type of gear you ultimately use, I think, definitely depends on what your are recording. Quiet ambience, gunshots/explosions, vehicles – all have their challenges and that definitely requires the right tool for the job for each of them.

DSR: What determines if you are going to record an effect in stereo or mono or both?

AM: I generally try to record everything in stereo regardless of the client requirements and convert it to mono during editing if needed. My thought is, if I’m making the effort to be out in the field to record sounds, I want to capture them in the most natural perspective which will give me the most flexibility when editing. It’s definitely easier to make a stereo recording mono than the other way around.

Now, that’s not to say that I don’t record in mono. If the client is very specific about how a sound is recorded, then I will definitely do as they ask. Typically when I’m doing multi-track recording of vehicles, for example, there will be mono recordings of the engine, exhaust, suspension and wheels. Since these will be mixed in some fashion as stereo or surround, mono works perfect. Weapons recordings are another good example as well, especially when capturing multiple perspectives to recreate localized shot sounds. Video games make use of mono sounds anchored to specific points which give a player clues to its location – and these need to be in mono.

DSR: As close to the sound as possible just doesn’t make sense to me sometimes. What distances do you record sounds from?

AM: There are many reasons why you’d record as close as possible – subtle sounds you want to emphasize, to exaggerate the sound more, to take advantage of the mics proximity effect, for perspective and for loud background sounds you want to minimize. For me, mic placement depends on the sound I’m trying to record and what my ultimate goal for the recording is. If I’m using it as an element in something else, the more it doesn’t sound like what it is, the better and ‘major’ close up is my choice. But, if my objective is to capture the sound realistically, then I’ll judge that distance at the time and do it from there.

Perspective is very important in field recording. Most sound library effects have been recorded at a close perspective because that is the most useable. But taking a sound recorded up close, lowering the volume, adding some space to it with delay or reverb, adding a touch of ‘ambience’ and rolling of a bit of the high end still won’t give you a believable ‘distant’ perspective. If I’m looking for that specifically, then I will record it that way.

DSR: What essential “industry standard” plug-ins and manufacturers are there, which are widely used especially for sound design work? Just like there are those “standard” plug-ins in music production and digital audio engineering like Waves stuff, Massey plug-ins etc.

AM: Personally, I don’t really see a difference between the two – music production and sound design are the same, well, at least to me anyway. I use the same equipment, software and plug-ins for both and what I use depends on what I’m trying to do with the sound. I’ve created sounds using everything from Nuendo, SONAR, Vegas Video and ACID to Audition and Sound Forge. I’ve even used ‘music’ programs like Kontakt and Cameleon for sound design. I consider everything a ‘tool’ and grab whatever I need out of the shed to stay fresh and unique.

DSR: The Neumann RSM191 seems very popular among the sound design professionals. Do you have any suggestions on reliable and good quality shotgun microphones currently on the market with maybe a more beginner-friendly price tag to start field recording with, like the Røde NTG-2? I guess the AKG 414 still goes as the all-around condenser? Cheers, -Just starting in sound design-

AM: Yeah, I love the 191 and always have really good luck with it. I highly recommend it! Budget wise, there are some very nice mics out there – the Røde NTG2 is definitely a good one for the price, I still use mine as well as the NTG3. The MXL FR303 and FR304’s are worth a look as well, especially for the price. They can’t compare to higher priced mics of course, but for $100, they at least get you started. Every mic has its own ‘character’ and ‘color’ and you should never select a mic with the ‘its expensive, it must be good’ attitude. I’ve had good results with some of MXL’s products so I’m not afraid to use ‘em. Plus, since they’re so economical, you can put them in places you wouldn’t dare put an expensive mic!

DSR: Mr. Marks, I have a question for you as well. I’ve just finished reading your “Complete Guide to Game Audio” and noticed that you refer to a sound designer and a composer as a sort of one-trick pony. You give justification for this on numerous occasions as having the ability to “provide a one-stop shop” for designers. Do you foresee this being the way audio is done for another good long while, or do you foresee a specialization in the industry like that of film, in that separate organizations/people are contracted/hired for sfx and music separately?

AM: That’s a great question and a difficult one to answer with any certainty. I have noticed a trend toward specialization and I’ve even adapted my own business profile to keep up. But, for every company I’ve done business with and job I’ve worked on that was compartmentalized, I’ve seen just as many instances where the one man band profile is very much alive and kicking. If you compose and do sound design, you have a great excuse to offer both as services. If you only do one, the other option is to partner with someone else who does the other very well and offer your services as a team. Either way, you’re covered and won’t lose jobs because you weren’t prepared. And, after all, you’re not only in this to contribute some great audio work but to make a career and a living at it as well so you shouldn’t let any opportunity slip away.

DSR: I also have a question I’m not entirely sure has a good answer. Say I have a particular passion for a specific type of audio (namely vehicles). Would it be unwise to tailor my demo reel to focus on this passion? I realize doing so would narrow down my potential employers by doing this, but I’m trying to gauge if that’s truly a bad thing.

AM: Specializing in anything is definitely a double-edged sword. The more services you can provide, of course, the more income potential you have. But, by doing one thing really, really well, so much so that you become the one person anyone thinks about when they need that specific thing, then you can monopolize that segment of the industry and do very well for yourself.

I know some sound designers and field recordists who specialize in vehicles or weapons and they definitely get a lot of jobs because of it. If you’ve got the confidence that you can truly be that good, I say go for it! But, in the meantime, be open to other things, keep the jobs coming in, learn additional skills which might translate to making your passion even better, reinvest your income on top-notch equipment and work towards that ultimate goal. There’s nothing that says you have to only do that ‘one’ thing until you become the master but definitely plan your path and stick to it if it means that much to you.

DSR: Finally, will you be at GDC this year? I’m trying to decide if it’s worth the money to go at this point in time.

AM: I always say I’m going but never know until a few days before whether I can get away – so, it’ll be a surprise. But, to help your decision, GDC has never been a waste of time for me. Getting out of the studio for a few days is always a good thing. Seeing old friends and making new ones is great. The conference is a great way to catch up on what’s happening in the industry and the expo floor is perfect for checking out new tools, current games and even job opportunities.

Just remember, if you do go, GDC is what YOU make of it. You can walk around for a week with your head down and waste your time OR you can mingle with all the other people passionate about the industry, talk to everyone (and I mean EVERYONE!) – especially the folks in line for coffee or sitting at your lunch table, pass out business cards, collect them and then follow up when you get home. I’m one of those people who are fairly introverted and I used to have to force myself to open up and be a part of the show. If you are a natural ‘people person’ then all the better, but work at it a bit if you aren’t and you’ll be very surprised with what can happen when you do.

Oh, and if you see me there, definitely wander over and say ‘hi’. I’d love to meet you!

There is a nice article at The Kartel, providing some notes and ideas given by Akira Yamaoka on his lecture at GDC 2010. Check:

Surival-horror fanatics, despite their differences, can all agree that the Silent Hill series was revolutionary for creating both an immersive and creepy environment. Through the use of haunting sound effects, and an original soundtrack, sound designer Akira Yamaoka was able to create a terrifying world that stood out to gamers for years to come. Attending GDC for the first time this year Yamaoka spoke on the power of sound, his theories on composing music, and his plans for the future.

Akira first introduced his culture, stating that the Japanese have a meticulous attention to detail, and are constantly seeking perfection. He explains that the recent advancements in technology have presented the opportunity to get much better results for sound design. He also expressed hope that future videogames will incorporate all five senses, to fully immerse the player (Can you imagine tasting a videogame? Yuck…Donkey Kong). Akira discussed his different opinions on music design, and some little known sound illusions below [...]

Read more…

There’s another interesting article at GameSpot featuring Yamaoka’s lecture…

Who Was There: Akira Yamaoka, who is best known for his music in the Silent Hill series, leveraged his 20 years of experience in video game production to discuss audio design in games from a producer’s perspective. He has worked not only as a composer, but also as a sound designer, sound director, and producer. Having recently left longtime employer Konami, Yamaoka is currently working with Suda51 on a new game at Grasshopper Manufacture that will be published by EA Partners.

What They Talked About: Yamaoka’s 2010 Game Developers Conference session began with a few slides that highlighted some facts of Japan, including the number of islands (6,852) and coastal line (comparable to that of Australia). Yamaoka wanted go over his background and heritage because the attention to detail that is part of the Japanese culture ties in closely to what he ultimately strives for: perfection.

Read more…

Recently, Dolby has unveiled two new technologies, both for films and video games.

The first surprise is the announce of Dolby Surround 7.1 at ShoWest 2010.

Dolby Laboratories, Inc. announced it is working with Walt Disney Pictures(R) and Pixar(R) Animation Studios to deliver a new audio format, Dolby(R) Surround 7.1. Disney(R) and Pixar have stated that Dolby Surround 7.1 will be launched in select theatres with the release of Toy Story(R) 3 in 3D this June.

• Brings a more exciting sensory experience to audiences for 2D and gives content creators control over audio placement in a theatre when mixing 3D movies. The ability to compose audio with visual elements of 3D allows content creators to immerse the audience deeper into the movie with dramatic realism.
• Provides content creators four surround zones to better orchestrate audio channels in a movie theatre environment. The four surround zones incorporate the traditional Left Surround and Right Surround with new Back Surround Left and Back Surround Right zones. The addition of the two Back Surround zones enhances directionality in panning 360 degrees around the theatre.
• Comprises 8 channels of audio and has the following channel layout: Left, Center, Right, Low-Frequency Effects (LFE), Left Surround, Right Surround, Back Surround Left (new), and Back Surround Right (new). In order for exhibitors to deliver the new format, Dolby will be providing Dolby Surround 7.1 playback capabilities in the Dolby CP650 and Dolby CP750 digital cinema audio processor lines.

More info… | Via: @filmsounddaily

The other new technology was announced at GDC. It’s called Axon, which provides innovative features to voice conversations in video games:

Dolby® Axon represents the future of voice communication for online games and virtual worlds. Crafted from the ground up specifically for game developers, Dolby Axon enables a realistic 3D voice experience that matches the game environment and greatly expands game-play possibilities.

A host of innovative features makes Dolby Axon unique and flexible, including 3D voice capabilities. Distance attenuation makes voices become louder as players approach one another within the game, or softer as the distance between players increases. Surround panning helps players locate competitors spatially within the game environment.

• Realistic 3D voice communication complete with surround panning and distance attenuation
• Advanced occlusion engine that maps voices to the game environment
• Enabled voice fonts for improved role-playing
• Proprietary audio technologies to deliver a clear voice experience
• Flexible API for creative integration and game-play options

More info…

Audio

Slides

A few notes abot the gun and animal shoots that are featured in the talk:

The gun shoot was done over 2 days in the fall of 2005 just outside of Los Angeles. It was for the Sony PS3 game Warhawk. There were 3 recordists on that shoot. John Fasal was the lead recordist. Chris Clanin (Warhawk audio lead) and I were the other 2. The samples presented in the talk were taken from all 3 recordists (John ran the most channels, so a good number of samples are probably his, the Nagra was definitely his). I don’t remember how many channels we had or which samples came from which recorder.

The animal shoot was in spring of 2005 in Big Bear, CA. It was for a PS2 project I was working on at the time called Neopets (don’t look it up, it’s terrible). I brought along 2 other sound designers from Sony (Jeff Darby and Brad Aldredge) to assist me in recording, each recordist ran their own machine (either in stereo or mono). I remember I used a Fostex FR2 at 24bit 96K with a Sennehiser MKH40 mic. All of the animal samples are from my recordings.

As here I sit four years later having been lucky enough to participate in the undercover debug of several titles, mouth still agape at the possibility to visualize sound and sound properties as a way to understand what is “going on” sound-wise at a given moment, I continue to be fascinated by these environments created by hand for the sake of debug. While this aspect of game audio may continue to be a closely held secret of developers leveraging internal pipelines and processes, a few screens have escaped which show off various functionality.

There may be more out there in the wild, but in my cursory scrubbing of conference presentations and recalled articles of the past – alas – this is all I’ve been able to turn up. (feel free to add links to other in game audio debug screen grabs in the comments section.)

In addition to presenting – what was then – state of the art examples of sound design and under the hood implementation considerations such as: speed of sound, distance propagation of sound, and reverb interpolation; he also went on to highlight an example of creative realtime DSP in Bizarre creations XBLA title Geometry Wars.

In a brilliant execution of a simple idea, Geometry Wars took a Low Pass Filter and applied it to the entire mix when dropping a bomb, submerging the high frequencies of both sound & music, and then releases the applied filter over the course of the next few seconds.

Listen for it at 2:27 in the following video:

Elegant in it’s simplicity and beautiful in it’s effectiveness, this creative use of DSP illustrated one of the strengths of the current generation to be able to effect sound dynamically, reactivity, and interactively as a way to further meld sound to gameplay. At the end of the day, finding ways to nurture and support the complex relationship of interactive audio in games is something we should all be reaching for.

The following article is an excerpt from “The future of sound design” lectures done during GDC, VFS and DFF between the periods of 2006-2007 by Charles Deenen. Rewritten for “DesigningSound.org”.

The Future of Sound Design in Video Games, Part 2

Every day in our life, sounds occur. Our brain thinks about virtually every sound we hear, and depending on how we’re connected with that sound, either (unknowing) react, translate, notice or feel something. On each sound we hear, we connect a space, event or happening to it, thereby learning how or when these sounds occur. Each human experiences sound every day, and learn about them, just like we learn a language. Some sounds are artificial, yet we connect a certain “moment” to them (i.e. computer graphic beeps in movies)

Now, you probably think “blah blah, yeah, yeah, where the heck are you going with this”. As a sound designer our job is to learn that language, transcribe it, understand it, and enhance it. You can easily make use of the human learned language of “experienced” sound. Fundamentals in sounds can be used together to form a combined emotion.

As an example, I’m sure you sometimes hear a distant fighter jet come by. By hearing it’s fantastic air-distorted rippling effect you instantly know it’s going incredibly fast without seeing it. When you hear a wasp buzzing, your instinct probably says “holy crap, swat it!”, all due to your brain associating a previous reaction/learning to that sound.

So, lets put this in practice by listening to some examples. The following is an example of standalone sweetners which would be used to add to parts of a car-by sound. It’s various ways of adding a “scream”, “howl”, “pain” or other sense of emotion to a car-by sound. Something probably that most of you have done, but why you did this is really the question:

As sound designers, we often try to emulate a certain sensation we’ve heard in real-life. Instead of using the real thing, we can reproduce the same effect by mimicking the feeling we want to chase. In this case we’re after alternate ways to add a sensation of speed to a car by. First sound is a regular Jet-by, followed by the pictured sounds, finished off by the car-by with the added ‘speed’ layers (example contains sounds from various sources)

To quickly grab a sound “feeling” in the future, one of the things I did a long time ago is make a list of various sounds which moved me somehow. Then I wrote down the feeling they impressed on me, and some ideas on how to use them.
Every time when I was in a rush, with little time to think about sound design and feelings, I pull up the list and scan it for my previous thoughts. Usually I find some ideas in it, especially in relationship to expressing a certain (combo) emotion.

• Make a list of “experienced” sounds and the emotion you feel when you hear them, with possible ideas on how to use them

“Fighter Jet fast By” – Speed sensation – Car by Sweetners

“Jail Cell Door close” – “Closed off feeling” – Logo slam

• Create some new sounds by using the character of what defines the “experienced” sound, in combo with some “plain” sounds.

S(t)imulating Learned Ear Deficiencies


The ear is an odd looking piece of human isn’t it? On some, it sticks out like a TV dish, shaped like a weird alien disformed growth, freezes when you’re in Montreal during the winter and oh yeah, does this awesome thing with sound. The ear behaves in weird ways sometimes. Now, this is not scientific in any way, so please don’t rant to me about it. It’s purely my own speculation on what happens.

Have you ever noticed that your ear seems to “compress” or partially shut-down the high-frequencies when it hears an instant loud sound? Or how it seems to “warble” when there’s a lot of low frequency build-up in the real world?

There’s a few tricks you can use (and maybe use already), which mimmic this ear behaviour, thereby tricking the listener into believing they are really hearing something much “bigger”

• ‘Low frequency Distortion’ is read as “loud” by the ear. It mimics the high frequency compression that’s happening when you’re listening to very loud sounds, and at the same time adds to the air distortion that happens between the source and your ear. The first sound in the below example is an indication of this (that sound btw was simply made by boosting the low-end by an incredible amount, running it through a crappy (behringer) compressor which freaked out about this low end, and then reducing the low-end back down.
• I believe your ear and brain have learned to translate spiked high-frequency sounds going into mid-range/low-end as “loud”. It’s a similar scenario that happens on guns & explosions. You can easily mimic these sounds by simulating an ear “shutdown” where it rejects higher frequencies after a brief moment. Lets call this the “Hi/Lo offset technique”; offsetting high frequency sounds, followed by a mid/low frequency sound. Try it out, and realize why does this sound “loud” to you?

Example: ear_def.mp3

Build-ups… ok, no rocket science here :) Any sound rises can be used to create a build-up of tension. There is a large variety of ways to do “rises”. Many of these techniques are used in Film Trailers daily. Not all are commonly used in gaming though;

• Music rises (the most obvious one)
• Building Rhythmic sound effect (i.e. pulse growing in size over time). Very effective to draw attention.
• Pitch / Frequency increase, effective in denser layers as you’ll certainly notice these through a dense layer of “noise”.
• Volume increasing over time
• Repetition frequency increase (i.e. pulse speeding up). The most overused sound attached to this is probably the heartbeat (yikes ! :)

In gaming I’m usually only hearing music used as a way to build intensity over time, yet the combo of delayed animation spawning combined with the above can create such a nice build-up of tension.
Now the cool thing is that the opposite (the descend), creates the opposite feeling. The feeling of calming down, loosing, “letting go” etc. can all be enhanced or created in that way.

Example of some “pitch” and “Volume” rises back to back. Some of these have been used in games. Starting with the most known “rise”.

rise.mp3

Hopefully some of this was useful or interesting to you. My hopes for this article are that you consciously use these “experienced sounds” in your daily sound design, and hopefully better understand why we actually use them, or shape certain sounds to begin with. Shoot me any comments you might have.

All movies and sound contained in this article are (c) Charles Deenen and cannot be distributed or used in any way without prior written consent.

The following article contains excerpts from the “Future of Sound Design” lectures at GDC, VFS and Dutch Film Festival originally presented in 2006. Please note that certain expressions are personal opinions, and cannot be read as “fact”.

In our endless passion to make games have a similar, or exceeding sound-scape experience in comparison to other media, we constantly try to find new ways and techniques to obtain this. Some people ask “why are we comparing ourselves to film sound design, we’re very different”. Other say “Film sound experiences are the ultimate goal”. I say both are right. But to really figure out what the future may hold, we have to first learn from the past to enable measurement of missing objectives and goals.

To answer, we have to being by asking ourselves some questions:

• What’s been done in the past?
• What’s broken/missing?
• How does this compare to Visuals?
• What about Emotions ?
• Is there a future for Audio?
• What about everything else ?

The Past – Evolution in Numbers

Technical hinderances, ever since the X360 and PS3, have been much less of a hurdle for a sound designer to create engaging soundscapes. Lets look at the history, based on the most popular game machine/console during each period.

As you can see in Fig 1 & 2, the amount of sound-data currently storable on the console is so much, that in comparison the old consoles barely show up on the graphs. Memory isn’t really a technological barrier anymore.

What’s broken/missing?

The obvious one to look at would be the Sound Designer/Artist. Are the requirements of creative vs technical understanding still too high? Are they a hurdle we still have to overcome? In fig 3 I’m showing my estimates of the job requirements of a sound artist/designer working in the video games industry, when looking at the biggest selling platforms. Funny enough, the industry seems to repeat itself. Consumers are now using phones and other small devices to play games. These devices seem to be equally powerful to game consoles 4-5 years ago, which brings back the same technical hurdles, all well known and documented.

The 2nd obvious one to look at is the “no boundary” story telling experience. During the years I’ve noticed that for some sound folks who grew up in the technically restricted era, its very hard to cross-over to new platforms. As an example this is one the reasons I originally stopped doing music in the mid 90’s. I was pretty good at making small processors like the C64 and SNES do things they weren’t meant to do, and therefor got an edge on making enjoyable music. With the introduction of Redbook (CD) audio, the playing field was open to everybody, and I was no longer able to take advantage of any hurdles others hadn’t overcome yet. It’s critical that these folks find ways to move to the story telling, un-inhibited way of thinking. They have to gain this experience, or they’ll be left behind.

So what does this lead us to? To me, the biggest general missing link in making games have equal or better sound experiences than film is an investment in emotionally believable audio followed by treating the player smart, in both gameplay and audio. Lets focus on the first one for now.

Too often when playing games (including our own), I still feel “disconnected” from the experience due to sound. Some games make a great attempt at it, but in the end, there’s always something happening which causes the de-focus from the experience. During the remainder of this article we’ll touch upon what causes these disconnects. To understand this better, lets also look at how audio and visuals work together

Visual Media

Look at the below pictures. You’ll probably have a different instant feeling or emotion about each. You can tell that approaching visual realism isn’t always a good thing (the “uncanny valley” effect). It distracts from the believability, and the connected emotion you’re supposed to have. We’ll see later that Dialog has a similar issue.

Another interesting effect is that feelings generated from visuals can be interpreted different from person to person and feelings created by visuals are culturally relevant at times. Images generate a feeling, a response that we learned during our life.

VISUALS and AUDIO –  THE MARRIED COUPLE

Now, lets do a quick exercise to see the relation-ship between visuals and audio (note: due to copyright, we can’t put this music here).

• Pick one of the above pictures. Look at it carefully. You’ll notice the feeling you had initially withers away quickly.
• After a few seconds, cue up your favorite rock piece near the chorus… Did the meaning of the picture change?
• Now cue a film-score (I like to use “seven pounds” as an example). How dramatically did the meaning of the picture change, but importantly, did the feeling it generated in combination with the picture sustain?

What can you conclude out of this? How does Audio fit into this picture?

• If picture gives you the instant feeling/reaction, audio maintains this feeling over time.
• Audio cues can change the expected emotion a picture generates
• Audio can enhance picture in more than a support role and change the emotional outcome
• Audio emotions take (usually) time to establish

A quick word about Feelings and Emotions

We have to understand when to say “Feeling” and when to say “Emotion”, as both are pretty different. It’ll also help us understand how audio plays a big part in this.

• Feelings are a learned response of the culture and your surroundings in which you grew up.
• Feelings are a subset of all your mind-body states (i.e. disappointment, hunger, hope etc.)
• A Feeling is the response part of the Emotion. (“I feel disappointed”… a resulting emotional reaction could be “I’m Angry”)
• Emotions are cross cultural – the same meaning all over the world
• Emotion is a chemical state in our brains. Those same chemicals inhibit our capabilities and limiting what we call rational thought

So how can a game use this ? If visuals and audio work together to establish feelings and emotions, you can use this to a certain degree to influence game-play:

• More emotion = less judgement
• If you want to remember something, get into the emotional mood you were in when you first experienced it.
• You’re likely to come back to a “liked” emotion. Some emotional states can be addictive.
• Person’s mood tends to follow that of the situation presented in front of them.

OK, so lets get back to Audio…..

The main ingredients of Audio in Game:

Everyone reading this probably knows the 4 main “technical” ingredients of audio in a game: Music, Dialog, Sound Effects and Mix. During the many lectures on this topic I always asked the question to the audience: “What is the most important element to a believable and emotionally engaging soundscape”? “What is the top contributor”, and “What is the top damager”. The answers usually ranged across the board, each picking their “favorite” one. Composers would pick music, sound editors would pick sound effects. Repetitive footsteps was often mentioned. Dialog was the most picked damaging ingredient…. Seldom the answer was “all”.
It’s probably obvious to you that every single ingredient is of equal importance to create an emotionally believable soundscape. You can’t approach a single ingredient in a lackluster way. Believability is key.

Believable Dialog

Dialog is still the #1 offender in believability area. I’ll get probably flamed for saying this, but I’ve yet to hear a single game which makes me believe I’m listening to the characters on-screen for an extended period of time. None have captured it as truly “believable” yet. Space, placement, acting, story, odd breaks, visual discrepancies etc. all contribute to dialog flaws. We’re running up against the same “uncanny valley” effect as visuals. We’re approaching reality and the human ear will now pick up every flaw, and is no longer forgiving. Yet if we’re very far from reality, we’ll believe it. Have you noticed you can watch a saturday morning cartoon and believe the characters?

What also isn’t helping us is that some characters in-game on-screen still look robotic (or don’t even exist), making us having to work even harder to make that certain voice believable in contrast to the visuals.

One issue I’m hearing quite a bit is the recording method used. Lots of dialog for games is still recorded in the traditional “music” way of placing a U87 (or similar) close-up to the actor. Often, introducing movement, space, air, body shocks is totally negated. 

Game Example: "NBA Street"

In 2005 we did a quick test for a game called “NBA street” for which no longer accepted this type of recording. We rigged up a lot of players with wireless lavs, and had them play for several hours while feeding them scenarios and lines. The resulting effect was a much higher degree of believability. Following is an example with the “U87” version back to back with the “Lavalier” version. On purpose, similar hokey lines were picked to illustrate the concept (this was not a concept of good acting). Which one is most believable to you?

Example: nba.mp3

#1 is the old U87 version, #2 is the Lavalier version, separated by a beep

Believability Gap

Believability “gaps” are the #2 offender. Everytime a player is jerked away from the game’s believability, it makes him realize it’s a game, and intensity lessens. When this happens the intensity buildup has to be restarted to a certain degree. The game will never be able to reach the full potential of engaging, emotionally believable audio. Some examples of these gaps are:

• Awkward forced Loading screens (i.e. “silence screen”)
• Repetition on dialog, sound effects, or anything else noticeable
• Un-natural imbalance (Vol / EQ / Space etc.)
• Non believable Dialog
• Anything which goes against “learned” sounds, if not on purpose (more on this later)

Audio Mix – “The Glue”

This is a large topic, one that’s too big for this article to cover, to which we’ll come back later. It’s the #3 offender to create sustaining believable soundscapes. Too many games still ship with the “wall of sound” approach. A player can only take this so long. There are many solutions to this problem. One of the causes still seems to be “producer X listening on his TV in a noisey floor area wanting to hear every detail”, which is a hard one to overcome.
With any of the solutions, the mix shouldn’t make a user notice what’s happening, yet get more engaged. If your producer is asking for the “wall” approach, he’s in reality not asking for this. He just couldn’t hear something he wanted to hear. It’s now up to you to refocus the mix constantly to allow him to hear what he wants to hear, yet doesn’t take away from the rest.
Much more to come on this topic as there are many tricks to accomplish this.

Excuses

One of the other blockers in achieving an emotionally engaging soundscape is excuses. Yeah, the ones that every Sound Designer or sound-sup makes when he can’t achieve the needed result. We’re all guilty of it (including me). Yet, it’s one of the hurdles we have to overcome if we ever want our industry to excel above the film media’s level of engagement. If you know you can’t mix for an emotionally engaging soundscape, don’t do it. Don’t pretend you can learn it overnight. If you don’t have enough money to achieve the result, scope down, sell your ideas to execs or whatever you have to do. Don’t use’ em as an excuse why you couldn’t succeed. There are many reasons, many which are direct blockers, and many which can be overcome with creative solutions. I often hear “well, we don’t have the tools that some others have”…. Tools are a means to get to a result, but not the only way. Come up with creative ways to get the tools you need.

Wrap up

So what is the future of sound design ? It’s not some sort of new tool. It’s not a new console, it’s not a new “cool way” of creating sound in real-time. It is purely us overcoming our hurdles to find new ways creating emotionally engaging and believable soundscapes. Breaking out of traditional ways, learning what the human reacts to. How feelings and emotions tie in with sound is a must-know Content is no longer king, technology is no longer the queen. The combo of all of it, and the stimulating, game supportive result is what the player will be experiencing and wanting.

–end of part 1– We’ll look at “ear deficiencies” and “experienced sound”, including methods on how to use those to build an emotional soundscape, in the next few days.

Written by Charles Deenen for Designing Sound

Four lectures of Rob Bridgett with information and good tips for game audio production, focused on the audio model on Scarface.

1. GDC CANADA 09 – POST PRODUCTION AUDIO PANEL

A moderated panel discussion on game audio between Rob Bridgett from Radical Entertainment (Scarface), Jason Ross from Relic Entertainment (Dawn of War II) and Gordon Durity from Electronic Arts Canada (Def Jam Vendetta) moderated by Leonard Paul of Lotus Audio (Facebreaker). Topics focus on the final stage of game audio schedule such as mixing, mastering, sound replacement, 7.1 surround and last-minute fixes before shipping.

• Intended Audience: Sound directors, sound designers, composers, audio producers and anyone directly involved in the audio production process will benefit from this session. A strong knowledge of game audio is advantageous.
• Idea Takeaway: Attendees learn how the game audio pros polish their audio in the final stages to achieve AAA title audio. An inside look into the techniques of advanced game audio for leading game companies is contrasted and demonstrated.

2. AES 2008 INTERACTIVE MIXING LECTURE

In video game development, audio postproduction is still a concept that is frowned upon and frequently misunderstood. Audio content often still has the same cut-off deadlines as visual and design content, allowing no time to polish the audio or to reconsider the sound in context of the finished visuals. This tutorial talks about ways in which video game audio can learn from the models of postproduction sound in cinema, allotting a specific time at the end of a project for postproduction sound design, and perhaps more importantly, mixing and balancing all the elements of the soundtrack before the game is shipped.

This tutorial will draw upon examples and experience of postproduction audio work we have done over the last two years such as mixing the Scarface game at Skywalker Sound and also more recent titles such as Prototype. The tutorial will investigate:

• Why cutting off sound at the same time as design and art doesn’t work
• Planning and preparing for postproduction audio on a game
• Real-time sound replacement and mixing technology (proprietary and middleware solutions)
• Interactive mixing strategies (my game is 40+ hours long, how do I mix it all?)
• Building/equipping a studio for postproduction game audio.

3. DEVELOP CONFERENCE 2007 – INTERACTIVE MIXING LECTURE

• Dynamic Range: A Study of Software DSP & Run-Time Mixing
• With reference to his work on ‘Scarface: The World Is Yours’,Bridgett will examine the absence of subtlety and silence in the audio of many recent video games, manifested by a lack of dynamic range, over-compression of sound and music assets, and leaving little potential for narrative tension and release

Specific topics will include:

• Production practices in which sound, music and dialogue have been consistently over-compressed at the individual asset level
• Interactive mixing and post-production as a valuable area where dynamics can be artistically controlled via such real-time DSP effects and run-time snapshot mixers to dynamically shape and prioritize the overall sound in an interactive environment
• Delegates will take away a greater understanding of how run-time mixing can re-define the dynamic range of game audio.

4. GDC SAN FRANCISCO 2007 – POST PRODUCTION AUDIO MODEL ON SCARFACE

• Session Description - Scarface used a unique model for its audio, combining conventional audio development from the beginning of the project and augmenting this with a movie-like post-production audio phase at Skywalker Sound. This essentially concentrated on two major elements, sound design and a final mix.
• Post production Sound Design - The audio development team spent a week with Oscar winning sound designer Randy Thom reviewing the game’s audio and took away assets just prior to alpha. Returning two months later with most of those sounds implemented for a final week of concentrated sound effects replacement.
• Post Production Mix - The entire audio development was also taken to a movie mix stage for three weeks where all the sound for the game was mixed. This used unique proprietary technology that allowed a mix control surface to be attached to all the sounds in the game and enabled the game to be mixed by a motion picture mixer Juan Peralta.

The lecture talks about the work flow aspects of taking the audio development off site, as well as the technical and aesthetic advancements made during the Scarface project.