Andy Farnell – a familiar name in computer audio – is a computer scientist, sound designer, author and a pioneer in the field of procedural audio. He is a visiting professor at several European Universities and a consultant to game and audio technology companies. His book, ‘Designing Sound‘, is a bible for procedural sound and should be on your bookshelf, if it isn’t already!

He was very kind to find time in his busy schedule when I visited London, and we talked about what procedural audio is, where it stands now and what it can be in the future. This article is a transcription of our conversation, which he was again very kind to edit along with me. It was no easy task because there was so much good content!

Thank you Andy!

DS: Where does Procedural Audio stand now? Would you say it is comparable to where CGI was in the 70s/80s, when computers weren’t powerful enough?

Andy: That is a central mythology – that the computers aren’t powerful enough to do it. This is often brought out as a straw man argument against Procedural Audio by skeptics. One of the things I did with my 2005 demo was to make all of the sounds (they weren’t very high in quality) that you would need for a first person shooter game – fire, water, wind, rain, some animals, some footsteps, some guns, some vehicles. This was 2005 and I had them all running on a 533 MHz processor generating a realistic-ish sort of soundscape to prove that if you had 1GHz processor and if you used half of it for the graphics then it would be quite possible to synthesise all the sounds using the remainder. Six years after doing that people would still come to me with this straw man argument, they would say, “You know Andy, we love this Procedural Audio stuff but there’s just not enough CPU available”. But we now have two to the five times more CPU than when I did my 2005 proof-of-concept demo. So, what’s behind that? Why are they saying that? It’s not true. What happens is the internal politics of resources. The requirements always expand to fit the resources available. The game worlds get bigger and bigger and the graphics get more and more demanding. The audio team will always have the least amount of CPU allocated to them as an afterthought, because in the current structural model of production sound is “post production”, and no body wants to commit to giving audio that much CPU bandwidth. I feel that is the real reason behind the argument. You often get these straw man arguments that enter in to a culture and just get recycled. People know that there is an argument and it comes to their tongue very quickly and they say “Yes we could do it but there is not enough CPU”. With the left over CPU on a modern games console I could provide you great procedural sound.  On an eight core architecture, we would need one or two CPU cores to give procedural sound. Even more interestingly is what happens when we run models in GPU, and many Procedural Audio models are inherently parallelisable. So, yes, Procedural Audio is somewhere in that era before the Tron movie, or before the Pixar CGI revolution, its possible, but not yet seen as viable, perhaps the shift is too painful for big companies to make.

DS: Have you tried doing a similar demo using today’s technology?

Andy: No. I just don’t have time at the moment. Life is moving so quickly and I’m involved in so many other interesting projects. Some of them to do with computer science, some of them to do with philosophy. Interesting times, but in the future I want to have a research department and have a bunch of guys, really smart guys who are just on this and want to do it and I can help direct their research. Because, when you look at what Procedural Audio breaks down in to, its actually very deep specialisations – just like CGI. There is room for intense talents within the area. Let us make use of CGI analogy again – if you are a real good texture artist you are great at looking at skin and saying, “That is the skin of a 40 year old, that is a skin of a particular kind of salamander, look at the way the bone structure moves underneath it, look at the way the light hits it”. You get specialisations within Procedural Audio which would be people who are very good at fluids, they are great at doing water falls and drops of water and boiling mud and lava. They understand that sound. They are able to model it and come up with great sounding objects and great processes that do it.

Before I took up this umbrella term, this banner of Procedural Audio, and tried to make a focussed idea out of it, I had mentors – I mean people I looked up to, leaders with ideas that nobody else was doing in industry or academia people like Perry Cook. He is like the grandfather of Procedural Audio. He was doing it in the early 90s, when the argument that there wasn’t enough CPU really was a good argument [laughs] and then after him came Dinesh Pai and Kees van den Doel and they worked on impacts and fluids. They did that as very narrow academic work. I don’t know if they saw (the generality of the possibilities) that the water could be taken and integrated with a glass so we get an object that could be filled up or emptied, or become raindrops in a particle based weather system that interacts with different objects the rain falls on… The object-object interaction based idea of “sounding objects” really came out the North Italian schools, otherwise we just have event driven sample playback . But, they did extremely good work and a lot of my stuff is just interpreting their work and generalising, extending it, and making a coherent philosophy of sound as process rather than data. We must always be mindful of that background to it. It didn’t just pop out of the air. It is a project that has been in the background (since Mathews in the 50′s)  growing slowly. If anything, I have  been a very vocal advocate of these ideas applied to the general case of everyday sonic simulation, and been instrumental in defining what procedural audio is.

DS: So the obstacles aren’t purely technological?

Andy: I don’t know what the “real obstacles” are now. I’ve said in another interview before, that around 2006-2007 it dawned on me that there weren’t any fundamental obstacles to radical technical progress. We could do this. The obstacles were structural and political. How do you introduce a new technology? How do you get people to take risks on that? One of the weaknesses of it, it’s a weakness but a very deep philosophical strength (and this is quite subtle), is that sound as data fits in to a capital model. Intellectual property allows you to own a sound asset. So if you record or create a sound, it is an asset that you own. You can trade assets. But procedural model breaks with an ownership model because what you are doing is you are substituting general sounding objects for something we can make million sounds in the future. There is no redundancy built in.

DS: Although, if a game developer spent time and resources building a procedural audio engine, as they would spend time building an audio engine, wouldn’t it be an asset that could be owned?

Andy: The observation there is that the code is the asset. But the code is useless without a group of people who understand how to make it sing. We move the value from residing in the thing itself to how it is used. I see procedural audio as an art to be practiced, not just an application layer to be built.

DS: So even with Procedural Audio you will need a sound designer to understand what it can sound like and how it can impact a player/end user?

Andy: I think this was on the game audio forum or something years back. Someone raised the accusation at me that, “Guys like you put people like us out of business and you are making technology that is going to replace our art”. I took that on board as a very valid point. Being a sound designer myself, the last thing I want to do is put other sound designers out of work. I see it as a liberating step – you have your sound samples and you have this. I always see it as a complimentary technology and not a replacing technology, that is point number one. Point number two that is more important is that every new technology that comes along generates a new requirement for skillets which the talented people in that business become really good at. So every Procedural Audio team would need a good sound designer.  I wouldn’t leave it to the programmers, I want somebody who has a great set of ears and I would actually put them in a higher position and get them to direct the programmers and say, “No its more like this, listen to these examples. I want to get this emotion across”, and they can direct it aesthetically. It’s not really putting sound designers out of work and it is not a totalitarian project. This is why I worry that the bean counters, the alienating/asset-oriented capitalists, are seduced by this kind of technology because they just think, “Well we plug that in and we get rid of the sound department”. That’s not what I want to see happen. One of the great advantages is that it gives 90% of your assets for free. You just put your objects in the world and you get default sounds. What that means is that you don’t have to worry about an asset-event matrix any more. You don’t have to worry that somebody has forgotten to put a sound on something because everything will be covered by default and now the sound designer is liberated not to be thinking up every single little rock sound but to go and focus on the emotionally significant sounds – the hero’s sword, the getaway car, the gun sound. They can put all their time and energy in to getting those right and not have to worry about the other stuff. That’s another argument for Procedural Audio. It raises the bar from where you start from.

DS: So, Procedural Audio is just another tool in the arsenal of a sound designer? A combination of different techniques?

Andy: Procedural Audio is a philosophy about sound being a process and not data. In its broadest sense, if I were to say it is the philosophy of sound design in a dynamic space, and the method is as irrelevant to Procedural Audio as whether you use oils or water colours is to painting. If you use papier-mâché and glue or whatever is to sculpture – the end is in the artist and not the method. So you can mix and match the methods, they exist separately. In the industry now, all the successful Procedural Audio is mixed methods of samples – granular methods with the exception of as far as I can see of Nick Fournel’s work which is basically a similar kind of resynthesis but it is phaselet or PVOC type re-synthesis. First lets see why that happens. There is obviously a clear bridge there between existing technologies and the direction that Procedural Audio can go in. It gives you an immediate start. You can use your existing sample libraries and your guys out in the field. I incorporate this in to my understanding of Procedural Audio as: your sound guys now do analytical recording, not for the purpose of using those as final products but for exposing and analysing the sound underneath so you can build your procedural model. Granular methods are a very direct way of doing that. You just take the input sound and bust it up in to its component waveforms and then you re-synthesise them as grain clouds in different ways. Or in the phase vocoder or linear predictor sense you split them up into transient-exciter components and resonant parts. In that sense it is a direct re-synthesis. Now these approaches that have a method and analyses part, in effect, have a one to one mapping. So you are doing re-synthesis but you can fiddle around with the parameters in the middle. I call this a shallow or phenomenal approach. What it means is that is that the way in which you can change the sounds is limited mostly by your understanding of the parametric interface of the re-synthesis method. The re-synthesis method doesn’t capture any of the physics, it doesn’t capture any of the process of the sounds. Whereas building a procedural model, where you got a model and a method which compliments the model, you are  interested in the behaviour of the sound that is built in to your model. That is not phenomenal. That is not surface, I call that essential or deep. Guys who are doing that kind of thing are like Zheng and James, they are using really accurate models. They are basically doing what computer scientists mean when they say computer modelling. They are using fluid dynamic models to model fluids. They need a rack of computers that take days to process a few seconds of sounds, its not practical Procedural Audio, as would be used in games, because it does not meet the real time criteria for a start. At least not yet.

But having said that I also think there is another side of (non  real-time) Procedural Audio which is not in computer games. It’s in animation, where the idea is that once we have introduced the sounding objects as models, into the scenes, sound ceases to be post production. You re-arrange the objects and their behaviour in your scene to do your visuals and the sounds come out for free. You can even change the location of the microphone virtually after the fact. This is the future of cinematic sound , CGA or computational audio. The best thing about this is that we can drop the real time constraint and  trade speed for quality.  Why not have thunder rendered with one with a million N-waves, which sounds more like the real thing but which takes a long time to compute because you have got a render farm and you are a million dollar Pixar type company? You need good model programs and lots of computing resources but you don’t care about the real time thing. I think Procedural Audio encompasses that as an art.  Skywalker once kindly offered me computing resources, I think Randy Thom set it up, but its hard to work remotely out of context, I would like to put a masters student or PhD on that one day, I grew up with that rather dangerous Radiophonic workshop ethos of Oram/Derbyshire  (two original women sound designers),  creative research and commercial production combined, it is an iconoclasm of experts, expectations and traditions. Much too risky for todays world.

DS: Much about sound design is about achieving hyper reality and not reality. You wouldn’t want a gun to sound like a real gun. So is Procedural Audio about creating realistic models and building on them to achieve hyper realism?

Andy: Amongst the many good sound designers that I have met, they all turn out to be really quite well rounded smart people. They aren’t purely phenomenal,  see the world not only through an artists gaze, but an informed, worldly apprehension. Sound is about going in to the world, what happens inside and outside things, its about deep knowledge about how things work and what does that mechanism mean to your emotions. I think the role of what a sound designer is becomes somebody who’s language is not about computers but understanding the mechanisms of sounds. And the  natural progression for sound designers, when they run out of all the shiny plugin technology in the world is Procedural Audio because it lets their art develop, connecting with sound and its causes and its propagation and reflection. Deep knowledge of sound is what a sound designer has, even when they can’t express or vocalise it. People who have become really talented at that often have hidden knowledge, ineffable knowledge, they don’t have a way to vocalise it or make it explicit but they understand things intuitively and you see them in the studio just do stuff and if you ask them how they did it, they would go “I don’t know, I knew thats what was needed” and it gives them, I don’t like the word, but it gives them a “rationale” to explicate their knowledge about sounds.

DS: How does Procedural Audio fit in to this? Most sound designers aren’t mathematicians or scientists, most of them have a good understanding of how objects react in the world but not necessarily the science behind it.

Andy: In the very long run and I’m talking about – I hope to see it in my life time – ten, twenty, thirty, forty years in the future I think all of this that we are talking about today will be well understood as part of the discipline of sound design. I think the philosophy of Procedural Audio will be a part just as much an artist now can talk about textures and different kinds of lighting. In a way sound is a real throwback, right now. Some people have said to me, “Sound technology is fifteen-twenty years behind graphics technology”. That’s crazy, all of the algorithms that made visuals possible came out of sound. One dimensional signals first and then two and three.. it comes from radio and radar. And, somehow culturally, sound got left behind. Because, we are visual beings and we put all our energy in to manipulating and creating ways to have a visual reality. Twenty-thirty years in the future all of what we are saying now will be a part of the language of sound design. But right now, theres a fork in the road between my understanding of it – the philosophy and the way it should go and the way it will go practically – and what is happening. So part of my philosophy in education is principles not products. It’s a reaction against the commodification of skills and people, techniques, arts being reduced to products. So if you look on the sound design list when somebody says, “How do I make such and such a sound”, and somebody else says, “Oh you need the zzaaaq plugin, that does that”, and they completely abdicate any desire for knowledge. They don’t even care (to pay for it) because they can get it off bit torrent anyway. Somebody else has packaged that capability and knowledge and given it to them. And by doing that they have robbed them of the knowledge, (Zarathustra says: be careful what I give you as a gift, because I may take something away from you) the knowledge is useless by itself but as for an artist, for your career, your development, for your ability to do things, that knowledge is important, its part of it. Both the commercial approaches to Procedural Audio at the moment, and this isn’t dissing these guys (in Audio Gaming and Sony… I am in touch with Amaury quite a bit and try to help them out recruiting and seeing the way ahead), what they are interested in producing is products (that is business). They want Procedural Audio models as drop in objects in the game. With this model, given to the current middleware developers, you’ll probably have opaque Procedural Audio objects, you cant see inside them, they have a few exposed parameters. Say you have a car and you can choose four or six cylinder engine, a bunch of different configurations for the exhaust, you can choose the body material, the tyre and that would be it. That is good! That is how you want your end user to see the object. At a certain level. So one approach would be to sell these as closed objects, with their functionality is hidden. But, for me as an “academic”, as a pioneer I am much more interested in enabling (my relationship is with the sound designer, with people), you should get in to this stuff. To me, the good stuff will be toolkits like the kind Dylan Menzies and others have proposed  (and using the pluggable physical components using FEM/discrete numerical difference schemes like Stefan Bilbao explicates) Now you have an engine model and its a part of the car and you can replace the engine – real world analogy here – you can tinker, you can take it apart and change the the way the camshaft and the pistons work. You can replace the method that is used for that engine with a subtractive method that you wrote in C++ and drop it in because there is a well defined interface. And the kind of well defined interfaces that work are like data flow interfaces – like Max/MSP and Pure Data type interfaces where you can just plum these objects together very quickly in the studio and test them in-world while playing. Ultimately, open ended data-flow user interfaces with efficient JIT /bytecode compilers are the future for creative Procedural Audio. Then you will have a very vibrant community, a vibrant ecosystem of programmer-sound designers and sound designer-programmers and people who work in teams and Procedural Audio will be a vibrant technology. But I think first it will go through what basically becomes a plugin (mystified) culture. Procedural Audio in my (wishful) philosophy is open and based around knowledge [laughs]. That will come with time after these products have driven a path and we will think about sound differently. The construction of sounds will be more technically informed and richer than saying, “Oh yeah, Hollywood Edge track 6 number 4, that’s the one you need”.

DS: Would you relate the questions raised about Procedural Audio to people talking about motion capture replacing actors?

Andy: I never made that connection, that’s really good. Mo-cap in relation to CGI is the same as what I am calling analytical recording for Procedural Audio. You go out in the field and try and look at the behavioural features and try and capture them and then use them as data.

DS: So would that be the important point to make then? To use the analysis as data and not exactly copy it, just as how mo-cap is used in CGI?

Andy: Yeah, I was talking to some guys who are in to this mo-cap and human face stuff recently and they were telling me some amazing things about how Tom Cruise voiced an animated character and they got all his face expressions – the eye brows and everything. The trouble was it was a dog character or something and when they played it back it looked too much like Tom Cruise [laughs] – and this was how you could extrapolate to hyper reality where you just scale everything so that the eye brows doubled the distance. So they got this super hyper real version of Tom Cruise in the character but somehow you still knew it was him. This is built in to procedural technologies and data analytical technologies. You can create models based on interpretations of real data and then extrapolate that off in to hyper reality and thats really powerful in film and games. You want that capability built in. They were doing something else really weird, like they were morphing characters, they were doing something like Sigourney Weaver and Tom Cruise and you make Sigourney Cruise. Like you do with hidden Markov models in composition where you can hybridise them and you can kind of have Beethoven and Shostakovich and get new ones, Shothoven or Beethovich (??) [laughs] or what ever your new composer is.

DS: The obvious use of Procedural Audio is gaming and animation, as you mentioned. Where else do you see it being applied? There has been some talk of it being used in electric cars to simulate engine sounds.

Andy: I can see it as a real interesting safety feature in a car. I would also worry about this becoming a real nuisance – imagine a car running along the street sounding like a clown’s car and the person is drunk and its 3 in the morning! I can see a lot of social tension about objects having arbitrary sounds. Sound can be very intrusive and it has very different implications across different cultures and age groups.

DS: I came across a Harmon owned company called HALOSonic that promises to deliver technology that seems to simulate engine sounds for ‘cars of the future’. While there is very little information on the actual technology being used, I wouldn’t be surprised if it used Procedural Audio.

Andy: Let us analyse that as a product and say why Procedural Audio is really powerful for something like this. On the face of it you would think of using samples. I am the CEO of the company, I want a cheap product and I want to get it out. RAM and disk space is so cheap. I’ll  get a bunch of sound designers, get them to make loops, load up my product with a thousand loops and plus I can now sell you extra ones. Why the hell would I be interested in Procedural Audio? There were two really powerful reasons – technical computer science reasons. Number one: If you haven’t got memory space like greeting cards, watches, mobile devices – one of the biggest errors of judgement in mobile/casual gaming and the whole mobile technology industry has been over estimating the available bandwidth. And what’s a good technology when you haven’t got any bandwidth? This is what is your biggest asset when you deliver procedural content. It occupies 4kb and plays for six hours. It has a million things that you can change all the time or download another one. So, as a technology procedural content is very powerful in situations where you’ve got limited bandwidth. With cars, another place where procedural technology is very powerful is where you want the sound to encode a large vector of changing parameters. Why is it useful to have a sound on a car? By listening to a car engine I can tell a lot about it – is it slowing down, speeding up, is it a large car or small car. I can localise it pretty well. So to replace a completely silent car engine what you want is a procedural sound object which behaves like the car (that is familiar to peoples expectations viz a viz reality – and hence safety) with engine, with tyre sounds, with exhaust simulation to delineate rear and front approach  In fact you could encode all kinds of other information about the car as a safety feature which people would quite quickly get used to. If it is a bus – it could be a bigger noise, if it is a bike it’s got a lighter sound. That would be difficult to do with a sample. So the procedural object would be more versatile and able to to encode more information. That would be argument number two. Argument number three might be that to develop a library of a thousand different car engines would be very expensive. But once Procedural Audio technologies mature I should be able to buy an engine model as a one piece of software and adapt it – I could commission it as a one of piece of software or buy it on a license, put in to my product and I have all the versatility of it. There comes a point I think where the code becomes cheaper than the recording, for a limited use case. Maybe. I’m not sure about the economics of that. I am interested to see how it turns out. If on the other hand there are a lot of people out there recording things and there is a very buoyant market in recordings….

DS: Do you see Procedural Audio code becoming a commodity that is built and sold?

Andy: I think so, I see that happening with the apps market. Production models have changed, and this is very much down to Apple. To their credit, I’m very anti-Apple at the moment (for ethical reasons), but to their credit one of the really good things that Apple has done is (by standardising experience and lowering expectations) accelerate the tool chain to the point where the production of an app is a half a day’s work sometimes. Cookie-cut template things that go out in the store and you download them. That is code. It’s the entire application. Whereas before, code was something that was produced over weeks or months, that still happens with bespoke apps but we’re seeing the trajectory towards commodity code like that. So yeah, if you are a great procedural sound designer why could you not be a guy who specialises in engines and you sold two to Volkswagen and one to Mercedes and you are working on a couple of other ones for some company? People come to you because they know you are the ‘engine guy’. Because its the standard API for interfacing with the procedural code you know at the end of the day you are going to get six floats or something – four rotational velocities for your wheels and you are going to get an engine speed and you are gonna get a bunch of stuff and you plug that in. I would like to be optimistic about the futures for these market places, I would like to think they they might work. That is very much my (humanist) philosophy as a person, I think that these things should create work which should create opportunities and they should create markets and they should create things that people can do and involve their talents and this is why I say you balance technology, art and business . And any one of those can become an overbearing dominant thing. The culture can override the technology and the business, the technology can become the “oh technology oh technology!” and stamp on the business but more often these days it’s about the business running out of control. If the business dictates the technology and the art, if any one of those three gets out to kill the other its bad. As for Apple, who want to commodify and control your creative experience for a profit motive – the absolute antithesis of their “nineteen eighty four position”. So what I would fear in the case that you are saying is that commercial attempts at pushing procedural audio initially turn out quite watered down,  neutered and over-packaged like supermarket food, in which case it will be procedural audio only in name, only in marketing speak. But then I’m a dreamer, 20 years ahead with this stuff in my head, disappointment with present reality is built in, its what keeps us pushing. Computational audio, more intelligent structuring of audio in games and cinema, will be a new creative frontier. But we need to get past a crisis of purpose with technology. The idea that easier is always better, that more is always better, dumbing down, disabling and concealing rather than opening up, enabling and enhancing. To paraphrase Laing who says “The emphasis is more and more on communication, but people have less and less to communicate”, we don’t really need more ways to do things, but better, more thoughtful and reflective ways to do things, to replace brute force and dizzying excess of choice from the previous epoch with more focussed and elegant ways. I hope some of the philosophies of procedural audio will shape sound design in a wider sense.

Today we’ll be touching on the interactive side with this months Featured Sound Designer David Sonnenschein regarding his Sonic Strategies: Animal Sounds Memory Game.

This is one of many Sound Games to be created by Sonnenschein that open ears and minds to hearing the world in new ways. Focusing on the neurobiology of audiovisual input and memory, the game draws upon film and music theory, and provides one of the cornerstones for creating story, character and emotion with audio. It uses the memory flip-card model as one example of gameplay.

This game challenges the player to move from visual to audio awareness and memory in four variations that gradually bridge one sensory input (sight) to another (hearing). See how fast you can complete each level, and how many cards you need to turn over each time. How does your performance compare when aided by sight and/or hearing?

Have fun! See if your friends have the same or different experience. This is the first of many Sound Games to come that will open your ears and mind to hearing the world in new ways and learning to create story, character and emotion with audio.

What follows is an discussion between myself (DK) and David Sonnenschein (DS) on the topic of sound interactivity and the work he is doing to further our understanding of how we related to the world around us with sound.

DK: There’s been a lot of documentation of your work on the Sound Design side of the world in Pro Audio, but not so much on interactive media and your Animal Sounds Memory Game specifically. So let’s kick off and talk a little bit about what the chain of events were that led you to pursue this avenue and how that got started.

David Sonnenschein: It’s always been an interactive nature for me in the world with sound and so the film world in a way has constrained me in my abilities and creativity because it’s so linear and once it’s there it’s locked in place, and that’s it. Similar to my music career where I was trained as a classical musician where I played for years in symphony orchestras where you would rehearse and then perform and that was it. You were aiming for, more or less, perfection and somewhat of an interpretation in that, but it didn’t have nearly the juice that I’d get with improvisation, for example, or just jamming with other musicians. That to me was something that really attracted me so much to doing interactivity and games and all that. I’m not a big gamer myself, simply because I don’t like to sit in front of a television screen for any length of time and once I get involved in a game it’s hours and hours, so I’ve kind of held myself back from being a real hardcore gamer because I have a tendency to get addicted. Other than that I really enjoy the interactivity, so I have always wanted to get much more interactive with the sound world. This is the beginning place of where I came into it, a gut level interest.

DK: I can see how the world, the intermittency, and randomness of your experience of sound has kind of led you to this interactivity where we’re trying to, in the same way, represent this reality of randomness, so to speak.

David Sonnenschein: Yeah, and not only randomness but interactivity, where we as players as opposed to simply audience are involved, and that to me is the key to what I would like to be doing more and more. So the person is not just watching or listening but actually making those sounds happen. Sometimes it’s choosing what sounds to have happen or interacting on a cerebral level and deciding on what you’re hearing or interpreting what you’re hearing, and sometimes it’s actually creating the sounds themselves and if you’re not a musician and trained, there’s plenty of sounds we all make. So this is the basis of my interest in going into interactivity with audio.

Now specifically this game I’ve created, called Animal Sounds Memory Game, is something that’s posted for everybody and anybody. My interest in this particular game was developed with Alan Sheltra, a programmer who was a student of mine in one of my sound design programs and also had skills specifically in Flash animation and programming. He already had a game that was programmed which we adapted into this audio game. His game was a very simple card game, only visually based, and we adapted it to what is currently a four-level game. The idea of this is that we are able to play with the sounds and images, in this case using our memory, to remember where the pairs of cards are, and it starts with pure visual memory and slowly moves to pure auditory memory through four levels.

LEVEL 1 requires recognizing and remembering the placement of pairs of animal images with no audio.
LEVEL 2 introduces the sounds of the animals together with the images.
LEVEL 3 replaces the animal images with the images of the audio waveforms of the animal sounds.
LEVEL 4 provides audio only for matching pairs of animal sounds.

DK: Was this the progression of levels something that you started out with or you felt like as the project developed this was what came out of your experiments with interactivity.

David Sonnenschein: This was conceived from the beginning and I wanted to give the players a hook for transforming their memory processing from audio to visual, so I conceptualized this as a way to do it and it works very nicely. What surprises me though is that not everybody is as strong in one level as another. Some people really lock into the visual right away and then have more difficulty with the audio. Some people have a much easier time with the audio and the visual doesn’t help them, and that indicates to me that everyone has their own strengths in perceptual and memory processing. I think that was an interesting discovery, and it’s really about letting you know where your strengths are. I also discovered that little kids, three year olds and four year olds, could play this game as well as adults, and in fact, as well as some of the most sophisticated adults who are professional sound editors and designers. There are two parameters to measure “success”: one is how fast you can complete the whole game with 24 cards or twelve pairs, and the other is how few cards you have to flip over to get the correct array. You can compete against yourself to see how you can improve, but you can also compare with other people, so that gaming element is there to get people interested in trying to win in one form or another or compete with another. But it’s also there in terms of diagnosing your strengths and also improving your abilities, and so to me that was a really successful exploration. This is one of many, it’s like a pilot program for a series of Sound Games.

DK: Do you see the goal of your Sound Games as partly educational, from the standpoint of introducing people to sound, as well as helping people understand how they process things differently between visual and auditory?

David Sonnenschein: This particular game, I would say both, it focuses specifically on auditory and visual. I have other games that are only auditory games, but I would say in a broader sense I have been aiming at several different possibilities of applying this game series. I’m currently teaching an online course with the Academy of Art University in San Francisco and I created a couple of very simple Flash based games for the class in sound design. One was layering sound, which I call Sound Objects, where the student are offered several sounds that they can then combine in different manners to create different effects. Another asks them to sequence different sound effects to create stories dependent on the order, which I call Sound Events. These are very simple, but are the basis of creating more sophisticated games. In this application it’s for educational use for professionals, and students who are going towards a professional career in sound.

Another form I’ve been working on is to make this educational but for children, to basically expand their perceptual awareness. I’ve prepared a whole series of games that is more playful, let’s say for kids, to be applied at an elementary school level which would be inside of their curriculum for expanding their perceptual abilities and this correlates to the context within the curriculum, whether it’s social studies or biology or art or reading. There are here’s various ways of entering into the school curriculum to justify this as a teaching tool. In any case, there are quite a few different ideas that I’ve proposed for this, and I’m seeking funding from educational sources for this purpose. For example, teaching students the different kinds of sound qualities like volume, pitch, speed, and rhythm and how to decipher and distinguish between these different qualities and their extremes of loud and soft, and being able to use that in their languaging, being able to identify the differences between those with game play.

DK: Sure because, throughout your childhood you’re constantly trying to figure out what it means to be too loud, that’s a constant challenge growing up: what is too loud. It peaks in your teenage years when you crank your guitar amp up to eleven. Throughout that process, it would seem like a good thing to have a measure for that, and I think that’s what your proposing through these playful perceptual awareness game.

David Sonnenschein: Exactly. Another example would be rhythm, if something is ordered and rhythmic versus chaotic or arrhythmic. That’s something that a child may not know intellectually but they could feel it and hear it, and then if it’s brought to their attention through game play, like a whole curriculum called Orff which has to do with body rhythms, this can be very effective. Orff is not inside of a computer, this is using musical instruments and body rhythms and things like that, so the children actually learn how to coordinate themselves and how to communicate better, like inside of game play in the computer.

I have literally dozens of games of this nature, in a proposal for a whole series that would be for school children to learn different areas. Some tie in, for example, with reading,understanding how letters sound, or how letters look. Visually we’re playing around with fonts all the time in our computers and sometimes one font might look really different to a kid and seem like it should sound different, but maybe it doesn’t. So there are a lot of ways we can play around with sound to help kids in this area. Another one is the sounds that we make with our voice, like ‘hiccup’. What words sound like the things they described? Onomatopoeia are what they’re called, and explores the origins of language, semantic and phonetic structure, and listening sensitivity.

There are other ones that explore the understanding experience of different listening modes.What I refer to as listening modes is hearing the sound qualities of a waveform, or reduced listening, like the loud/soft or high/low pitches, or the sound source which I refer to in my classes as causal listening, what do we call it, where does it come from, what’s making the sound; it’s a dog, it’s a bird; or the meaning of the sound, called semantic listening, whether the dog is hungry or the dog is angry. Of course the meanings when we speak are all about the words too, but it’s also about the tone of our voices, so it’s teaching kids how to distinguish between different levels of listening. I could go on and on with this, but this is one area that I feel is very fertile and I’m excited to explore for educational use, for professionals and sound editors, but also for kids in general.

DK: That sounds like a great pursuit and a fantastic start to the series. Raising that awareness of sound in our lives and helping to define it for people educationally is a noble pursuit and it sounds like you have some great ideas about how people already interact with sound and ways that you can help define that for them through play. There’s so much latent learning going when we’re playing that it doesn’t necessarily have to be explicit in what you’re teaching, it sometimes creeps in around the corners, and that’s when games really succeed at teaching when it’s not necessarily being taught explicitly but just latently experienced. I think that is a side of learning that interactive really has as a leg up on some other typed of learning.

David Sonnenschein: Absolutely, and part of what you’ve been describing is another whole audience which is really not labeled as educational at all, but as pure gameplay. In that area I have a project that approaches this with the same elements but for a different languaging: it’s really about fun. It’s about engaging people for the interest in doing it for itself, not because a teacher told you that you had to do it. The experience of the “Ah Ha!” moment where something happens to you and you go, “Wow, that is so cool,” and that in and of itself is attractive to anybody who just wants to play. I have a background in neurobiology as well and I’m very conscious of what happens in the brain when those “Ah Ha!” moments happen and we can talk about left an right hemispheres of the intellectual and intuitive brain functions kind of joining together and bringing the player into this conscious awakening to their sonic world. That’s a really high goal of mine, to get people really into their listening powers and what they can do in their own environment and their own lives.

DK: Absolutely, and that “Ah-Ha!” moment that you’re talking about is a parallel that runs through every person, and with sound, for a lot of people, it’s very abstract when those things happen. Unless it’s seeing something happening and hearing the sound in response to seeing that, let’s say instead it’s an out-of-sight sound of a train in the distance, these kind of abstract sound experiences that we have that might lead to the “Ah Ha!” moments are a little bit harder for people to grasp. It seems like another way that we have to educate people as far as the role that sound plays in their life.

David Sonnenschein: Yeah, and putting it inside of game play is a motivation to get involved initially. If there’s something to win or something to improve upon, or some goal, that in and of itself I think is really important for any age. If you have someone who recommended it, a testimonial like “Hey this is so cool, just try it”, that may be enough to get people going and hopefully those numbers will grow. Then there’s something like Guitar Hero, which is an audio game that requires extreme skill but it starts off with people learning something very slow and it has different levels and of course it links into popular songs, so that’s really the “catch” for that game. I really find that extremely beneficial to introduce the interactivity of music to people who are not musicians so they can start doing something on that level.

One of the things that I’m attracted to as a filmmaker, director, and writer is storytelling, and that’s a whole other level that I’d like to share in this interview. All of these are little pieces of a bigger game that I’ve been formulating and developing over the years, and what I want to do is tie many of these small games together into a larger environment and gameplay so there’s a purpose to each one. In other words, with each game you can gain some more knowledge, power or tools, whatever you want to call them, to move along in the gameplay to another level.There’s a building of expertise and knowledge within the game so you can reach a higher level and experience things inside your own perceptual awareness and your own brain, but also interacting with the game itself and perhaps other players. This is a much bigger project than just the little games, but that’s kind of where I’m going with this, and like any film or game producer it requires a fair amount of structure and development and financial support, so I’m working on this as a long term project as I do with my films that require, often, a ton of money to get things going. I want to show people what it’s about, and that’s a longer term goal that I have.

DK: Sure, and these bigger projects take more people and more resources, and they become greater than the sum of their parts over time. But they’re all seeded in these little games that you’re producing and working with, and kind of rooted in your experiences with sound in our life.

David Sonnenschein: That’s exactly it, and it excites me to see the gaming world expanding and including audio more and more. I’ve been preparing the second edition of my book with some chapters on interactive media and in so doing I’ve had wonderful interviews with some of the top game sound designers. They know this stuff really well and they’ve been doing it for years and years and some of them are other featured sound designers as a matter of fact on DesigningSound.org including Charles Deenan who is one of them. We are collaborating with one another on a knowledge set as well, where is it going, not just technically, but in terms of gameplay, in terms of the ability to use technology for increasing people’s interactivity, perceptual awareness and ultimately the story, the gameplay itself. That is something where I have a very lofty goal because I use sound not just for education, and not just for entertainment, but I’m trained as a sound healer as well. When I use those words, it’s a very broad term, I refer to the use of sound for transformation, on a psychological, emotional, and/or physical level. So I would like to incorporate all of that in the game play as well.

DK: I think about the marriage of those areas in games, like you said, the diversification of games, different game types, what constitutes a game, who is a gamer and I think that all of these things are expanding and can encompass all of these different areas that you’re talking about. Whereas once upon a time games may have just represented something very playful and less serious, I think we are gaining in our ability to look at these things more as interactive pursuits, interactive experiences and with that you can tie together all of these different areas whether it’s sound healing, education, perceptual awareness, all of these things can now be encompassed in this idea of interactive experiences. I think we really are standing in a place where you can accomplish this goal of seeing those things dovetail.

David Sonnenschein: Yeah, I’m very excited about what you’re talking about. Similar to what film has been doing in terms of genres, we have extremely commercially successful films in so many different genres. Of course the more expensive ones are going to be the action flicks, the sci-fi, the fantasy and big blockbuster types. Some of the most successful have been serious dramas or ones that were family oriented, for example some of the Pixar films, which I think are extraordinary for all ages. I think that what has happened historically with games is that we’ve had a lot of teenage boy shoot-em-up kind of things, and they’re still strong, but I think they’re expanding much more into the female audience, much more into a lot of different ages, people who have been playing games for the last twenty to thirty years are ready for other kinds of games. As their own lives progress, now they’re parents with little kids, there are a lot of thing evolving in the industry itself. I’m excited to be on that interface in what I’m doing in film and joining the interactive world. I’m fairly new to it compared to people who’ve been doing it for decades. I’m extremely excited to enter in with this specific focus of audio. Another very interesting niche market is the blind audience, to do purely audio for them, and it’s a wonderful thing to see some of the things that are coming out with that and I hope that I can provide something for them as well.

DK: Right, well it’s people coming from other disciplines, like yourself in sound or in film, to the game industry who are helping, I think, to diversify things from that teenage bedroom mentality that has been somewhat wrongly shouldered with the stereotypes, which exist for a reason. I think that people have always played games but I think that electronic games have really been cultivated in the bedrooms of earth over the past thirty years, and that has certainly changed over time with the coming of casual games. I think that when people come from outside of the game industry and recognize the power of interactivity, that is helping everyone move that stereotype further into the back and bring electronic gaming or interactive experiences to a broader audience.

David Sonnenschein: Yeah, and I feel very gratified that it’s getting closer to what I said in the beginning of our talk on the interactivity with improvisation in music, and that I’m returning, in some ways, to some roots that I have in my musical background, and the feeling that, we’re using Skype right now to interact and record and there’s all sorts of things we can do all over the world interactively with people now, and talking is obviously one of them but there’s so much more. This is the world that we’re talking about today and I’m very enthusiastic for any of the readers of what we’re talking about to give feedback about their ideas on where interactive audio is going.

Here is the final interview with Rob Bridgett, about Prototype, talking about the sound of the cinematics, the mixing process, and more!

Designing Sound: First of all tell us something about what was your contribution on Prototype and what do you did for the sound of the game?

Rob Bridgett: In late 2007, the audio director for the project, Scott Morgan, asked if I could get involved and help out with the game mid-production. Cory Hawthorne was working as Technical Sound Designer and Implementer on the project which meant I had the opportunity to cover two areas on the game, one was as cinematics sound designer and implementer and the other was as game mixer. In terms of the first role, I was responsible for the sound effects, Foley, dialogue editing and mix of all the cut scenes in the game. The music was edited and supervised by the sound director for the project, Scott Morgan, and once all the components were assembled I would provide a mix automation pass before the finished file went into the game.

The second role, that of mixer, was one that came into play only during the post-production sound beta phase of the project’s development, in which Scott and I spend four weeks mixing the entire game in Radical’s 7.1 mix room. I always welcome the opportunity to help out on projects like this as it offers a break from being an audio director and allows a lot more time to concentrate more fully on one or two areas in particular.

DS : Can you tell us something about the process for the cinematics sound production?

RB: Sure. I’ll talk you through a typical set-up and process that I use on cinematics. The actual work on the cut-scenes starts fairly early in production. Once a script has been approved for production, placeholder dialogue is recorded here, for this we typically just use members of the team to read out the dialogue. We record this, edit it and give those files to the animation team so that they can begin their storyboarding process. They use these placeholder files to come up with very rough timings and shot lists which really gets the whole process kick started. Usually during this time, the actors are cast for the cinematics and they are recorded which eventually means that after a couple of months you have the real dialogue takes to work with and the animation team can start being more accurate with their timings.

Up until that stage, Scott Morgan, the game’s audio director had pretty much run the process, I was myself at this time finishing up the 50 cent game. I rolled onto the project in January 2008, at this point Scott had all the dialogue recorded and the cinematics team had some very rough avi files of the various cinematic scenes, so this was a good time to actually start building up the sound elements and structural foundations of the cinematics.

The first thing that I do is create a seperate Nuendo session for each scene. I typically do this from a cinematics template that I have created in Nuendo, which basically is an empty project with pre-assigned tracks and folder tracks.

• Dialogue Folder Track containing six mono tracks all assigned to CENTRE only
• SFX Folder Track containing five mono tracks all assigned to CENTRE only plus five stereo LR tracks
• Foley Folder track containing ten mono tracks all assigned to CENTRE only
• Ambience Folder Track containing four stereo tracks all panned LR and slightly LS RS
• Music Folder Track containing four LR stereo tracks and two 5.1 music tracks
• LFE folder track containing 4 mono tracks all assigned to LFE only.

These templates provide very quick structure to the whole project which is easy to navigate and expand upon. I recommend this for anyone getting into a new cinematics audio project as getting organized at the earliest stages like this saves tons of time later on.

As we had the dialogue ready and recorded, one of the first tasks for me to do was to go through all the scenes and ‘worldize’ the voices – rather than re-recording, I used Altiverb VST for each different room or physical space depicted in the scenes. The roomverb was panned mainly to the fronts (LCR) but also to the rears in order to give the sense of the listener being inside the room and surrounded by the reflections off the walls. This is quite a subtle effect, yet it adds a great deal of realism to dialogue that is recorded close-mic in an ADR room. Further to this some low-end was also rolled off the dialogue in order to simulate more of a distant mic / location sound feel. Having done this treatment on each of the cinematic scenes, it was time to move onto the second phase of building up the sound, adding roomtone and BG ambience.

For each scene and for each cut, I added roomtone that I had recorded here in the various spaces at Radical. There is a lot of AC in Radical and it makes for some useful roomtone source recordings, this meant that I had a ready to use library of roomtone beds which I could quickly edit into the scenes. For each camera cut in perspective the volumes of the roomtones were changed to ensure they corresponded to the listener position and point of view of the characters.

Scott Morgan had also been on location to New York to gather exterior ambience for the game, and it is these recordings that I was able to quickly use and edit together for any of the exterior scenes in the game. In fact, in the end I mainly relied on the actual background ambience file from the game for these ambience beds, as this would mean there was continuity between cut-scenes and game. In some of the scenes we also let the ambience present in the game continue throughout the cut-scene in order to maintain complete continuity from game to cut-scene and back to game again, for these instances, it just meant muting the ambience folder track on export.

With all the reverb and backgrounds built up, some effort could be put into sound effects design. An initial pass was done just concentrating on big fx moments like explosions or body impacts, also because the movies were low in detail at this point it could not be seen what materials or detail would be present in the final movies. All of the cut-scenes in the game used the in-game engine, so the full detail could only be seen at run-time in the game.

For the Foley in the cinematics we contracted Sharpe Sound here in Vancouver to cover the movements for all our cinematics scenes. The Foley was returned to us un-edited so the next phase of my work was to edit all the Foley and premix this so it sat well with the other effects and backgrounds. During all this work, the movies were constantly being iterated upon, receiving a lot of editing work and often large sequences would be re-cut and even deleted entirely. This meant lots of rounds of re-syncing dialogue and effects to the latest cuts of the movies.

By the time we reached Alpha and the work on the cut-scenes was locked down, I had two weeks in which to complete the final effects pass and mix on all the movies, matching the vo, music and effects levels for all of the movies. There were around 30 movies in total, around 45 minutes of in game rendered cut-scenes. Scott and I then reviewed all the cut scenes with the rest of the cinematics team and made notes of a few final tweaks before sign off.

DS: Did you record all the sound effects of the cinematics… what are the sources?

RB: We do have our own sound library here at Radical, in which we have archived many of our sources for other games such as Crash and Scarface. This library is invaluable in quickly getting sounds that I know will work. I think the key to good, fast work is actually knowing your library really well and being able to access exactly what you want quickly. The Foley, as I said, was all recorded fresh for this project, but the majority of the effects, the bodyfalls, punches and transformation sounds were all recorded here specifically for the project.

Perhaps the best example of some of the cinematic sound design we’ve been talking about is the ‘intro cinematic’ for the game which can be viewed online here…

DS: How about the mix for the game? Can you tell us something about the process involved for that?

RB: We have a proprietary run-time mixing system that enables us to do this attached to Mackie hardware control surfaces, the same one used on Scarface that I have talked about in depth elsewhere. For the mix we spent a total of four weeks, this time was broken down into a few different phases.

The first week of the mix was probably the most critical because it was where we set the overall output levels of the game. The first thing we did was to bring the whole output of all the channels down by around -6dB. This is because that when we started listening and mixing at reference listening level of 79dB, the game was incredibly loud. What tends to happen during development is that sounds are turned up and up so that you can hear them while you are populating the game with them, this approach is fine while in development, but at some point you have to reset the whole board and start from scratch again. This is what we did in the first few days of the mix. Getting the dialogue to a decent level and then ‘mixing around’ it is the approach we have been taking. So, once we’ve set our dialogue level, the music will be determined in relation to that, as with the effects and so on. Intelligibility of dialogue is really at the centre of most mixes to be honest, I still hear so many games today where you actually cannot audibly hear what is being said by certain characters because guns are being fired etc. This is perhaps one of the many areas where the styles of mixing in cinema is an influence.

Anyway, once the overall listening level is set, it is a matter of playing through the entire game, identifying key mix moments, mainly dialogue or mission related, but often tied to some in-game feature or effect like the thermal vision in prototype, for which we pitch down the ambience and add a low pass filter to many of the sounds in the game. Similarly for Infected Vision, where all sounds are given a muted treatment except for infected who remain clear and unprocessed during this mode. We also tweak every individual sound to make sure it is not too quiet or too loud. This is what takes the majority of time on a game mix, up to two weeks in this case, and all the time being aware of keeping the overall listening level tolerable for the player at home. Another major thing is to maintain the levels of sound, particularly dialogue and music, throughout cinematics and gameplay so that there isn’t a jarring disconnect between the two modes of exposition.

The final week of mixing we used to test how the game sounded on various mixdown configurations, such as stereo TV and all the various output configurations on the various consoles. The mix is tweaked at this point to ensure that users who listen on a tv set only are able to hear what they should be hearing, usually in the form of a few minor tweaks to music levels and dialogue levels but nothing too significant that it will adversely affect the surround mix.

DS: It’s a video game for PC, PS3 and Xbox 360… to work with sound… what platform you prefer?

RB: A tricky question, as a developer of multi-platform games I do have some opinions from a mixer’s point of view. The 360 is certainly the least complicated in terms of outputs, it supports Dolby Digital 5.1 and stereo (via optical and HDMI) as well as an analogue stereo output so it is kind of the easier to work with in terms of options and checking the mix. However the PS3 has discreet 7.1 support as well as a whole host of audio output options including DTS and PCM as well as Dolby Digital, which does make it more complex for checking and testing, but also provides more options for the user, particularly the higher-end HD audiophile user. As for the PC, this is potentially the most complicated of the platforms to mix and test for, because you can have any soundcard on the market connected to your PC which means we have to test on a wide variety of cards but can’t always be sure of what end users will be hearing. Having mixed the Xbox version of Prototype first, we then cloned all our mix settings and did a mix pass on the PS3 – fortunately the mix translated very well and I think we only made one or two very minor adjustments. The biggest difference being the difference between our two different compression codecs used: XMA on the 360 and MP3 on the PS3.

DS: I saw an interview with Mark Tuffy of DTS who said that Prototype was the first Xbox 360 game with 7.1 sound.. It’s about neural surround… what do you know about the implementation of that process in Prototype?

RB: I actually know very little about the implementation of this in the game from a technical point of view. What I do know is that it is running the DTS Neural surround code on the Xbox360 (there is an option in the sound menu in the game to turn this on or off) and that it is outputting a 7.1 mix of the game when listening through a receiver with neural enabled. The receiver then basically decodes the extra two back surround channels from the Left and Right surround channels of the regular 5.1 outputs. We mixed the game while monitoring in 7.1 on the Xbox, while always checking how the sound folded down to both 5.1 Dolby Digital and Stereo. The game also runs in 7.1 PCM on the PS3.

DS: In terms of interactive mixing, what aspects would you highlight as most important in the mixing and implementation of interactive audio on Prototype?

RB: Some of the action gets pretty intense pretty quickly in this game. Strike teams are sent into heavily infected zones, the amount of sound playing back is huge and there is essentially a sound for every event and collision occurring, so the game needs to be able to deal with this. It isn’t really part of the mixing system but it plays into it, there are limits specified in our engine on the amount of certain types of sounds that can be played back at any one time, such as dialogue or gun shots, and there is a priority system which gives precedence to some sounds over others. To add to this, the mixer system allows us to finesse certain events such as the shot from the thermobaric tank, whereby we duck down most other sounds to foreground this one huge tank weapon ejection to make it seem like it is much louder than it really is!

DS: I think one of the best features of Prototype sound are the ambiences, there are a lot of those and too much recording of many places… Why does the sound team gave much importance to the ambiences? What is the importance of these in Prototype?

RB: You know, the audio director Scott, has written a superb and detailed article on the ambiences in prototype here that can best answer your question… I really recommend it as there is quite a unique approach to ambience in this game, which I agree, works really well in conveying the feeling of New York.

Prototype Official Website