Article by Jeff Seamster.

Jeff had originally wanted to include this during his feature month this past December. Circumstances conspired against it, but thankfully he wasn’t willing to give up. So, now we have another great article to share with you. Enjoy!

Make a quick list of your 5 favorite video games or films in terms of sound design. I’ll wager they all have something in common; strong audio direction and a cohesive audio aesthetic. This isn’t due to some happy accident or last minute thinking during the post-production process. A successful audio aesthetic requires thoughtful planning and documentation coupled with deliberate execution and course correction.

Why is it important?

Discussions of games like BioShock, Ico, Limbo or movies like Blade Runner, The Matrix and Toy Story will inevitably include references to strong visual direction and style. That strong and consistent direction makes it easy for an audience to interpret, absorb and connect with their favorite works. There’s no reason this can’t be true for audio as well. Achievements in audio aesthetic don’t get as much coverage as those in the visual arts, but there are concepts and practices that can guide audio professionals toward that same level of achievement.

Cohesive aesthetic is our best strategy for creating memorable experiences. By wrapping our audience in a well-conceived and unified soundscape, the work at large becomes memorable even if that soundscape seems complex or alien at first. Repetition and shared vocabulary are tools that can be used to reinforce that influence. We’re all trying to create an aural experience that results in immediate recall of our titles years after the initial experience.

The narrative and visual teams on any given project will have their own agendas and aesthetic motivations. When integrated with these other disciplines in a game or film production, cohesive audio aesthetic will result in a work that is stronger and more memorable overall. This is how our touchstone achievements are born.

What makes up an audio aesthetic?

An audio aesthetic is constructed from sound principles, reference and guidelines that inform design and content creation decisions all the way from the overarching project down to the level of individual assets. When talking with other audio professionals, I refer to these as our design pillars, common vocabulary, and style bible.

The first task in developing an audio aesthetic is defining design pillars for the entire project. These are the types of content, techniques and styles upon which the game or film will rely most heavily. Content types may include character voiceover, ambient sound and music. Pillars of technique might include stylistic counterpoint, hyper-realism and anthropomorphism. A project will typically have one or two style pillars, often tied to the project’s visual style. These style pillars could include directions such as “Cyberpunk” or “Gothic”. Altogether, design pillars should put you in a position to describe the audio aesthetic for your title succinctly. If your design includes more than a handful of these pillars, it might be worth performing a reassessment to make sure the design isn’t becoming muddled stylistically.

In order to communicate about aesthetic effectively with your audio team, it’s worth developing a library of reference and common vocabulary early on in the project. This vocabulary is used to further refine the project’s sonic identity and can be extracted from multiple points of reference including films, games, books and music. Your references can and probably should be shared by multiple disciplines on the project. Descriptions for common vocabulary would read something like “Gritty explosions: (see Battlefield 3)”, “Visceral fight scenes like the Bourne movies” or “The foreboding ambience of Se7en”.

Once design pillars and common vocabulary have been established, I like to condense these ideas into a “style bible” that can be easily understood and referenced by the existing audio team, newly hired audio personnel and external, non-audio teams alike. The style bible provides detail on the project’s aesthetic direction as well as guides and constraints for content creation. Detailed aesthetic direction should include elaboration upon design pillars as well as specific points of reference within the project’s common vocabulary. These specific points of reference will take the form of video clips and audio files including a brief writeup of how the material in these clips applies to the project. Content creation guidelines can direct toward style and technique (i.e. “50’s sci-fi”, “compress, but leave some breathing room”) or even away from them (i.e. “avoid synthesized sounds”, “no spring reverb”). A comprehensive style bible will reach beyond the field of sound design and into the areas of voiceover and music.

Documenting your vision

As I mentioned earlier, the documentation of an audio aesthetic should be simple and easily understandable by all disciplines on a project, not just the audio team. But it should be easy to express your direction without a document as well. If the audio direction for a project can’t be conveyed verbally and clearly to other team members and creative leadership, then it may need refinement. Periodic meetings with the audio team that specifically discuss aesthetic will help with this refinement while keeping the team informed of stylistic shifts and changes.

Once the direction has been refined, it’s time to begin construction of your style bible in a format that’s easy to edit and easy to share. The written portion could be a source-controlled Word document or Google Doc if you have multiple editors working on the document simultaneously. If you have a team in which members are given categorical focus (i.e. voiceover, combat, ambience), invite each member to review and develop the section upon which he or she will be focused.

Design pillars will act as the commandments of your style bible and should be documented with no more than a couple of sentences. After you’ve got those down, document your common vocabulary and stylistic references with links to video and audio files. Keep reference material local whenever possible as you never know when an online video capture or sound file will suddenly disappear. If a reference from a game, movie, or television show is given solely as an audio file, try to include at least a screencap of the video source to help team members get a frame of reference. Finally, document your content creation guidelines with write-ups for each category of sound along with corresponding processing and mastering techniques. For added clarity, link your document to some individual, mastered assets along with the audio projects used to create them. These projects should be clearly organized and they should serve as examples of departmental standards, both technical and creative.

A very popular practice that should be borrowed from our visual counterparts in game development is that of the art room. Game artists create an art room as a fully realized example of a game’s style in terms of setting, scale, lighting, decoration, coloration and more. Game audio professionals can mirror this practice by creating a menagerie of game characters and settings with sound or even a fully playable level with comprehensive audio treatment including mastering and mix. For those of us in film sound, the art room can be a 5-10 minute segment that is representative of the overall audio aesthetic for the production. Whatever the format of your art room, it must be an easy to use and accurate reference for those creating additional content for the project.

When should you create your audio aesthetic?

The short answer is “As early as possible”. If you have the good fortune to be involved on a game or film during the pre-production process, experiment and isolate your aesthetic during this time to free up the rest of the project for pure content creation. For those of you strictly working in post-production, spend at least a little time assembling, if not a style bible, a style handbook to inform your team and your decisions during the intense post-production window. My short answer continues with “and as you go.” Whether you are involved from the start or only at the tail end, aesthetic is a moving target that can evolve, even if just slightly, right up until the project is wrapped.

Keep it flexible. Keep it safe.

Even if you and your team are confident in the direction for a project, that direction should be flexible. You may realize later in the development cycle that your aesthetic isn’t necessarily working and the creative heading for a project can change, sometimes drastically. When developing your sound and compiling your style bible, keep in mind they should never be so static that one or more components cannot be replaced, shifted or removed. Yes, it can be a pain to retread work that is already done, but preparation for design shifts outside of audio will lessen the sting if and when it actually happens.

Equally important as being flexible is being critical and honest about your own work. Think of it like a voice actor spot checking accent over the course of a production. Regular confirmation and course correction of your audio aesthetic is the only way to guarantee that your vision will make it through to the final product. Naturally, this is a more time consuming aspect of the job on a 20+ hour game than it is on a 2-hour film, but it is no less valuable a practice in either format.

Not just a big budget issue

If you’re thinking to yourself, “This seems like more of an issue for major motion picture and large scale games”, I encourage you to reconsider. It could be said that aesthetic development is even more important for smaller productions since they have less production time and a shorter window with their audiences to establish a unique sonic identity. Regardless of the budget, duration and scale of your project, developing cohesive aesthetic is a practice that pays off every time. Start small, keep it tight and go make history.

It’s still Elliott Koretz’ month here at Designing Sound and here’s a story on one of his most prolific collaborations, with director Michael Mann. In this interview, Elliott shares stories about working methods, the use of music and silence and Mann’s tireless search for perfection. Enjoy!

How did you and Michael Mann meet the first time?

My association with Michael actually goes way back to “Miami Vice” the tv show. I was an editor on it for the pilot and for two seasons. But my relationship as a supervisor started with his later tv show, “Robbery Homicide Division”. I interviewed with him and ended up getting the show. I then went on to “Collateral” and “Miami Vice”. We had a number of discussions about “Public Enemies” but ultimately I had another commitment on a different film that prevented me from doing it.

How would you describe him as a collaborator and filmmaker?

I think his track record in both tv and film speaks for itself. He has been an innovator for many years. I think one of the marks of greatness is when you are channel surfing and come across a film and even though you may have seen it countless times you stop and watch it. I find I do that with many of Michael’s films. It’s very exciting working with him. I think those of us that work with him are drawn in knowing every project has that potential for greatness. Michael is the definition of a tireless worker and the challenge is to keep up with him. He works himself harder than any director I know. His demands are many and it’s important to be mentally prepared to present him your best at all times. He has many ideas about sound and expects them to be addressed as quickly as possible. It’s very important for me to be clearly communicating this information to my crew. No question it is truly challenging working for and with him but I do understand that his intensity is in the quest for his ideal soundtrack.

You’ve collaborated on two feature films. Quite often, sound can be very tricky to talk about – how did you communicate about sound and how did your dialogue evolve throughout the process of making the two movies?

That’s a good question. Before I started designing on “Collateral” Michael called me into his office one day. He was still shooting the movie and wanted me to hear the production sound on a certain scene. It was the scene in the alleyway where Jaimie Foxx is tied to the steering wheel of his taxi and Tom Cruise comes out of a building and finds that some petty criminal has stolen his briefcase from the back seat. Tom ends up shooting the guy. The alleyway was between a tall apartment building and a large above ground parking structure. The resonant sounds of the gunshots were amazing even with blanks. Michael said, “This is the sound I want here”. I did some sweeetning to the impact of the shots but the final mix contained that same production echo. It’s really cool. I think this is a good example of Michael’s clarity in certain aspects of the sound track. He will be very clear and specific about what he is looking for. To that end he has been very generous with giving me access to do extensive field recordings for his films to facilitate that and I will speak on that subject more later on. But to address the question directly, the process I found worked best with Michael was to introduce sound elements via the Avid as early as possible. I would cut sequences, crash them down to make them Avid friendly and get that material over to the picture department to integrate into their cut. I think watching the film and hearing my material in that environment got him comfortable with the sound early in the post-production process. If he embraced it in the cutting room it really helped on the dub stage later. We would also have more traditional “spotting” sessions. My crew and I would screen reels with Michael and he would give notes along the way.

Michael Mann’s films are often very dependent on music and sound for their emotional impact. How much sound is written into his scripts?

Well the scripts were a good starting point for me to start thinking about what I needed to do. They were often very descriptive about the literal sound of the film. By that I mean very specific vehicles, weapons, and devices used in the story as well as the environments of each scene. I think the subjective design aspects of each film came more from viewing the movie and getting a sense of the flow of the picture editing.

How early on were you involved in the films? And what were your schedules?

I was fortunate that I started both “Collateral” and “Miami Vice” while they were still shooting. That gave me the ability to do a couple of very important things. First, to start getting my design sounds into the Avid very early in the process and secondly to get out to sets and have access to record. Both films were long post schedules and I think I was on for just about six months on each.

Mann actually started out working in documentaries. How important is realism to the sound design of Mann’s movies?

Very important. And that leads to my field recording. On “Collateral” we recorded every gun in the movie. We recorded the taxi. We recorded the metro trains for the film’s ending. These all were very detailed recording sessions with many mikes and multiple recorders. They were expensive to do but of incredible value to the sound of the movie. One of the more unusual outings occurred when I went out with my assistant, Bruce Barris, for an all night recording session in downtown Los Angeles. Our goal was to get recordings of the sounds one might hear at 3am. Now let me say 3am in downtown Los Angeles is quite interesting. There are many homeless people who sleep on the streets. It is an unfortunate reality and we tried to not intrude on anybody’s dignity. We recorded at a discreet distance the voices of the night. This material was used in the movie during the scenes after the first kill when Jamie realizes he is driving around with an assassin and the tension is high.

On “Miami Vice” I had the opportunity to fly down to Miami while they were shooting and do extensive recordings. We rented an airstrip and used it to record all the major autos in the film including a brand new Ferrari that wasn’t even available in the US yet. We had a few days with the speedboats and staged our own races recording bys and onboards. On one of the days at sea the lead speedboat driver (who turned out to be a reigning world champion) offered to run the boat out to Bimini for lunch. It’s 53 miles off the Miami coast and at full throttle it wouldn’t take very long to get there. Ultimately the assistant director accompanying us said no because he thought Michael wouldn’t approve. That and the fact we didn’t have our passports ended that little adventure. We also did some onboard recordings of a unique experimental airplane used in the film.

Recording the plane is an amusing story to share. Let me first describe the situation. This plane, an Adams A-500 was a very cool looking small airplane that had a featured role in a scene where our undercover heroes are smuggling some drugs into the country. Because it was so small we couldn’t be onboard while they were shooting but we did rig up our recorders and mikes and instructed the actors how to hit record and stop on the machines. The pilot (a real pilot made up to look like one of our leads) was very impatient and wanted us off the plane so he could get going. We finished what we needed to do, though not soon enough for him, and left to get to an airstrip where the plane was to fly over at what we were told treetop level for part of the smuggling sequence. We had two positions to record from at the field. My assistant was on one side of the runway and I stood in the middle but back towards the end of it. The plane came in and headed straight towards me. I stood with my shotgun mike in hand. The plane was so low I could clearly see the pilots face. Evidently he had decided that treetop level wasn’t good enough he had dropped down and came right at me just a few feet above the ground. I held my stance for as long as possible and then hit the deck. However being the professional that I am I held the mike up and got the recording. I heard later that the radio chatter from the pilot indicated he knew exactly what he was doing.

Once I edited out the “Oh Shit” it was a great close up recording.

These types of recording opportunities do not happen very often and I benefited from the fact that both movies had pretty decent budgets. I think this gives you a little insight into what Michael prioritizes when thinking about sound in his film.

As songs and music have such an important role in Mann’s movies do you collaborate more closely with the music supervisors and editors than on the usual Hollywood films where the two departments rarely meet until the mix? I read that on Collateral the song Shadow On the Sun was used already when shooting the scene with the coyotes in front of the cab.

You are correct. Music is a huge part of the sound of Michael’s movies. There was quite a bit of collaboration between our departments.We were all in the same building actually.  It’s really critical to know where the music is working and in what frequencies. Then we can compliment each other and not fight for the same sonic space.

I’ve read that he’s really hands-on in the mixing process. What is he like on the dub stage?

Michael comes to the stage really well prepared. He has notebooks put together in preproduction and during the filming that have his thoughts behind every scene. He references those throughout the dub. He works very closely with the dialog mixer and is active in crafting the subtle nuances that a db up or down can help sell. I think it’s fair to say it’s an intense experience for the mixers because of this attention to detail.

I like the way you utilize silence or near-silence in subjective ways in both Collateral and Miami Vice. How many of these decisions are made during the sound editing and how many during the mixing?

A little of both I think. There are definitely moments that are designed up front for dramatic effect. Silence preceding a big action piece is quite effective. I think it draws the audience into the world we have created. And so some of that is indeed planned. But there is always room for experimentation on the dub stage. Sometimes Michael would have new ideas for us to pursue. Sometimes a change in music would lead to a new sonic direction with a scene.

At the opposite end of silence, Miami Vice had some of the most intense, hefty gun sounds I can remember – actually one of the few other sequences that equal those is the shootout in Heat. What is it about Mann and gun sounds? And how did you get those Miami Vice effects?

The guns in Vice were actually a blend of production guns, library fx and a reworking of the Collateral gun recording sessions. For the big shootout scene near the end of the film our assistants went through every production take from the shootout and build a cutting library. That is what the bed of o/s shots were made from as well as some of the close up weapons. The design was inspired by the sounds of the news footage of the North Hollywood shootout. It was very much a news/documentary type feeling. The close up guns also had some layered library sounds. Then add the ricco’s, zip bys, impacts, debris and it’s a really effective sequence.

Both Collateral and Miami Vice are very nocturnal movies. Did that influence the use of sound?

Yes. We were extremely specific with every sound in both movies particularly with the night sequences. What I mean by specific is each sound cut in these scenes was really thought out. No background recordings just rolled in. Nothing superfluous. We would ask ourselves what was the motivation behind each of these elements? What really happens at night? With the long schedule we were able to fine tune and really think out what was important. I think this process is a great way to design in general, and certainly for the night time scenes the spare nature of the tracks added a great deal of suspense to the overall mood.

Has Mann mentioned some specific inspirations for his use of sound?

Not really. I know he is quite a student of film history but our conversations were much more directed to the specifics of what he wanted for the particular film and not so much about his motivation behind it.

He seems to be extremely committed to his work. Inspiring or scary?

Yes.

Seriously, as you may have gathered and to repeat what I said before, he is most definitely a challenging guy to work for. I think he is always searching for perfect moments. He pushes himself extremely hard and you as a supervisor have to understand how to keep your crew sharp through the long hours and through the pressure packed needs to feed the stage with up to the minute conformed materials. It’s a pretty relentless pursuit for long stretches of time and you really need to be psychologically ready to be onboard for the ride.

I am very appreciative of the opportunities Michael has given me. I am very proud of the work we have done and I leave open the door to possible future collaborations.

The Sound of Hotel for Dogs.

Andy Farnell – a familiar name in computer audio – is a computer scientist, sound designer, author and a pioneer in the field of procedural audio. He is a visiting professor at several European Universities and a consultant to game and audio technology companies. His book, ‘Designing Sound‘, is a bible for procedural sound and should be on your bookshelf, if it isn’t already!

He was very kind to find time in his busy schedule when I visited London, and we talked about what procedural audio is, where it stands now and what it can be in the future. This article is a transcription of our conversation, which he was again very kind to edit along with me. It was no easy task because there was so much good content!

Thank you Andy!

DS: Where does Procedural Audio stand now? Would you say it is comparable to where CGI was in the 70s/80s, when computers weren’t powerful enough?

Andy: That is a central mythology – that the computers aren’t powerful enough to do it. This is often brought out as a straw man argument against Procedural Audio by skeptics. One of the things I did with my 2005 demo was to make all of the sounds (they weren’t very high in quality) that you would need for a first person shooter game – fire, water, wind, rain, some animals, some footsteps, some guns, some vehicles. This was 2005 and I had them all running on a 533 MHz processor generating a realistic-ish sort of soundscape to prove that if you had 1GHz processor and if you used half of it for the graphics then it would be quite possible to synthesise all the sounds using the remainder. Six years after doing that people would still come to me with this straw man argument, they would say, “You know Andy, we love this Procedural Audio stuff but there’s just not enough CPU available”. But we now have two to the five times more CPU than when I did my 2005 proof-of-concept demo. So, what’s behind that? Why are they saying that? It’s not true. What happens is the internal politics of resources. The requirements always expand to fit the resources available. The game worlds get bigger and bigger and the graphics get more and more demanding. The audio team will always have the least amount of CPU allocated to them as an afterthought, because in the current structural model of production sound is “post production”, and no body wants to commit to giving audio that much CPU bandwidth. I feel that is the real reason behind the argument. You often get these straw man arguments that enter in to a culture and just get recycled. People know that there is an argument and it comes to their tongue very quickly and they say “Yes we could do it but there is not enough CPU”. With the left over CPU on a modern games console I could provide you great procedural sound.  On an eight core architecture, we would need one or two CPU cores to give procedural sound. Even more interestingly is what happens when we run models in GPU, and many Procedural Audio models are inherently parallelisable. So, yes, Procedural Audio is somewhere in that era before the Tron movie, or before the Pixar CGI revolution, its possible, but not yet seen as viable, perhaps the shift is too painful for big companies to make.

DS: Have you tried doing a similar demo using today’s technology?

Andy: No. I just don’t have time at the moment. Life is moving so quickly and I’m involved in so many other interesting projects. Some of them to do with computer science, some of them to do with philosophy. Interesting times, but in the future I want to have a research department and have a bunch of guys, really smart guys who are just on this and want to do it and I can help direct their research. Because, when you look at what Procedural Audio breaks down in to, its actually very deep specialisations – just like CGI. There is room for intense talents within the area. Let us make use of CGI analogy again – if you are a real good texture artist you are great at looking at skin and saying, “That is the skin of a 40 year old, that is a skin of a particular kind of salamander, look at the way the bone structure moves underneath it, look at the way the light hits it”. You get specialisations within Procedural Audio which would be people who are very good at fluids, they are great at doing water falls and drops of water and boiling mud and lava. They understand that sound. They are able to model it and come up with great sounding objects and great processes that do it.

Before I took up this umbrella term, this banner of Procedural Audio, and tried to make a focussed idea out of it, I had mentors – I mean people I looked up to, leaders with ideas that nobody else was doing in industry or academia people like Perry Cook. He is like the grandfather of Procedural Audio. He was doing it in the early 90s, when the argument that there wasn’t enough CPU really was a good argument [laughs] and then after him came Dinesh Pai and Kees van den Doel and they worked on impacts and fluids. They did that as very narrow academic work. I don’t know if they saw (the generality of the possibilities) that the water could be taken and integrated with a glass so we get an object that could be filled up or emptied, or become raindrops in a particle based weather system that interacts with different objects the rain falls on… The object-object interaction based idea of “sounding objects” really came out the North Italian schools, otherwise we just have event driven sample playback . But, they did extremely good work and a lot of my stuff is just interpreting their work and generalising, extending it, and making a coherent philosophy of sound as process rather than data. We must always be mindful of that background to it. It didn’t just pop out of the air. It is a project that has been in the background (since Mathews in the 50′s)  growing slowly. If anything, I have  been a very vocal advocate of these ideas applied to the general case of everyday sonic simulation, and been instrumental in defining what procedural audio is.

DS: So the obstacles aren’t purely technological?

Andy: I don’t know what the “real obstacles” are now. I’ve said in another interview before, that around 2006-2007 it dawned on me that there weren’t any fundamental obstacles to radical technical progress. We could do this. The obstacles were structural and political. How do you introduce a new technology? How do you get people to take risks on that? One of the weaknesses of it, it’s a weakness but a very deep philosophical strength (and this is quite subtle), is that sound as data fits in to a capital model. Intellectual property allows you to own a sound asset. So if you record or create a sound, it is an asset that you own. You can trade assets. But procedural model breaks with an ownership model because what you are doing is you are substituting general sounding objects for something we can make million sounds in the future. There is no redundancy built in.

DS: Although, if a game developer spent time and resources building a procedural audio engine, as they would spend time building an audio engine, wouldn’t it be an asset that could be owned?

Andy: The observation there is that the code is the asset. But the code is useless without a group of people who understand how to make it sing. We move the value from residing in the thing itself to how it is used. I see procedural audio as an art to be practiced, not just an application layer to be built.

DS: So even with Procedural Audio you will need a sound designer to understand what it can sound like and how it can impact a player/end user?

Andy: I think this was on the game audio forum or something years back. Someone raised the accusation at me that, “Guys like you put people like us out of business and you are making technology that is going to replace our art”. I took that on board as a very valid point. Being a sound designer myself, the last thing I want to do is put other sound designers out of work. I see it as a liberating step – you have your sound samples and you have this. I always see it as a complimentary technology and not a replacing technology, that is point number one. Point number two that is more important is that every new technology that comes along generates a new requirement for skillets which the talented people in that business become really good at. So every Procedural Audio team would need a good sound designer.  I wouldn’t leave it to the programmers, I want somebody who has a great set of ears and I would actually put them in a higher position and get them to direct the programmers and say, “No its more like this, listen to these examples. I want to get this emotion across”, and they can direct it aesthetically. It’s not really putting sound designers out of work and it is not a totalitarian project. This is why I worry that the bean counters, the alienating/asset-oriented capitalists, are seduced by this kind of technology because they just think, “Well we plug that in and we get rid of the sound department”. That’s not what I want to see happen. One of the great advantages is that it gives 90% of your assets for free. You just put your objects in the world and you get default sounds. What that means is that you don’t have to worry about an asset-event matrix any more. You don’t have to worry that somebody has forgotten to put a sound on something because everything will be covered by default and now the sound designer is liberated not to be thinking up every single little rock sound but to go and focus on the emotionally significant sounds – the hero’s sword, the getaway car, the gun sound. They can put all their time and energy in to getting those right and not have to worry about the other stuff. That’s another argument for Procedural Audio. It raises the bar from where you start from.

DS: So, Procedural Audio is just another tool in the arsenal of a sound designer? A combination of different techniques?

Andy: Procedural Audio is a philosophy about sound being a process and not data. In its broadest sense, if I were to say it is the philosophy of sound design in a dynamic space, and the method is as irrelevant to Procedural Audio as whether you use oils or water colours is to painting. If you use papier-mâché and glue or whatever is to sculpture – the end is in the artist and not the method. So you can mix and match the methods, they exist separately. In the industry now, all the successful Procedural Audio is mixed methods of samples – granular methods with the exception of as far as I can see of Nick Fournel’s work which is basically a similar kind of resynthesis but it is phaselet or PVOC type re-synthesis. First lets see why that happens. There is obviously a clear bridge there between existing technologies and the direction that Procedural Audio can go in. It gives you an immediate start. You can use your existing sample libraries and your guys out in the field. I incorporate this in to my understanding of Procedural Audio as: your sound guys now do analytical recording, not for the purpose of using those as final products but for exposing and analysing the sound underneath so you can build your procedural model. Granular methods are a very direct way of doing that. You just take the input sound and bust it up in to its component waveforms and then you re-synthesise them as grain clouds in different ways. Or in the phase vocoder or linear predictor sense you split them up into transient-exciter components and resonant parts. In that sense it is a direct re-synthesis. Now these approaches that have a method and analyses part, in effect, have a one to one mapping. So you are doing re-synthesis but you can fiddle around with the parameters in the middle. I call this a shallow or phenomenal approach. What it means is that is that the way in which you can change the sounds is limited mostly by your understanding of the parametric interface of the re-synthesis method. The re-synthesis method doesn’t capture any of the physics, it doesn’t capture any of the process of the sounds. Whereas building a procedural model, where you got a model and a method which compliments the model, you are  interested in the behaviour of the sound that is built in to your model. That is not phenomenal. That is not surface, I call that essential or deep. Guys who are doing that kind of thing are like Zheng and James, they are using really accurate models. They are basically doing what computer scientists mean when they say computer modelling. They are using fluid dynamic models to model fluids. They need a rack of computers that take days to process a few seconds of sounds, its not practical Procedural Audio, as would be used in games, because it does not meet the real time criteria for a start. At least not yet.

But having said that I also think there is another side of (non  real-time) Procedural Audio which is not in computer games. It’s in animation, where the idea is that once we have introduced the sounding objects as models, into the scenes, sound ceases to be post production. You re-arrange the objects and their behaviour in your scene to do your visuals and the sounds come out for free. You can even change the location of the microphone virtually after the fact. This is the future of cinematic sound , CGA or computational audio. The best thing about this is that we can drop the real time constraint and  trade speed for quality.  Why not have thunder rendered with one with a million N-waves, which sounds more like the real thing but which takes a long time to compute because you have got a render farm and you are a million dollar Pixar type company? You need good model programs and lots of computing resources but you don’t care about the real time thing. I think Procedural Audio encompasses that as an art.  Skywalker once kindly offered me computing resources, I think Randy Thom set it up, but its hard to work remotely out of context, I would like to put a masters student or PhD on that one day, I grew up with that rather dangerous Radiophonic workshop ethos of Oram/Derbyshire  (two original women sound designers),  creative research and commercial production combined, it is an iconoclasm of experts, expectations and traditions. Much too risky for todays world.

DS: Much about sound design is about achieving hyper reality and not reality. You wouldn’t want a gun to sound like a real gun. So is Procedural Audio about creating realistic models and building on them to achieve hyper realism?

Andy: Amongst the many good sound designers that I have met, they all turn out to be really quite well rounded smart people. They aren’t purely phenomenal,  see the world not only through an artists gaze, but an informed, worldly apprehension. Sound is about going in to the world, what happens inside and outside things, its about deep knowledge about how things work and what does that mechanism mean to your emotions. I think the role of what a sound designer is becomes somebody who’s language is not about computers but understanding the mechanisms of sounds. And the  natural progression for sound designers, when they run out of all the shiny plugin technology in the world is Procedural Audio because it lets their art develop, connecting with sound and its causes and its propagation and reflection. Deep knowledge of sound is what a sound designer has, even when they can’t express or vocalise it. People who have become really talented at that often have hidden knowledge, ineffable knowledge, they don’t have a way to vocalise it or make it explicit but they understand things intuitively and you see them in the studio just do stuff and if you ask them how they did it, they would go “I don’t know, I knew thats what was needed” and it gives them, I don’t like the word, but it gives them a “rationale” to explicate their knowledge about sounds.

DS: How does Procedural Audio fit in to this? Most sound designers aren’t mathematicians or scientists, most of them have a good understanding of how objects react in the world but not necessarily the science behind it.

Andy: In the very long run and I’m talking about – I hope to see it in my life time – ten, twenty, thirty, forty years in the future I think all of this that we are talking about today will be well understood as part of the discipline of sound design. I think the philosophy of Procedural Audio will be a part just as much an artist now can talk about textures and different kinds of lighting. In a way sound is a real throwback, right now. Some people have said to me, “Sound technology is fifteen-twenty years behind graphics technology”. That’s crazy, all of the algorithms that made visuals possible came out of sound. One dimensional signals first and then two and three.. it comes from radio and radar. And, somehow culturally, sound got left behind. Because, we are visual beings and we put all our energy in to manipulating and creating ways to have a visual reality. Twenty-thirty years in the future all of what we are saying now will be a part of the language of sound design. But right now, theres a fork in the road between my understanding of it – the philosophy and the way it should go and the way it will go practically – and what is happening. So part of my philosophy in education is principles not products. It’s a reaction against the commodification of skills and people, techniques, arts being reduced to products. So if you look on the sound design list when somebody says, “How do I make such and such a sound”, and somebody else says, “Oh you need the zzaaaq plugin, that does that”, and they completely abdicate any desire for knowledge. They don’t even care (to pay for it) because they can get it off bit torrent anyway. Somebody else has packaged that capability and knowledge and given it to them. And by doing that they have robbed them of the knowledge, (Zarathustra says: be careful what I give you as a gift, because I may take something away from you) the knowledge is useless by itself but as for an artist, for your career, your development, for your ability to do things, that knowledge is important, its part of it. Both the commercial approaches to Procedural Audio at the moment, and this isn’t dissing these guys (in Audio Gaming and Sony… I am in touch with Amaury quite a bit and try to help them out recruiting and seeing the way ahead), what they are interested in producing is products (that is business). They want Procedural Audio models as drop in objects in the game. With this model, given to the current middleware developers, you’ll probably have opaque Procedural Audio objects, you cant see inside them, they have a few exposed parameters. Say you have a car and you can choose four or six cylinder engine, a bunch of different configurations for the exhaust, you can choose the body material, the tyre and that would be it. That is good! That is how you want your end user to see the object. At a certain level. So one approach would be to sell these as closed objects, with their functionality is hidden. But, for me as an “academic”, as a pioneer I am much more interested in enabling (my relationship is with the sound designer, with people), you should get in to this stuff. To me, the good stuff will be toolkits like the kind Dylan Menzies and others have proposed  (and using the pluggable physical components using FEM/discrete numerical difference schemes like Stefan Bilbao explicates) Now you have an engine model and its a part of the car and you can replace the engine – real world analogy here – you can tinker, you can take it apart and change the the way the camshaft and the pistons work. You can replace the method that is used for that engine with a subtractive method that you wrote in C++ and drop it in because there is a well defined interface. And the kind of well defined interfaces that work are like data flow interfaces – like Max/MSP and Pure Data type interfaces where you can just plum these objects together very quickly in the studio and test them in-world while playing. Ultimately, open ended data-flow user interfaces with efficient JIT /bytecode compilers are the future for creative Procedural Audio. Then you will have a very vibrant community, a vibrant ecosystem of programmer-sound designers and sound designer-programmers and people who work in teams and Procedural Audio will be a vibrant technology. But I think first it will go through what basically becomes a plugin (mystified) culture. Procedural Audio in my (wishful) philosophy is open and based around knowledge [laughs]. That will come with time after these products have driven a path and we will think about sound differently. The construction of sounds will be more technically informed and richer than saying, “Oh yeah, Hollywood Edge track 6 number 4, that’s the one you need”.

DS: Would you relate the questions raised about Procedural Audio to people talking about motion capture replacing actors?

Andy: I never made that connection, that’s really good. Mo-cap in relation to CGI is the same as what I am calling analytical recording for Procedural Audio. You go out in the field and try and look at the behavioural features and try and capture them and then use them as data.

DS: So would that be the important point to make then? To use the analysis as data and not exactly copy it, just as how mo-cap is used in CGI?

Andy: Yeah, I was talking to some guys who are in to this mo-cap and human face stuff recently and they were telling me some amazing things about how Tom Cruise voiced an animated character and they got all his face expressions – the eye brows and everything. The trouble was it was a dog character or something and when they played it back it looked too much like Tom Cruise [laughs] – and this was how you could extrapolate to hyper reality where you just scale everything so that the eye brows doubled the distance. So they got this super hyper real version of Tom Cruise in the character but somehow you still knew it was him. This is built in to procedural technologies and data analytical technologies. You can create models based on interpretations of real data and then extrapolate that off in to hyper reality and thats really powerful in film and games. You want that capability built in. They were doing something else really weird, like they were morphing characters, they were doing something like Sigourney Weaver and Tom Cruise and you make Sigourney Cruise. Like you do with hidden Markov models in composition where you can hybridise them and you can kind of have Beethoven and Shostakovich and get new ones, Shothoven or Beethovich (??) [laughs] or what ever your new composer is.

DS: The obvious use of Procedural Audio is gaming and animation, as you mentioned. Where else do you see it being applied? There has been some talk of it being used in electric cars to simulate engine sounds.

Andy: I can see it as a real interesting safety feature in a car. I would also worry about this becoming a real nuisance – imagine a car running along the street sounding like a clown’s car and the person is drunk and its 3 in the morning! I can see a lot of social tension about objects having arbitrary sounds. Sound can be very intrusive and it has very different implications across different cultures and age groups.

DS: I came across a Harmon owned company called HALOSonic that promises to deliver technology that seems to simulate engine sounds for ‘cars of the future’. While there is very little information on the actual technology being used, I wouldn’t be surprised if it used Procedural Audio.

Andy: Let us analyse that as a product and say why Procedural Audio is really powerful for something like this. On the face of it you would think of using samples. I am the CEO of the company, I want a cheap product and I want to get it out. RAM and disk space is so cheap. I’ll  get a bunch of sound designers, get them to make loops, load up my product with a thousand loops and plus I can now sell you extra ones. Why the hell would I be interested in Procedural Audio? There were two really powerful reasons – technical computer science reasons. Number one: If you haven’t got memory space like greeting cards, watches, mobile devices – one of the biggest errors of judgement in mobile/casual gaming and the whole mobile technology industry has been over estimating the available bandwidth. And what’s a good technology when you haven’t got any bandwidth? This is what is your biggest asset when you deliver procedural content. It occupies 4kb and plays for six hours. It has a million things that you can change all the time or download another one. So, as a technology procedural content is very powerful in situations where you’ve got limited bandwidth. With cars, another place where procedural technology is very powerful is where you want the sound to encode a large vector of changing parameters. Why is it useful to have a sound on a car? By listening to a car engine I can tell a lot about it – is it slowing down, speeding up, is it a large car or small car. I can localise it pretty well. So to replace a completely silent car engine what you want is a procedural sound object which behaves like the car (that is familiar to peoples expectations viz a viz reality – and hence safety) with engine, with tyre sounds, with exhaust simulation to delineate rear and front approach  In fact you could encode all kinds of other information about the car as a safety feature which people would quite quickly get used to. If it is a bus – it could be a bigger noise, if it is a bike it’s got a lighter sound. That would be difficult to do with a sample. So the procedural object would be more versatile and able to to encode more information. That would be argument number two. Argument number three might be that to develop a library of a thousand different car engines would be very expensive. But once Procedural Audio technologies mature I should be able to buy an engine model as a one piece of software and adapt it – I could commission it as a one of piece of software or buy it on a license, put in to my product and I have all the versatility of it. There comes a point I think where the code becomes cheaper than the recording, for a limited use case. Maybe. I’m not sure about the economics of that. I am interested to see how it turns out. If on the other hand there are a lot of people out there recording things and there is a very buoyant market in recordings….

DS: Do you see Procedural Audio code becoming a commodity that is built and sold?

Andy: I think so, I see that happening with the apps market. Production models have changed, and this is very much down to Apple. To their credit, I’m very anti-Apple at the moment (for ethical reasons), but to their credit one of the really good things that Apple has done is (by standardising experience and lowering expectations) accelerate the tool chain to the point where the production of an app is a half a day’s work sometimes. Cookie-cut template things that go out in the store and you download them. That is code. It’s the entire application. Whereas before, code was something that was produced over weeks or months, that still happens with bespoke apps but we’re seeing the trajectory towards commodity code like that. So yeah, if you are a great procedural sound designer why could you not be a guy who specialises in engines and you sold two to Volkswagen and one to Mercedes and you are working on a couple of other ones for some company? People come to you because they know you are the ‘engine guy’. Because its the standard API for interfacing with the procedural code you know at the end of the day you are going to get six floats or something – four rotational velocities for your wheels and you are going to get an engine speed and you are gonna get a bunch of stuff and you plug that in. I would like to be optimistic about the futures for these market places, I would like to think they they might work. That is very much my (humanist) philosophy as a person, I think that these things should create work which should create opportunities and they should create markets and they should create things that people can do and involve their talents and this is why I say you balance technology, art and business . And any one of those can become an overbearing dominant thing. The culture can override the technology and the business, the technology can become the “oh technology oh technology!” and stamp on the business but more often these days it’s about the business running out of control. If the business dictates the technology and the art, if any one of those three gets out to kill the other its bad. As for Apple, who want to commodify and control your creative experience for a profit motive – the absolute antithesis of their “nineteen eighty four position”. So what I would fear in the case that you are saying is that commercial attempts at pushing procedural audio initially turn out quite watered down,  neutered and over-packaged like supermarket food, in which case it will be procedural audio only in name, only in marketing speak. But then I’m a dreamer, 20 years ahead with this stuff in my head, disappointment with present reality is built in, its what keeps us pushing. Computational audio, more intelligent structuring of audio in games and cinema, will be a new creative frontier. But we need to get past a crisis of purpose with technology. The idea that easier is always better, that more is always better, dumbing down, disabling and concealing rather than opening up, enabling and enhancing. To paraphrase Laing who says “The emphasis is more and more on communication, but people have less and less to communicate”, we don’t really need more ways to do things, but better, more thoughtful and reflective ways to do things, to replace brute force and dizzying excess of choice from the previous epoch with more focussed and elegant ways. I hope some of the philosophies of procedural audio will shape sound design in a wider sense.

First: a BIG thank you to all the contributors, this wouldn’t have been possible without the sounds!

How did this work?

The Max/MSP patch works by playing random files in a random sequence and from random points within the files. If left by itself it can play these files back in this random order to create a never-ending soundscape. Although, there are a few controls to help design the way it sounds:

• Play length of each file (Eg.: If it’s set at 4000ms, it would play back a sound for 4000ms and then crossfade into the next sound)
• Fade length – crossfade length (from 0ms to 1ms less than the play length)
• Speed/pitch: Vari-speed playback control
• Reverse
• Random type: Urn (random without repeats), Drunk (Randomised but ‘drunken’), Random (random with unpredictable repeats), Counter (sequential playback)

The outputs were connected to a channel strip type interface, from which the signal was sent (parallel) to a reverb and delay plugin.

What you hear below are sounds that were contributed (thank you again!) and then generated and ‘performed’ using the above mentioned controls with a MIDI controller. I recorded a few takes and this was the one I preferred the most. I’ve also included a screengrab of the patch at work, for the curious.

Have a great sounding year ahead!

Thanks to:
Eckhard Kuchenbecker
Echo Collective
Hrishikesh Dani
Chris Nealy
Angel Perez Grandi
Jaydea Lopez
Ana Roman
Péter Terner
Anna Papaioannou

Year End Sonic-Mash on Soundcloud

Click here to view the embedded video.

Here is the first interview with this month’s special guest Elliot Koretz, talking about general aspects of his career.

How did you get started in sound design?

My first industry job was as an apprentice editor in the shipping room at Disney Studios. I was exposed to all types of editing (picture, music, and sound) but I was attracted to sound for not only what I saw as the ability to be very creative but for the autonomy of working independently of the director and producers who seemed to be always in the picture editors room. At Disney I met a sound editor who was also moonlighting at Neiman-Tillar, a leading independent sound house back in the day. He saw my interest in wanting to advance to editor a little quicker than what was the norm at Disney and offered to put in a good word for me there. I was offered assistant editors position and took it. While there I was first introduced to electronic editing. This was approximately 1980 and they had, as far as I know, the first system that was used for this, ACCESS. That’s really pretty amazing for so long ago. I think the first show I ever cut on electronically was a tv show, “Aloha Paradise” It was a kind of “Love Boat” on land and the sound needed was pretty straight forward fx. But I do remember one particular episode where the story line had a man who was interested in a divorced woman with a young child. The kid was opposed to this relationship and at one point bites the guy on the leg in kind of a comical manner. This lead to what I believe may have been the first “design” moment of my career. I layered a celery snap with some sort of other big crunch and………I was off and running as a designer.

After that I moved around landing at a number of post facilities for a while. I was an editor at Stephen Cannell, which turned out to be a great place to learn to cut action sequences. On shows like “The A-Team” you had a week to cut an entire reel (approx 12 min) of Dia, FX, BG’s and Foley. And inevitably you had a scene like this: Our heroes were in some sort of large vehicle, traveling pretty fast on a rough surface, being chased by a helicopter that was shooting at them. They meanwhile had constructed some sort of rapid firing gun that was shooting nails or some other projectiles……..and little to none of this could be created just straight out of the sound library.

These kinds of sequences needed multi-layered design and remember this was on film. Many units and also much of the final result of my work couldn’t be heard played together until the dub stage. On an old fashion film sync block you could only hear three or four “channels” at once. Anything wider than that and you had only your experience and imagination to visualize the combined sound.

I think doing this kind of design work way back then really helped me understand how to efficiently combine elements to get the sound I wanted.

I spent some time at Soundelux when the company was still pretty young and while there moved into cutting sound on features. (Still editing on film). I did return to tv editing and ended up working first as an editor then as supervisor on the show, “MacGyver”. It was another busy design show with the lead character always inventing something to beat the bad guys that required creative design work. After a successful first season the producers wanted to change to an all-electronic post. Soundelux at that time was not prepared for the huge investment in equipment and ultimately the show was moved to a newly created facility, Modern Sound. Over that summer they built a new mix stage, foley stage, and editing rooms using both Synclavier and 24 track editing systems. I was offered to continue as the supervisor of the show and accepted. After a very brief training period at the offices of New England Digital (the creators of the Synclavier) I jumped into the world of electronic post again.

The problems we faced were immense. This was 1986 and the technology was still in it’s infancy. There were not yet sound libraries that were “digital” and the decision was made to purchase a copy of the library of a leading sound supervisor at the time, Fred Brown. Then the issue was storage. The best we could do at the time was to digitize onto floppy discs. They could only hold a few seconds of sound each so you can imagine the challenges that caused. This was truly the bleeding edge of technology.

It was at times very exhilarating but often very frustrating to be at the forefront of this transition. There were times we struggled to achieve what was extremely easy to accomplish on film and other times we saw how cool it was to work in a non destructive environment with new tools to manipulate the sound.

After that season I moved around again to a couple of different facilities but then found what turned out to be a long-term home at Weddington Productions. The three owners at that time (Steve Flick, Richard Anderson, and Mark Mangini) were doing some of the most creative sound design anywhere. There is no question that was the turning point in my becoming a much more accomplished designer. Working with the talented people at Weddington constantly challenged me to step up my game and really think hard about what I could do to impact the movie sonically in every detail.

While there I made the full time transition to ProTools and it’s world of opportunities that cutting digitally has brought to all of us.

All these pieces of the puzzle have helped form what I do today. At Universal where myself and my crew have 5.1 editing suites and all sorts of plug in devices I reference all that experience from both the film and digital worlds when conceptualizing the design work I do.

How has been the evolution of your work and how your approach to sound has changed over the years?

Well, in some ways it’s changed dramatically and in others not so much. In a practical sense I mentioned the switch from film to digital. I really embraced it and all the flexibility it gave me while staying in my editing environment and not having to wait for a reprint of something or a specialist for processing. It’s just more efficient and much easier for me to experiment with sounds. In a more subjective perspective I think I grow after every film I do. I am a very hands on supervisor and I feel that one of the perks of being in charge is that I get to choose which elements of the project I will personally handle. I still try (time and budget permitting) to be very old school in my method. I like to pull and organize the fx and bg’s my editors will work from (I always encourage and give them the option of going beyond the pull) and give them a “cut list”. I think that method lends itself more to continuity and flow of the sound of the film. If I can’t do that then I meet with the editors, run the reels and give as much info as I can to them and review the work later. I think one of the bigger changes in my approach in recent years has been to make a concerted effort to co ordinate with the composer more. We all have been in the situation in a mix where we are fighting for the same sonic space with the music tracks. If I know where the music is working and in what frequencies and what type of rhythm I can attempt to compliment it and not fight it.

I wonder how sound design has changed the way you listen.

I think I listen and think about emotions. What are we trying to say to the audience? Like with music I don’t want to fight the dialog so I see fx, bg’s and design as a tool, sometimes very subtle other times not, to promote the directors’ vision. I listen for bridging opportunities to use sound to connect scenes. I always remember on the dub stage for the movie “Speed” Greg Landaker (not sure about spelling) the lead fx mixer suggested some great ideas to do just that. The frenetic pace of the film lent itself to fast whooshing elements to bridge cuts. It was just one more layer to make it a more finished and cohesive movie.

How has been your work with directors? any particular story on that?

You touch on a very important question. I think that just as important as my design work on the film or maybe sometimes even more important is my rapport with the director (and the picture editor as well). We as supervisors and designers need to be very politically astute and sensitive to the personalities we work with. Some clients like a “take charge guy” who they are counting on to lead the way in the sound post. Some want a person that gives them exactly what they ask for…and nothing more. I guess what I am saying is that we need to size up who we are working with and as early as possible give them what they need. As wildly creative as we are we can’t lose sight that we are a service. I don’t believe that one style will fit all.

I have been very fortunate to work with some amazingly talented directors. When people look at my resume they usually want to know about Michael Mann. In addition to working on and supervising some of his television shows I supervised and did the design work on both “Collateral” and “Miami Vice”. Michael is without question a creative genius and a visionary that has given us some amazing tableaus. The challenge is that he is so demanding of himself, often working 20 hours days for seemingly months on end and he expects his team to keep up with him at all times. I think he has his ideal of the visual and sonic harmony he wants and has little tolerance if you are not on board with him. If you understand that it makes your job less difficult. People always want to hear horror stories, the truth is that the hours were long and tough but as I was mentioning in the previous question when you understand who you are working with and what they expect of you then you as a supervisor can depersonalize challenging situations for you and your crew and keep everyone on point.

I did a film with the amazing Irish director Jim Sheridan. He was a very easygoing guy with me and my crew and regaled the dub stage with wonderful tales, as is the tradition for storytellers like him. His style was more to allow me the freedom to bring design ideas to the stage and then he would give input.

I love when a director really understands and supports what sound and sound design will bring to their film. I worked with Gavin O’Connor on the film “Miracle” a few years back. He wanted realism throughout his film. He wanted hockey players that could act as opposed to actors that could skate a bit and for sound he wanted the most realistic sounding sports movie ever. We did extensive recordings of skating and hockey crowds and then mixing with Mike Minkler and Myron Nettinga we got a terrific soundtrack. Gavin was so incredibly appreciative of the work we all did and that’s always refreshing and nice to have.

Another great collaboration has been with the director Thor Freudenthal. His name may not be familiar to everyone but I think it soon will be. He is a very talented young director. I worked on “Hotel for Dogs” and “Diary of a Wimpy Kid” with him. Both films presented unusual design challenges and Thor was very supportive to make that sonic space that we all hope for available in the final mix.

All these directors I just mentioned understood the value of doing field recording for their films. Whether it’s getting out to Miami and recording onboard speed boats at over 135mph (Yikes!), Directing a crowd of 5,000 people chanting “USA, USA” or dog ADR sessions (Story on that to follow), working with someone that gets the concept of what we can bring to the film by doing these things is always a bonus.

I wonder how you approach the different roles you can play on a film, such as sound designer or supervisor. Any preference?

I really enjoy them all. Unless a particular film is just too demanding a job for me to exclusively hold both titles I will try most often and handle those myself. To accomplish that I am fortunate enough to have worked with for almost ten years one of the best assistants (who also happens to be one of the best field recordists, great editor and also talented mixer) Bruce Barris. His wide range of skills allows me the freedom to be creative while he has handled some of the other aspects of the workflow. He has been an invaluable partner in the design process.

And speaking of that I do see the work we do as a collaborative effort. I am most definitely the point man with the client but it is the entire team that I count on. With the budgets so tight these days my crew is often small. Everyone has to be really capable. I try and spend quite a bit of time with each member keeping them up to date with as much info as I can.

On some of my films for one reason or another I have assumed the role of ADR supervisor as well. I do really enjoy getting the opportunity to work with the actors.

So I guess that although design is probably my favorite part of the job, as I like to say “it’s full service” and I’m good with hands on the other tasks as well.

What are your favorite tools to work with?

I think it’s rare that I use any effect out of the library without doing some sort of tweaking to it. I use quite a bit of the standard plug ins that are included in ProTools and also the Waves bundle, Izotope (particularly Trash), AltiVerb, and Speakerphone to name a few. Multiple layers of sounds addressing different frequencies are the key. I look for new plug ins and applications all the time as they are rapidly growing.

Do you have any special method for dealing with deadlines/creative challenges?

Well, that’s not very easily accomplished. It’s time management. I think one of the most important skills in that regard is having the dub stage experience to really understand what will play and what will be less important in the overall mix. Sizing up the key sequences and looking at how much time you have to spend on them is crucial. I find this does not come naturally to everyone and I help my crew know what areas to concentrate on.

Is there any of your projects that you remember for being the most challenging or favorites?

Which of your children do you like the best, eh? So hard to answer. I will pick out one but I probably could find examples in almost all my films.

A couple of years ago I did a family film, “Hotel for Dogs”. On the surface it seemed like a fairly simple movie to do. There were some Rube Goldberg like mechanical inventions to design but otherwise I didn’t at first see any challenges or possible difficulties. Was I off the mark…..by a mile. The movie had many dogs in it (it was titled Hotel for Dogs…..right?) a number of them feature performers. The conceit was that they sounded like normal dogs. Nothing comical or unrealistic in their performance. It turned out that in every single bit of production the tracks were filled with the sound of the various trainers urging on their dogs to perform with whistles, clickers, and other devices that basically made the original sound track unusable.

So now I was faced with the reality that I had to replace every single sound all the dogs made for the entire movie. There was no library in town that has such a variety and complete sets for all these dogs. I was in serious trouble until an incredibly serendipitous event occurred. Some of my crew members and I were walking to lunch. We were working at Universal and sometimes we would cut through the theme park to eat up above us at City Walk. As we walked through the park I noticed there was a stage with the sign that read “animal act”. There was a worker standing in front with a dog beside her. I told her I was a fellow employee and what I was working on and asked, “Do the dogs in the act follow commands to bark?” She assured me they did and led me to backstage to meet the trainers. Turned out they had worked on my movie and actually some of the same dogs were here in this live show. After discussing what was needed with the trainers we set up a date and brought the dogs down to the foley stage for a “doggy” ADR session. Each dog responded to silent commands and barked, whined, sniffed and growled as we recorded them. I now had my kits for each of the main dogs in the movie.

Cutting their tracks was like doing voice replacement for about eight actors throughout an entire movie. Dogs never stop making sounds. They are always panting and licking and doing something that required considerable thought. I would find the most evocative material while still “keeping it real”.

The satisfaction came that in the final product my work was truly invisible. The dog vocals fit perfectly (being from the same dog in many instances) and no one would ever suspect that what they were hearing was not production. The work did not call attention to itself but never the less was some of the best sound work I’ve done recently.

What are your favorite films for sound?

That’s another loaded question. There is such great work out there. I go all the way back to classics like “Shane” and “Forbidden Planet” as early examples. And certainly I used to try and destroy my speakers playing “Top Gun” and then “Days of Thunder” at dangerous volume levels. The work of Ben Burtt, Gary Rydstrom, Randy Thom, Ren Klyce….I could go on and on. When I worked with at Weddington the movies that we were doing, Die Hard, Apollo 13, Speed, all the Joe Dante films……..were so incredibly well done. And recently my colleague at Universal Scott Hecker has put out some of the coolest tracks (300, Watchmen, and Suckerpunch) with Chris Jenkins and Frankie Montano mixing. I thought Avatar was an incredible piece of work knowing the difficulties in having to conceptualize design when you may still be working against a storyboard. I love movies. Always been a film fan and it’s just too hard to narrow the field on my favorites.

This is a good point to mention mixers. To understand how to collaborate and help them do their job is huge. I can’t emphasis enough my belief that it’s a team effort and although I do plenty of premixing back in the editing room I love that another set of very talented ears listens to the material and can add their expertise to it. I always try and meet with the team as early as possible and include them in the process.

Is there any advice you’d like to give to other sound designers out there?

Well, I think the key is to think divergently. Outside the box. Don’t be confined by the laws of nature. That’s how little kids think and that ability seems to disappear as we grow up. I know that there are sounds that have to be exact and correct but emotional sound has a huge role in design. And the practical advise is to put your ego aside and listen to what the filmmaker is saying and present yourself in a manner that instills confidence that you are the right person for the job. One of my favorite stories that help bring that point home is this. Walter Murch and Randy Thom were participating in a forum about sound. When Randy was speaking he told a story of how when he meets with the director he regales him or her with visions of incredible design to come with all sorts of amazing nuance and the client is wowed. They know they have he right person. The the meeting ends, Randy goes into the privacy of his editing room and says to himself, “How the f*ck am I going to do it?”

So don’t let them see you sweat. Bring your best attitude to your meetings……and then go back to your room and start panicking!

Seriously, this has been a lot of fun. Thanks for the opportunity to share some stories. I hope this has been informative and a little entertaining.

New year, new month, and of course, a new featured sound designer. This month our special guest is sound designer Elliot Koretz.

I was born and raised just outside Boston and got my start in the industry very early. I guess I was always fascinated with tv and movies. When I was fourteen a cable tv/public access facility opened in my hometown and I got a job there (sometimes volunteer sometimes paid) working in tv production. It was great exposure and training. A number of years later when my dad got a job transfer to the Los Angeles area I moved out to L.A. with the thought that this really is the ultimate place to be for the film industry. I had a degree in film and went looking for work. After what turned out to be a very short search I landed an entry level position at Walt Disney Studios. I worked in the mail room for and then I had an opportunity to get into the editing dept which started me on my my way to being a sound editor and designer.

I am married with two children, two birds and a dog.

Credits @ IMDb

Those of us working in the field of audio production are accustomed to either a lengthy haul of design and content creation starting in pre-production or a series of intense sprints from one project’s post-production to another. Whichever camp you might fall into, the time and focus required by our craft make it easy to fall behind the latest developments in our field and fall out of touch with our network of peers. There are steps that both aspiring and seasoned audio professionals can take to stay current and stay connected.

To Get Ahead, Start By Looking Back

If you’re like me, you find it sometimes painful to look back at your previous work. There’s always something you’d have done differently knowing what you do now. But looking to your past is a useful exercise in refining your sonic identity while isolating techniques and aesthetic choices upon which you may be relying too heavily. Once you’ve put your ideas out into the wild, they instantly begin to decay and you need somewhere else to turn while you wait for everything old to become new again. Keeping your sound fresh and growing as an artist requires breaking out of your comfort zone, sometimes forcibly. But where to begin?

Experiment and Deconstruct

When you’re doing project-based work all the time, it’s not hard to forget the importance of raw experimentation and exercises in deconstruction. You wouldn’t be hearing “the new sound” in current media without that sort of experimentation and there’s no reason you can’t be leading the charge or at least staying in stride. Try some new mic techniques, some non-literal/nontraditional designs, or some new plug-ins to evolve your sound and sharpen your skills. Speaking of plug-ins, take advantage of the many affordable (and even free) plug-ins and sound manipulation programs available on the web. If you need a couple to get you started, check out Paul’s Extreme Stretch and the SIR2 convolution reverb. You can lose yourself for hours using either one.

Deconstruction is an equally important skill that can be difficult at first, but it gets easier the more you do it. Try to recreate unique sounds that you’ve heard in a recent movie or game and try to understand why certain sound design choices and techniques are successful. Can you replicate the sound of a Recognizer from Tron Legacy? Can you explain why the dynamic dialogue system in Bastion succeeded where similar systems that came before failed? Neither of those examples has a simple solution and your conclusions might be different from those of another sound designer. I raise the point because you can bet that at some point in your career, you’ll be asked to replicate an audio experience or specific sound from another production. Regular practice in deconstruction will help you do deliver on such requests quickly. Now where can you go for reference material?

Dig Into Some Indie

Naturally we need to keep tabs on what’s happening in the worlds of big budget film and triple-A game development. But sound designers for independent film and games are also constantly pushing the limits of their media. Sometimes they’re going for a nontraditional sound to meet a new visual style or, in the case of games, a new game mechanic or play style. Other times they’re innovating out of necessity since they’ve been asked to do more with less. Someday you may be asked to do the same. In any case, there are lessons to be learned from the indie scene so try to keep at least one finger on its pulse. For games, you can easily find new releases around the web on sites like IndieGames.com. If you’re not close to a theater that screens independent films, services like Netflix and the iTunes store are starting to carry a fairly deep catalog of independent feature-length movies and short films.

Take a Trip

There can be vast differences in audio aesthetic from one region to the next, with some of the most drastic differences found in comparing films and games from Asia to those of the West. The aesthetic of one region can give surprising and compelling results when integrated into the traditional aesthetic of another. Try to find places where the aesthetics of two or more regions can work together even if you’re leveraging their points of contrast. Those of us interested in freelance sound design should be well versed in regional aesthetic to meet the needs and expectations of our international clientele. Game releases are hit or miss when it comes to availability outside their country of origin, but the biggest hits are typically available around the globe. Foreign movies, on the other hand, are readily available and online movie outlets are again proving an excellent source for your research.

Tune In

A modern sound designer should feel comfortable bringing musical character to his or her work. We’re hearing less wall-to-wall music in film, games, and television lately. This creates new opportunities for a sound designer to support narrative and drive mood in places where underscore would have traditionally been used. A sound designer should also be familiar with the trends and styles in modern music. Placement of source music will often fall into a sound designer’s responsibilities and familiarity with the latest developments in the music scene will allow for informed choices in that placement. Electronic music tends to lead the charge with novel technique in DSP and production. If you want to hear where DSP is headed next, look to the likes of BT, Richard Devine and Amon Tobin. As for keeping tabs on the rest of the music scene, tap into outlets like Spotify and Pitchfork to find out what’s trending in music.

Take Notes

It can be difficult to keep all this information in your head, so get into the habit of keeping notes about your work and the work of those around you. When I find a game or movie with particularly interesting sound design, I keep a notebook handy to jot down timings for points of interest. This is especially useful when working on a team where there’s a lot of reference material being passed around. Note taking is also important during your exercises in experimentation and deconstruction. Knowing how to get back to your best results is just as important as the experimentation itself.

Build Your Network

One of the best resources for any creative professional is a strong network of similarly focused individuals. For the sound designer, this network would naturally include other sound designers and audio professionals. It should also include professionals from a variety of complementary disciplines such as studio engineering, field recording and film-making. This network will serve as additional pairs of ears in your hunt for new and interesting sound design.

One of the most useful developments in the past few years is the level of access we have to each other via Twitter. Professionals who were otherwise unconnected or unreachable five years ago are now a tweet away from your questions and communiques. When limited to 140 characters, a lot of people are willing to (briefly) open up about the work they’re doing. If you’ve got a question for a sound designer, filmmaker or editor whose work intrigues you, send them a tweet and many times you’ll get an answer the same day. That tweet could lead to a more involved collaboration in the future.

Make The Time, But Know Your Limits

One of the most important things to remember in your efforts to stay current is that your time and your capacity for information are limited. The topics I’ve mentioned require effort on your part and trying to do it all at once will only burn you out. Prioritize the material most relevant to your industry and choose your battles when consuming media.

I’m focused on game development, so the majority of my time is spent checking out the latest developments in game audio and the gaming community. Every week I play several new game demos and retail releases realizing of course that I can’t finish every game I get my hands on. My Netflix instant movie queue is always growing with recommendations from colleagues and friends so I try to knock out a couple of movies each week as well. One of the greatest sacrifices we make as sound designers is that, for the most part, we can’t listen to music while we work. Most of my music intake happens during implementation work, my commute, and while reading the news. On that front, there are a handful of audio and gaming news sites I check in with daily, a job made much easier with Google Reader and RSS feeds. Where in all this do I find time for the experimentation and deconstruction that I value so highly? Wherever I can. And this brings up an important point: Experimentation and deconstruction are often spontaneous acts that happen when inspiration strikes. Keep your audio workstation and gear ready to go at a moment’s notice.

These practices work well for me, but everyone has different needs. Consult your professional and personal networks to inform your workflow decisions and to keep you posted on new media, new techniques, new websites, etc. Above all, remember that this isn’t an extracurricular pursuit; this is part of the job. Find the time to do it, proceed wisely and stay at the front of the pack.

Since it is that time of the year when most people are going to be celebrating/holidaying/having-a-good-time in some form or the other, I thought it would a good idea if we could create a sonic-mash of these sounds by getting everyone to contribute.

How? Record anything – voices, ambience, fireworks, bottle opening, shouts of joy, sounds of calm, carols, the local pub, a fireplace – anything that you might experience from now till the 9th of January. It could be recordings off a mobile phone, handheld recorder or the new microphone you gifted yourself for Christmas/end of the year. I will close all upload channels on January 9th and on the 13th I’ll post a few of these generative soundscapes made up off all the sound we’ve recorded.

Restrictions

Minimum Duration: 4 seconds
Maximum Duration: 10 seconds
Format: WAV/AIFF
Sample Rate: 48/44.1KHz (nothing higher if possible, I don’t want to fry my RAM!)

Where to upload?

I’ve created a SoundCloud group which you can access here. Please make sure the files are downloadable.

Or by FTP:
FTP Username: sonicmash@re-sounding.com
FTP Server: ftp.re-sounding.com
FTP Server Port: 21
SFTP Server Port: 2222
Password: sound123

Or you can email me: varun[at]designingsound[dot]org

This should be fun!

Meanwhile, have a good Christmas/New Year/Season’s Greetings/just-a-good-time!

DS: Like many other audio professionals that I know, you’ve got a background in music. Do you consider that the driving force behind entering an audio profession? How did that background in musical performance affect your first foray into audio production?

JS: Since musicians are attuned to their aural surroundings, either naturally or through training, they’re also instinctively aware of the importance and influence of sound in media like video games and film. This awareness led me to the field of sound design and that seems to be the case with almost every other audio professional I’ve met. It’s certainly helpful to work with other musicians because we’ve all inherited a vocabulary of articulation, dynamics and tempo that can be applied directly to sound design, editing and mix.

My sense of phrasing and dynamics in sound design is largely due to my background in music. Like many sound designers, when describing the design of a sound or scene, I often find myself “conducting” my way through a verbalization of what I’m hearing in my head. This is why sound designers don’t talk about their work in public; people might think we’re crazy. Most importantly, performing has given me a natural education in mix, especially my performances with orchestra and chorale.

DS: You’ve also studied computer science. What was the motivation for that, and how did you envision these two fields merging in your life?

JS: When I was preparing to enter the field of game development, I realized that many game audio professionals were expected to implement and troubleshoot their work at the code level. During my studies in programming, I concentrated on building a simple audio engine and then integrating it into a larger development project. Meanwhile, my friends in Berklee’s Music Synthesis program were using the Csound programming language and early versions of Max/MSP to create sounds unlike anything I’d heard before. I wanted in on the action and my studies in C/C++ and procedural programming helped me hit the ground running.

With the robust offerings in audio middleware available today, sound designers typically don’t need to dig down to the code level to get their work done. Still, game engines and tools are growing increasingly complex to meet the scale and scope of modern titles. I consider a strong technical background to be a huge benefit for keeping up with the constant changes and for interfacing with the tech team.

DS: How would you describe your transition from music into sound design? It seems like there’s been a fairly linear progression through the games you’ve worked on.

JS: To be honest, I was concerned that moving into sound design would be more of a fork in the road than a transition, almost as though I’d have to sacrifice music to pursue sound design or vice versa. This has proven true to some extent because on large-scale projects, no one person has the bandwidth to do it all. The good news is that the modern aesthetic in game audio and cinema is blurring the lines between sound design and underscore. It’s exciting for me to hear the fields of sound design and music complementing each other, playing off one another, and arriving at a soundscape that wouldn’t have been possible given a more traditional separation of the two disciplines. I’ve embraced this aesthetic entirely.

Day-to-day, I find that I create individual sounds, compose scenes, and mix in a very musical fashion. When I look back at the work I’m happiest with, it’s always a place where I’ve developed the sound or scene “musically” in terms of voicing, articulation, and timbre.

DS: What, or who, have been some of your biggest influences sonically? What do you bring from those influences to your work?

JS: Wow, there are so many personalities and projects that have shaped my sonic style. My strongest influences come from the worlds of music and film sound. On the music side, Björk and Amon Tobin have had a huge impact on my sound design and personal aesthetic. I love the way Björk’s songwriting and vocal performances drift between haunting and visceral. If I can capture that same emotional range in my work, I know I’m doing something right. I consider Amon Tobin to be equal parts sound designer and musician. Deconstructing his music is some of the best sound design education you can get.

In the field of film sound, I’ve been heavily inspired by the work of Ben Burtt, Walter Murch, and Randy Thom. All three of them are incredibly committed to detail and quality, but more importantly they design, edit and mix thoughtfully, always complementing the overarching narrative of their films. Another thing that draws me to these three is their ability to articulate how and why they arrive at their conclusions and practices. Their lectures and articles have proven more valuable than any other resource in my career.

DS: Is there anything you do to find new influences and inspiration? What do you prioritize in continuing to hone your craft?

JS: One of the biggest challenges for an in-house game audio professional is staying current in popular culture and in the trends of audio itself. In an age of unprecedented and convenient access to games, film, music and literature, that challenge gets a lot more manageable. I draw a lot of inspiration from music and film. Spotify and sites like nme.com help me keep tabs on the latest developments in music. Netflix is my go-to outlet for discovering independent and foreign films that are hard to come by in the theaters. Most importantly, my friends and colleagues who share those same influences keep me informed of their latest discoveries.

One thing I keep in the back of my head while working on a multi-year project is that the tastes and trends in audio are always changing. I know that the sonic expectations of the entire world are being shaped by the latest releases in gaming, film and music. If I hear something unique in a new game, for instance, I immediately ask myself “Could I create that if I needed to?” because chances are, that sound will be referenced by someone on a future project. If the answer to my question is even a “maybe”, I get to work trying to recreate the scene or individual sound effect. It’s not just a good exercise professionally, it’s the type of discovery and experimentation that keeps things interesting, challenging and fun.

DS: In your career thus far, what has been your favorite challenge?

JS: My favorite challenge has been developing and adapting my sound design to meet the needs of multiple game genres. So far I’ve worked in the genres of real-time strategy, city building, sports, fighting and first-person shooter. Each genre has brought with it a tremendous amount of discovery and a new set of audio challenges. The perspective shift from top down to first person, for example, has been a huge eye opener in terms of new avenues and constraints for audio.

One thing that I always enjoy is deconstructing the legacy of each genre and the expectations from core players of that genre. I like to identify those expectations early on to ensure I’m meeting, and hopefully exceeding, what fans have come to expect from the aesthetic and player feedback of each genre.

DS: What kind of things do you do to advocate for audio throughout the rest of the development team?

JS: I learned early on that without advocacy, audio can easily fall off the radar for the rest of the studio. It doesn’t help that the audio team is typically working behind closed doors. I’ve made a concerted effort to promote the audio team within each studio by reaching out to other departments and developing the mutually beneficial relationship that can, and should, exist between them and the audio team. The key here is that talking is not enough. I show artists and designers how audio can develop their ideas with prototypes rather than trying to explain it with words or on paper. And I’ve yet to meet an artist or designer who isn’t excited to see his or her work come alive after an audio pass. Many times those prototypes lead to iteration on the original artistic or design concept resulting in a stronger end result.

I’m also a strong promoter for audio being included in the concept and pre-production phases of a development cycle. Most modern game projects are too large for audio to be treated strictly as a post-production department. I feel that the most memorable and best sounding games are those where audio is a contributing voice from the start.

DS: What developments in game audio make you excited for the future of that industry?

JS: Two things: The availability of incredible middleware and the independent game development scene. With industrial-strength audio middleware freely available to professional sound designers and students alike, new discoveries are being made all the time in terms of implementation. It means that students are going to come into the game development field already trained up on the tools we use day to day. Middleware still comes with its own technical challenges, but for game audio professionals, I think the scales are now weighted much more heavily on the side of creativity.

The independent film scene has brought us some of the most experimental and creative audio in the field of cinema. I believe independent games are headed in that same direction. Just as independent film sound is informing the sound of major motion picture, I believe the sound of independent games will influence the work of game audio professionals everywhere.

DS: What are you currently working on?

JS: I’m working on Bioshock Infinite at Irrational Games.

DS: Do you have any advice for people interested in entering game audio as a profession?

JS: First and foremost: No reel, no deal. I’m consistently surprised when I meet people who want to get into the audio field, game-related or otherwise, and haven’t created a demo reel of any kind. You live and die by your latest work in this industry and a reel is required for even most internships. There are free audio/music apps available for download and there are plenty of places to host your content once it’s ready.

Next, and this might seem obvious, I advise aspiring game audio professionals to play games! Knowing both the pitfalls and the paths to success that have come before you is invaluable. More importantly, a sound designer well-versed and well-played in video games holds a common vocabulary that can be used to interact with all team members at a game studio.

And finally, I encourage aspiring game audio professionals to use the copious amounts of information and software available on the net to learn the tools of the trade. The same tools we use every day, both audio middleware and game engines, are available to anyone with an internet connection. Download them, start learning, and start experimenting!