Categories Menu

Posted by on Jun 13, 2012 | 1 comment

AudioGaming – An Interview

We’ve had some inspiring, insightful and practical thoughts about procedural audio in the past few interviews. We’ve not only looked at what can ideally be possible with it, but also what its limitations can be. For those of you who keep up with game audio news, AudioGaming must not be new to you. They are a few years old with the sole purpose of developing next generation audio tools in the realm of procedural audio. After spending these past few years researching and developing their infrastructure, they have just released AudioWeather – a procedural wind and rain solution available both as VSTs/AUs and within FMOD Studio.

The interview below is a transcription of a conversation I had with Amaury LaBurthe, the founder and CEO of AudioGaming. What was immediately apparent through our conversation was that they are a group of highly driven and talented people who want to empower sound designers with better workflows and tools, while being very realistic and open about what they do.

For those of you who are interested in trying out their VSTs/AUs, scroll down to the end. They were kind enough to give us all a discount code!

DS: Let’s start this off with a background of AudioGaming. What led you to form the company?

Amaury: We started the company in 2009, at that time I was still working for Sony R&D (Sony-ComputerScienceLAboratory, a small Sony lab in Paris dedicated to research), which was mainly DSP and interactive audio research. Before that I was working for Ubisoft and before that I was working for Sony in research so I had experience in both sides – research at Sony and creative/sound design when I switched to Ubisoft as I was audio lead on several games. At some point when I was working for Ubisoft I thought it was a shame that I needed tools that didn’t exist. Because of my background in R&D I knew some of it was possible. In 2009 we mainly thought about the project as I was still working for my former lab (Sony) through AudioGaming. We started full time at the end of 2009. We also won two national contests for innovation in France where we got grants in 2009 and 2010 that helped us a lot in starting the company.

We spent almost two years working on the technology and the R&D side. We finished with a pipeline that allows us to create efficient interactive audio tools. This led us to create our first plugin – AudioWeather. It is a real time synthesiser dedicated to the generatation of wind and rain sounds. There are no samples and you can control most of the parameters of the sound that is generated because it is a model. This is becoming available through FMOD (and as a VST), so it runs on PC, Mac, PS3 and the XBOX. Before that we created AudioGesture as a first large scale test of our pipeline on consoles. It generates sounds in real-time based on the movements made with a Wiimote.

DS: What is your approach? Do you follow the age old cyclic method of recording -> analysing -> synthesising -> listening?

Amaury: There is definitely a lot of recording, analysis, synthesising and as you said there is a circle where we always start from the recordings. But, we also start from the usage. What do you want to do and how do you want to interact with it? This is very important because you can either take the sampling approach – like sampling a piano where you would sample all the keys with different velocities and you end up with ten gigabytes of sounds and you have a great sounding thing or you can look at it from the interaction perspective and think about what you want to do with it. For the wind and rain we decided we would like something that is dynamic and interactive by using a simple slider – from no wind and rain to full wind and rain. Then we asked ourselves what kind of parameters we would like to control dynamically and what kind of parameters we would like to control statically.

Once we had a model of what we’d like to have, we switched on the analysis phase, where we recorded a lot of wind and analysed them. We also approached the French meteorology department and asked them if they had recordings of wind. They didn’t have audio recordings but what was good was that they had lots of data about the way it behaves. They gave us precise recordings of the evolution of the speed of the wind amongst other things. We analysed this statistical data and we modelled not only the sound but also the statistical evolution of the wind. How does it evolve over time? What is it that makes it sound like the wind and not the sea? The sea has a back and forth movement while the wind has a very precise statistical behaviour with various characteristics like gusts and squalls. We analysed all of that data to get behaviour models that would help differentiate the behaviour between the sea and the wind. When you are dealing with interactive behaviour its not only the sound itself but how it behaves over time.

DS: As it has been discussed previously, with procedural techniques it seems like you have limitless possibilities and you have to setup limitations for yourself to understand what you really want.

Amaury: Its a bit like the ocean of sound when the first synthesisers appeared. You have so many possibilities that it’s a bit scary (laughs). You can do anything and in the end you do nothing. It is like Max/MSP or Pure Data, if you don’t set any goal and very clear limitations you probably won’t end up anywhere. That is why we usually start from the sound, behaviour and interaction. That gives a very precise idea of what we want to achieve. If you don’t do that it is very hard and we have learnt that the hard way. With AudioGesture for example, we had no clear outcome in mind so we spent a lot of time working on it but it did not end up as something very creative. We had a lot of different models but they were not very easy to use. A part of the difficulty with procedural audio is that because a lot of people don’t know about it, they don’t project any usage and you have to find the usage for them. But, you might be wrong. You have to prove it is useful in scenarios that people don’t have in mind. It is tricky. This is why we went from broad things like gesture to more focussed things like wind and rain and now we are even more focussed and working on the footstep machine where the usage is very very clear and obvious.



DS: A lot of questions have built up about procedural audio over time – can it sound good? Will it sound good? Does it take up too much CPU? Have you faced any of these questions?

Amaury: Not really! (laughs) But, its very hard. We are being judged in comparison with images right now. Nobody is saying that image synthesis quality is bad and we are going to use pictures or videos instead. Nobody asks these questions. When the first games started to use image synthesis the quality was poor. But nobody ever questioned that. On the other hand, when audio samples became useable in games and when CD quality audio arrived with the PlayStation, there was no reason for people to care about synthesis anymore. So it has been hard for audio synthesis to compete with CD audio quality when we should have evolved just like images. It would have sounded crap in the beginning and then would have sounded better and better. People would then never question procedural audio, it would be natural. But the process did not happen. And now people question it, which is very valid.

We have tools like FMOD and Wwise or internal engines that are becoming very good at making things that are static by nature very dynamic. We have established workflows with a very good amount of control over the final audio experience you propose. The tools and the pipeline exists so why use procedural audio? There are several arguments that push for procedural audio. If you have big open worlds it is a nightmare to sound everything. When you have gestures it is difficult to sound them in real time. But, do you really want to sound gestures in real time? Sounding animations on big games is a big thing. What if you could sound anything that is moving almost automatically but not lose the sound design control? Just that, part of the hard work will be done automatically and you will control the generation of the sound and its adaptation to the movements. I think the good way to push it now is not in terms of opposition but in terms of complementarity. It can bring new things to existing pipelines and it can save time. But, it won’t replace several parts of the work. If you take the example of recording, even with procedural audio you always need to record things, either to analyse them or to use them directly in the synthesis because some synthesis techniques use samples. It’s just new tools that break some of the limitations. I agree with Nicolas Fournel when he says It’s not meant to replace anything or break what exists already. I think it’s the correct way to push it. But it’s true, it’s quite hard and people expect a perfect result right from the start.

If you think about music, at some point we had sound banks made of real waves and used through midi partitions. It’s crazy the flexibility such a pipeline gives. You can write hours of music in midi with many layering possibilities, on the fly alteration of the partition, slowing or speeding up music with no CPU cost or bad sound quality. Your whole set of music with its variations, its special events or alterations, etc… would be stored in a few kilobytes. You can interrupt the music very gracefully because you know the intrinsic property of the composition, etc… But rather than that, because it was sounding more poor than a real recording at that time we went for the full audio route. Which is insanely harder to make dynamic in real-time. But every game is doing image rendering using textures (sound bank?) with real-time processing. And now companies like Allegorithmic propose procedural textures which seems very natural. Hopefully the same can now happen for audio. We’re only 20 or 30 years late.

YouTube Preview Image

DS: Of course. It is also not something to replace everything we know about sound.

Amaury: Our goal is really to create a community and catalyse different approaches where interactive/real-time audio is at the heart of it. You can get away from the time line and think more of the interaction and become more of a crafter where your experiences with how a sound behaves and how you control it makes it evolve and fit with what is happening on the screen rather than browsing through thousands of files and finding one that fits. It can’t replace people. It just gives us more ways to create and manipulate sound

Apart from that, As Nicolas Fournel says in a very cool and precise interview on DS, we really need to think about the business model and link that with what sound designers need and want. If we take the example of AudioWeather, the original intent is really not to provide a closed model with only a few exposed parameters. We don’t “own” the truth of what wind is and how it should sound. But we’re not in the position right now where we can create an ultimate sound design tool, easy to manipulate as a simple VST but as powerful and limitless as Pd. We have to start somewhere. What we’ve tried with AudioWeather and AudioWind/AudioRain is to decompose the model in to separate layers and give access to meaningful parameters. This is adding a layer of difficulty because you need to map between synthesis parameters and low-level synthesis parameters, but we hope that can somehow circumvent the limitations that can arise from having a model pre-determined, as the variety of sounds you can get is pretty insane. We’ve added a randomize button by the way which sets random numbers on all the controls. This is the most fun tool I’ve been using recently! It’s crazy to browse randomly through the “ocean of sounds” it can create and I had a small Schaeffer-ien experience for a few minutes.

The good thing is that we’ll hopefully be able to build from that and propose tools that are more and more open. And we would be tremendously happy to get all sorts of feedback to help create tools that are more linked to everyday creative needs and can help get rid of some limitations.

DS: Introducing such solutions within existing tools like FMOD and even the release of VSTs seems like a good method to break down barriers. If it exists within your every day tools, you might just try it out.

Amaury: Yeah, thats what we hope. The guys at FMOD are really great. They have been very supportive and liked the idea. It is really great that we are able to work with them. Just like you said, if something exists from the start you will want to test it. If its behaving in a cool way, why not use it? We think its a good first approach. You can use it directly in the game and see if it fits or not. Our implementation in Fmod studio is rather limited for now, but hopefully we’ll release an updated version where you’ll have access to a lot of control parameters. Also in France we are participating in the video game master programme at a French university where we explain new ways of using and producing audio so that people will broaden their view on interactive audio and they will hopefully over time begin to consider this as another tool that can do things that other tools can’t to and conversely.

DS: Going back to your tools and workflow, do you still use tools like Pd or Max/MSP to prototype ideas or do you have complete control over the tools you use?

Amaury: Both actually. We use Pd a lot. It’s there, it’s working and it’s fast. You can put ideas together quickly. It’s a good tool in that perspective. We also spent quite some time building our own infrastructure, where it is a lot less friendly (unlike Pd), it’s more of a coding infrastructure where we’ve got optimised tools and we can take in to account the specificities of the platforms and everything. We have the framework to be efficient on the different hardware that we want to support. We need to have both because we need to be able to prototype and test ideas but then when it comes to the hard part you need to be able to control every bit and every sample. We need to have our own infrastructure.

DS: The VSTs are due to release soon. What else are you working on?

Amaury: The AudioWind and AudioRain VSTs should be released by the end of this month. We are finishing the beta so it should be there soon. We are writing manuals and debugging a few things. [Ed: This was at the time of the interview. The VSTs were released on 12 June] It is also a nice compliment to the video game version because that way people can try it in their DAW and if they find some usefulness in that environment it can become more natural to have that in game. The in-game version comes with the additional benefit of being able to control from a zero to hundred percent wind or rain. Whereas in the VST you don’t have the same interaction – you can control all parameters but to get from a zero to hundred percent you will need to automate some of the things.

I think it’s natural to have both options. You won’t be using them in the same context but if people are really used to having a VST they can use, then it is easier to think about its possibilities in a game. It is more natural this way. Other than us, most of the companies entering the video game market are people who are coming from the post production world – McDSP, iZotope.. all of them are big players in the post production world but they are new comers in the video game industry. Whereas, we are coming from the video game industry so it was natural for us to start there.

We also are working a lot on the footsteps engine. We’ve just finished a coproduction called Chicken Doom with a French studio (called DogBox). It’s a smaller game for mobile and tablets. We are using Unity to implement the procedural content and its working really well.

DS: Anything interesting to share about the future, or is all top secret?

Amaury: [laughs] We don’t have too many secrets. We are finishing the VSTs, we have just finished Chicken Doom and we are going to work on a game called Holy Shield, again by DogBox (crazily talented guys giving real importance to sound) where we will be implementing procedural audio with Unity. We are working a lot on the footstep machine. Apart from that we are still working on AudioGesture with the Kinect and Playstation Move and different devices. We also have several prototypes, but they are only prototypes and there is not much to show right now. We also have a musical interaction game prototype which we’d like to push, but it’s not something we can do on our own. You can control the composition of the music depending on how well you play the game, maybe this can emerge at some point (anybody open for a co-production with a kickstarter?? ;) . That’s pretty much it.

DS: Any plans of opening up such solutions to mobile devices?

Amaury: We’ve done an integration of AudioWeather in Unity. It’s working and we’ve got good performances, so it makes it available to most mobile platforms, but, because it is a .dll and not directly in C# it is not available in native clients. It can be used on all the other platforms though. We will probably release of version of AudioWeather in Unity’s asset store. Our pipeline enables that so it’s more a matter of needs for now. What we hope is that mobile games are going to evolve and we will eventually find more AAA games on mobile platforms. It’s going to happen and the needs are going to evolve so it probably will be a natural evolution for us too. This is also why we are going to work on Holy Shield because the goal is to have a AAA like production for mobile & tablets and we will be able to showcase the procedural stuff you can do in Unity.

You can watch videos and download trials/plugins here.
If you access the store, you can use this code to get both plugins at the price of one: DSEXTRABUNDLEPRICE

Read More

Posted by on Jun 4, 2012 | 4 comments

Procedural Audio: An Interview with Nicolas Fournel

Nicolas Fournel has had a long career with game audio and has been at the forefront in the research and development of audio tools. He maintains, a repository of procedural audio information and the recently created the Procedural Audio Interest Group. He was more than happy to add his voice to the procedural audio series based on his varied experiences in developing such tools in the ‘field’.

DS: Your work has taken you all over the world with companies like EA, Konami and Sony. Tell us about it.

NF: I guess the professional “game audio” aspect of my career really started in the early 90’s. I was developing commercial audio software on Amiga and PC and later created a company in Paris which was specialized in audio synthesis. One of the products was Virtual Waves, a software modular synthesizer which included many types of synthesis, processing and analysis modules. That was long before Reaktor (or Generator as it was first called) and this kind of software. Instead of coming with instrument sounds, Virtual Waves shipped with a lot of patches generating sound effects, so among our clients were a lot of French game studios and supporting them was really interesting. In a sense, it was already “procedural audio” for games. Besides that, at the time I was also building alternative controllers like the Semekrys, which was an instrument built from 4 large touch-screens in glass, mounted on a big plexiglass structure.

Anyway, in 2000 there was an opportunity to work on the audio of the forthcoming GameCube console and thus I moved to San Francisco to join Factor 5. They were located on the same road than the Skywalker Ranch and worked closely with Lucas Arts. They were famous for Rogue Squadron on the N64 (and before that, Turrican etc…) but they were also very involved in the audio system of the future GameCube. I worked on MusyX (which was the audio tool / run-time shipped by Nintendo with the GameCube devkits). I also designed a couple of resampling filters for the ROM of the DSP of the GameCube itself as well as some other weird things like a very crude HRTF system. Because there was so little space left in the ROM it was pretty rough and never got used I think. At the end, all we got left was something like 10 bytes in the ROM and we just put our initials in there…

Read More

Posted by on Apr 7, 2012 | 0 comments

RjDj – Crafting ‘Dimensions’

Robert Thomas and Joe White from RjDj, known for crafting interactive sound-musical worlds on iOS devices, were kind enough to spend an afternoon sharing their thoughts on interactive soundscapes and music, technical and creative limitations, Pure Data, procedural techniques and their latest app Dimensions. You most definitely would have heard of their Inception App‘, released over a year ago.

DS: Tell us about RjDj

Robert: RjDj was formed in 2008 but was actually initially conceptualised as far back as the 1990s. The first app that was released was very unusual for that time because it was doing things with sound from the microphone and using the accelerometer in ways that was not really supported in the SDK. Since then we have done a wide range of different apps, which are about exploring interactive music or reactive music or augmented music and that has gone from RjDj to Rj Voyager through working on projects like the Inception app to Dimensions, the latest one, which is veering more in to the world of games. RjDj was formed by Michael Breidenbruecker, who is one of the cofounders of and he is the CEO and driving force behind the company. My role is more in terms of music composition and looking after the sonic side of things, and Joe is also working in that area and a bit more oriented toward the reactive infrastructure of how we make music happen in relation to events. He also works a lot more on sound design that I do.

DS: Most of what you do is in Pure Data (Pd). Do you begin the composition and design process in a regular DAW and then migrate to Pd?

Robert: We use a conventional DAW of some kind and export bits that get reassembled in real time based on whatever rules are appropriate for that situation. Much of Dimensions was done in Cubase and some of it was done in Logic and then exported as tracks which are some times stacked one on top of each other and fade in and out based on different user interaction. There are individual hits that get algorithmically put back together on the fly – drum patterns and things like that, and layers of reactively triggered synthesis.

Joe: In our earlier projects we were trying a lot more to reconstruct all the music inside Pd but we obviously had performance hits and we had to optimise it all. In Dimensions, we created all the assets in the DAW and mixed it as well as we could so that we could concentrate on how it played back in Pd. How we construct the assets in Pd is just as important, as it needs to be simple while achieving the same sort of experience.

Robert: In 2009 we did a project that was extremely ambitious. Almost everything in there was algorithmically put together in the mix. It was all stemmed material or individual tracks that were put back together in Markov states and various trees of possibility and based on the user behaviour and all kinds of things. We found it very difficult to achieve a very good level of quality if we were that ambitious with the structure. Also, a lot of the end listeners couldn’t understand the level of complexity we were producing in terms of the structure and the variants of the music. For them it was just music. Now we concentrate on making one very obvious reactive thing over more linear music.

Read More

Posted by on Jan 18, 2012 | 9 comments

Procedural Audio: Interview with Andy Farnell

[Continuing with the procedural audio series…]

Andy Farnell – a familiar name in computer audio – is a computer scientist, sound designer, author and a pioneer in the field of procedural audio. He is a visiting professor at several European Universities and a consultant to game and audio technology companies. His book, Designing Sound‘, is a bible for procedural sound and should be on your bookshelf, if it isn’t already!

He was very kind to find time in his busy schedule when I visited London, and we talked about what procedural audio is, where it stands now and what it can be in the future. This article is a transcription of our conversation, which he was again very kind to edit along with me. It was no easy task because there was so much good content!

Thank you Andy!

DS: Where does Procedural Audio stand now? Would you say it is comparable to where CGI was in the 70s/80s, when computers weren’t powerful enough?

Andy: That is a central mythology – that the computers aren’t powerful enough to do it. This is often brought out as a straw man argument against Procedural Audio by skeptics. One of the things I did with my 2005 demo was to make all of the sounds (they weren’t very high in quality) that you would need for a first person shooter game – fire, water, wind, rain, some animals, some footsteps, some guns, some vehicles. This was 2005 and I had them all running on a 533 MHz processor generating a realistic-ish sort of soundscape to prove that if you had 1GHz processor and if you used half of it for the graphics then it would be quite possible to synthesise all the sounds using the remainder. Six years after doing that people would still come to me with this straw man argument, they would say, “You know Andy, we love this Procedural Audio stuff but there’s just not enough CPU available”. But we now have two to the five times more CPU than when I did my 2005 proof-of-concept demo. So, what’s behind that? Why are they saying that? It’s not true. What happens is the internal politics of resources. The requirements always expand to fit the resources available. The game worlds get bigger and bigger and the graphics get more and more demanding. The audio team will always have the least amount of CPU allocated to them as an afterthought, because in the current structural model of production sound is “post production”, and no body wants to commit to giving audio that much CPU bandwidth. I feel that is the real reason behind the argument. You often get these straw man arguments that enter in to a culture and just get recycled. People know that there is an argument and it comes to their tongue very quickly and they say “Yes we could do it but there is not enough CPU”. With the left over CPU on a modern games console I could provide you great procedural sound.  On an eight core architecture, we would need one or two CPU cores to give procedural sound. Even more interestingly is what happens when we run models in GPU, and many Procedural Audio models are inherently parallelisable. So, yes, Procedural Audio is somewhere in that era before the Tron movie, or before the Pixar CGI revolution, its possible, but not yet seen as viable, perhaps the shift is too painful for big companies to make.

Read More

Posted by on Jan 16, 2012 | 0 comments

The Sound of ‘Pugs Luv Beats’

[This is a first of a series of interviews/articles on procedural/generative sound]

Pugs Luv Beats is a hilarious music composition game for iOS devices developed by Edinburgh based studio Lucky Frame. It’s about guiding pugs (in costumes) around a galaxy of worlds, whilst creating an endless variety of music. It sounds fantastic and runs on a generative sound/music engine developed in Pure Data.

Lucky Frame is Yann Seznec (artist, musician and sound designer), Jonathan Brodsky (artist, designer, musician, coder) and Sean McIlroy (illustrator and print maker). Jon and Yann were kind enough to make some time right after the release of the game to talk about the sounds and technology behind Pugs Luv Beats.

DS: How did Pugs Luv Beats come together?

Yann: Jon and I made an app called Mujik a couple of years ago. A lot of people downloaded it and there were a lot of good reviews. It was basically a different approach to music on a mobile interface. After playing around with that for a while we started thinking about how much further we could take the idea and Jon started getting into the idea of making games. So, we started thinking about how we could really bridge that gap between music and games. What if you could use a game interface to create music rather than to play music that is already there? That was the starting point. Jon made a demo which we dubbed ‘Space Hero’. The idea was that you were controlling a little ship that was shooting enemies. As the enemies came on screen they made a sound and as you destroyed them they made a sound, with the twist being you could edit how the enemies came after you so it was like a piano roll hybrid drum sequencer. It was more of a proof-of-concept than anything else. We took that to Channel 4 and to make a very long story short they ended up eventually telling us that they liked the idea and that they wanted to invest in it. Interestingly they told us, ‘We want to invest in the concept but don’t make that game’ [laughs]. So we started making various different prototypes for what became Pugs Luv Beats.

Read More