Audio Implementation Greats #10: Made for the Metronome

[Written by Damian Kastbauer for Designing Sound]

If you talk to anyone in game audio today about successful tempo synced synergy between music and sound effects it wont take long for your discussion to end up at REZ and the work of  Tetsuya Mizuguchi. The quintessential poster boy for synesthesia in video games and a stunning example of overt spontaneous interactive music creation.

But imagine for a moment stripping away the throbbing electronica pulse and replacing it with an organic instrument-based soundtrack created by one of the foremost prodigy of curiously inspired noise making bass thumpers, with sound effects locked to a groove oriented metronome, and you’ve got makings of a monster.

“Whereas many games today occupy free-formed soundtracks that respond entirely to the player, Mushroom Men is recorded to a beat. “You have sparks sparking in time to the music, and there are moments when the background music backs out and you hear the cricket cricking on beat,” says Jimi Barker, another sound designer with Gl33k.

“Piersall continues, “You want to make it seem like the world plays to a beat.””

–Austin Chronical Article: Making Mushrooms Dance

[youtube]http://www.youtube.com/watch?v=LqqFKIroxUU[/youtube]

This brief overview sparked a fascination in me regarding the potential for experimentation in the realm of reactive and interactive music. I was lucky enough to end up down in Austin, TX for the 2009 Game Developers Conference and had a chance to connect to the fine fellow’s at Gl33k, the audio house behind the sound for Mushroom Men, The Maw, Comic Jumper, Ghostbusters (Wii), and more recently Epic Mickey. Not only did they unlock the secret door to the Austin underground, they also proved to be unmatchable hosts and a great group of guys with their sights firmly set on revolutionizing outsourced audio.

I had a chance to chat recently with Matt Piersall, Pres and New School Beast Handler at Gl33k, about everything from Drinks at the Death Star to syncopated bee buzzing. The full audio proceedings of this meeting of the minds can be found at GameAudioPodcast.com and a transcription has been provided by esteemed game audio professional Michael Taylor.

[DK]: Damian Kastbauer
[MP]: Matt Piersall

DK: Mushroom Men, man, that was a beautiful thing with regards to the way that the music came together, and that was the first title for Red Fly. Can you talk a little about how you guys got involved with that project, and what the first steps ended up being?

MP: Yeah, basically the way that we got involved was, my very first game job was at Terminal Reality as a Contractor. A guy there that (has since) left, his name was Jeff Mills, he was a Level Designer. He left Terminal Reality. They did Ghostbusters, and have tech they built called the Infernal Engine,  which we still use on some projects, and is pretty cool. What ended up happening was that I ran into him, like four or five years later at GDC. I had a drink with him at, you know that Death Star bar.

DK: Yeah! At the top of the Marriott?

MP: Yeah, we had a drink there, and I gave him a USB key demo, and was like, I really wanna…I saw the project, and I thought it was gonna be really cool. It just seemed like something I always wanted to do, in terms of establishing a new IP. Everything I had worked on up to that had had been a sequel;  a two, or a three; something was wasn’t as completely creative. That’s how we initially got involved.

I love sound design, and it had always been a goal of mine to work really closely with a developer. Basically what Gl33k does now is support developers with outsourced implementation, sound design, and audio direction. I sent in a bid to Red Fly, and they were like “Oh, can we get some music for this amount of money, because that is basically all of our budget for sound” and I said: “Yeah, that sounds really cool. I’d love to make some music” They seemed like a young enough studio, open to new ideas so I was like “ Yeah, lets make some music”

So, we ended up doing a couple of tracks, and there wasn’t a lot of movement at first, and then they came down to Dallas, where we were originally located. When they came down to Dallas, we got margaritas, and we started discussing ideas. I started talking about how I had always wanted to do a musical video game. Not as in playing music, but as in, literally a musical, people are singing and stuff like that.

DK: Like ‘Westside Story: The Videogame’?

MP: Exactly! [laughs] So, we had this long talk, and everybody got kinda tanked, and they agreed at that meeting the we should have this Metronome system. After that meeting, months went by, things were very slow moving. They didn’t have a way to get us builds in Perforce, so they had to package them up (via FTP). This is really early y’know! We were downloading 4-5Gb of data, and not understanding how things worked at all, and really having very, very little communication with the team.

We came to (their studio up in) Austin one time, and I did a mock-up: I captured some footage from one of the builds that I had, and I made some music, and I made an environment noise in Ableton, and the environment noise was rhythmic. So I did all this all this rhythmic stuff, and then I comped a video together, and I show them and they loved it. They were like “Alright, cool. We’re definitely gonna move on this, and make this metronome thing a reality.”

In that same meeting, they said; “You guys should move to Austin, what are you doing in Dallas?” I was like; “You know what? That’s a really good question!” At the time, all of our clients except one were either in Austin, or LA . We had five months of work, and I was like “Let’s move to Austin!”. In retrospect, that was really crazy. I’m trying to be more, I guess careful, so I’m trying to plan six moths out,  if I don’t know what’s going on in six months, I start to freak out!

DK: Awesome. And that kind of positioned you closer to Red Fly, your studio is right next to their development house now, and kinda that’s how you stepped into that pond up in Austin?

MP: Yes, actually Austin has a lot of developers, maybe around fifty or sixty? And its helped us out a lot. We always work with Retro Studios, we work with Disney… they’re all in town. I think where we are there are four developers on the same street.

DK: Not to jump the shark on this, but as an outsourced audio company, you have this autonomy and location (theoretically) becomes less of an issue. But I think that, maybe where your driving with the Mushroom Men story, the move to Austin, and your proximity to Red Fly, it must have really helped with development.

MP: Well, at the time, just to be completely real about our experience level at that time, we had mainly delivered assets. We delivered them kinda over the fence, like a lot of sound houses do it. The proximity thing was completely necessary. I believe that you can be remote and do a great job, because we’ve had to do that before, but what you don’t have at a distance is that you don’t get the same camaraderie with the team. You have to have a lot of face-time, even if you live thousands of miles away. In this case, I loved Austin, and always wanted to live here, it just has a cool life to it, and so many developers. The game community and the development communities here are really active. I don’t think we could have accomplished what we did had we not moved.

DK: Sure. It sounds like you could pass builds over the wall, you could step next door and talk with the guy who was doing the wiring under the hood, because you were not using any middleware for Mushroom Men, it was all proprietary Red Fly developed action.

MP: Yeah, it was the old Infernal Engine. It’s been updated a lot since and is really great. The old engine, from a level design standpoint was pretty cool, but from an audio standpoint, we didn’t even have things like pitch randomisation. It was very, very bare bones. But when we got here, it was great. We had Perforce access, and everything happened so much faster. We could attend meetings, and meet some of the guys doing animations, camera, effects.. We just sort of became a part of their team.

DK: Right on. From that point, once you had moved across the state, how did things come together technically? Over drinks you had decided this was going to be a metronome based system, that had repercussions that cascaded throughout the entire game design, because now you are not only syncing audio, but you are also syncing visuals, animation and VFX.

MP: The reality is that I wished that we had used it in a more extreme way. We got the functionality, and from the audio side we used it all over the place, and we did a lot of experimentation with that. From the design side, a lot of people used it, but it was so subtle that you barely notice it. In retrospect, I would have said we should really play it up with camera cuts, and UI elements coming in on beat, which those do actually, but its super subtle. Animation on beat was a little more difficult, because they had to build the animations to a beat. Particle effects were fairly easy, because we had emitters that would fire them off on beat. Essentially the whole system was kind of like a sequencer. Kind of like an 808, it would listen and fire off events.

I look at Mushroom Men as a great learning experience. We worked on it for about a year, but we did most of the work after beta. We had nine months of preproduction, then we worked for three months! [laughs]

DK: That sounds about right! Was there quantisation used as those events were fired off? I mean, did they have to do a lot of juggling of those events in order to line them up? That’s got to be part of the system, right?

MP: Yeah, its totally part of the system. The way the system worked was that you would have a stream. The stream was your backing track, that was the main music that made everything fall on beat. So we started that paused, and then we waited and we kicked out an event to everything. Everything would update every second on a tempo of 120 BPM to recheck that is was still on beat. It was really cool, and I wished we had used it even more. I ended up getting in a little trouble, I took too much liberty and snuck in and synced some things that weren’t originally synced. You just kind of had to do that. It ended up making the end product cooler for sure.

DK: You bet! Can you give us some examples of things for people who may be picking up the game for the first time after reading this article? What kinds of things should they be listening for that you think exemplify the system and its capabilities?

MP: Well, actually there are about three or four things that I’m into, and I hate everything else. [laughter] Ok, so there’s these bees in one of the levels, and we made a track that was the theme for the bees that was like [sings/buzzes theme]. We tied this into the bees particle effect so that its in sync with the…[buzzzes like a raver bee] So yeah, the bees are buzzing on beat, which I thought was really fun. Actually, every ambience in the whole game is rhythmic. I took everything into Ableton and made all looping, streamed ambience rhythmic somehow, even if its not real obvious. You know how in Ableton you can do these ‘stutter’ edits? It’s really subtle, but we did a lot of that with volume ducking for the streamed ambience., and we kicked off wood creaks and crickets and all the insects you hear which are making a beat, and every single localised and spatialized emitter based ambient sounds are on beat too. The only sounds in the game that aren’t on beat are the enemy fighting sounds, because they didn’t want to change their game design too much to support this Metronome thing, because that was not what the game was originally pitched as.

DK: Totally. So is this a good point to segway into talking about how you roped in Les Claypool to lay down some magic on this? This is maybe a side topic to the actual Metronome system, but it plays a huge role, and I feel like its one of the few crossover examples of popular music contributing to games in a very unique way.

MP: Yeah, totally. The way Les Claypool got involved was through the guys at Red Fly, who are huge Claypool fans. They got some cash together and approached Les and said “Hey, do you want to do music for this video game?” I don’t think he was that interested at first, as I don’t think he digs games that much, but his son loves games, and convinced him to do it. Actually, his son, who is like seven or eight, actually plays on the Mushroom Men soundtrack. I think his daughter does too, I think they play percussion on it, which is really cool. But basically he ended up getting involved, and I met him, his wife and his kids and we talked about what we wanted. I didn’t really feel like I could give him much direction, but I said “Hey you’re gonna do what you do, but if you could just write everything at 120BPM that would really help me out.” and he said “Sure, no problem”. In addition to that, I also requested stems, because you have to have that. We had written a bunch of themes at Gl33k, and we had done this one thing that was really, really fun, and I would love to do this on a bigger scale. We went to this small college town in Texas called Denton, and we recorded with this band called Midlake. So we just holed up in their studio for three days, and just recorded a ton of material. And so we went and remixed some stuff and got some tracks out of that material. Then we got Les’ stuff, and we remixed that and mixed it with what we already had and oddly enough it all kind of worked.

DK: That’s such a great story because I think what you’re talking about is angling toward the way a lot of game music is coming together these days, with regard to having composers composing music, and then having to rip it apart for the various systems and remixing it to achieve the different intensity levels for instance.

MP: Absolutely. Its cool too, because I could never do what Les Claypool does, and I couldn’t do what the Midlake guys have great guitar players, a great drummer and a great keyboard player, and they added some little bits in there. Basically what I ended up being was a remix artist, which is more my background; electronic music stuff. Basically Mushroom Men is a huge remix album, with a bunch of random sources.

DK: Remix and Orchestrator, while your designing and interactive orchestrator of sorts, which the Metronome system is, you have to be the maestro of that system and build content to support that. I think that is a great concept and a great analogy for the way a lot of game music comes together these days

MP: Yeah, it’s really cool, and is kind of exciting to me. I’ve been guilty of it in some of the games I’ve been involved in – the wallpaper music. Sometimes you can’t get away from it, because time is always an issue. And if you don’t plan for it,   or set out to say you’re gonna do this crazy interactive music thing, then you’re not going to accomplish it. It’s not one of those things where during alpha you’re like “Oh by the way, we need a intensive dynamic music system.” I love Wwise so much, because you can kind of fake some of that if you get in late in the game. But you still need to sit down with your programmers and the tech people and get a cool system in there earlier or it’s not going to happen.

DK: Right. It’s the two-way street of communication. In order to be interactive music, it has to interact with the game. If the game isn’t sending it information to react to or interact with, then you can’t truly have interactive music.

MP: Exactly

DK: I hear you. You have to plan for it. It has to be part of the design, and I think that when you do approach it holistically like that as part of the design, then you can better communicate that to the composer and take it to the next level with through remixing to make sure it hits all the right spots.

MP: Definitely.  One of the biggest things that I found remixing did for us was that it just gave us so much content, and that’s one of the things that makes it work. It was easy to make new content, a new track would take only a couple of hours rather than produce a minute every day, or two days.

DK: Do you ever see re-visiting the Metronome, and the idea of beat-synced sound? Is there a place for it?

MP: There’s absolutely a place for it. I really wanted to explore this for Epic Mickey, but I got involved a little to late to really push forward with that. Hopefully, maybe on a future project we could maybe do something along those lines.

I think the place for it is that it’s not just for cartoon stuff, or weird, psychedelic worlds. I think you can use it in a lot of ways. You could use to enhance any moment. The gamer doesn’t necessarily always want everything on beat, but even having something on rhythm in a super realistic, really intense first-person-shooter could be really effective if it’s used right.

Exploring new systems and concepts like these is one of my passions, and keeps me interested in this business. The reason that I got into game audio was that I felt that it was a way to let people experience this creative outlet that you have, but you don’t have to be there. And so, the more creative and interesting the system, and I guess I’d call it ‘audio-driven design’  where its not necessarily that the audio drives the entire design, it’s just that the audio provides a bigger role than just a supporting one, like supporting player feedback. There should be bi-directional communication; the audio should communicate the the game, and the game should communicate to the music. Right now, a lot of games are starting to do it, and the quality bar of that has risen so much.

DK: Right, and at the end of the day, you still have to serve the game, and the style of game. But also having this two-way street of communication to allow for greater interactivity. I think we’re taking slow steps towards that.

MP: To be completely open about the Mushroom Men stuff, we just scratched the surface. We had the chance to do something kind of interesting, and it turned out to be pretty cool. I really loved working on it, and working on a system like that, which ended up being kind of a feature. We got to do some cool stuff with ‘Splosion Man too, so we’ve had a few chances to do some cool things. It’s not always an easier sell though, because sometimes they call audio teams only when things are on fire [laughter].

DK: [laughter] I saw that coming a million miles away! That’s when they call all right!

MP: I can understand though, you don’t want to waste a bunch of money on an audio team if you aren’t ready for them yet. But at the same time, I’d rather have a small team the whole way through, and then go from two people to twelve people at the end of a project.

DK: Beautiful thing! This has been a great talk about Mushroom Men. I feel like we covered a ton of space on it, and it’s great to hear about it all. It’s such a unique moment in time that you captured through both the implementation, the remix and the composition of Les Claypool’s work and that synergy that came together for Mushroom Men that was extremely unique in my opinion.

MP: We did some cool stuff, but if there’s anything I’d take from it, it’s that it’s a way that I like to work. It proves that you can outsource and bring in specialists to do certain things for a project. It basically proved that the way I like to work is viable from a critical standpoint.

DK: For sure. And creatively satisfying?

MP: Oh my gosh, yes! I miss that part of it. It was the most fun I’ve ever had on a project. It was a great time of life, we had just moved here, I lived across the street from the studio and there are 90 bars within a square mile! Great times!

Gameplay footage focusing on synced ambient sound @ 6:20 and with music @ 8:00:

[youtube]http://www.youtube.com/watch?v=PC0-6T9N1R8[/youtube]

Gameplay footage focusing on Bee combat:

[youtube]http://www.youtube.com/watch?v=h5Uoh_C5EQY[/youtube]

Stick around after the interview for a discussion on the late Austin GDC audio track, AES Game Audio, Footsteps experiences, and general banter between friends.

So this is the New Year…

The turning of the new year marks the one year anniversary of both the Audio Implementation Greats Series here at Designing Sound, as well as the Game Audio Podcast which I co-founded with Anton Woldhek around the same time. Over the course of the year we’ve covered topics that are relevant to professionals working in the industry such as: Footstep Design, Interactive Music, and Procedural Audio, and will be pressing into the new year with exciting new topics.

Beginning in 2011 we’ll be kicking out a fantastic discussion about Interactive and Dynamic Mixing with David Mollerstedt (DICE), Rob Bridgett (Radical Entertainment), Kristoffer Mellroth (Microsoft Game Studio). It’s out hope that these will continue to fill in the relevant gaps of information between other articles, dev diary’s, and interviews out there and bring some new insights into the art of interactive sound.

If you have any examples of outstanding audio in games, or interesting techniques being used, please drop a line!

Special thanks to Michael Taylor for transcription of the interview which can be found over at the Game Audio Podcast. “Michael Taylor is an audio designer, composer and proficient tea-boy and can be found at www.stomp224.co.uk “

ShroomMan & BossShroom Artwork © Aaron Armstrong