3 Deaf Mice: A sound-music game

David Sonnenschein, the author of Sound Design: The Expressive Power of Music, Voice, and Sound Effects in Cinema and founder of SoundDesignForPros.com, has a new project. 3 Deaf Mice is an audio-music discovery game (initially targeted for PCs and Macs) where the mice need help in designing their next hit song. It aims to enhance listening skills and creativity with sonic treasure hunts and puzzles. Thanks to Aurelien Folie for this interview with David about the game and the sounds that make it.

3 Deaf Mice just launched as a Kickstarter project. Show your support by checking it out!

What is 3 Deaf Mice?

3 Deaf Mice is a game aimed toward our sound design community and beyond, where you discover the root of listening and how we create through sound. We do it through the story of the 3 Deaf Mice, who have been playing loud music for too long and have lost their hearing, so the player has to help them make their next hit song. To do this, the player has to perfect their abilities to listen to the world in different ways.  By solving sonic treasure hunts and audio puzzles, they’ll be able to piece together a song called “Cheatin’ the Trap”, which consists of 10 verses, 10 tracks, and 10 game play levels. Once the player has made it all the way through the game, unlocking all 10 tracks and 10 verses, they’ll be able to create their own remix by manipulating all the elements of the song. We’ll also expand this into contests for the community to interact with one another.

What got you interested in designing such a game?

I’ve been teaching the fundamentals of sound design to filmmakers at university and pro training schools for many years, and I feel it’s time to bring this skill set and knowledge base to a larger public, especially kids. The way kids respond positively to new ideas is through games. If we tell kids they have to study something, they’ll usually run away or procrastinate, because they’ve already got enough to do at school. Instead, we want them to find pleasure through the game play, where they will not only learn aspects of mindfulness, concentration and discernment for their personal growth, but also acquire skills that will be handy in communication with their peers, especially using audiovisual media that is so accessible nowadays with mobile phones and Youtube.

Is there a storyline? How does it contribute to the game and learning?

The first part is embedded in the title: The Mice have gone deaf from listening and playing music too loud. It’s a bit of a cautionary tale to warn kids, without too much hammering, that they should protect their ears, which when damaged, often do not repair. So we want them to learn to enjoy sound and music without harming their health.

The story of the Mice is told in the lyrics of the song that will be discovered during the game play.  We learn their origins and specialties, with each having mastered different areas of the world of sound that they will share with the player.  Ratatat, the albino Punk drummer, is the speediest physics professor around, revealing vibrational mysteries from the atom to the universe. Shred, the Metal guitarist, takes you on a psychedelic acoustic journey of the brain and how it processes and converts noise into art.  Woofer, the Rasta bassist, expresses the spirit of musical invention, connecting with animal communication and the sonic story of humanity. We’ll be going on this adventure with them, how they got together, formed their band, became famous and had to confront the challenges of stardom.

Why do you think sound and music are so important to teach?

I think we as a species have obviously used sound to communicate since we evolved into creatures of expression, with our voice, or hitting sticks against rocks, or whatever. It became more sophisticated as language became the center of our communication skills, but in fact we have an extraordinary way of communicating that goes beyond the meaning of words, with intonation and melody, called prosody. The way we express ourselves with anger, fear, or humor, is actually much more than through just words. Besides our non-verbal voice, we can do this through environmental sounds or music. Music is one of the most potent forms of emotional communication, able to elicit triumph, tears or laughter in a film. The neurological aspect is important too, as sound travels first through our emotional brain area before it reaches the cognitive center where we understand the meaning of the words.  So we are susceptible to the emotional content of the sound much more so than the visual input, which goes directly from the eyes to the visual brain center. This is a general reason why it’s important to teach sound and music for communication and understanding of our own world.

How are you combining sound, music and game play?

This particular game has been designed so that we create music out of sound through game play. The final song will have ten tracks and ten verses.  There are ten game levels, each opening up one more track and verse. Each game level will have three stages to accomplish the opening of a new track and verse. In these progressive stages of the game, the player will be challenged and rewarded for successfully recognizing different qualities of sound. We’re referring to these as Source, Shape, and Meaning, which in more academic terms are known as causal, reduced, and semantic listening modes. The first stage takes us through the Source to help us identify where a sound is coming from and what is its origin. Then we reach a stage where we identify the Shape of the sound wave, defining its pitch, timbre, reverb and so on. And then into the DSP transformation, where we are actually manipulating the sound shape or waveform to create a specific result, which results in a piece of the sonic puzzle. Here we enter into the Meaning of the sound, as we begin putting together these puzzle pieces into the complete ten tracks and ten verses of the song mix.

What makes this approach to sound design unique?

I think what makes this somewhat pioneering is that it takes a technological and psychological knowledge base and turns it into a really exciting game with rewards that opens up the individual’s creativity towards sound and music. We’re taking the player through a whole process of identification, pattern recognition and construction of small audio building blocks forming larger and larger pieces into a final complete composition that ultimately will become available to restructure into remixes of the player’s own version of the song. I’ve been searching for this kind of audio play for quite awhile and not found anything quite like it.  My hope is that 3 Deaf Mice will generate lots of interest in this kind of exploration in the future from other music, sound and game people excited to develop this whole world of audio play.

Music and sound design, where is the border?

I think the border has to do with how we define music, which is really a sense of order and parameters of how the sound is being perceived. Most often we refer to musical elements of rhythm, tempo, melody, harmonic structure, and timbre of the individual instruments. When we say that’s a piano, or that’s a flute, we’re talking about timbre, as well as attack. The sound of a flute, as opposed to the sound of a trashcan, is distinguished partly by its simpler timbre and stronger fundamental tone than a trash can. But nowadays, that order is being blurred because we sample sounds and can play them on a keyboard, or we go to a performance like Stomp that uses all kind of objects like brooms or newspaper, everyday objects, to create rhythm and melody. So that’s an example where the border has been blurred, and that’s also what we are doing in this game.

Why rock as a genre?

I think that rock music has established itself in the popular psyche as something very accessible, and it has been around for 50, 60 years now, so it can appeal to all age levels. I find that even young people really like classic rock from the 60’s. So that’s one of the major reasons to choose this particular music style. The other reason is based on the story. Rock has traditionally been played at very loud volumes, and therefore has this related health issue about hearing loss. So I am using it as this pivot point to say that we can listen to rock music without burning out our ears.  Let’s listen to it in other ways than how loud it can be; let’s listen to the different elements of it, the drums, the guitars, the bass, the vocals. But let’s say we can extend that further, because this is not just straight up rock ‘n roll, as I’ve been influenced by the Beatles, Pink Floyd, beat box, hip hop, and all kinds of more current sounds. This is giving an opportunity to kind of update the classic rock genre by using sound effects that we will be discovering within the game play, incorporating those inside the music itself.

Tell us about the composition process

This particular game song began with the lyrics, telling the story of the Mice’s trajectory to rock fame, overindulgence and the consequences of playing super loud music. Once I had created the traditional structure of verses and refrain, I began to compose the music itself with guitar, bass and drums parts. But it is still in the process of being refined because while we develop this whole game, we will be discovering how sound effects fit into the music.  With each verse we’ll find the sound effects that will be embedded in the game play, the story and each the specific environment.  Then we will be selecting and shaping those sound effects to support the musical composition, making them into beats, stings, and melodies.

The other important aspect of composing this music is that it will be interactive in the more advanced stages of the game, so that the components will need to combine in various ways. For example, the vocals, instruments and sound effects may be rematched from one verse to another so that the actual story and feeling of a remixed verse will be shifted according to the player’s own creativity.

What are the sound design challenges for such a game?

One challenge in particular that I have already encountered is a level of the game where we will be taking a sound effect and processing that sound effect through several filters to create a phoneme of a part of the verse of the song. The phoneme is the most elemental piece of the human language, when put together creates syllables, and then words and phrases. So it’s been a challenge to find out how to manipulate the sounds so that they become recognizable phonemes, transforming the sound effect into speech. I’m analyzing artificial speech creation and methodologies, and then embedding that into the game play so that it’s completely accessible and fun for the player.

Another overall challenge is to make the game extremely engaging, and not just a learning process, so the player is motivated to keep going and rewarded each time they accomplish a level and a challenge. That’s a very different way of creating sound than for a passive medium like film, where the audience is just taken for a ride. This is much more about the player with their hands on several of the controls, driving the game at their own speed and with their own enthusiasm level.

Do you have any further thoughts about developing this game?

I am so excited about starting a new career after working for decades in film, sound and music. This seems like a perfect combination of challenges for me to create a form that can be enjoyed by so many people, and it’s that joy of sound that I can share with many more people than ever before because I don’t have to define the phrase “sound design”, I just have to say it’s a game and invite people to try it out. Sound design has been a very esoteric study and discipline that only a few of us in the professional world have been aware of and understand, knowing what it means to be a sound designer. Now we can make it just a great game and people can have fun and learn those things that I’ve been teaching for all these years without having to make it academic, without having to get a degree, or get a job. They can just have fun with it and then all those other things can come as benefits if they really get turned on.

Check out the 3 Deaf Mice kickstarter page.