Want to see something that’ll mess with your head?
[youtube]http://youtu.be/73LE1vKGfy4[/youtube]
Now, you may not have noticed anything all that strange watching the video, but mute the sound and watch it again. After that, close your eyes and listen to just the audio. Notice anything strange now? You’ve just witnessed one of the more interesting perceptual illusions, the McGurk effect.
Named after psychologist Harry McGurk, who first documented the phenomenon with his assistant John McDonald, the McGurk effect demonstrates the intrinsic link between aural and visual cues when perceiving speech. It was discovered during a study to understand how speech is perceived by infants in various stages of cognitive development. McGurk and McDonald found that if you combine a sound recording of one phoneme (for example, “ba”) with a video recording of someone mouthing another phoneme (“ga”), the brain combines the two into a completely different phoneme, “da.” Different combinations of spoken and mouthed phonemes can produce different results, and while the effect’s intensity can vary from language to language, further research has found the McGurk effect to be a fairly universal occurrence. In 1976, McGurk and McDonald released their findings in a paper titled “Hearing Lips and Seeing Voices.”
This audio-visual link has a lot of bearing on what we do as sound designers. As dialog is often the driving force of a narrative, understanding the links between visual cues and aural information is incredibly important. When you have characters with obscured faces, like those wearing masks or hoods, it can be very easy for the viewer to start having a hard time perceiving words that would otherwise be understood clearly; the visual cues can help the brain perceive the correct sounds, even if the aural aspects aren’t the clearest.
We can also use this effect when doing line replacements. Typically, these line changes will be done when a character’s back is turned, but sometimes it’s necessary to change words being spoken by someone in full view. While this isn’t always successful, understanding how certain phoneme sounds interact with certain phoneme movements can greatly increase the brain’s “acceptance” of this new dialog. Single words are generally more believable than entire lines; I’ve once swapped “at” for “to” in an onscreen line, and the change was imperceptible. This is also something to consider when it comes to foreign language dubbing; careful synchronization of dialog in another language helps the brain link the disparate visuals and sounds together, making it feel more natural even if the sync isn’t always perfect.
The best part? Even after one has been made aware of the McGurk effect, the phenomenon still has the same impact. Go ahead, check out that video one more time. Chances are you’re still going to hear it the same way as you did the first time.