• Home
  • About
  • Site Policies
  • Contact

Designing Sound

Art and technique of sound design

  • All Posts
  • Featured
  • News
  • Interviews
  • Reviews
  • Tutorials
  • Resources
    • VR Audio Resources
    • Independent SFX Libraries
    • Events Calendar
  • Series Archives
    • Featured Topics
    • Featured Sound Designers
    • Audio Implementation Greats
    • Exclusive Interviews
    • Behind the Art
    • Webinar/Discussion Group Recordings
    • Sunday Sound Thought
    • The Sound Design Challenge

“Hearing Lips and Seeing Voices” – The McGurk Effect

December 15, 2014 by Samuel Ejnes

Want to see something that’ll mess with your head?

[youtube]http://youtu.be/73LE1vKGfy4[/youtube]

Now, you may not have noticed anything all that strange watching the video, but mute the sound and watch it again. After that, close your eyes and listen to just the audio. Notice anything strange now? You’ve just witnessed one of the more interesting perceptual illusions, the McGurk effect.

Named after psychologist Harry McGurk, who first documented the phenomenon with his assistant John McDonald, the McGurk effect demonstrates the intrinsic link between aural and visual cues when perceiving speech. It was discovered during a study to understand how speech is perceived by infants in various stages of cognitive development. McGurk and McDonald found that if you combine a sound recording of one phoneme (for example, “ba”) with a video recording of someone mouthing another phoneme (“ga”), the brain combines the two into a completely different phoneme, “da.” Different combinations of spoken and mouthed phonemes can produce different results, and while the effect’s intensity can vary from language to language, further research has found the McGurk effect to be a fairly universal occurrence. In 1976, McGurk and McDonald released their findings in a paper titled “Hearing Lips and Seeing Voices.”

This audio-visual link has a lot of bearing on what we do as sound designers. As dialog is often the driving force of a narrative, understanding the links between visual cues and aural information is incredibly important. When you have characters with obscured faces, like those wearing masks or hoods, it can be very easy for the viewer to start having a hard time perceiving words that would otherwise be understood clearly; the visual cues can help the brain perceive the correct sounds, even if the aural aspects aren’t the clearest.

We can also use this effect when doing line replacements. Typically, these line changes will be done when a character’s back is turned, but sometimes it’s necessary to change words being spoken by someone in full view. While this isn’t always successful, understanding how certain phoneme sounds interact with certain phoneme movements can greatly increase the brain’s “acceptance” of this new dialog. Single words are generally more believable than entire lines; I’ve once swapped “at” for “to” in an onscreen line, and the change was imperceptible. This is also something to consider when it comes to foreign language dubbing; careful synchronization of dialog in another language helps the brain link the disparate visuals and sounds together, making it feel more natural even if the sync isn’t always perfect.

The best part? Even after one has been made aware of the McGurk effect, the phenomenon still has the same impact. Go ahead, check out that video one more time. Chances are you’re still going to hear it the same way as you did the first time.

Filed Under: featured Tagged With: article, audio, aural perception, congnition, dialog, mcgurk, perception, psychoacoustics, psychoacoustics 2014, sonic perception, sound, sound design, vocalization

Social Media

  • Twitter
  • YouTube

Posts By Month

#sounddesign on Twitter

#sounddesign

tunepocketTunePocket Music@tunepocket·
26 May 2020

Download royalty free gunshot sound effects, incl. one shot gun sounds, gun cocking sounds, gunshot with silencer, machine guns, and more #gunshot #sfx #sounddesign #videoediting #gamedev #royaltyfree https://t.co/bTTDQkveKF

Reply on Twitter 1265206307927162880Retweet on Twitter 1265206307927162880Like on Twitter 1265206307927162880Twitter 1265206307927162880
Reel2MReel2Media@Reel2M·
26 May 2020

At Reel2Media we are harnessing the power of smart speaker technology, to deliver cutting edge skills that solve problems and add value. To find out more, click here https://t.co/VAAw7vTW1V #sounddesign #smartspeaker #alexaskill #amazonalexa

Reply on Twitter 1265206269255847936Retweet on Twitter 1265206269255847936Like on Twitter 1265206269255847936Twitter 1265206269255847936
SoniKSoundLibSoniK Sound Library@SoniKSoundLib·
26 May 2020

Listen to #soundcloud preview of our next release! #np https://t.co/S3bxxrnqrh #audiopost #soundlibrary #soundediting #sounddesign #ambisonics #spatial #surround #ambience #roomtones

Reply on Twitter 1265204382779559936Retweet on Twitter 12652043827795599361Like on Twitter 12652043827795599361Twitter 1265204382779559936
asoundeffectA Sound Effect@asoundeffect·
26 May 2020

Want to know what the independent sound effects community has been up to? Hear the very latest SFX libraries here: https://t.co/OUBM2SqMoP #sounddesign #soundeffects #soundlibraries #indieSFX #gameaudio #filmsound #filmmaking #gamedev #indiedev #indiefilm

Reply on Twitter 1265199255838511105Retweet on Twitter 1265199255838511105Like on Twitter 1265199255838511105Twitter 1265199255838511105
OblikLinesOblik Lines@OblikLines·
26 May 2020

Working on new tracks with many instances of Misty Valley ! ⁠
Misty Valley is free for all NI Reaktor users⁠
You can get it here: https://t.co/bb53Gtkw5e

#reaktor6 #reaktor #nativeinstruments #synths #vst #sounddesign #virtualinstrument #musicproducer #obliklines #mistyvalley

Reply on Twitter 1265198647307907073Retweet on Twitter 1265198647307907073Like on Twitter 1265198647307907073Twitter 1265198647307907073
Load More...

Copyright Info

All content on Designing Sound is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

Copyright © 2023 · Magazine Pro Theme on Genesis Framework · WordPress · Log in