Helping visually impaired children through audio – An interview with Monica Gori from ABBI

Acknowledging a lack of solutions to help visually impaired children to apprehend their movements and surroundings, a team lead by IIT-researcher Monica Gori created the ABBI project. Built with young children in mind, this bracelet is generating audio based on body movement and spatial localisation and, therefore, helping them interacting with other and their surroundings.

Where did the idea for ABBI came from?

The idea of ABBI came from the necessity of developing new rehabilitative device for young children with visual impairment. In 2014 the number of blind children below the age of 15 years was estimated to be 19 million. Psychophysical evidence suggests that some audio skills in congenital blind individuals result enhanced. On the other hand, recent psychophysical works have pointed out that some other forms of auditory perception in visually impaired individuals result compromised. The creation of new technological devices to be used early in life is a must. However, despite the huge improvement of technological devices specifically designed for visually impaired users, many of these solutions are not suitable for young children and are not meant for rehabilitation purposes. They are too complex because they imply the need of learning a new language, following long training programs and integrating multiple sensory signals.

Starting from these premises, we developed ABBI (Audio Bracelet for Blind Interaction) that can be used by children with visual impairments from few years of age. ABBI is based on the idea of using the auditory modality to convey spatial information about the movement of the person’s main effectors/limbs (such as the wrists and feet). The movement of the visually-impaired person is therefore associated with an auditory feedback that provides spatial information related to the position of the body in the space (an information which is usually conveyed through vision). Research has shown that the association of a sound source with the movement of the person is a natural way to link action and perception, which is fundamental for the development of the sense of space. These sources of sounds allow the visually-impaired child or adult to associate body movements with the position in space of the limbs and, consequently, to build a representation of his/her body movement in the space in an intuitive and direct manner. In addition, sound sources placed on other persons provide a better sense of the events taking place in the environment and to improve social skills of these children and adults (Figure 1 on the right). By tackling the impact that blindness might cause on cognitive processes that contribute to the normal development of the sense of space, it is possible to alleviate problems that might derive from having an impaired sense of space, such as a lack of mobility, poor navigation skills, limited interaction with others.

We tested this device in 40 children from 3 to 17 years of age by performing a longitudinal three months rehabilitation training. During this period, half of the subjects (experimental group) performed the training with ABBI for a total of 65 hours while the other group (control group) performed the typical rehabilitation without the audio-motor training with ABBI. Results suggest that the use of the audio-motor training with ABBI helps rehabilitating the sense of space and motor skills and that the improvement is maintained after one year of rehabilitation training. These findings show that an audio-motor training can be helpful to compensate for the lack of visual experience with long lasting effects.

How many people are working on ABBI? What type of profession can be found in your team?

There are more than 25 scientists and technicians (list of names at the end of the document) working on ABBI. The team is composed by psychologists expert on psychophysics, biomedical, mechanical and electronic engineers, medical doctors, rehabilitators, experts of user studies and experts on software.

What type of technology is in use in ABBI (hardware, software, programming language, … )?

ABBI Main Functions:

ABBI has the following characteristics:

Audio-motor association. The core functionality of the ABBI device is to produce a sound that can be triggered and/or modulated by the motion of the user. For example, ABBI can playback a recorded sound files when a motion is detected, modulate a synthetic sound as a function of the user movement to give sounds to movement, etc.

Communication and control. Besides producing sounds, the ABBI device can store in a log the activity of the user, send notifications about the user motion and communicate with any recent smartphone and computer. All these functions can be configured to be tailored to the application and, possibly, to the preferences of the user.

Web access. Through a smartphone or a PC, remote access to the ABBI device might be possible from the Internet to ensure, for example, that the rehabilitation or training protocol is well followed.

Versatility. The ABBI circuit can be mounted in various configurations to accommodate the needs of the different applications; it is small and light enough to be used by a one-year old child and might also be mounted within objects (e.g. toys) to make them audible to the visually impaired population.

Simplicity and user friendliness. Simplicity was an important goal in the design of the ABBI system. Once it has been configured, the ABBI device can function independently, without any particular intervention from the user. Both the hardware and firmware are designed with the idea that ABBI would “wake up” when the user wears it, and would “fall asleep” when left alone. This simplicity makes it possible to use ABBI from the first year of life.

Hardware:

The ABBI device encompasses a custom-designed electronic circuit, a battery and micro-speaker and/or headset connector (see pictures below – Fig.1). The electronic circuit has been designed by the Italian Institute of Technology for the ABBI project. This high-density six-layer circuit has a small size (25 by 24 mm) and includes a powerful micro-controller (Arm Cortex M3), an audio amplifier, a low-energy (“Smart”) Bluetooth communication module that can communicate with any recent smartphone and computer, an Inertial Motion Unit (IMU) that includes an accelerometer, gyroscope and magnetometer to sense motion, a 16 Mb Flash Memory to store sound files and/or log the activity of the user, and a micro-USB connector to charge the battery, update the firmware and upload/download large files.

Figure 1: ABBI device and ABBI battery and micro-speakerSoftware :

The ABBI system includes several applications, which provide various degrees of control of the device such as ABBI Apps for smartphones and PC applications to control advanced functions of the device. For example, the user might use a common smartphone application to monitor the battery level of the device or to change the volume of the sound. More advanced applications can be used by the rehabilitator and/or scientist to tune the functions of the ABBI device or retrieve information stored in it.

Key technological aspects:

• The ABBI approach is based on state-of-the-art understanding of how spatial cognition develops and the possible long lasting effects of blindness during the most critical years of sensory, motor and cognitive development. The ABBI technology was developed by a team of psychologists, experts in the cognitive and sensory development of visually impaired people and professionals that interact everyday with blind people.
• The ABBI device is simple to use. Many Sensory Substitution Devices (SSD) are difficult to use because they overwhelm the human brain processing capacities. Blind individuals using SSD can be overloaded by the wide amount of acoustical and tactile signals coming from these devices. Experience and many user studies show that the systems must be relatively simple to use in order to be adopted such as, for example, audio books, white canes, guide dogs and echo-locating using tongue clicks or cane tapping sounds.
• ABBI uses the natural abilities of the human brain to process the position and movement of the sound sources in space and, therefore, it does not require the user to learn a new “language” to understand substitutive signals. It exploits information that can be naturally decoded by the hearing system. Positions, motions and activities of the persons or objects associated to these sounds sources are learned implicitly, unconsciously through everyday natural training. In that sense, ABBI is a simple interface, like the white cane, which is more natural than other more technologically advanced devices.
• The ABBI technology does not require learning complex language to work, in order to be used by very young children, from their first years of life (as opposed to other sensory substitution devices that are introduced in late childhood or adulthood). Moreover, ABBI can be also easily integrated in the daily life of children at home.

ABBI Applications:

This section illustrates several needs and applications currently under development for the ABBI system in collaboration with professionals in various rehabilitation fields:

• Spatial cognition training: In this application, the ABBI system is used to provide audio feedbacks about the movement of the child by positioning a sound source on his/her wrist or ankle. In this configuration, the ABBI system could be used by young children (one to five years old), at an age when such a training is most beneficial.
• Balance and mobility training: Vision also provides information that is used to maintain balance. The ABBI system can be used to train visually-impaired persons to maintain their balance by providing audio feedbacks about the movement of their body.
• Navigation training: Visually-impaired people hesitate to move and explore their surroundings. Equipped objects and people in the room will encourage visually impaired people to explore their surroundings and help them to develop a representation of the space around.
• Social interaction training: Visual impairments make it difficult, if not impossible, to interact with distant events, people and objects that do not produce any sound. Many joint actions like lifting something together, joint play etc… rely on visual feedbacks. In this application, several people will wear an ABBI, which provides feedbacks about their position in the room and their activity. Special games have been developed to encourage playful activity in small groups of children.

What type of researches have been done in order to make Abbi safe and useable by children?

The ABBI system and its applications are being developed with the active involvement and participation of rehabilitators and therapists, who have extensive experience in working with visually impaired people. In particular it is worth mentioning, an ongoing, long lasting collaboration with Istituto David Chiossone in Genoa and the joint lab established in their premises to assess the effectiveness of new devices and rehabilitation.

In this context, the ABBI system is intended to be used as a complement to existing tools and techniques currently used by rehabilitators, both at home and in public contexts.

IIT survey with stakeholders and opinion leaders (institutions, professionals, etc..) working in this sector suggest that there is a strong unmet medical need for new technologies and services to integrate existing methods used to help visually impaired people, which has yet to be fulfilled. The validity of the system is supported by experimental studies demonstrating that the association between the sensory and motor systems is vital for the development of visual-like spatial abilities: this is a central characteristic of the ABBI approach.

What was your first « wow » moment during tests, regarding a child performing an interaction previously impossible for them?

The first wow moment was when I saw a blind child running and having fun (it is very rare to see a blind child run) by following the rehabilitator wearing the ABBI device and playing with her.

With the ABBI device it was possible to develop more than 20 audio games that are usually made with vision such as “Capture the flag” or “Home base/Home plate”. Children had a lot of fun playing these games.

These games readapted with ABBI are described below:

• Capture the flag: All children are assigned to two groups except one child who is the leader. Children in each group are assigned to a number and a specific sound. The leader has the flag and chooses a specific sound of ABBI, different from the other children participating. The leader calls a number and the children corresponding to that number in each group has to localize the sound produced by the ABBI belonging to the leader, run towards the leader and capture the flag. The main objective is to localize a single sound embedded in a complex sonorous context.
• Home base/Home plate: All children are assigned to two groups, each group having a leader who chooses the preferred sound of ABBI. At the same time both leaders turn on their ABBIs and the children belonging to their team have to localize the sound and reach the leader. The exercise ends when all children reach the correct position of the sound/leader.

What were the main difficulties in order to create such a life-changing device for children? / What were the main sonic difficulties encountered while developing ABBI?

One of the main difficulties was related to the identification of the best sounds for ABBI. We performed specific workshops with children, families and rehabilitators to define them in a user-centered manner.

Another difficulty was related to the identification of the best psychophysical tests to monitor the improvement after the ABBI use.

From a hardware point of view, some difficulties have been found regarding the hardware miniaturization.

More info

More information about ABBI initiatives and about how ABBI device works can be found in this video or on the official website.

Team:

Coordinator: Monica Gori

Co.Coordinator: Gabriel Baud-Bovy

U-VIP, IIT: Finocchietti Sara, Cappagli Giulia, Cuturi Luigi, Cuppone Anna Vera

RBCS, IIT: Sandini Giulio, Ben Porquis Lope

EDL (IIT): Zini Giorgio, Ancarani Fabio, Torazza Diego, Nervi Stefano, Ilengo Stefano

University of Hamburg: Roder Brigitte, Rogge Ann’Kathrine

Istituto Chiossone: Cocchi Elena, Capris Elisabetta, Campana Paola, Gilio Carla, Grammatico Debora, Rolando Marika

University of Lund: Magnusson Charlotte, Caltenco A. Hector, Bitte Riderman

University of Glasgow: Brewster Stephen, Wilson Graham, Euan Freeman

Publications:

• Devices for visually impaired people: high technological devices  with low user acceptance and no adaptability for children. Gori, Cappagli, Tonelli, Baud-Bovy and Finocchietti 2016 Neuroscience and Biobehavioral Reviews impact factor 10.5
• From science to technology: orientation and mobility in blind children and adults. Luigi Cuturi, Elena Aggius-Vella, Claudio-Campus, Alberto Parmiggiani and Monica Gori Neuroscience and Biobehavioral Reviews impact factor 10.5
• Auditory spatial localization: Developmental delay in children with visual impairments  Cappagli, G., Gori, M. 2016 Research in Developmental Disabilities impact factor 2,18
• Depth echolocation learnt by novice sighted people  Tonelli, A., Brayda, L., Gori, M. 2016, PLoS ONE impact factor 3.057
• Early visual deprivation severely compromises the auditory sense of space in congenitally blind children  Vercillo, T., Burr, D., Gori, M. 2016 Developmental Psychology impact factor 3.116
• Evaluation of the Audio Bracelet for Blind Interaction for improving mobility and spatial cognition in early blind children – A pilot study  Finocchietti, S., Cappagli, G., Porquis, L.B., (…), Cocchi, E., Gori, M. 2015 Proceedings of the IEEE Engineering in Medicine and Biology Society, EMBS impact factor 3.05
• Encoding audio motion: spatial impairment in early blind individuals. 6:1357. [doi:10.3389/fpsyg.2015.01357] Finocchietti S., Cappagli G. and Gori M. (2015) Frontiers in Psychology impact factor 2.843
• Auditory and proprioceptive spatial impairments in blind children and adults. Developmental Science (in press). Cappagli, G., Cocchi, E. & Gori, M. (2015). Developmental Science impact factor 3,98
• Co-located games created by children with visual impairments  Magnusson, C., Rydeman, B., Finocchietti, S., (…), Baud-Bovy, G., Gori, M. 2015 MobileHCI 2015 – Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct

Conference articles in proceedings:

• G. Wilson, S. Brewster, S. FINOCCHIETTI, G. Baud-Bovy, M. Gori, H. Caltenco, C. Magnusson. Effects of Sound Type on Recreating the Trajectory of a Moving Source. Conference on Human Factors in Computing Systems (CHI) (2015), Seoul, South Korea.
• Magnusson, C., Caltenco, H., Gori, M., Finocchietti, S. and G. Cappagli. What do you like? Early design explorations of sound and haptic preferences. International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileCHI), Copenhagen, Denmark.