Smartwatch app alerts deaf people to everyday sounds
Image credit: Dreamstime
Researchers in the US have developed a smartwatch app for deaf and hard-of-hearing people who want to be aware of nearby sounds.
Developed by a team at the University of Washington, the app called SoundWatch picks up and identifies a sound the user is interested in – examples include a siren, a microwave beeping, or a bird chirping – sends a buzz on the watch along with information about the sound.
“This technology provides people with a way to experience sounds that require an action – such as getting food from the microwave when it beeps. But these devices can also enhance people’s experiences and help them feel more connected to the world,” said Dhruv Jain, a doctoral student in the School of Computer Science & Engineering.
He added: “I use the watch prototype to notice birds chirping and waterfall sounds when I am hiking. It makes me feel present in nature. My hope is that other d/Deaf and hard-of-hearing people who are interested in sounds will also find SoundWatch helpful.”
The team started this project by designing a system for people who wanted to be able to know what was going on around their homes. “I used to sleep through the fire alarm,” said Jain, whom himself was born hard of hearing.
The first system, called HomeSound, uses Microsoft Surface tablets scattered throughout the home which act as a network of interconnected displays. Each display provides a basic floor plan of the house and alerts a user to a sound and its source. The displays also show the sound’s waveforms, to help users identify the sound, and store a history of all the sounds a user might have missed when they were not home.
The researchers tested HomeSound in the Seattle-area homes of six d/Deaf or hard-of-hearing participants for three weeks. For this experiment, participants were instructed to go about their lives as normal and complete weekly surveys.
Based on feedback from the participants, a second prototype used machine learning to classify sounds in real-time. Here, the researchers created a dataset of over 31 hours of 19 common home-related sounds such as a dog bark or a cat meow, a baby crying and a door knock.
“People mentioned being able to train their pets when they noticed dog barking sounds from another room or realising they didn’t have to wait by the door when they were expecting someone to come over,” Jain explained. “HomeSound enabled all these new types of interactions people could have in their homes. But many people wanted information throughout the day, when they were out in their cars or going for walks.”
Following this prototype, the researchers then directed their attention toward a smartwatch system that allows users to get sound alerts wherever they are, even in places they might not have their phones, for example at the gym.
The researchers tested the SoundWatch app in March 2020 – before Washington’s stay-at-home order – with eight d/Deaf and hard-of-hearing participants in the Seattle area. Users tested the app at three different locations on or around the UW campus: in a grad student office, in a building lounge, and at a bus stop.
As a result, people found the app was useful for letting them know if there was something that they should pay attention to. For example: that they had left the tap running or that a car was honking. However, the participants said the app sometimes misclassified sounds (labelling a car driving by as running water) or was slow to notify users (one user was surprised by a person entering the room before the watch sent a notification about a door opening).
Another current focus is developing a method to pick out specific sounds from background noise and identifying the direction a sound, such as a police siren, is coming from.
“Disability is highly personal, and we want these devices to allow people to have deeper experiences,” Jain said. “We’re now looking into ways for people to personalise these systems for their own specific needs. We want people to be notified about the sounds they care about – a spouse’s voice versus general speech, the back door opening versus the front door opening, and more.”
Sign up to the E&T News e-mail to get great stories like this delivered to your inbox every day.