The notion of invisibility differs from person to person. This concept gains a particular meaning for people with restricted sensory capabilities, such as color-blind, blind or individuals experiencing hearing loss. In this work, we focus on a Machine Learning (ML)-based solution to enable people with hearing loss to sense nature through a portable actuated cardboard device. The device captures different nature sounds, such as bird sounds and classifies these into categories. This information is then used to generate specific vibration patterns of the box. The users are presented with additional information such as pictures of the recognized birds and information about the species. In this way, we bring the feeling of relaxation in nature closer to people experiencing hearing loss.
Studies have shown that being in nature reduces anger, fear and relieves stress and contributes to a better overall wellbeing. However, people with hearing loss are unable to experience the positive influence of connecting with nature in this fashion. For this reason, we propose “Nature in a Box” (NiB), a portable sensing device for deaf people to be used in nature. For instance, Jane is an individual with hearing loss. It’s a nice sunny day and she sits in the park and enjoys the view. But the visible is not the only thing she can experience now. She takes out her NiB device and turns it on using a touch sensor. She places her hands on the left and right side of the box using the designated areas. Through a microphone, the device starts recording different sounds of the environment. When the device recognizes a sound emitted by an animal, it classifies the species it belongs to using a sound- classifying ML model. Based on this information, the device retrieves an image and a description of the animal. Subsequently, it shows this information on the display. At the same time, a generative ML model transforms the retrieved sound into a pattern of vibrations. These are used to actuate the box using the in-built vibration modules. As such, Jane can perceive her surroundings through haptic and visual modalities in near real-time. The NiB device is portable, small and easy-to-use for any age group, children as well. It can be charged any time via a USB connector and you can use it anywhere. For privacy reasons, when the recording is performed the device flashes a status LED to inform users and other people. Connecting with nature plays an important role in human life and it is a source of positive emotions and helps your overall wellbeing. Using NiB, we want to extend this experience to include people who may otherwise experience difficulties to fully immerse themselves in nature due to their restricted sensory capabilities.