What You Hear is What You See

January 24, 2014




Sound is an aspect too often forgotten in the UI design process. Most devices we use are capable of producing sound, and we see no reason not to put these senses to good use. When designing for the eye, we usually need to think of data placement and the amount of information the user is seeing. This is primarily due to the fact that our eyes can only focus on one place at a time. More than half of our surrounding is out of our field of view. With the help of small eye movements, we gather information about the subject at hand.

However, the basics of design for the ear are fundamentally different. Our ears gather information all the time, and this cannot be stopped unless we physically block our ears. Even when we selectively focus on a single source of sound, we never cease hearing other things as well. Due to these differences, human beings use their eyes and ears simultaneously to benefit from the best qualities of both senses. Traditionally, the ear tells where the eye should focus on. An unexpected sound grasps our attention and leads it to whatever is causing the sound. The division of work between our senses has constructed human hearing to support emotion, while the more factual data is often gathered by vision. This multimodal approach is essential when dealing with design for audiovisual media.

In UI design, sound must be used carefully and with respect. Sensing events all around us without a need for focused attention makes sound an excellent tool to process observation and alerting. However, typical users of today’s digital media are used to suppressing excessive visual stimuli and can get irritated by sonic announcements that cannot be skipped or browsed the same way visual material can. Therefore, sound should carry information on the occasion when the user’s eyes are busy or unable to see the screen.

Classic examples of intuitive multimodal interfaces can be found in hand held geiger counters and metal detectors. These devices provide a perfect opportunity for users to get readings through a sonic feedback instead of keeping an eye constantly on a screen. The best results for multimodal UI design are obtained by designing for not only the eye but also for the ear. This is the reason UX designers would benefit greatly from understanding the basics of human perception for both vision and hearing.

–Antti Mäki, UX and Sound Designer, Idean