How Much Effort Does it Take to Just Listen? — BrainPost | Easy-to-read summaries of the latest neuroscience publications

What is listening effort?

We don’t often think about how much of our mental space we reserve for listening to speech—for many people, it feels effortless. However, there are often obstacles: background noises, distracting conversations, or age-related hearing loss, to name a few. According to some models, we have a limited amount of mental resources, and the more we spend trying to decipher speech, the less we have left over for critical thinking, memory, and other high-level processing. However, a nebulous concept such as “effort” isn’t easy to quantify, and scientists have tried a number of approaches, from self-report questionnaires to full-sized brain scanners. Here’s a run-down of all the techniques used to measure listening effort.

Using self-report measures

One way to measure effort on a task is to ask people about it directly. This is the simplest method, but it can get tricky because people may have different interpretations of what “effort” means. To be more precise, some recommend breaking down effort into sub-components, like mental effort, physical effort, time pressure, or frustration. The NASA task load index is one such breakdown. However, a recent study suggests that we should ask about tiredness, a question that is not present in the NASA task load index. In that study, people’s tiredness ratings during speech listening were shown to correlate with the next method of measuring effort: pupil size.

Using the size of the pupil (pupillometry)

Our pupils dilate under states of mental arousal or effort, and specifically when listening conditions are worsened. Using video recordings of the eye or special glasses with infra-red cameras, we can measure the size of a person’s pupil as they hear and respond to sounds—this is called pupillometry. It is less subjective than self-report, and we can evaluate the effort someone expends on a task without forcing them to stop and reflect. However, pupillometry currently needs specific lighting conditions, and sometimes the pupil response can plateau in complex tasks. So, there are still some challenges to using this method.

Using brain activity (EEG, fMRI, fNIRS)

Electroencephalography, or EEG, measures electrical activity in the brain and can be used as another way to tap into listening effort. Among other EEG measures, the N100 response to sound is one index of this effort. This automatic response happens 100 milliseconds after the onset of a sound, and it becomes bigger when the speech is made less intelligible. Another EEG measure of effort is alpha power. If we take the activity in the alpha range and sum up its power over the course of an experiment, we can see when more effort is being expended.

Blood flow to different brain areas has long been used as a proxy for brain activity in those areas. In particular, blood flow to the left inferior frontal region of the brain (close to the temples) and the superior temporal gyrus (just above the ears) can give us a hint about how much effort is being exerted. This can be done in a magnetic scanner that detects the magnetic properties of blood (fMRI) or using a cap with small infra-red lights pointed at the scalp (fNIRS). These blood flow methods are a little slower than EEG, but fMRI, in particular, can pinpoint the location of activity in the brain with better accuracy, and fNIRS is advantageous because it doesn’t interfere with hearing aids or other devices.

What’s the bottom line?

Hearing is a crucial aspect of health: hearing loss has a large societal burden and may contribute to the risk of dementia later in life. Armed with multiple tools to measure listening effort, we can study how it varies in different conditions and populations, and better understand the link between hearing and cognition.

References

Pichora-Fuller, M. K. et al. Hearing Impairment and Cognitive Energy. Ear Hear. 37, 5S-27S (2016).

Hart, S. G. & Staveland, L. E. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Adv. Psychol. 52, 139–183 (1988).

McGarrigle, R., Rakusen, L. & Mattys, S. Effortful listening under the microscope: Examining relations between pupillometric and subjective markers of effort and tiredness from listening. Psychophysiology 58, 1–22 (2021).

Zekveld, A. A., Kramer, S. E. & Festen, J. M. Pupil response as an indication of effortful listening: The influence of sentence intelligibility. Ear Hear. 31, 480–490 (2010).

Koo, M. et al. Effects of noise and serial position on free recall of spoken words and pupil dilation during encoding in normal-hearing adults. Brain Sci. 11, 1–14 (2021).

Zhang, Y., Lehmann, A. & Deroche, M. Disentangling listening effort and memory load beyond behavioural evidence: Pupillary response to listening effort during a concurrent memory task. (2020). doi:10.1101/2020.05.04.076588

Obleser, J. & Kotz, S. A. Multiple brain signatures of integration in the comprehension of degraded speech. Neuroimage 55, 713–723 (2011).

Obleser, J., Wöstmann, M., Hellbernd, N., Wilsch, A. & Maess, B. Adverse listening conditions and memory load drive a common alpha oscillatory network. J. Neurosci. 32, 12376–12383 (2012).

Wild, C. J. et al. Effortful listening: The processing of degraded speech depends critically on attention. J. Neurosci. 32, 14010–14021 (2012).

Kousaie, S. et al. Language learning experience and mastering the challenges of perceiving speech in noise. Brain Lang. 196, 104645 (2019).

Zekveld, A. A., Heslenfeld, D. J., Johnsrude, I. S., Versfeld, N. J. & Kramer, S. E. The eye as a window to the listening brain: Neural correlates of pupil size as a measure of cognitive listening load. Neuroimage 101, 76–86 (2014).

Rovetti, J., Goy, H., Pichora-Fuller, M. K. & Russo, F. A. Functional Near-Infrared Spectroscopy as a Measure of Listening Effort in Older Adults Who Use Hearing Aids. Trends Hear. 23, 233121651988672 (2019).

Ford, A. H. et al. Hearing loss and the risk of dementia in later life. Maturitas. 112, 1–11 (2018).