Non-Spatial Replay Is Informed by Previous Experience

Post by Sarah Hill

What's the science?

During rest, real and imagined stimuli are sequentially replayed in the brain as spontaneous rapid-fire sequences independent of sensory input. This phenomenon, termed 'replay', has been widely studied in rodent hippocampal place cells following a spatial navigation task. Replay has also been observed in non-spatial scenarios, such as when making inferences about how objects relate to one another. For example, after observing a broken vase next to a guilty-looking dog, one might mentally replay a scenario in which the dog knocks over the vase, despite not having witnessed said scenario. Relational knowledge abstracted from previous experience is theorized to guide future inference of relational information, though how this occurs and how it might be used in replay is poorly understood. This week in Cell, Liu and colleagues demonstrate that non-spatial scenarios are replayed in sequences inferred from existing knowledge of relationships between objects.

How did they do it?

To investigate how replay facilitates rapid inference of relational information, the authors carried out two studies using magnetoencephalography (MEG) to record spontaneous neural activity in human subjects. The first study was designed to test whether replay is derived in part from knowledge obtained from past experience. On day 1 of this experiment, participants were shown eight pictures from two different sequences, having been told the order the pictures were presented in was a scrambled version of the 'true' order. For example, pictures that were shown in the order [YZ, Y'Z'], [XY, X'Y'], and [WX, W'X'] implied two sequences WXYZ and W'X'Y'Z. On day 2, participants underwent MEG scanning while being presented with eight new pictures from another two sequences (termed the 'applied learning' phase): [CD, C'D'], [AB, A'B'], and [BC, B'C'] from the sequences ABCD and A'B'C'D'. After a 5 minute rest period, the subjects were shown that one of the terminal stimuli - either D or D' - was associated with a reward. This served to characterize one sequence as rewarding and the other as neutral (termed the 'value learning' phase). After another 5 minute period of rest, participants were randomly presented with pictures from the two sequences and asked to indicate the sequence to which the stimulus belonged. During the first 5 minute rest period, neural representations of each picture were decoded from the MEG recordings to decipher the sequence in which they were replayed. The decoding of these stimuli was based on a functional localizer task performed prior to the main task, in which participants simply viewed the images they would see later.

The second study was designed to rule out the possibility that sequential replay is a consequence of a simple Hebbian-like associative mechanism (based on the strength of association between the neurons encoding the stimuli). This would occur if participants encoded individual transitions (e.g. AB) rather than applying abstract knowledge (true replay). In this experiment, subjects took part in a 2-day task similar to that of the first experiment, except the pictures were no longer presented in pairs. Instead, they were explicitly told that the order in which pictures were presented mapped to the sequence of pictures in the 'true' order, and the same mapping pattern would be used for pictures shown on day 2. On day 1, subjects were presented with pictures individually in the order [Z' X Y' Y] and [Z W' W X'] corresponding to the sequences [WXYZ] and [W'X'Y'Z'], and told the sequence position of the stimuli (e.g. “the first stimuli to be presented (Z’) is last in the second sequence”). On day 2, they were first permitted to rest for 5 minutes while in the MEG scanner and then presented with eight new pictures that mapped to the 'true' order in the same way as day 1: [D' B C' C] and [D A' A B'] mapped to the sequences [ABCD] and [A'B'C'D']. After another 5 minute rest period, participants were shown the pictures in a randomized order and asked to indicate either the position of the stimulus in its respective sequence or the sequence to which the picture belonged. Finally, neural representations of each stimulus, as well as the position and sequence identity for each stimulus, were again decoded from the MEG recordings during the first rest period. 

What did they find?

In the first study, the authors observed that participants mentally replayed the novel pictures (ABCD and A'B'C'D) in the 'true' inferred order rather than the order visually experienced. During the 5 minute rest period following the applied learning phase, spontaneous neural activity encoding the new stimuli replayed the picture sequences predominantly in the forward direction. However, following the value learning phase, a spontaneous replay of the rewarding sequence occurred in the reverse direction (while the neutral sequence continued to replay in the forward direction). This is consistent with a previous study that found reward increases the relative frequency of reverse replay in rodents.

Neural replay during the 5 minute rest period was also observed in the second study following the applied learning phase. However, because the pictures were no longer shown in pairs, the order in which they were neurally represented could not have resulted from an associative mechanism. This suggests instead that relational information acquired from day 1 was used to inform the sequential replay of stimuli on day 2. Intriguingly, relational and sensory information about each stimulus were represented in distinct spatial and temporal domains. While each stimulus was neurally represented in occipital regions, corresponding position and sequence information were represented in posterior temporal regions. Furthermore, during sequential replay, spontaneous neural activity encoding the position and sequence of each stimulus occurred 40-60 ms before representation of the stimulus itself, demonstrating that relational information about an object is represented prior to representation of the object during rest.           

What's the impact?

This study provides further evidence that replay is not restricted solely to spatial processes and can be recorded non-invasively in humans. These findings suggest that spontaneous replay of stimulus sequences serves to aid the construction of internal relational models that are used for making future inferences about how objects relate to one another. More broadly, it seems possible that replay may be a brain-wide mechanism present in spatial and non-spatial learning and inference processes.

Liu et al. Human Replay Spontaneously Reorganizes Experience. Cell (2019). Access the original scientific publication here.