How the Orbitofrontal Cortex Learns How to Learn
Post by Rachel Sharp
The takeaway
The orbitofrontal cortex (OFC) plays a crucial role in the brain's ability to learn and adapt to new situations. By studying mice and computational models, researchers show how the OFC supports meta-reinforcement learning, a process where learning evolves to incorporate better strategies and allow for better decision-making.
What's the science?
Meta-learning, or 'learning to learn', has been a fundamental concept in psychology and artificial intelligence (AI), enabling humans and AI models to quickly acquire new skills based on generalized knowledge from past experiences. In the AI field, this concept has been used to enhance deep learning models, allowing them to refine their learning algorithms across multiple episodes. Similarly, in neuroscience, meta-reinforcement learning (meta-RL) involves the quick neural computation of multiple reinforcement learning (RL) processes at parallel, but different timescales. This week in Nature Neuroscience, Hattori and colleagues unravel how the brain, specifically the orbitofrontal cortex (OFC), manages this complex form of learning. The authors explore whether, and how, synaptic plasticity (a mechanism for adaptive learning at the cellular level) and neural activity-based mechanisms in the OFC collaborate to facilitate meta-RL, using a combination of mouse models and deep RL algorithms.
How did they do it?
The researchers trained mice on a probabilistic reversal learning task, where mice had to choose between two options with varying reward probabilities. This task was designed to mimic the complexities of decision-making in changing environments. Additionally, they focused on the OFC as a key area for mediating meta-RL. To test the role of synaptic plasticity in the OFC, they used paAIP2, a light-inducible inhibitor of CaMKII kinase activity, which blocks synaptic plasticity without affecting pre-existing neural connectivity. Put another way, they injected a virus containing an inhibitor of synaptic plasticity into neurons in the OFC. They were then able to control when this inhibitor was active or not by shining a light at these neurons. When they applied a light to the neurons, paAIP2 would become active, and inhibit the activity of CaMKII.
They then trained deep RL models on the same task, using a meta-RL framework. This meta-RL framework allowed the models to update their strategies across sessions, similar to how the mice learned. The researchers then examined the neural mechanisms behind this learning process, focusing on the OFC in mice, and comparing their findings with the computational models.
What did they find?
The study revealed that synaptic plasticity in the OFC is essential for efficient across-session meta-RL in mice. When synaptic plasticity was inhibited in the OFC, mice showed delayed learning and stabilization of reward-based choices. However, once the mice became task experts, blocking OFC plasticity did not affect their performance, indicating that OFC activity (as opposed to plasticity) is required for fast or immediate RL These results when taken together, suggest that CaMKII-mediated plasticity in the OFC is necessary for meta-learning of RL which supports long-term strategy development, while OFC activity supports more immediate decision-making. The findings also demonstrate that both mice and deep RL models improved their decision-making over time, adapting their choices based on reward history, indicating the OFC's vital role in conducting meta-RL.
What's the impact?
This study sheds light on the dual role of the orbitofrontal cortex (OFC) in managing both slow and fast reinforcement learning (RL). The OFC uses CaMKII-dependent synaptic plasticity for slow, across-session meta-learning, in which generalized knowledge is accumulated and stored over long periods. For immediate, trial-by-trial decision-making, the OFC relies on its neural activity. The study's findings align closely with deep reinforcement learning models, demonstrating a remarkable parallel between artificial and biological learning systems. The insights from this study not only enhance our understanding of the brain's learning mechanisms but also pave the way for more sophisticated AI models.