Brown University researchers have shown that reinforcement learning and working memory — two distinct brain systems — work hand-in-hand as people learn new tasks.
The study, published in Proceedings of the National Academy of Sciences, focused on the interplay of two very different modes of learning a new task: reinforcement learning and working memory. Reinforcement learning is an “under-the-hood” process in which people gradually learn which actions to take by processing rewards and punishments at the neural level, and then choosing the one that works best on average — even if the person is not aware of it. In contrast, working memory involves keeping previous actions and their outcomes in mind to more rapidly and flexibly improve performance.
“People have largely interpreted these systems as working independently or as competing with each other in the learning process,” said Michael Frank, a professor in Brown’s Department of Cognitive, Linguistic and Psychological Sciences and co-author of the paper.
“But we show that the two work together, with neural signals underlying working memory helping to guide those that support reinforcement learning.”
Anne Collins, an assistant professor at the University of California, Berkeley, led the work when she was a postdoctoral researcher working with Frank, who directs the Initiative for Computation in Brain and Mind in the Brown Institute for Brain Science. Collins and Frank developed an experimental method designed to isolate the brain signals associated with each of the two systems.
For the study, 40 participants were shown a series of symbols on a screen and asked, for each symbol, to press a particular button on a keyboard. They weren’t told which key was the right one for each symbol. They had to learn it. When they got it right, they were rewarded with points. Over repeated trials, the participants came to learn which keys corresponded with which symbols.
In order to distinguish the contributions from reinforcement learning and working memory, the researchers set up problems with different numbers of symbols, ranging from two to six, and participants had to learn which button to press for each of them. Generally, people can only hold three or four items in working memory at a time, and only for short periods of time. So when the number of symbols or the delay increases, the contribution of working memory to the learning process should diminish.
As the participants performed the tasks, an EEG cap recorded signals from the brain, and the authors applied statistical methods to extract those signals related to one learning system or the other.
The study showed that when memory demands were high, the signals in the brain correlated to reinforcement learning actually got stronger. In other words, when the working memory system was overtaxed, the reinforcement learning system became more important in the learning process. In contrast, when participants could hold information in mind, signals associated with reinforcement learning were weaker, suggesting an increased role for working memory.
The researchers also found that they could decode from the brain signals in a particular trial whether information was likely to be in memory or not. That too traded off with the neural marker of reinforcement learning.
Those findings, the researchers say, suggest that the two systems aren’t working independently.
“If they were completely independent of each other, we’d expect the signals associated with reinforcement learning to stay the same regardless of memory demands,” Frank said.
“But that’s not we see, and that’s a sign that the two systems are interacting.”
But on its own, that finding didn’t reveal the nature of that interaction — whether it’s cooperative or competitive. Was working memory shoving the reinforcement learning into the background in trials when the information could be readily accessible in mind? Or could it be that working memory helps to augment reinforcement learning? To figure that out, the researchers looked how the brain signals associated with reinforcement learning changed as the learning process unfolded from trial to trial.
The reinforcement learning system is driven by what’s known as “reward prediction error” or RPE, and it’s the signal the researchers used to track the reinforcement learning process. RPE represents the extent to which the reward that results from an action exceeds one’s expectations. Take for example a study participant trying to figure out which button to press when they see a given symbol. If they happen to guess right and get rewarded with points, that outcome is surprisingly good and produces a high RPE.
In the brain, the reinforcement learning system uses the neurotransmitter dopamine to encode RPE. A high RPE — meaning a surprisingly good outcome — is associated with a large release of dopamine. The reinforcement learning system uses that dopamine flood as a signal to update our understanding of what actions we should take to get a given reward. When we repeat that action subsequently, we’re less surprised by the reward and so the RPE is lower. As RPE continues to diminish, the system eventually stops updating, and in so doing, settles upon an appropriate action.
One scenario for how working memory could be interacting with reinforcement learning is by attenuating reward expectations, making them more quickly come into line with actual rewards. In that way, working memory could be working cooperatively to speed the reinforcement learning process.
The study found strong evidence for just that scenario. During repeated trials at small set sizes where working memory is active, brain signals associated with RPE started out high in the first few trials, and then quickly dropped off — a sign that cognitive processes are informing the neural signalling associated with reinforcement learning. In contrast, if working memory were merely suppressing reinforcement learning, one wouldn’t expect to see the quick drop in RPE.
The results, Frank said, provide some of the first concrete evidence for cooperation between these two systems.
“Thinking of these not as separate systems but as one big integrated system changes our understanding of the basic science of how people and animals learn,” Frank said.
“It might help us make better predictions about how the overall learning process is affected in people who have deficits in either of these systems.”
And that, Frank said, could one day lead to better treatments for learning impairments.
Focused Delivery For Brain Cancers
A person’s brainstem controls some of the body’s most important functions, including heart beat, respiration, blood pressure and swallowing. Tumor growth in this part of the brain is therefore twice as devastating. Not only can such a growth disrupt vital functions, but operating in this area is so risky, many medical professionals refuse to consider it as an option.
New, interdisciplinary research in Washington University in St. Louis has shown a way to target drug delivery to just that area of the brain using noninvasive measures, bolstered by a novel technology: focused ultrasound.
The research comes from the lab of Hong Chen, assistant professor of biomedical engineering in the School of Engineering & Applied Science and assistant professor of radiation oncology at Washington University School of Medicine. Chen has developed a novel way in which ultrasound and its contrast agent — consisting of tiny bubbles — can be paired with intranasal administration, to direct a drug to the brainstem.
The research, which also included faculty from the Mallinckrodt Institute of Radiology and the Department of Pediatrics at the School of Medicine, along with the Department of Energy, Environmental & Chemical Engineering in the School of Engineering & Applied Science, was published online this week and will be in the Sept. 28 issue of the Journal of Controlled Release.
This technique may bring medicine one step closer to curing brain-based diseases such as diffuse intrinsic pontine gliomas (DIPG), a childhood brain cancer with a five-year survival rate of a scant two percent, a dismal prognosis that has remained unchanged over the past 40 years. (To add perspective, the most common childhood cancer, acute lymphoblastic leukemia, has a five-year survival rate of nearly 90 percent).
“Each year in the United States, there are no more than 300 cases,” Chen said.
“All pediatric diseases are rare; luckily, this is even more rare. But we cannot count numbers in this way, because for kids that have this disease and their families, it is devastating.”
Chen’s technique combines Focused UltraSound with IntraNasal delivery, (FUSIN). The intranasal delivery takes advantage of a unique property of the olfactory and trigeminal nerves: they can carry nanoparticles directly to the brain, bypassing the blood brain barrier, an obstacle to drug delivery in the brain.
This unique capability of intranasal delivery was demonstrated last year by co-authors Ramesh Raliya, research scientist, and Pratim Biswas, assistant vice chancellor and chair of the Department of Energy, Environmental & Chemical Engineering and the Lucy & Stanley Lopata Professor, in their 2017 publication in Scientific Reports.
“At the beginning, I couldn’t even believe this could work,” Hong said of delivering drugs to the brain intranasally.
“I thought our brains are fully protected. But these nerves actually directly connect with the brain and provide direct access to the brain.”
While nasal brain drug delivery is a huge step forward, it isn’t yet possible to target a drug to a specific area. Chen’s targeted ultrasound technique is addressing that problem.
When doing an ultrasound scan, the contrast agent used to highlight images is composed of microbubbles. Once injected into the bloodstream, the microbubbles behave like red blood cells, traversing the body as the heart pumps.
Once they reach the site where the ultrasound wave is focused, they do something unusual.
“They start to expand and contract,” Chen said.
As they do so, they act as a pump to the surrounding blood vessels as well as the perivascular space — the space surrounding the blood vessels.
“Consider the blood vessels like a river,” Chen said.
“The conventional way to deliver drugs is to dump them in the river.”
In other parts of the body, the banks of the river are a bit “leaky,” Chen said, allowing the drugs to seep into the surrounding tissue. But the blood brain barrier, which forms a protective layer around blood vessels in the brain, prevents this leakage, particularly in the brains of young patients, such as those with with DIPG.
“We will deliver the drug from the nose to directly outside the river,” Chen said, “in the perivascular space.”
Then, once ultrasound is applied at the brain stem, the microbubbles will begin to expand and contract. The oscillating microbubbles push and pull, pumping the drug toward the brainstem. This technique also addresses the problem of drug toxicity — the drugs will go directly to the brain instead of circulating through the whole body. In collaboration with Yongjian Liu, an associate professor of radiology, and Yuan-Chuan Tai, an associate professor of radiology, Chen used positron emission tomography (PET scan) to verify that there was minimal accumulation of intranasal-administered nanoparticles in major organs, including lungs, liver, spleen, kidney and heart.
So far, Chen’s lab has had success using their technique in mice for the delivery of gold nanoclusters made by the team led by Liu.
“The next step is to demonstrate the therapeutic efficacy of FUSIN in the delivery of chemotherapy drugs for the treatment of DIPG,” said Dezhuang Ye, lead author of the paper, who is Chen’s graduate student from the Department of Mechanical Engineering & Materials Science.
The lab has also teamed up with Biswas to develop a new aerosol nasal delivery device to scale up the technique from a mouse to a large animal model.
Chen’s lab collaborated on this research with pediatric neuro-oncologist Joshua Rubin, MD, PhD, a professor of pediatrics at the School of Medicine who treats patients at St. Louis Children’s Hospital. Chen said the team hopes to translate the findings of this study into clinical trials for children with DIPG.
There are difficulties ahead, but Chen believes researchers will need to continue to innovate when it comes to solving such a difficult problem as treating DIPG.
A targeted inspiration
Hong Chen’s lab collaborated with Joshua Rubin, MD, PhD, a professor of pediatrics at the School of Medicine on this research. And it all started with a couple of colleagues talking one day:
“My work in this field started with a conversation with him,” Chen said.
“He said, ‘Wow, this would be a perfect technique for treating this deadly disease.’ Without him to point me in this direction, I probably wouldn’t have known this application existed.
“That’s why I consider the Washington University environment, and the School of Engineering & Applied Science, so unique. It provides you so much opportunity to work with people from different backgrounds. It allowed me to expand my research scope and to be able to work on clinically relevant questions.”
Secret Tunnels Discovered Between The Skull And The Brain
Bone marrow, the spongy tissue inside most of our bones, produces red blood cells as well as immune cells that help fight off infections and heal injuries. According to a new study of mice and humans, tiny tunnels run from skull bone marrow to the lining of the brain and may provide a direct route for immune cells responding to injuries caused by stroke and other brain disorders. The study was funded in part by the National Institutes of Health and published in Nature Neuroscience.
“We always thought that immune cells from our arms and legs traveled via blood to damaged brain tissue. These findings suggest that immune cells may instead be taking a shortcut to rapidly arrive at areas of inflammation,” said Francesca Bosetti, Ph.D., program director at the NIH’s National Institute of Neurological Disorders and Stroke (NINDS), which provided funding for the study.
“Inflammation plays a critical role in many brain disorders and it is possible that the newly described channels may be important in a number of conditions. The discovery of these channels opens up many new avenues of research.”
Using state-of-the-art tools and cell-specific dyes in mice, Matthias Nahrendorf, M.D., Ph.D., professor at Harvard Medical School and Massachusetts General Hospital in Boston, and his colleagues were able to distinguish whether immune cells traveling to brain tissue damaged by stroke or meningitis, came from bone marrow in the skull or the tibia, a large legbone. In this study, the researchers focused on neutrophils, a particular type of immune cell, which are among the first to arrive at an injury site.
Results in mouse brains showed that during stroke, the skull is more likely to supply neutrophils to the injured tissue than the tibia. In contrast, following a heart attack, the skull and tibia provided similar numbers of neutrophils to the heart, which is far from both of those areas.
Dr. Nahrendorf’s group also observed that six hours after stroke, there were fewer neutrophils in the skull bone marrow than in the tibia bone marrow, suggesting that the skull marrow released many more cells to the injury site. These findings indicate that bone marrow throughout the body does not uniformly contribute immune cells to help injured or infected tissue and suggests that the injured brain and skull bone marrow may “communicate” in some way that results in a direct response from adjacent leukocytes.
Dr. Nahrendorf’s team found that differences in bone marrow activity during inflammation may be determined by stromal cell-derived factor-1 (SDF-1), a molecule that keeps immune cells in the bone marrow. When levels of SDF-1 decrease, neutrophils are released from marrow. The researchers observed levels of SDF-1 decreasing six hours after stroke, but only in the skull marrow, and not in the tibia. The results suggest that the decrease in levels of SDF-1 may be a response to local tissue damage and alert and mobilize only the bone marrow that is closest to the site of inflammation.
Next, Dr. Nahrendorf and his colleagues wanted to see how the neutrophils were arriving at the injured tissue.
“We started examining the skull very carefully, looking at it from all angles, trying to figure out how neutrophils are getting to the brain,” said Dr. Nahrendorf.
“Unexpectedly, we discovered tiny channels that connected the marrow directly with the outer lining of the brain.”
With the help of advanced imaging techniques, the researchers watched neutrophils moving through the channels. Blood normally flowed through the channels from the skull’s interior to the bone marrow, but after a stroke, neutrophils were seen moving in the opposite direction to get to damaged tissue.
Dr. Nahrendorf’s team detected the channels throughout the skull as well as in the tibia, which led them to search for similar features in the human skull. Detailed imaging of human skull samples obtained from surgery uncovered the presence of the channels. The channels in the human skull were five times larger in diameter compared to those found in mice. In human and mouse skulls, the channels were found in the both in the inner and outer layers of bone.
Missing Immune Cells That Could Fight Lethal Brain Tumors
Glioblastoma brain tumors can have an unusual effect on the body’s immune system, often causing a dramatic drop in the number of circulating T-cells that help drive the body’s defenses.
Where the T-cells go has been unclear, even as immunotherapies are increasingly employed to stimulate the body’s natural ability to fight invasive tumors.
Now researchers at Duke Cancer Institute have tracked the missing T-cells in glioblastoma patients. They found them in abundance in the bone marrow, locked away and unable to function because of a process the brain stimulates in response to glioblastoma, to other tumors that metastasize in the brain and even to injury.
The findings, published online Aug. 13 in the journal Nature Medicine, open a new area of exploration for adjunct cancer drugs that could free trapped T-cells from the bone marrow, potentially improving the effectiveness of existing and new immunotherapies.
“Part of the problem with all these immunotherapies — particularly for glioblastoma and other tumors that have spread to the brain — is that the immune system is shot,” said lead author Peter E. Fecci, M.D., Ph.D., director of the Brain Tumor Immunotherapy Program in Duke’s Department of Neurosurgery.
“If the goal is to activate the T-cells and the T-cells aren’t there, you’re simply delivering therapy into a black hole.”
Fecci said the research team began its search for the missing T-cells after observing that many newly diagnosed glioblastoma patients have the equivalent immune systems of people with full-blown AIDS, even before they undergo surgery, chemotherapy and radiation.
Where most people have a CD-4 “helper” T-cell count upwards of 700-1,000, a substantial proportion of untreated glioblastoma patients have counts of 200 or less, marking poor immune function that makes them susceptible to all manner of infections and potentially to progression of their cancer.
Initially, the researchers hunted for the missing T-cells in the spleen, which is known to pathologically harbor the cells in certain disease states. But the spleens were abnormally small, as were the thymus glands — another potential T-cell haven. They decided to check the bone marrow to see if production was somehow stymied and instead found hordes of T-cells.
“It’s totally bizarre — this is not seen in any disease state,” Fecci said.
“This appears to be a mechanism that the brain possesses for keeping T-cells out, but it’s being usurped by tumors to limit the immune system’s ability to attack them.”
When examining the stashed T-cells, Fecci and colleagues found that they lacked a receptor on the cell surface called S1P1, which essentially serves as a key that enables them to leave the bone marrow and lymph system. Lacking that key, they instead get locked in, unable to circulate and fight infections, let alone cancer.
Fecci said the research team is now working to learn exactly how the brain triggers the dysfunction of this S1P1 receptor. He said the current theory is that the receptor somehow is signaled to retract from the cell surface into the cell interior.
“Interestingly, when we restore this receptor to T-cells in mice, the T-cells leave the bone marrow and travel to the tumor, so we know this process is reversible,” Fecci said.
His team is collaborating with Duke scientist Robert Lefkowitz, M.D., whose 2012 Nobel Prize in Chemistry honored discovery of the class of receptors to which S1P1 belongs. They are working to develop molecules that would restore the receptors on the cells’ surface.
“We are hopeful that this finding provides a missing element that would enable more immunotherapies to be effective for more people,” Fecci said. He said the finding could also work in reverse, offering a new approach to quell auto-immune disorders by activating the T-cell sequestration.
Like Us on Facebook
Report Reveals Link Between Air Pollution And Increased Risk For Miscarriage
Air quality has been associated with numerous adverse health outcomes from asthma to pre-term birth. Researchers at University of Utah...
Alcohol Intake May Be Key to Long-term Weight Loss for People with Diabetes
Research shows that losing weight can help prevent or delay the onset of diabetes. While best practice for weight loss...
Hang In There. As Couples Age, Humor Replaces Bickering
Honeymoon long over? Hang in there. A new UC Berkeley study shows those prickly disagreements that can mark the early...
Prenatal Exposure To Chemicals In Personal Care Products May Speed Puberty In Girls
Girls exposed to chemicals commonly found in toothpaste, makeup, soap and other personal care products before birth may hit puberty...
Spinal Injuries Throw Body Clocks Off-Schedule, New Study Shows
In the hours and days following a spinal cord injury, the gears that control the body’s internal clocks fall profoundly...
When Storing Memories, Brain Prioritizes Those Experiences That Are Most Rewarding
The brain’s ability to preserve memories lies at the heart of our basic human experience. But how does the brain’s...
Parental ‘Feeding Styles’ Reflect Children’s Genes
New research from King’s College London and UCL challenges the idea that a child’s weight largely reflects the way their...
LISTS10 months ago
12 Types of Consultants You Will Meet
LISTS7 months ago
9 Medical Terms You Can Use Everyday To Sound Smarter 🤓
NEUROLOGY8 months ago
Soccer Heading Shown To Impair Players Cognitively
LISTS9 months ago
13 Types of Students You Meet at Medical School
LISTS10 months ago
13 Things You Did The First Time You Started Seeing Patients
LISTS7 months ago
10 Types of Medical Students During Exams
ENDOCRINOLOGY9 months ago
Researchers Identified The Body’s Glucostat
LISTS9 months ago
13 Things You Thought at Your First Lecture in Medical School