Researchers assume that an effective type of AI model called a transformer might be implemented in the brain through networks of nerve cell and astrocyte cells. The work could offer insights into how the brain works and help researchers understand why transformers are so reliable at machine-learning jobs. Credit: MIT News with figures from iStock
A new study bridging neuroscience and machine learning uses insights into the prospective function of astrocytes in the human brain.
Artificial neural networks are ubiquitous machine-learning models that can be trained to complete many jobs. Their name comes from the reality that their architecture is influenced by the method biological nerve cells process details in the human brain.
Researchers discovered a brand-new type of more powerful neural network model called a transformer about 6 years earlier. These designs can attain unmatched efficiency, such as by generating text from triggers with near-human-like precision. A transformer underlies AI systems such as OpenAIs ChatGPT and Googles Bard, for example. While extremely effective, transformers are also strange: Unlike with other brain-inspired neural network models, it hasnt been clear how to build them using biological components.
Bridging Biology and Transformers
Now, researchers from MIT, the MIT-IBM Watson AI Lab, and Harvard Medical School have produced a hypothesis that may explain how a transformer could be developed using biological elements in the brain. They recommend that a biological network composed of nerve cells and other brain cells called astrocytes could carry out the very same core calculation as a transformer.
Current research study has shown that astrocytes, non-neuronal cells that are abundant in the brain, interact with nerve cells and play a function in some physiological processes, like controling blood circulation. However researchers still lack a clear understanding of what these cells do computationally.
With the brand-new research study, released recently in open-access format in the Proceedings of the National Academy of Sciences, the scientists explored the function astrocytes play in the brain from a computational point of view, and crafted a mathematical design that shows how they might be utilized, together with nerve cells, to construct a biologically plausible transformer.
Their hypothesis provides insights that might stimulate future neuroscience research into how the human brain works. At the exact same time, it could help machine-learning scientists explain why transformers are so effective throughout a varied set of complex tasks.
” The brain is far superior to even the very best synthetic neural networks that we have developed, but we dont truly understand exactly how the brain works. There is scientific worth in believing about connections in between biological hardware and massive expert system networks. This is neuroscience for AI and AI for neuroscience,” says Dmitry Krotov, a research employee at the MIT-IBM Watson AI Lab and senior author of the term paper.
Signing up with Krotov on the paper are lead author Leo Kozachkov, a postdoc in the MIT Department of Brain and Cognitive Sciences; and Ksenia V. Kastanenka, an assistant teacher of neurobiology at Harvard Medical School and an assistant investigator at the Massachusetts General Research Institute.
A Biological Impossibility Becomes Plausible
Transformers operate differently than other neural network models. For circumstances, a persistent neural network trained for natural language processing would compare each word in a sentence to an internal state identified by the previous words. A transformer, on the other hand, compares all the words in the sentence at the same time to create a forecast, a process called self-attention.
For self-attention to work, the transformer must keep all the words prepared in some type of memory, Krotov describes, but this didnt seem biologically possible due to the method nerve cells interact.
Nevertheless, a few years ago researchers studying a slightly various type of machine-learning design (called a Dense Associated Memory) realized that this self-attention mechanism could take place in the brain, but just if there were interaction in between at least 3 nerve cells.
” The number three really popped out to me due to the fact that it is understood in neuroscience that these cells called astrocytes, which are not neurons, form three-way connections with neurons, what are called tripartite synapses,” Kozachkov states.
When two neurons communicate, a presynaptic nerve cell sends out chemicals called neurotransmitters across the synapse that connects it to a postsynaptic nerve cell. Sometimes, an astrocyte is also linked– it covers a long, thin tentacle around the synapse, creating a tripartite (three-part) synapse. One astrocyte might form countless tripartite synapses.
The astrocyte collects some neurotransmitters that stream through the synaptic junction. At some time, the astrocyte can signify back to the nerve cells. Because astrocytes operate on a much longer time scale than nerve cells– they develop signals by slowly raising their calcium reaction and after that reducing it– these cells can hold and integrate details communicated to them from nerve cells. In this method, astrocytes can form a kind of memory buffer, Krotov says.
” If you consider it from that perspective, then astrocytes are extremely natural for exactly the calculation we need to perform the attention operation inside transformers,” he adds.
Constructing a Neuron-Astrocyte Network
With this insight, the scientists formed their hypothesis that astrocytes might play a role in how transformers compute. They set out to build a mathematical design of a neuron-astrocyte network that would operate like a transformer.
They took the core mathematics that make up a transformer and established basic biophysical designs of what nerve cells and astrocytes do when they communicate in the brain, based upon a deep dive into the literature and guidance from neuroscientist collaborators.
Then they combined the models in certain methods up until they came to an equation of a neuron-astrocyte network that describes a transformers self-attention.
” Sometimes, we discovered that particular things we wished to be true couldnt be plausibly executed. So, we had to believe of workarounds. There are some things in the paper that are extremely mindful approximations of the transformer architecture to be able to match it in a biologically possible way,” Kozachkov says.
Through their analysis, the researchers revealed that their biophysical neuron-astrocyte network theoretically matches a transformer. In addition, they carried out mathematical simulations by feeding images and paragraphs of text to transformer designs and comparing the reactions to those of their simulated neuron-astrocyte network. Both reacted to the prompts in comparable methods, verifying their theoretical model.
” Having stayed electrically quiet for over a century of brain recordings, astrocytes are among the most plentiful, yet less explored, cells in the brain. The capacity of letting loose the computational power of the other half of our brain is huge,” states Konstantinos Michmizos, associate professor of computer system science at Rutgers University, who was not included with this work. “This study opens up an interesting iterative loop, from understanding how intelligent behavior may genuinely emerge in the brain, to equating disruptive hypotheses into new tools that show human-like intelligence.”
The next step for the researchers is to make the leap from theory to practice. They hope to compare the models predictions to those that have been observed in biological experiments, and utilize this understanding to improve, or potentially negate, their hypothesis.
In addition, one ramification of their study is that astrocytes may be associated with long-lasting memory, since the network needs to keep info to be able act on it in the future. Additional research could examine this idea even more, Krotov says.
” For a lot of reasons, astrocytes are very crucial for cognition and habits, and they run in essentially various ways from nerve cells. My most significant hope for this paper is that it catalyzes a lot of research study in computational neuroscience towards glial cells, and in specific, astrocytes,” adds Kozachkov.
Reference: “Building transformers from astrocytes and neurons” by Leo Kozachkov, Ksenia V. Kastanenka and Dmitry Krotov, 14 August 2023, Proceedings of the National Academy of Sciences.DOI: 10.1073/ pnas.2219150120.
This research study was supported, in part, by the BrightFocus Foundation and the National Institute of Health.
Scientist assume that a powerful type of AI design understood as a transformer might be carried out in the brain through networks of neuron and astrocyte cells. The work could use insights into how the brain works and assist researchers understand why transformers are so reliable at machine-learning tasks. Scientists discovered a new type of more effective neural network design known as a transformer about six years earlier. While exceptionally reliable, transformers are also mystical: Unlike with other brain-inspired neural network designs, it hasnt been clear how to develop them utilizing biological components.
In addition, they carried out numerical simulations by feeding images and paragraphs of text to transformer designs and comparing the actions to those of their simulated neuron-astrocyte network.
