BERT

In a groundbreaking development, researchers at Sungkyunkwan University in South Korea have unveiled a novel memory system inspired by the principles of Hebbian theory. This cutting-edge innovation promises to significantly enhance the performance of transformer-based machine learning models, particularly in tasks involving the processing of long data sequences. The research, published on the arXiv preprint server, introduces “Memoria,” a general memory network that leverages Hebbian theory to improve long-term dependencies in neural networks.
While transformers have proven to be formidable tools in processing sequential data, they have limitations when it comes to handling longer sequences. Transformers’ restricted storage capacity impedes their ability to effectively process extended input sequences, leading to a decline in performance. Unlike <a href="https://www.cryptopolitan.com/ai-chatbots-show-high-creativity/" rel="noopener noreferrer nofollow">humans</a>, who selectively remember and utilize relevant information from inputs, transformers process all raw data from start to end, making them less efficient in handling lengthy sequences.
<h2 class="wp-block-heading" id="a-hebbian-memory-for-transformers">A Hebbian memory for transformers</h2>
Researchers Sangjun Park and JinYeong Bak’s primary objective was to design a system that could overcome these limitations and empower transformer models with improved capabilities. Drawing inspiration from Hebbian theory, which posits that neurons and cells that are repeatedly activated together tend to form associations that lead to learning, they introduced Memoria.
<blockquote class="wp-block-quote">
 “Memoria stores and retrieves information called engram at multiple memory levels of working memory, short-term memory, and long-term memory, using connection weights that change according to Hebb’s rule,”
</blockquote>
explain Park and Bak in their paper.
In a series of experiments, the researchers evaluated the effectiveness of their Hebbian memory system, and the results were highly promising. Memoria demonstrated its ability to significantly enhance the performance of transformer models across a range of tasks that involve processing lengthy data sequences.
<blockquote class="wp-block-quote">
 “Through experiments with popular transformer-based models like BERT and GPT, we present that Memoria significantly improves the ability to consider long-term dependencies in various tasks,”
</blockquote>
the researchers reported.
<blockquote class="wp-block-quote">
 “Results show that Memoria outperformed existing methodologies in sorting and language modeling, and long text classification.”
</blockquote>
<h2 class="wp-block-heading" id="open-source-code-for-wider-adoption">Open-Source code for wider adoption</h2>
In a move aimed at fostering wider adoption and collaboration, the code developed by Park and Bak is open-source and readily accessible on GitHub. The researchers also deployed Memoria using an independent Python package, making it user-friendly for developers around the world. This open-access approach is expected to encourage other research groups to explore the potential of Memoria in enhancing the performance of their transformer-based models. 
The groundbreaking memory architecture developed by Sungkyunkwan University researchers is poised for further exploration in complex tasks, and its potential applications are vast. As more research groups worldwide experiment with Memoria, we can expect a surge in the development of more powerful and efficient transformer-based <a href="https://www.cryptopolitan.com/researchers-on-efficient-machine-learning/" rel="noopener noreferrer nofollow">machine learning</a> models.

Revolutionizing AI: Memoria Unleashes Transformers’ Hidden Power

Table of Contents

Cryptopolitan_News

Other articles published on Oct 19, 2023

Bella Protocol (BEL) Price Prediction 2024, 2025, 2030, 2040, and 2050

Palantir Stock Price Falls 3.59%: Can PLTR Stock Retest 50 EMA?

Why is Ardor (ARDR) Coin Pumping? A Breakout Above This Level Suggests It’s Just the Beginning

Sam Bankman-Fried's lawyers can't stop making mistakes

DEX Platform SynFutures Secures $22M in Series B Funding Round Led By Pantera Capital

Doge Uprising ($DUP) Announces Presale Launch: A Trailblazing Crypto Project Uniting Manga, Web3,...

Join our free newsletter for daily crypto updates!