Sparse Butterfly Matrix Attention for Enhancing Transformer Performance

NCUIR > College of Electrical Engineering & Computer Science > Graduate Institute of Computer Science and Information Engineering > Electronic Thesis & Dissertation > Item 987654321/95530

Please use this identifier to cite or link to this item: http://ir.lib.ncu.edu.tw/handle/987654321/95530

Title:	Sparse Butterfly Matrix Attention for Enhancing Transformer Performance
Authors:	邱柏瑋;Chiu, Po-Wei
Contributors:	資訊工程學系
Keywords:	深度學習;DeepLearning
Date:	2024-07-22
Issue Date:	2024-10-09 16:59:29 (UTC+8)
Publisher:	國立中央大學
Abstract:	隨著處理長序列數據需求的增加，許多研究集中於提高模型性能和效率。在這項研究中，我們提出了一種基於 Transformer 架構的高效模型——Butterflyer。Butterflyer 旨在處理圖像、文本、路徑數據和數學運算等長序列數據。我們引入了一種創新的 Butterfly Attention 機制，該機制利用 Butterfly 矩陣的計算來替代傳統的自注意力機制，從而計算和捕捉不同的交互模式，提高了模型的性能。通過採用 Sophia 優化器，我們進一步提升了 Butterflyer 的訓練效率和性能。我們使用 Long Range Arena (LRA)基準數據集來評估 Butterflyer 的性能。實驗結果顯示，Butterflyer 在各種應用中表現出色，尤其在文本分類、圖像分類和文檔檢索等任務中，超越了目前最先進的模型。 ;With the increasing demand for handling long sequential data, many studies have focused on improving model performance and efficiency. In this research, we propose Butterflyer, an efficient model based on the Transformer architecture. Butterflyer is designed to process long sequential data such as images, text, path data, and mathematical operations. We introduce an innovative Butterfly-Attention mech anism that replaces the traditional self-attention mechanism, utilizing Butterfly Matrices to compute and capture different interaction patterns, thereby enhancing computational efficiency and model performance. By employing the Sophia opti mizer, we further improve the training efficiency and performance of Butterflyer. We evaluate Butterflyer’s performance using the Long Range Arena (LRA) bench mark dataset. Experimental results show that Butterflyer demonstrates superior performance across various applications, particularly in tasks such as text classifi cation, image classification, and document retrieval, outperforming state-of-the-art models.
Appears in Collections:	[Graduate Institute of Computer Science and Information Engineering] Electronic Thesis & Dissertation

Files in This Item:

File	Description	Size	Format
index.html		0Kb	HTML	36	View/Open

社群 sharing

Loading...