結合跨尺度自注意力與分割混合層之輕量化分類網路;CSASML: Combining Cross-Scale Attention with Split-and-Mixed Layer for Lightweight Classification Network

NCU Institutional Repository > 資訊電機學院 > 資訊工程研究所 > 博碩士論文 > Item 987654321/95423

jsp.display-item.identifier=請使用永久網址來引用或連結此文件: http://ir.lib.ncu.edu.tw/handle/987654321/95423

题名:	結合跨尺度自注意力與分割混合層之輕量化分類網路;CSASML: Combining Cross-Scale Attention with Split-and-Mixed Layer for Lightweight Classification Network
作者:	廖柏諭;Liao, Po-Yu
贡献者:	資訊工程學系
关键词:	輕量化;自注意力;Lightweight;Vision Transformer;Self-attention
日期:	2024-07-02
上传时间:	2024-10-09 16:47:44 (UTC+8)
出版者:	國立中央大學
摘要:	近幾年神經網路的崛起打開人工智慧與產業的結合為人類文明邁入下一個里程碑，其中卷積神經網路與Transformer的能力讓世界有目共睹，但後者比起前者龐大的計算量導致其無法在移動端設備上順暢執行，因此輕量化技術成為熱門研究目標。針對Transformer輕量化主要有幾個方向，像是自注意力機制計算複雜度由二次方變成線性，做法為減少需計算的輸入特徵降低計算量，但我們拋棄的特徵也有相關性，因此輕量化後勢必會降低模型預測能力，需要設計一個強化特徵表示能力的方法補足模型精度，如何平行兩者讓整體效能維持不變或是提升的情況下浮點數計算變少將是我們的目標。我們提出輕量化分割網路方法Split-and-Mixed Module與提取跨尺度特徵方法Cross-Scale Attention Module，針對分類任務設計出骨架網路CSASML，經過實驗證明我們能降低25%計算量在CIFAR100資料集提升1.8%分類準確度，這些模組可加進其他分類網路也能達到同樣效能，泛用性高。;In recent years, the rise of neural networks has ushered in a new milestone in the integration of artificial intelligence with industry, advancing human civilization to the next stage. The capabilities of convolutional neural networks (CNNs) and Transformers have been widely recognized, but the latter′s massive computational requirements hinder smooth execution on mobile devices. Consequently, lightweight techniques have become a popular research focus. There are several approaches to light-weighting Transformers, such as reducing the computational complexity of the self-attention mechanism from quadratic to linear. This can be achieved by decreasing the input features that need to be computed, thereby reducing the computational load. However, the discarded features also hold relevance, meaning that light-weighting inevitably diminishes the model′s predictive capability. It is necessary to devise a method to enhance feature representation to compensate for the loss in model accuracy. Our goal is to balance both aspects, maintaining or improving overall performance while reducing the number of floating-point operations. We propose a lightweight split network method called Split-and-Mixed Module, along with a method for extracting cross-scale features, the Cross-Scale Attention Module. For classification tasks, we designed CSASML backbone. Experimental results demonstrate that we can reduce computational load by 25% and improve classification accuracy by 1.8% on the CIFAR100 dataset. These modules can also be integrated into other classification networks to achieve similar performance improvements, highlighting their high versatility.
显示于类别:	[資訊工程研究所] 博碩士論文

文件中的档案:

档案	描述	大小	格式	浏览次数
index.html		0Kb	HTML	30	检视/开启

在NCUIR中所有的数据项都受到原著作权保护.

社群 sharing

数据加载中.....