在資源受限的邊緣裝置上執行語言模型(Language Model， LM)微調時，主要瓶頸在於反向傳播過程涉及龐大的梯度運算與儲存需求；此外，模型內含的倒數平方根、指數函數及SiLU等非線性超越函數，亦高度依賴專用硬體。針對上述挑戰，本論文基於TinyLlama 1.1B模型與量化感知低秩自適應(Quantization-Aware Low-Rank Adaptation， QA-LoRA)技術，提出一個客製化的反向傳播框架，將低精度量化、非線性函數近似與稀疏化策略整合至單一反向運算流程中進行驗證。實作上，本研究以矩陣形式首先推導各子模組的反向傳播路徑，確保資料格式轉換、近似函數及稀疏化機制能精確嵌入底層運算。在資料格式方面，提出具備動態指數偏置的動態對齊浮點數格式(Dynamically aligned Floating-Point format， DFP12)，在12位元精度下仍可維持模型收斂。在非線性運算上，針對倒數、倒數平方根、Softmax與SiLU採用了硬體友善的近似設計，並透過硬體資源共用策略，旨在徹底免除對超越函數專屬硬體的需求。運算量最佳化方面，則導入兼具粗、細粒度的稀疏化策略，並深入剖析SiLU誘導稀疏性與淺層凍結的耦合效應。實驗結果驗證了本框架的硬體效益與模型表現：在架構投影評估下，Softmax近似設計以每個元素額外一次乘法，換取約九次加法之免除，且查表記憶體僅需32 Bytes；SiLU近似函數的導數均方誤差改善約3.5 倍，並省下高達484 MB的激活值儲存空間。將整體框架應用於BoolQ與WinoGrande任務中，反向傳播計算量分別大幅減少41.11%與58.46%，而準確率仍保持在80.18%與71.03%，與BF16自動微分基準的差距控制在1%以內。此外，在生成式摘要任務Xsum中，計算量減免為38.2%，ROUGE-L分數由29.84%提升至31.28%，充分展現所提架構優異的泛化能力。;Fine-tuning Language Models (LMs) on edge devices is bottlenecked by massive gradient computations and storage demands during backpropagation. Additionally， transcendental functions (e.g.， inverse square root， exponentials， SiLU) heavily rely on dedicated hardware. We propose a customized backpropagation framework based on TinyLlama 1.1B Model and Quantization-Aware Low-Rank Adaptation (QA-LoRA)， seamlessly integrating low-precision quantization， non-linear approximations， and sparsity strategies into a unified backward propagation flow. By manually deriving matrix-form backward paths， data conversions， approximations， and sparsity mechanisms are precisely embedded into underlying computations. We propose a dynamically aligned floating-point format (DFP12) with dynamic exponent bias to maintain model convergence at 12-bit precision. Hardware-friendly approximations for reciprocal， inverse square root， Softmax， and SiLU leverage resource sharing to eliminate dedicated transcendental hardware. Workloads are optimized via coarse- and fine-grained sparsity， alongside analyzing the coupling effect of SiLU-induced sparsity and shallow-layer freezing. Results verify this framework′s efficiency and performance. Softmax approximation trades one multiplication for the elimination of nine additions， requiring only 32 Bytes of Look-Up Table (LUT) memory. SiLU derivative approximation improves mean squared error (MSE) by 3.5×， saving up to 484 MB of activation memory footprint. On BoolQ and WinoGrande tasks， backward computations are reduced by 41.11% and 58.46%， maintaining 80.18% and 71.03% accuracy， respectively， bounding the gap with the BF16 automatic differentiation baseline to within 1%. Furthermore， the computational load of Xsum task was reduced by 38.2%， and ROUGE-L score increases from 29.84% to 31.28%， demonstrating excellent generalization capability.
