LoLDU：透過下三角-對角矩陣-上三角分解進行低秩適應，以實現參數高效微調。

LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning

October 17, 2024

作者: Yiming Shi, Jiwei Wei, Yujia Wu, Ran Ran, Chengwei Sun, Shiyuan He, Yang Yang

cs.AI

摘要

模型規模的快速增長需要大量的計算資源進行微調。現有方法如低秩適應（LoRA）已經開始解決處理完全微調中的大量更新參數的問題。然而，LoRA利用隨機初始化和低秩矩陣的優化來近似更新權重，這可能導致次優的收斂和準確性差距，相較於完全微調。為了解決這些問題，我們提出了LoLDU，一種參數高效微調（PEFT）方法，與常規PEFT方法相比，可將可訓練參數減少2600倍，同時保持可比的性能。LoLDU利用下三角-對角-上三角分解（LDU）來初始化低秩矩陣，以實現更快的收斂和正交性。我們專注於優化對角矩陣以進行比例轉換。據我們所知，LoLDU在所有PEFT方法中具有最少的參數。我們在4個指示遵循數據集、6個自然語言理解（NLU）數據集、8個圖像分類數據集和多個模型類型（LLaMA2、RoBERTa、ViT和Stable Diffusion）的圖像生成數據集上進行了廣泛實驗，提供了全面和詳細的分析。我們的開源代碼可在以下網址訪問：https://github.com/SKDDJ/LoLDU。

English

The rapid growth of model scale has necessitated substantial computational resources for fine-tuning. Existing approach such as Low-Rank Adaptation (LoRA) has sought to address the problem of handling the large updated parameters in full fine-tuning. However, LoRA utilize random initialization and optimization of low-rank matrices to approximate updated weights, which can result in suboptimal convergence and an accuracy gap compared to full fine-tuning. To address these issues, we propose LoLDU, a Parameter-Efficient Fine-Tuning (PEFT) approach that significantly reduces trainable parameters by 2600 times compared to regular PEFT methods while maintaining comparable performance. LoLDU leverages Lower-Diag-Upper Decomposition (LDU) to initialize low-rank matrices for faster convergence and orthogonality. We focus on optimizing the diagonal matrix for scaling transformations. To the best of our knowledge, LoLDU has the fewest parameters among all PEFT approaches. We conducted extensive experiments across 4 instruction-following datasets, 6 natural language understanding (NLU) datasets, 8 image classification datasets, and image generation datasets with multiple model types (LLaMA2, RoBERTa, ViT, and Stable Diffusion), providing a comprehensive and detailed analysis. Our open-source code can be accessed at https://github.com/SKDDJ/LoLDU{https://github.com/SKDDJ/LoLDU}.