在聯邦學習中的低秩適應中的選擇性聚合

Selective Aggregation for Low-Rank Adaptation in Federated Learning

October 2, 2024

作者: Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, Liangqiong Qu

cs.AI

摘要

我們透過對學習的 A 和 B 矩陣的不對稱分析，探討了在聯邦學習中 LoRA 的情況。在這個過程中，我們發現 A 矩陣負責學習一般知識，而 B 矩陣則專注於捕捉客戶特定知識。基於這一發現，我們提出了Federated Share-A Low-Rank Adaptation（FedSA-LoRA），該方法採用兩個低秩可訓練的矩陣 A 和 B 來建模權重更新，但只有 A 矩陣與伺服器共享進行聚合。此外，我們深入研究了在其他 LoRA 變體（如 rsLoRA 和 VeRA）中學習的 A 和 B 矩陣之間的關係，揭示了一致的模式。因此，我們將我們的 FedSA-LoRA 方法擴展到這些 LoRA 變體，得到了 FedSA-rsLoRA 和 FedSA-VeRA。通過這種方式，我們建立了一個將 LoRA 與 FL 整合的通用範式，為未來在後續 LoRA 變體與 FL 結合的工作提供指導。在自然語言理解和生成任務上的大量實驗結果證明了所提方法的有效性。

English

We investigate LoRA in federated learning through the lens of the asymmetry analysis of the learned A and B matrices. In doing so, we uncover that A matrices are responsible for learning general knowledge, while B matrices focus on capturing client-specific knowledge. Based on this finding, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices A and B to model the weight update, but only A matrices are shared with the server for aggregation. Moreover, we delve into the relationship between the learned A and B matrices in other LoRA variants, such as rsLoRA and VeRA, revealing a consistent pattern. Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL. Extensive experimental results on natural language understanding and generation tasks demonstrate the effectiveness of the proposed method.