联邦素描 LoRA：大型语言模型的设备端协作微调

摘要

在设备上对大型语言模型（LLMs）进行微调越来越受到关注。最近的研究将低秩适应（LoRA）技术与联邦微调相结合，以减轻与设备模型大小和数据稀缺性相关的挑战。然而，计算资源的异质性仍然是一个关键瓶颈：尽管更高秩的模块通常会提高性能，但不同的设备能力限制了LoRA的可行秩范围。现有方法试图解决这个问题，但要么缺乏分析上的合理性，要么会增加额外的计算开销，为高效且理论基础的解决方案留下了很大的空间。为了解决这些挑战，我们提出了联邦草图LoRA（FSLoRA），它利用草图机制使设备能够选择性地更新服务器维护的全局LoRA模块的子矩阵。通过调整决定设备上子矩阵秩的草图比例，FSLoRA能够灵活适应设备特定的通信和计算约束。我们对FSLoRA进行了严格的收敛分析，描述了草图比例如何影响收敛速度。通过在多个数据集和LLM模型上进行全面实验，我们展示了FSLoRA相对于各种基线的卓越性能。

English

Fine-tuning large language models (LLMs) on devices is attracting increasing interest. Recent works have fused low-rank adaptation (LoRA) techniques with federated fine-tuning to mitigate challenges associated with device model sizes and data scarcity. Still, the heterogeneity of computational resources remains a critical bottleneck: while higher-rank modules generally enhance performance, varying device capabilities constrain LoRA's feasible rank range. Existing approaches attempting to resolve this issue either lack analytical justification or impose additional computational overhead, leaving a wide gap for an efficient and theoretically-grounded solution. To address these challenges, we propose federated sketching LoRA (FSLoRA), which leverages a sketching mechanism to enable devices to selectively update submatrices of global LoRA modules maintained by the server. By adjusting the sketching ratios, which determine the ranks of the submatrices on the devices, FSLoRA flexibly adapts to device-specific communication and computational constraints. We provide a rigorous convergence analysis of FSLoRA that characterizes how the sketching ratios affect the convergence rate. Through comprehensive experiments on multiple datasets and LLM models, we demonstrate FSLoRA's superior performance compared to various baselines.

联邦素描 LoRA：大型语言模型的设备端协作微调

Federated Sketching LoRA: On-Device Collaborative Fine-Tuning of Large Language Models

摘要

Summary

Support