LIBMoE：一个用于在大型语言模型中进行混合专家全面基准测试的库。

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models

November 1, 2024

作者: Nam V. Nguyen, Thong T. Doan, Luong Tran, Van Nguyen, Quang Pham

cs.AI

摘要

专家混合模型（MoEs）在更高效和有效的大型语言模型（LLMs）的发展中发挥着重要作用。由于巨大的资源需求，研究大规模MoE算法仍然对许多研究人员不可及。本研究开发了LibMoE，这是一个全面且模块化的框架，旨在简化MoE算法的研究、训练和评估。基于三个核心原则：（i）模块化设计，（ii）高效训练；（iii）全面评估，LibMoE通过标准化训练和评估流程，使MoE在LLMs中更容易接触到广泛的研究人员。利用LibMoE，我们在零样本设置下对三种不同LLMs和11个数据集上的五种最先进的MoE算法进行了广泛基准测试。结果表明，尽管具有独特特征，所有MoE算法在广泛任务范围内平均表现大致相似。凭借模块化设计和全面评估，我们相信LibMoE将对研究人员朝着下一代MoE和LLMs取得有意义进展具有重要价值。项目页面：https://fsoft-aic.github.io/fsoft-LibMoE.github.io。

English

Mixture of Experts (MoEs) plays an important role in the development of more efficient and effective large language models (LLMs). Due to the enormous resource requirements, studying large scale MoE algorithms remain in-accessible to many researchers. This work develops LibMoE, a comprehensive and modular framework to streamline the research, training, and evaluation of MoE algorithms. Built upon three core principles: (i) modular design, (ii) efficient training; (iii) comprehensive evaluation, LibMoE brings MoE in LLMs more accessible to a wide range of researchers by standardizing the training and evaluation pipelines. Using LibMoE, we extensively benchmarked five state-of-the-art MoE algorithms over three different LLMs and 11 datasets under the zero-shot setting. The results show that despite the unique characteristics, all MoE algorithms perform roughly similar when averaged across a wide range of tasks. With the modular design and extensive evaluation, we believe LibMoE will be invaluable for researchers to make meaningful progress towards the next generation of MoE and LLMs. Project page: https://fsoft-aic.github.io/fsoft-LibMoE.github.io.