LIBMoE：一個用於在大型語言模型中進行綜合基準測試的專家混合庫

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models

November 1, 2024

作者: Nam V. Nguyen, Thong T. Doan, Luong Tran, Van Nguyen, Quang Pham

cs.AI

摘要

專家混合（MoEs）在更高效和有效的大型語言模型（LLMs）的發展中扮演著重要角色。由於巨大的資源需求，研究大規模MoE算法對許多研究人員來說是難以接觸的。本研究開發了LibMoE，一個全面且模塊化的框架，以簡化MoE算法的研究、訓練和評估。基於三個核心原則：（i）模塊化設計，（ii）高效訓練；（iii）全面評估，LibMoE通過標準化訓練和評估流程，使MoE在LLMs中更容易接觸到廣泛的研究人員。使用LibMoE，我們在零-shot設置下對三種不同的LLMs和11個數據集上廣泛評估了五種最先進的MoE算法。結果顯示，儘管具有獨特特性，所有MoE算法在廣泛任務範圍內平均表現大致相似。憑藉模塊化設計和全面評估，我們相信LibMoE將對研究人員朝著下一代MoE和LLMs取得有意義進展具有重要價值。項目頁面：https://fsoft-aic.github.io/fsoft-LibMoE.github.io。

English

Mixture of Experts (MoEs) plays an important role in the development of more efficient and effective large language models (LLMs). Due to the enormous resource requirements, studying large scale MoE algorithms remain in-accessible to many researchers. This work develops LibMoE, a comprehensive and modular framework to streamline the research, training, and evaluation of MoE algorithms. Built upon three core principles: (i) modular design, (ii) efficient training; (iii) comprehensive evaluation, LibMoE brings MoE in LLMs more accessible to a wide range of researchers by standardizing the training and evaluation pipelines. Using LibMoE, we extensively benchmarked five state-of-the-art MoE algorithms over three different LLMs and 11 datasets under the zero-shot setting. The results show that despite the unique characteristics, all MoE algorithms perform roughly similar when averaged across a wide range of tasks. With the modular design and extensive evaluation, we believe LibMoE will be invaluable for researchers to make meaningful progress towards the next generation of MoE and LLMs. Project page: https://fsoft-aic.github.io/fsoft-LibMoE.github.io.