小型語言模型研究概況

A Survey of Small Language Models

October 25, 2024

作者: Chien Van Nguyen, Xuan Shen, Ryan Aponte, Yu Xia, Samyadeep Basu, Zhengmian Hu, Jian Chen, Mihir Parmar, Sasidhar Kunapuli, Joe Barrow, Junda Wu, Ashish Singh, Yu Wang, Jiuxiang Gu, Franck Dernoncourt, Nesreen K. Ahmed, Nedim Lipka, Ruiyi Zhang, Xiang Chen, Tong Yu, Sungchul Kim, Hanieh Deilamsalehy, Namyong Park, Mike Rimer, Zhehao Zhang, Huanrui Yang, Ryan A. Rossi, Thien Huu Nguyen

cs.AI

摘要

由於小型語言模型（SLMs）在各種語言任務中以最少的計算資源展現出的效率和性能，因此它們變得越來越重要，適用於各種場景，包括設備內部、移動設備、邊緣設備等。本文提出了一份關於SLMs的全面調查，著重於它們的架構、訓練技術和模型壓縮技術。我們提出了一個新的分類法，用於將用於優化SLMs的方法進行分類，包括模型壓縮、修剪和量化技術。我們總結了用於對SLMs進行基準測試的基準數據集，以及常用的評估指標。此外，我們強調了尚待解決的關鍵挑戰。我們的調查旨在成為對開發和部署小型而高效的語言模型感興趣的研究人員和從業者的寶貴資源。

English

Small Language Models (SLMs) have become increasingly important due to their efficiency and performance to perform various language tasks with minimal computational resources, making them ideal for various settings including on-device, mobile, edge devices, among many others. In this article, we present a comprehensive survey on SLMs, focusing on their architectures, training techniques, and model compression techniques. We propose a novel taxonomy for categorizing the methods used to optimize SLMs, including model compression, pruning, and quantization techniques. We summarize the benchmark datasets that are useful for benchmarking SLMs along with the evaluation metrics commonly used. Additionally, we highlight key open challenges that remain to be addressed. Our survey aims to serve as a valuable resource for researchers and practitioners interested in developing and deploying small yet efficient language models.