M2rc-Eval：大規模多語言存儲庫級別的程式碼完成評估

M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation

October 28, 2024

作者: Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai, Yanan Wu, Ke Jin, Ge Zhang, Zekun Wang, Guoan Zhang, Bangyu Xiang, Wenbo Su, Bo Zheng

cs.AI

摘要

在軟體工程領域，存儲庫級別的程式碼自動完成引起了廣泛關注，並引入了幾個基準數據集。然而，現有的存儲庫級別的程式碼自動完成基準通常僅專注於有限數量的語言（<5），這無法評估現有程式碼大型語言模型（LLMs）在不同語言間的一般程式碼智能能力。此外，現有的基準通常報告不同語言的整體平均分數，忽略了在不同完成情境中的細粒度能力。因此，為了促進多語言情境下程式碼LLMs的研究，我們提出了一個涵蓋18種程式設計語言的大規模多語言存儲庫級別程式碼自動完成基準（稱為M2RC-EVAL），並提供了兩種細粒度註釋（即桶級別和語義級別）在不同完成情境下，我們基於解析的抽象語法樹獲取這些註釋。此外，我們還整理了一個大規模多語言指令語料庫M2RC-INSTRUCT數據集，以提高現有程式碼LLMs的存儲庫級別程式碼自動完成能力。全面的實驗結果證明了我們的M2RC-EVAL和M2RC-INSTRUCT的有效性。

English

Repository-level code completion has drawn great attention in software engineering, and several benchmark datasets have been introduced. However, existing repository-level code completion benchmarks usually focus on a limited number of languages (<5), which cannot evaluate the general code intelligence abilities across different languages for existing code Large Language Models (LLMs). Besides, the existing benchmarks usually report overall average scores of different languages, where the fine-grained abilities in different completion scenarios are ignored. Therefore, to facilitate the research of code LLMs in multilingual scenarios, we propose a massively multilingual repository-level code completion benchmark covering 18 programming languages (called M2RC-EVAL), and two types of fine-grained annotations (i.e., bucket-level and semantic-level) on different completion scenarios are provided, where we obtain these annotations based on the parsed abstract syntax tree. Moreover, we also curate a massively multilingual instruction corpora M2RC- INSTRUCT dataset to improve the repository-level code completion abilities of existing code LLMs. Comprehensive experimental results demonstrate the effectiveness of our M2RC-EVAL and M2RC-INSTRUCT.