Bielik 7B v0.1：一种波兰语言模型 -- 开发、见解和评估

Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation

October 24, 2024

作者: Krzysztof Ociepa, Łukasz Flis, Krzysztof Wróbel, Adrian Gwoździej, Remigiusz Kinas

cs.AI

摘要

我们介绍了Bielik 7B v0.1，这是一个70亿参数的生成文本模型，用于波兰语处理。该模型在经过筛选的波兰语语料库上进行训练，通过创新技术解决了语言模型开发中的关键挑战。这些技术包括加权指令交叉熵损失，平衡不同指令类型的学习，以及自适应学习率，根据训练进展动态调整学习率。为了评估性能，我们创建了Open PL LLM排行榜和波兰MT-Bench，这是评估各种自然语言处理任务和对话能力的新框架。Bielik 7B v0.1展示了显著的改进，与Mistral-7B-v0.1在RAG Reader任务中的平均得分相比，提高了9个百分点。它在波兰MT-Bench中表现出色，特别是在推理（6.15/10）和角色扮演（7.83/10）类别中。该模型代表了波兰语言人工智能领域的重大进步，为各种语言应用提供了强大工具，并在该领域设立了新的基准。

English

We introduce Bielik 7B v0.1, a 7-billion-parameter generative text model for Polish language processing. Trained on curated Polish corpora, this model addresses key challenges in language model development through innovative techniques. These include Weighted Instruction Cross-Entropy Loss, which balances the learning of different instruction types, and Adaptive Learning Rate, which dynamically adjusts the learning rate based on training progress. To evaluate performance, we created the Open PL LLM Leaderboard and Polish MT-Bench, novel frameworks assessing various NLP tasks and conversational abilities. Bielik 7B v0.1 demonstrates significant improvements, achieving a 9 percentage point increase in average score compared to Mistral-7B-v0.1 on the RAG Reader task. It also excels in the Polish MT-Bench, particularly in Reasoning (6.15/10) and Role-playing (7.83/10) categories. This model represents a substantial advancement in Polish language AI, offering a powerful tool for diverse linguistic applications and setting new benchmarks in the field.