Bielik 7B v0.1：一個波蘭語言模型--開發、洞察和評估

Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation

October 24, 2024

作者: Krzysztof Ociepa, Łukasz Flis, Krzysztof Wróbel, Adrian Gwoździej, Remigiusz Kinas

cs.AI

摘要

我們介紹了Bielik 7B v0.1，一個擁有70億參數的波蘭語生成文本模型，用於波蘭語言處理。通過在經過精心策劃的波蘭語語料庫上進行訓練，該模型通過創新技術應對語言模型開發中的關鍵挑戰。這些技術包括加權指令交叉熵損失，平衡不同指令類型的學習，以及自適應學習率，根據訓練進度動態調整學習率。為了評估性能，我們創建了Open PL LLM Leaderboard和Polish MT-Bench，這是評估各種自然語言處理任務和對話能力的新框架。Bielik 7B v0.1展示了顯著的改進，與Mistral-7B-v0.1在RAG Reader任務上相比，平均分數提高了9個百分點。它在Polish MT-Bench中表現優異，特別是在推理（6.15/10）和角色扮演（7.83/10）類別中。這個模型代表了波蘭語言人工智慧領域的重大進步，為各種語言應用提供了一個強大工具，並在該領域設定了新的基準。

English

We introduce Bielik 7B v0.1, a 7-billion-parameter generative text model for Polish language processing. Trained on curated Polish corpora, this model addresses key challenges in language model development through innovative techniques. These include Weighted Instruction Cross-Entropy Loss, which balances the learning of different instruction types, and Adaptive Learning Rate, which dynamically adjusts the learning rate based on training progress. To evaluate performance, we created the Open PL LLM Leaderboard and Polish MT-Bench, novel frameworks assessing various NLP tasks and conversational abilities. Bielik 7B v0.1 demonstrates significant improvements, achieving a 9 percentage point increase in average score compared to Mistral-7B-v0.1 on the RAG Reader task. It also excels in the Polish MT-Bench, particularly in Reasoning (6.15/10) and Role-playing (7.83/10) categories. This model represents a substantial advancement in Polish language AI, offering a powerful tool for diverse linguistic applications and setting new benchmarks in the field.