OLMoTrace：將語言模型輸出追溯至數萬億訓練詞元

OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens

April 9, 2025

作者: Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, YenSung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Cheng, Karen Farley, Sruthi Sreeram, Taira Anderson, David Albright, Carissa Schoenick, Luca Soldaini, Dirk Groeneveld, Rock Yuren Pang, Pang Wei Koh, Noah A. Smith, Sophie Lebrecht, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi, Jesse Dodge

cs.AI

摘要

我們推出OLMoTrace，這是首個能夠即時追蹤語言模型輸出至其完整、多兆詞元訓練數據的系統。OLMoTrace能夠發現並展示語言模型輸出片段與訓練文本語料庫中文件之間的字面匹配。借助於infini-gram（Liu等人，2024）的擴展版本，我們的系統能在幾秒內返回追蹤結果。OLMoTrace有助於用戶透過其訓練數據的視角來理解語言模型的行為。我們展示了如何利用它來探索事實核查、幻覺以及語言模型的創造性。OLMoTrace已公開並完全開源。

English

We present OLMoTrace, the first system that traces the outputs of language models back to their full, multi-trillion-token training data in real time. OLMoTrace finds and shows verbatim matches between segments of language model output and documents in the training text corpora. Powered by an extended version of infini-gram (Liu et al., 2024), our system returns tracing results within a few seconds. OLMoTrace can help users understand the behavior of language models through the lens of their training data. We showcase how it can be used to explore fact checking, hallucination, and the creativity of language models. OLMoTrace is publicly available and fully open-source.