OLMoTrace：将语言模型输出追溯至数万亿训练标记

OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens

April 9, 2025

作者: Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, YenSung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Cheng, Karen Farley, Sruthi Sreeram, Taira Anderson, David Albright, Carissa Schoenick, Luca Soldaini, Dirk Groeneveld, Rock Yuren Pang, Pang Wei Koh, Noah A. Smith, Sophie Lebrecht, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi, Jesse Dodge

cs.AI

摘要

我们推出OLMoTrace，这是首个能够实时追踪语言模型输出至其完整、数万亿token训练数据的系统。OLMoTrace能够发现并展示语言模型输出片段与训练文本语料库中文献之间的逐字匹配。依托于infini-gram（Liu等人，2024）的扩展版本，我们的系统能在数秒内返回追踪结果。OLMoTrace有助于用户通过训练数据的视角理解语言模型的行为。我们展示了如何利用它来探索事实核查、幻觉现象以及语言模型的创造力。OLMoTrace现已公开，且完全开源。

English

We present OLMoTrace, the first system that traces the outputs of language models back to their full, multi-trillion-token training data in real time. OLMoTrace finds and shows verbatim matches between segments of language model output and documents in the training text corpora. Powered by an extended version of infini-gram (Liu et al., 2024), our system returns tracing results within a few seconds. OLMoTrace can help users understand the behavior of language models through the lens of their training data. We showcase how it can be used to explore fact checking, hallucination, and the creativity of language models. OLMoTrace is publicly available and fully open-source.