下一个标记预测中的物理学

Physics in Next-token Prediction

November 1, 2024

作者: Hongjun An, Yiliang Song, Xuelong Li

cs.AI

摘要

我们发现了Next-token Prediction（NTP）中的基础物理学。我们确定了NTP中信息守恒定律，并提出了信息容量第一定律（IC-1），证明自回归模型中智能出现的本质基本上是信息传递的过程。我们还将兰道尔原理引入NTP，制定了信息容量第二定律（IC-2），建立了自回归模型训练与能量消耗之间的关系。此外，我们提出了几个推论，对生产实践具有实际意义。最后，我们验证了我们的发现与现有理论的兼容性和互补性。

English

We discovered the underlying physics in Next-token Prediction (NTP). We identified the law of information conservation within NTP and proposed the First Law of Information Capacity (IC-1), demonstrating that the essence of intelligence emergence in auto-regressive models is fundamentally a process of information transfer. We also introduced Landauer's Principle into NTP, formulating the Second Law of Information Capacity (IC-2), which establishes the relationship between auto-regressive model training and energy consumption. Additionally, we presented several corollaries, which hold practical significance for production practices. Finally, we validated the compatibility and complementarity of our findings with existing theories.