當以快速思考和慢速思考訓練LLM的層時發生了什麼：一個梯度的觀點

What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective

October 31, 2024

作者: Ming Li, Yanhong Li, Tianyi Zhou

cs.AI

摘要

在LLM後訓練中有何不同？我們通過梯度的角度研究大型語言模型（LLMs）中不同層的訓練模式，當使用不同回應和初始模型進行訓練時。我們特別關注快速思考與慢速思考如何影響層級梯度，鑒於最近在推理路徑（如CoT和過程獎勵）上訓練LLMs變得流行。在我們的研究中，沒有CoT的快速思考導致較大的梯度和跨層之間較大的梯度差異，這表明後者帶來的學習穩定性。此外，預訓練的LLMs受快速思考的不穩定性影響較小，而經過指導調整的LLMs則受到較大影響。此外，我們研究了當使用慢速思考路徑訓練不同LLMs時，梯度模式是否能反映回應的正確性。結果顯示，慢速思考的梯度可以區分正確和無關的推理路徑。作為比較，我們對非推理知識學習任務進行了類似的梯度分析，然而，在這些任務中，單純增加回應長度並不會導致慢速思考的類似行為。我們的研究加強了對LLM訓練的基本理解，並對其效率和穩定性提供了新的見解，為構建可泛化的System-2代理奠定了基礎。我們的代碼、數據和梯度統計可在以下鏈接找到：https://github.com/MingLiiii/Layer_Gradient。

English

What makes a difference in the post-training of LLMs? We investigate the training patterns of different layers in large language models (LLMs), through the lens of gradient, when training with different responses and initial models. We are specifically interested in how fast vs. slow thinking affects the layer-wise gradients, given the recent popularity of training LLMs on reasoning paths such as chain-of-thoughts (CoT) and process rewards. In our study, fast thinking without CoT leads to larger gradients and larger differences of gradients across layers than slow thinking (Detailed CoT), indicating the learning stability brought by the latter. Moreover, pre-trained LLMs are less affected by the instability of fast thinking than instruction-tuned LLMs. Additionally, we study whether the gradient patterns can reflect the correctness of responses when training different LLMs using slow vs. fast thinking paths. The results show that the gradients of slow thinking can distinguish correct and irrelevant reasoning paths. As a comparison, we conduct similar gradient analyses on non-reasoning knowledge learning tasks, on which, however, trivially increasing the response length does not lead to similar behaviors of slow thinking. Our study strengthens fundamental understandings of LLM training and sheds novel insights on its efficiency and stability, which pave the way towards building a generalizable System-2 agent. Our code, data, and gradient statistics can be found in: https://github.com/MingLiiii/Layer_Gradient.