關於大型語言模型在邏輯推理中的記憶化

On Memorization of Large Language Models in Logical Reasoning

October 30, 2024

作者: Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, Ravi Kumar

cs.AI

摘要

大型語言模型（LLMs）在具有挑戰性的推理基準上表現出色，但也可能出現基本推理錯誤。當涉及理解LLMs推理能力背後的機制時，這種對比行為令人困惑。一個假設是，對常見推理基準的性能日益高且幾乎飽和可能是由於記憶類似問題所致。本文系統地探討了這一假設，使用基於“騎士和說謊者”（K&K）謎題的動態生成的邏輯推理基準，量化測量了推理任務中的記憶現象。我們發現，LLMs在微調後可以插值訓練謎題（實現接近完美的準確性），但在這些謎題稍作變動時失敗，表明模型在解決這些訓練謎題時嚴重依賴記憶。另一方面，我們展示，雖然微調導致大量記憶，但也始終改善了泛化性能。通過擾動測試、跨難度級別可轉移性、探測模型內部和使用錯誤答案進行微調的深入分析，我們表明LLMs學會在K&K謎題上推理，盡管訓練數據存在記憶現象。這種現象表明LLMs展現出記憶和真正推理能力之間的複雜相互作用。最後，我們的逐樣本記憶分數分析揭示了LLMs在解決邏輯謎題時如何在推理和記憶之間切換。我們的代碼和數據可在https://memkklogic.github.io 上獲得。

English

Large language models (LLMs) achieve good performance on challenging reasoning benchmarks, yet could also make basic reasoning mistakes. This contrasting behavior is puzzling when it comes to understanding the mechanisms behind LLMs' reasoning capabilities. One hypothesis is that the increasingly high and nearly saturated performance on common reasoning benchmarks could be due to the memorization of similar problems. In this paper, we systematically investigate this hypothesis with a quantitative measurement of memorization in reasoning tasks, using a dynamically generated logical reasoning benchmark based on Knights and Knaves (K&K) puzzles. We found that LLMs could interpolate the training puzzles (achieving near-perfect accuracy) after fine-tuning, yet fail when those puzzles are slightly perturbed, suggesting that the models heavily rely on memorization to solve those training puzzles. On the other hand, we show that while fine-tuning leads to heavy memorization, it also consistently improves generalization performance. In-depth analyses with perturbation tests, cross difficulty-level transferability, probing model internals, and fine-tuning with wrong answers suggest that the LLMs learn to reason on K&K puzzles despite training data memorization. This phenomenon indicates that LLMs exhibit a complex interplay between memorization and genuine reasoning abilities. Finally, our analysis with per-sample memorization score sheds light on how LLMs switch between reasoning and memorization in solving logical puzzles. Our code and data are available at https://memkklogic.github.io.