選擇性注意力提升了Transformer效能。

Selective Attention Improves Transformer

October 3, 2024

作者: Yaniv Leviathan, Matan Kalman, Yossi Matias

cs.AI

摘要

在注意力機制的背景中不必要的元素會降低性能。我們引入了選擇性注意力，這是對標準注意力機制的一個簡單且無需參數的改變，可減少對不需要的元素的關注。選擇性注意力提高了各種模型大小和上下文長度下的語言建模性能。例如，在C4上以語言建模目標訓練的一系列變壓器，搭配選擇性注意力的性能與標準變壓器相當，而後者在其注意力模塊中擁有約2倍的頭數和參數。選擇性注意力還允許減少注意力上下文緩衝區的大小，在推論過程中降低了記憶體和計算需求。例如，在C4上訓練的具有1億參數的變壓器，當配備選擇性注意力時，其注意力模塊的記憶體需求分別比沒有選擇性注意力的模型少了16倍、25倍和47倍，並且具有相同的驗證困惑度。

English

Unneeded elements in the attention's context degrade performance. We introduce Selective Attention, a simple parameter-free change to the standard attention mechanism which reduces attention to unneeded elements. Selective attention improves language modeling performance in a variety of model sizes and context lengths. For example, a range of transformers trained with the language modeling objective on C4 with selective attention perform equivalently to standard transformers with ~2X more heads and parameters in their attention modules. Selective attention also allows decreasing the size of the attention's context buffer, leading to meaningful reductions in the memory and compute requirements during inference. For example, transformers with 100M parameters trained on C4 with context sizes of 512, 1,024, and 2,048 need 16X, 25X, and 47X less memory for their attention module, respectively, when equipped with selective attention, as those without selective attention, with the same validation perplexity.