通过推测性拒绝实现快速的前N个解码

Fast Best-of-N Decoding via Speculative Rejection

October 26, 2024

作者: Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, Andrea Zanette

cs.AI

摘要

大型语言模型（LLMs）的安全有效部署涉及一项关键步骤，称为对齐，该步骤确保模型的响应符合人类偏好。流行的对齐技术，如DPO、PPO及其变体，通过在后训练阶段改变预训练模型权重来对齐LLMs。虽然主流，但这些后训练方法在LLMs部署前增加了相当复杂性。推理时对齐方法避免了复杂的后训练步骤，而是偏向于生成与人类偏好一致的响应。最著名的推理时对齐方法称为Best-of-N，其效果与最先进的后训练程序一样。不幸的是，Best-of-N在推理时需要比标准解码策略更多的资源，这使其在计算上不可行。在这项工作中，我们引入了一种计算上可行的推理时对齐算法，名为Speculative Rejection。它根据给定的奖励模型生成高分响应，类似于Best-of-N，同时在计算效率上更高，效率提高了16到32倍。

English

The safe and effective deployment of Large Language Models (LLMs) involves a critical step called alignment, which ensures that the model's responses are in accordance with human preferences. Prevalent alignment techniques, such as DPO, PPO and their variants, align LLMs by changing the pre-trained model weights during a phase called post-training. While predominant, these post-training methods add substantial complexity before LLMs can be deployed. Inference-time alignment methods avoid the complex post-training step and instead bias the generation towards responses that are aligned with human preferences. The best-known inference-time alignment method, called Best-of-N, is as effective as the state-of-the-art post-training procedures. Unfortunately, Best-of-N requires vastly more resources at inference time than standard decoding strategies, which makes it computationally not viable. In this work, we introduce Speculative Rejection, a computationally-viable inference-time alignment algorithm. It generates high-scoring responses according to a given reward model, like Best-of-N does, while being between 16 to 32 times more computationally efficient.