NoisyRollout：通过数据增强强化视觉推理

NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation

April 17, 2025

作者: Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, Michael Qizhe Shieh

cs.AI

摘要

近期强化学习（RL）的进展显著增强了视觉-语言模型（VLMs）的推理能力。然而，在VLMs中，如何提升策略探索以更有效地扩展测试时计算资源仍待深入探索。此外，VLMs在应对不完美的视觉感知方面持续面临挑战，这进而影响了后续的推理过程。为此，我们提出了NoisyRollout，一种简单而有效的RL方法，它通过混合来自清晰图像和适度失真图像的轨迹，在视觉感知及由此产生的推理模式中引入有针对性的多样性。无需额外训练成本，NoisyRollout通过融入视觉导向的归纳偏置，增强了VLMs的探索能力。此外，NoisyRollout采用了一种噪声退火调度策略，在训练过程中逐步降低失真强度，确保早期从噪声信号中获益，同时保持后期训练的稳定性和可扩展性。仅使用2.1K训练样本，NoisyRollout在涵盖推理与感知任务的5个域外基准测试中，实现了开源RL调优模型中的最先进性能，同时保持了相当甚至更优的域内性能。

English

Recent advances in reinforcement learning (RL) have strengthened the reasoning capabilities of vision-language models (VLMs). However, enhancing policy exploration to more effectively scale test-time compute remains underexplored in VLMs. In addition, VLMs continue to struggle with imperfect visual perception, which in turn affects the subsequent reasoning process. To this end, we propose NoisyRollout, a simple yet effective RL approach that mixes trajectories from both clean and moderately distorted images to introduce targeted diversity in visual perception and the resulting reasoning patterns. Without additional training cost, NoisyRollout enhances the exploration capabilities of VLMs by incorporating a vision-oriented inductive bias. Furthermore, NoisyRollout employs a noise annealing schedule that gradually reduces distortion strength over training, ensuring benefit from noisy signals early while maintaining training stability and scalability in later stages. With just 2.1K training samples, NoisyRollout achieves state-of-the-art performance among open-source RL-tuned models on 5 out-of-domain benchmarks spanning both reasoning and perception tasks, while preserving comparable or even better in-domain performance.