並行自回歸視覺生成

Parallelized Autoregressive Visual Generation

December 19, 2024

作者: Yuqing Wang, Shuhuai Ren, Zhijie Lin, Yujin Han, Haoyuan Guo, Zhenheng Yang, Difan Zou, Jiashi Feng, Xihui Liu

cs.AI

摘要

自回歸模型已成為視覺生成的強大方法，但由於其逐個預測過程，推理速度較慢。在本文中，我們提出了一種簡單而有效的並行自回歸視覺生成方法，提高了生成效率，同時保留了自回歸建模的優勢。我們的關鍵洞察是，並行生成取決於視覺標記之間的依賴關係-具有弱依賴性的標記可以並行生成，而具有強依賴性的相鄰標記難以一起生成，因為它們的獨立抽樣可能導致不一致。基於這一觀察，我們開發了一種並行生成策略，可以並行生成具有弱依賴性的遠程標記，同時對具有強依賴性的本地標記進行順序生成。我們的方法可以無縫集成到標準自回歸模型中，而無需修改架構或標記器。在ImageNet和UCF-101上的實驗表明，我們的方法實現了3.6倍的加速，並在圖像和視頻生成任務中實現了最多9.5倍的加速，而品質幾乎沒有下降。我們希望這項工作能激發未來在高效視覺生成和統一自回歸建模方面的研究。項目頁面: https://epiphqny.github.io/PAR-project.

English

Autoregressive models have emerged as a powerful approach for visual generation but suffer from slow inference speed due to their sequential token-by-token prediction process. In this paper, we propose a simple yet effective approach for parallelized autoregressive visual generation that improves generation efficiency while preserving the advantages of autoregressive modeling. Our key insight is that parallel generation depends on visual token dependencies-tokens with weak dependencies can be generated in parallel, while strongly dependent adjacent tokens are difficult to generate together, as their independent sampling may lead to inconsistencies. Based on this observation, we develop a parallel generation strategy that generates distant tokens with weak dependencies in parallel while maintaining sequential generation for strongly dependent local tokens. Our approach can be seamlessly integrated into standard autoregressive models without modifying the architecture or tokenizer. Experiments on ImageNet and UCF-101 demonstrate that our method achieves a 3.6x speedup with comparable quality and up to 9.5x speedup with minimal quality degradation across both image and video generation tasks. We hope this work will inspire future research in efficient visual generation and unified autoregressive modeling. Project page: https://epiphqny.github.io/PAR-project.