使用分段交叉注意力和内容丰富的视频数据整理生成长视频传播

Long Video Diffusion Generation with Segmented Cross-Attention and Content-Rich Video Data Curation

December 2, 2024

作者: Xin Yan, Yuxuan Cai, Qiuyue Wang, Yuan Zhou, Wenhao Huang, Huan Yang

cs.AI

摘要

我们介绍了Presto，这是一种新颖的视频扩散模型，旨在生成具有长程连贯性和丰富内容的15秒视频。将视频生成方法扩展到长时间段以保持情景多样性带来了重大挑战。为了解决这个问题，我们提出了分段交叉注意（SCA）策略，它将隐藏状态沿时间维度分割为段，使每个段可以与相应的子标题进行交叉注意。SCA不需要额外的参数，可以无缝地融入当前基于DiT的架构中。为了促进高质量的长视频生成，我们构建了LongTake-HD数据集，包括261k个内容丰富的视频，具有情景连贯性，并附带整体视频标题和五个渐进式子标题。实验表明，我们的Presto在VBench语义得分上达到了78.5%，在动态度上达到了100%，优于现有最先进的视频生成方法。这表明我们提出的Presto显著增强了内容丰富性，保持了长程连贯性，并捕捉了复杂的文本细节。更多详细信息请查看我们的项目页面：https://presto-video.github.io/。

English

We introduce Presto, a novel video diffusion model designed to generate 15-second videos with long-range coherence and rich content. Extending video generation methods to maintain scenario diversity over long durations presents significant challenges. To address this, we propose a Segmented Cross-Attention (SCA) strategy, which splits hidden states into segments along the temporal dimension, allowing each segment to cross-attend to a corresponding sub-caption. SCA requires no additional parameters, enabling seamless incorporation into current DiT-based architectures. To facilitate high-quality long video generation, we build the LongTake-HD dataset, consisting of 261k content-rich videos with scenario coherence, annotated with an overall video caption and five progressive sub-captions. Experiments show that our Presto achieves 78.5% on the VBench Semantic Score and 100% on the Dynamic Degree, outperforming existing state-of-the-art video generation methods. This demonstrates that our proposed Presto significantly enhances content richness, maintains long-range coherence, and captures intricate textual details. More details are displayed on our project page: https://presto-video.github.io/.