VidTok：一种多功能且开源的视频分词器

VidTok: A Versatile and Open-Source Video Tokenizer

December 17, 2024

作者: Anni Tang, Tianyu He, Junliang Guo, Xinle Cheng, Li Song, Jiang Bian

cs.AI

摘要

将视频内容编码为紧凑的潜在标记已成为视频生成和理解中的基本步骤，这是因为需要解决像素级表示中固有的冗余。因此，随着视频中心研究日益突出，对高性能、开源视频标记器的需求不断增长。我们介绍了 VidTok，这是一种多功能视频标记器，在连续和离散标记化方面均提供了最先进的性能。VidTok相较于现有方法融合了几项关键进展：1）模型架构，如卷积层和上/下采样模块；2）为解决传统矢量量化（VQ）常见的训练不稳定性和码书崩溃问题，我们将有限标量量化（FSQ）融入到离散视频标记化中；3）改进的训练策略，包括两阶段训练过程和使用降低帧率。通过整合这些进展，VidTok在现有方法基础上取得了显著的改进，在标准化评估设置下，在多个指标（包括PSNR、SSIM、LPIPS和FVD）上展现出卓越的性能。

English

Encoding video content into compact latent tokens has become a fundamental step in video generation and understanding, driven by the need to address the inherent redundancy in pixel-level representations. Consequently, there is a growing demand for high-performance, open-source video tokenizers as video-centric research gains prominence. We introduce VidTok, a versatile video tokenizer that delivers state-of-the-art performance in both continuous and discrete tokenizations. VidTok incorporates several key advancements over existing approaches: 1) model architecture such as convolutional layers and up/downsampling modules; 2) to address the training instability and codebook collapse commonly associated with conventional Vector Quantization (VQ), we integrate Finite Scalar Quantization (FSQ) into discrete video tokenization; 3) improved training strategies, including a two-stage training process and the use of reduced frame rates. By integrating these advancements, VidTok achieves substantial improvements over existing methods, demonstrating superior performance across multiple metrics, including PSNR, SSIM, LPIPS, and FVD, under standardized evaluation settings.