In-2-4D：从两张单视图图像到四维生成的中间帧生成

In-2-4D: Inbetweening from Two Single-View Images to 4D Generation

April 11, 2025

作者: Sauradip Nag, Daniel Cohen-Or, Hao Zhang, Ali Mahdavi-Amiri

cs.AI

摘要

我们提出了一项新任务——In-2-4D，旨在从极简的输入设置中生成四维（即三维加运动）插帧：仅需两幅单视角图像，分别捕捉物体在两个不同运动状态下的瞬间。给定代表运动物体起始与终止状态的两幅图像，我们的目标是生成并重建其四维运动轨迹。我们采用视频插值模型来预测运动，但帧间大幅度的运动可能导致解释上的模糊性。为此，我们采用分层策略，识别出视觉上接近输入状态且展现显著运动的关键帧，随后在这些关键帧之间生成平滑的片段。对于每个片段，我们利用高斯溅射技术构建关键帧的三维表示。片段内的时间帧引导运动，通过变形场将其转化为动态高斯分布。为了提升时间一致性并优化三维运动，我们扩展了多视角扩散模型在时间步上的自注意力机制，并应用刚体变换正则化。最后，我们通过插值边界变形场并优化其与引导视频的对齐，将独立生成的三维运动片段合并，确保过渡平滑无闪烁。通过大量定性、定量实验及用户研究，我们验证了该方法及其各组成部分的有效性。项目页面详见https://in-2-4d.github.io/。

English

We propose a new problem, In-2-4D, for generative 4D (i.e., 3D + motion) inbetweening from a minimalistic input setting: two single-view images capturing an object in two distinct motion states. Given two images representing the start and end states of an object in motion, our goal is to generate and reconstruct the motion in 4D. We utilize a video interpolation model to predict the motion, but large frame-to-frame motions can lead to ambiguous interpretations. To overcome this, we employ a hierarchical approach to identify keyframes that are visually close to the input states and show significant motion, then generate smooth fragments between them. For each fragment, we construct the 3D representation of the keyframe using Gaussian Splatting. The temporal frames within the fragment guide the motion, enabling their transformation into dynamic Gaussians through a deformation field. To improve temporal consistency and refine 3D motion, we expand the self-attention of multi-view diffusion across timesteps and apply rigid transformation regularization. Finally, we merge the independently generated 3D motion segments by interpolating boundary deformation fields and optimizing them to align with the guiding video, ensuring smooth and flicker-free transitions. Through extensive qualitative and quantitiave experiments as well as a user study, we show the effectiveness of our method and its components. The project page is available at https://in-2-4d.github.io/