A recent frontier in computer vision has been the task of 3D video generation, which consists of generating a time-varying 3D representation of a scene. To generate dynamic 3D scenes, current methods explicitly model 3D temporal dynamics by jointly optimizing for consistency across both time and views of the scene. In this paper, we instead investigate whether it is necessary to explicitly enforce multiview consistency over time, as current approaches do, or if it is sufficient for a model to generate 3D representations of each timestep independently. We hence propose a model, Vid3D, that leverages 2D video diffusion to generate 3D videos by first generating a 2D "seed" of the video's temporal dynamics and then independently generating a 3D representation for each timestep in the seed video. We evaluate Vid3D against two state-of-the-art 3D video generation methods and find that Vid3D is achieves comparable results despite not explicitly modeling 3D temporal dynamics. We further ablate how the quality of Vid3D depends on the number of views generated per frame. While we observe some degradation with fewer views, performance degradation remains minor. Our results thus suggest that 3D temporal knowledge may not be necessary to generate high-quality dynamic 3D scenes, potentially enabling simpler generative algorithms for this task.
Our work is based on the following great papers:
Stable Video Diffusion introduces an open-source image to 2D video model with high-quality video generation.
V3D introduces functionality for generating multi-views from individual images, which we base our codebase upon.
Gaussian Splatting achieves state-of-the-art performance on 3D generation and enables fast rendering.
@article{parthasarathy2024vid3d,
author = {Rishab Parthasarathy and Zachary Ankner and Aaron Gokaslan},
title = {Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion},
journal = {arXiv preprint arXiv:2406.11196},
year = {2024},
}