AIM 2024 稀疏神經渲染挑戰賽：數據集與基準測試

AIM 2024 Sparse Neural Rendering Challenge: Dataset and Benchmark

September 23, 2024

作者: Michal Nazarczuk, Thomas Tanay, Sibi Catley-Chandar, Richard Shaw, Radu Timofte, Eduardo Pérez-Pellitero

cs.AI

摘要

最近在可微分和神經渲染方面的發展在各種2D和3D任務中取得了令人印象深刻的突破，例如新視角合成、3D重建。通常，可微分渲染依賴於對場景進行密集的視角覆蓋，以便從僅外觀觀察中可以將幾何形狀與外觀區分開來。當只有少數輸入視圖可用時，通常會出現一些挑戰，這通常被稱為稀疏或少樣本神經渲染。由於這是一個過度約束的問題，大多數現有方法引入了正則化的使用，以及各種學習和手工製作的先驗知識。稀疏渲染文獻中一個經常出現的問題是缺乏一個統一、最新的數據集和評估協議。儘管高分辨率數據集在密集重建文獻中很常見，但稀疏渲染方法通常使用低分辨率圖像進行評估。此外，數據拆分在不同手稿中不一致，測試的真實圖像通常是公開可用的，這可能導致過度擬合。在這項工作中，我們提出了稀疏渲染（SpaRe）數據集和基準。我們引入了一個新的數據集，遵循DTU MVS數據集的設置。該數據集由基於合成高質量資產的97個新場景組成。每個場景最多有64個相機視圖和7種照明配置，分辨率為1600x1200。我們釋出了82個場景的訓練拆分，以促進通用方法的發展，並為驗證和測試集提供了一個在線評估平台，其真實圖像保持隱藏。我們提出了兩種不同的稀疏配置（分別為3和9個輸入圖像）。這為可重現的評估提供了一個強大且便利的工具，並使研究人員能夠輕鬆訪問具有最先進性能分數的公共排行榜。網址：https://sparebenchmark.github.io/

English

Recent developments in differentiable and neural rendering have made impressive breakthroughs in a variety of 2D and 3D tasks, e.g. novel view synthesis, 3D reconstruction. Typically, differentiable rendering relies on a dense viewpoint coverage of the scene, such that the geometry can be disambiguated from appearance observations alone. Several challenges arise when only a few input views are available, often referred to as sparse or few-shot neural rendering. As this is an underconstrained problem, most existing approaches introduce the use of regularisation, together with a diversity of learnt and hand-crafted priors. A recurring problem in sparse rendering literature is the lack of an homogeneous, up-to-date, dataset and evaluation protocol. While high-resolution datasets are standard in dense reconstruction literature, sparse rendering methods often evaluate with low-resolution images. Additionally, data splits are inconsistent across different manuscripts, and testing ground-truth images are often publicly available, which may lead to over-fitting. In this work, we propose the Sparse Rendering (SpaRe) dataset and benchmark. We introduce a new dataset that follows the setup of the DTU MVS dataset. The dataset is composed of 97 new scenes based on synthetic, high-quality assets. Each scene has up to 64 camera views and 7 lighting configurations, rendered at 1600x1200 resolution. We release a training split of 82 scenes to foster generalizable approaches, and provide an online evaluation platform for the validation and test sets, whose ground-truth images remain hidden. We propose two different sparse configurations (3 and 9 input images respectively). This provides a powerful and convenient tool for reproducible evaluation, and enable researchers easy access to a public leaderboard with the state-of-the-art performance scores. Available at: https://sparebenchmark.github.io/