TryOffDiff：使用擴散模型進行高保真度服裝重建的虛擬試穿

TryOffDiff: Virtual-Try-Off via High-Fidelity Garment Reconstruction using Diffusion Models

November 27, 2024

作者: Riza Velioglu, Petra Bevandic, Robin Chan, Barbara Hammer

cs.AI

摘要

本文介紹了虛擬試穿（VTOFF），這是一項新穎的任務，專注於從穿著衣物的個人的單張照片生成標準化的服裝圖像。與傳統的虛擬試穿（VTON）不同，後者是將服裝數字化穿在模特兒身上，VTOFF的目標是提取一個標準的服裝圖像，這在捕捉服裝形狀、紋理和精細圖案方面提出了獨特的挑戰。這個明確定義的目標使VTOFF在評估生成模型中的重建保真度方面特別有效。我們提出了TryOffDiff，這是一個適應Stable Diffusion和基於SigLIP的視覺條件的模型，以確保高保真度和細節保留。在修改後的VITON-HD數據集上進行的實驗表明，我們的方法在基於姿勢轉移和虛擬試穿的基準方法上表現優越，並且需要更少的預處理和後處理步驟。我們的分析顯示，傳統的圖像生成指標未能充分評估重建質量，促使我們依賴於DISTS進行更準確的評估。我們的結果突顯了VTOFF在增強電子商務應用中的產品圖像、推進生成模型評估以及激發未來高保真度重建工作的潛力。演示、代碼和模型可在以下網址找到：https://rizavelioglu.github.io/tryoffdiff/

English

This paper introduces Virtual Try-Off (VTOFF), a novel task focused on generating standardized garment images from single photos of clothed individuals. Unlike traditional Virtual Try-On (VTON), which digitally dresses models, VTOFF aims to extract a canonical garment image, posing unique challenges in capturing garment shape, texture, and intricate patterns. This well-defined target makes VTOFF particularly effective for evaluating reconstruction fidelity in generative models. We present TryOffDiff, a model that adapts Stable Diffusion with SigLIP-based visual conditioning to ensure high fidelity and detail retention. Experiments on a modified VITON-HD dataset show that our approach outperforms baseline methods based on pose transfer and virtual try-on with fewer pre- and post-processing steps. Our analysis reveals that traditional image generation metrics inadequately assess reconstruction quality, prompting us to rely on DISTS for more accurate evaluation. Our results highlight the potential of VTOFF to enhance product imagery in e-commerce applications, advance generative model evaluation, and inspire future work on high-fidelity reconstruction. Demo, code, and models are available at: https://rizavelioglu.github.io/tryoffdiff/