DiffSensei：連接多模式LLM和擴散模型以進行定制漫畫生成

DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation

December 10, 2024

作者: Jianzong Wu, Chao Tang, Jingbo Wang, Yanhong Zeng, Xiangtai Li, Yunhai Tong

cs.AI

摘要

故事視覺化是從文字描述中創建視覺敘事的任務，隨著文本到圖像生成模型的進展而有所進步。然而，這些模型通常缺乏對角色外觀和互動的有效控制，特別是在多角色場景中。為了應對這些限制，我們提出了一個新任務：定制漫畫生成，並引入DiffSensei，一個專門設計用於生成具有動態多角色控制的漫畫的創新框架。DiffSensei將基於擴散的圖像生成器與多模態大型語言模型（MLLM）相結合，後者充當文本兼容的身份適配器。我們的方法採用遮罩交叉注意力，無縫地融入角色特徵，實現精確的布局控制，而無需直接像素傳輸。此外，基於MLLM的適配器調整角色特徵以與面板特定的文本提示對齊，從而允許在角色表情、姿勢和動作中進行靈活調整。我們還介紹了MangaZero，這是一個針對此任務量身定制的大規模數據集，包含43,264頁漫畫和427,147個帶注釋的面板，支持在連續幀中可視化各種角色互動和動作。廣泛的實驗表明，DiffSensei優於現有模型，在漫畫生成方面取得了重大進展，實現了可通過文本調整的角色定制。項目頁面為https://jianzongwu.github.io/projects/diffsensei/。

English

Story visualization, the task of creating visual narratives from textual descriptions, has seen progress with text-to-image generation models. However, these models often lack effective control over character appearances and interactions, particularly in multi-character scenes. To address these limitations, we propose a new task: customized manga generation and introduce DiffSensei, an innovative framework specifically designed for generating manga with dynamic multi-character control. DiffSensei integrates a diffusion-based image generator with a multimodal large language model (MLLM) that acts as a text-compatible identity adapter. Our approach employs masked cross-attention to seamlessly incorporate character features, enabling precise layout control without direct pixel transfer. Additionally, the MLLM-based adapter adjusts character features to align with panel-specific text cues, allowing flexible adjustments in character expressions, poses, and actions. We also introduce MangaZero, a large-scale dataset tailored to this task, containing 43,264 manga pages and 427,147 annotated panels, supporting the visualization of varied character interactions and movements across sequential frames. Extensive experiments demonstrate that DiffSensei outperforms existing models, marking a significant advancement in manga generation by enabling text-adaptable character customization. The project page is https://jianzongwu.github.io/projects/diffsensei/.