VLSBench：揭示多模态安全中的视觉泄漏
VLSBench: Unveiling Visual Leakage in Multimodal Safety
November 29, 2024
作者: Xuhao Hu, Dongrui Liu, Hao Li, Xuanjing Huang, Jing Shao
cs.AI
摘要
在各种应用中，多模态大型语言模型（MLLMs）的安全性问题逐渐成为一个重要问题。令人惊讶的是，先前的研究表明了一个反直觉的现象，即使用文本去对齐MLLMs可以达到与使用图像文本对训练的MLLMs相当的安全性能。为了解释这种反直觉的现象，我们发现了现有多模态安全基准中的一种视觉安全信息泄漏（VSIL）问题，即图像中的潜在风险和敏感内容已经在文本查询中被揭示。这样，MLLMs可以根据文本查询轻松拒绝这些敏感的文本-图像查询。然而，在现实场景中，没有VSIL的图像文本对是常见的，但被现有多模态安全基准所忽视。因此，我们构建了多模态视觉无泄漏安全基准（VLSBench），防止图像到文本查询的视觉安全泄漏，其中包括2.4k个图像文本对。实验结果表明，VLSBench对于包括LLaVA、Qwen2-VL、Llama3.2-Vision和GPT-4o在内的开源和闭源MLLMs都构成了重大挑战。本研究表明，对于存在VSIL的多模态安全场景，文本对齐已经足够，而对于不存在VSIL的多模态安全场景，多模态对齐是一个更有前景的解决方案。请访问我们的代码和数据：http://hxhcreate.github.io/VLSBench
English
Safety concerns of Multimodal large language models (MLLMs) have gradually
become an important problem in various applications. Surprisingly, previous
works indicate a counter-intuitive phenomenon that using textual unlearning to
align MLLMs achieves comparable safety performances with MLLMs trained with
image-text pairs. To explain such a counter-intuitive phenomenon, we discover a
visual safety information leakage (VSIL) problem in existing multimodal safety
benchmarks, i.e., the potentially risky and sensitive content in the image has
been revealed in the textual query. In this way, MLLMs can easily refuse these
sensitive text-image queries according to textual queries. However, image-text
pairs without VSIL are common in real-world scenarios and are overlooked by
existing multimodal safety benchmarks. To this end, we construct multimodal
visual leakless safety benchmark (VLSBench) preventing visual safety leakage
from image to textual query with 2.4k image-text pairs. Experimental results
indicate that VLSBench poses a significant challenge to both open-source and
close-source MLLMs, including LLaVA, Qwen2-VL, Llama3.2-Vision, and GPT-4o.
This study demonstrates that textual alignment is enough for multimodal safety
scenarios with VSIL, while multimodal alignment is a more promising solution
for multimodal safety scenarios without VSIL. Please see our code and data at:
http://hxhcreate.github.io/VLSBenchSummary
AI-Generated Summary