V2V-LLM：基于多模态大语言模型的车车协同自动驾驶

V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multi-Modal Large Language Models

February 14, 2025

作者: Hsu-kuang Chiu, Ryo Hachiuma, Chien-Yi Wang, Stephen F. Smith, Yu-Chiang Frank Wang, Min-Hung Chen

cs.AI

摘要

当前，自动驾驶车辆主要依赖其独立传感器来理解周围环境并规划未来轨迹，然而当传感器出现故障或被遮挡时，这种依赖便显得不可靠。为解决这一问题，基于车对车（V2V）通信的协同感知方法被提出，但这些方法多集中于检测与跟踪领域，它们对整体协同规划性能的贡献仍有待深入探索。受近期利用大型语言模型（LLMs）构建自动驾驶系统进展的启发，我们提出了一种新颖的问题设定，将LLM融入协同自动驾驶中，并推出了车对车问答（V2V-QA）数据集及基准。同时，我们提出了基线方法——车对车大型语言模型（V2V-LLM），该模型利用LLM融合来自多辆联网自动驾驶车辆（CAVs）的感知信息，以回答驾驶相关的问题：包括场景理解、显著物体识别及规划。实验结果表明，我们提出的V2V-LLM作为一种统一模型架构，在协同自动驾驶中执行多种任务方面展现出潜力，并优于采用不同融合策略的其他基线方法。我们的工作还开辟了一个新的研究方向，有望提升未来自动驾驶系统的安全性。项目网站：https://eddyhkchiu.github.io/v2vllm.github.io/。

English

Current autonomous driving vehicles rely mainly on their individual sensors to understand surrounding scenes and plan for future trajectories, which can be unreliable when the sensors are malfunctioning or occluded. To address this problem, cooperative perception methods via vehicle-to-vehicle (V2V) communication have been proposed, but they have tended to focus on detection and tracking. How those approaches contribute to overall cooperative planning performance is still under-explored. Inspired by recent progress using Large Language Models (LLMs) to build autonomous driving systems, we propose a novel problem setting that integrates an LLM into cooperative autonomous driving, with the proposed Vehicle-to-Vehicle Question-Answering (V2V-QA) dataset and benchmark. We also propose our baseline method Vehicle-to-Vehicle Large Language Model (V2V-LLM), which uses an LLM to fuse perception information from multiple connected autonomous vehicles (CAVs) and answer driving-related questions: grounding, notable object identification, and planning. Experimental results show that our proposed V2V-LLM can be a promising unified model architecture for performing various tasks in cooperative autonomous driving, and outperforms other baseline methods that use different fusion approaches. Our work also creates a new research direction that can improve the safety of future autonomous driving systems. Our project website: https://eddyhkchiu.github.io/v2vllm.github.io/ .