EvMic：基于事件的有效时空建模实现非接触式声音恢复

EvMic: Event-based Non-contact sound recovery from effective spatial-temporal modeling

April 3, 2025

作者: Hao Yin, Shi Guo, Xu Jia, Xudong XU, Lu Zhang, Si Liu, Dong Wang, Huchuan Lu, Tianfan Xue

cs.AI

摘要

当声波撞击物体时，会引发振动，产生高频且细微的视觉变化，这些变化可用于恢复声音。早期研究常面临采样率、带宽、视野范围及光路简洁性之间的权衡。近年来，事件相机硬件的进步展现了其在视觉声音恢复应用中的巨大潜力，因其在捕捉高频信号方面具有卓越能力。然而，现有基于事件的振动恢复方法在声音恢复方面仍不尽如人意。本研究提出了一种全新的非接触式声音恢复流程，充分利用事件流中的时空信息。首先，我们通过创新的模拟流程生成了大规模训练集。随后，设计了一个网络，利用事件的稀疏性捕捉空间信息，并采用Mamba模型来建模长期时间信息。最后，训练了一个空间聚合模块，以整合来自不同位置的信息，进一步提升信号质量。为了捕捉由声波引发的事件信号，我们还设计了一套采用激光矩阵的成像系统，以增强梯度，并收集了多组数据序列用于测试。在合成数据与真实世界数据上的实验结果验证了本方法的有效性。

English

When sound waves hit an object, they induce vibrations that produce high-frequency and subtle visual changes, which can be used for recovering the sound. Early studies always encounter trade-offs related to sampling rate, bandwidth, field of view, and the simplicity of the optical path. Recent advances in event camera hardware show good potential for its application in visual sound recovery, because of its superior ability in capturing high-frequency signals. However, existing event-based vibration recovery methods are still sub-optimal for sound recovery. In this work, we propose a novel pipeline for non-contact sound recovery, fully utilizing spatial-temporal information from the event stream. We first generate a large training set using a novel simulation pipeline. Then we designed a network that leverages the sparsity of events to capture spatial information and uses Mamba to model long-term temporal information. Lastly, we train a spatial aggregation block to aggregate information from different locations to further improve signal quality. To capture event signals caused by sound waves, we also designed an imaging system using a laser matrix to enhance the gradient and collected multiple data sequences for testing. Experimental results on synthetic and real-world data demonstrate the effectiveness of our method.