Depth-Aware Audio Visual Segmentation with Geometry-Heuristic Cross Attention

The current state-of-the-art in Audio Visual Segmentation (AVS) has demonstrated successful milestones in performing pixel-level sounding object segmentation. However, they faced significant issues of misalignment and performance degradation in complex settings, such as in substandard lighting conditions and cluttered environments. To address those challenges, we introduce Depth-aware Audio Visual Segmentation (Depth AVS) to enhance the capability of current transformer-based AVS. This paper proposes three main contributions: first, a Geometry-Heuristic Cross Attention (GHCA) as a novel method to suppress irrelevant distant features from the main object of interest, thus enhancing the robustness of audio-visual cross-attention fusion in cluttered and inadequate lighting conditions. Second, an Intermediate Fusion module for integrating depth and RGB features to enrich our model’s learning with non-redundant visual features. Third, a depth-aware segmentor that outputs not only a binary mask but also a segmented depth mask. We experimented using the S4 AVSBench-Object dataset against the current state-of-the-art in AVS, AVSegFormer. Our experiments demonstrate that our Depth AVS surpasses the performance of the AVS baseline method using only small input sizes. Our Depth AVS also extends the capability of AVS using distance estimation with a small error rate.