11institutetext: University of Surrey, UK 22institutetext: Imperial College
London, UK

# Unsupervised Audio-Visual Segmentation with Modality Alignment

Swapnil Bhosale 11 Haosen Yang 11 Diptesh Kanojia 11 Jiankang Deng 22 Xiatian
Zhu 11

###### Abstract

Audio-Visual Segmentation (AVS) aims to identify, at the pixel level, the
object in a visual scene that produces a given sound. Current AVS methods rely
on costly fine-grained annotations of mask-audio pairs, making them
impractical for scalability. To address this, we introduce unsupervised AVS,
eliminating the need for such expensive annotation. To tackle this more
challenging problem, we propose an unsupervised learning method, named
Modality Correspondence Alignment (MoCA), which seamlessly integrates off-the-
shelf foundation models like DINO, SAM, and ImageBind. This approach leverages
their knowledge complementarity and optimizes their joint usage for multi-
modality association. Initially, we estimate positive and negative image pairs
in the feature space. For pixel-level association, we introduce an audio-
visual adapter and a novel pixel matching aggregation strategy within the
image-level contrastive learning framework. This allows for a flexible
connection between object appearance and audio signal at the pixel level, with
tolerance to imaging variations such as translation and rotation. Extensive
experiments on the AVSBench (single and multi-object splits) and AVSS datasets
demonstrate that our MoCA outperforms strongly designed baseline methods and
approaches supervised counterparts, particularly in complex scenarios with
multiple auditory objects. Notably when comparing mIoU, MoCA achieves a
substantial improvement over baselines in both the AVSBench (S4: +17.24%; MS3:
+67.64%) and AVSS (+19.23%) audio-visual segmentation challenges.

##  1 Introduction

Audio-visual segmentation (AVS) accurately identifies pixel-level objects
producing specific sounds in videos. Previous studies focused on audio-visual
signal intersection using self-supervised learning methods [1, 41], but face
limitations in real-world applications like video editing and robotics.
Supervised learning with pixel-level annotated video-audio pairs is intuitive
[52], but scaling annotation is challenging, as object segmentation demands
over $100\times$ more effort than classifying or bounding boxes [53]. In
complex backgrounds, ground-truth segmentation labels may overlap or be ill-
defined, requiring fine-grained boundaries for differentiation among
surrounding objects.

Figure 1:  Emergence of coarse and noisy audio-pixel association in
ImageBind’s [11] multimodal feature space. (a) Raw audio waveform, (b) input
RGB, (c) using frozen ImageBind features, (d) MoCA (ours) – generating finer
masks and particularly only in the presence of sounding objects (see $2^{nd}$
frame, MoCA generates no mask due to silent audio).

In this work, we advocate for addressing the practical and scalable
unsupervised AVS problem, eliminating the need for exhaustive pixel
annotation. This choice is influenced by notable advancements in purpose-
generic image-level audio-visual representation learning, as demonstrated by
ImageBind [11]. Empirically, we observe the emergence of coarse cross-modal
associations between audio and image pixels, as depicted in Figure 1. However,
despite this progress, the acquired knowledge remains insufficient for
achieving an accurate unsupervised solution.

To address the mentioned limitation, we propose Modality Correspondence
Alignment (MoCA), an unsupervised audio-visual segmentation framework that
efficiently integrates off-the-shelf foundation models using a minimal number
of learnable parameters. Our approach initiates with unsupervised pairing of
training images, leveraging image-level visual features such as those provided
by DINO [3]. This process augments the original audio-image pairing
relationships by introducing additional positive and negative pairs at a
coarse image level, thereby establishing the underlying one-audio-to-multi-
object-instances mapping information. To establish precise audio-pixel
associations, we introduce a novel audio-visual adapter and pixel matching
aggregation within an audio-enhanced visual feature space. This strategy
facilitates the correlation of audio signals and image pixels within a
conventional image-level contrastive learning framework. Specifically, we
aggregate the similarity of pixel pairs across both the original and
additionally paired images, effectively addressing imaging variations like
translation and rotation, while allowing for the emergence of desired audio-
pixel associations. To further refine object boundaries, we incorporate
capabilities from open-world object detection and segmentation models (e.g.,
SAM [25]), aligning them with the derived audio-pixel associations. This
integration ensures accurate delineation of object boundaries in the
synchronized audio-visual context.

Our contributions are summarized as follows:

  * •

We address the more challenging unsupervised Audio-Visual Segmentation (AVS)
timingly, aiming to scale up cross-modal association capabilities for broader
practical applications.

  * •

We propose a novel and efficient unsupervised framework, named MoCA, which
systematically incorporates the off-the-shelf capabilities of pre-trained
foundation models with a minimal number of additional parameters. This is
facilitated primarily by our innovative pixel matching aggregation, enabling
the automatic emergence of audio-pixel associations based on coarse image-
level pairwise information.

  * •

We benchmark the performance of unsupervised AVS by introducing two robust
baseline methods. Our extensive experiments demonstrate that MoCA
significantly outperforms these baselines on both the AVSBench (S4: +17.24%;
MS3: +67.64%) and AVSS (+19.23%) datasets in terms of mIoU, thereby narrowing
the performance gap with supervised AVS alternatives.

##  2 Related Work

Audio-Visual Semantic Segmentation Existing AVS methods use fully supervised
models to identify audible visual pixels associated with a given audio signal
[50, 33, 26, 17, 43, 35]. These models are typically trained on thousands of
manually annotated segmentation masks [52]. However, their resource-intensive
nature for supervised training presents challenges for AVS systems, especially
in large-scale applications with diverse scenes and multiple audible objects.
To address these challenges, we propose an unsupervised AVS approach in this
work, eliminating the need for exhaustive audio-mask pairs.

Self-Supervised Audio-Visual Feature Learning There have been various self-
supervised approaches for learning audio-visual correspondence. Two primary
methods, namely reconstruction by masked autoencoders (MAEs) [10, 37] and
contrastive learning [6, 30, 15, 48], have been widely explored. Recent
methodologies, such as MAVIL, integrate both MAE and contrastive learning to
efficiently generate audio-visual representations for tasks like audio-video
retrieval [22]. More recently, ImageBind, as presented in [11], focuses on
learning a joint embedding across six modalities, enhancing zero-shot
capabilities for large-scale vision foundation models. These methods primarily
address coarse relationships in a cross-modal manner. However, in the context
of Audio-Visual Segmentation (AVS), our goal is to pinpoint object boundaries
at the fine pixel level. Achieving this demands a more detailed and accurate
understanding in the visual domain, and in this work, we address this
challenge for the first time using an unsupervised approach.

Unsupervised Image Semantic Segmentation Our challenge revolves around
unsupervised image semantic segmentation, where the goal is to accurately
isolate objects of interest within an image. Many approaches in the literature
address this problem, primarily focusing on enhancing pre-trained self-
supervised visual features. One such method is SegSort [23], which employs an
iterative process to refine visual features within a spherical embedding
space. MaskContrast [46] takes a different approach by leveraging an off-the-
shelf saliency model to generate binary masks for each image. The model then
contrasts learned features within and across these saliency maps. STEGO [16]
addresses the problem by distilling the correspondence of visual features
through self-correlation within the same image and cross-correlation across
another similar image, along with complex post-processing steps.

It is worth noting that these methods specifically tackle mono-modality
challenges and lack considerations for the cross-modal association between
audio and pixel data—a unique aspect our work aims to address.

##  3 Method

Given a video sequence $\\{I_{t}\\}_{t=1}^{T}\in\mathcal{R}^{H\times W}$
comprising $T$ non-overlapping continuous image frames and an accompanying
audio sequence $\\{A_{t}\\}_{t=1}^{T}$, our objective is to generate object
segmentation masks $\\{G_{t}\\}\in\mathcal{R}^{H\times W}$ that correspond to
the audio content. Here, $H$ and $W$ represent the height and width of each
frame. These masks label individual pixels, highlighting the sound-producing
object in $A_{t}$ within the frame $I_{t}$.

Figure 2:  Overview of our proposed MoCA: (Left) In training, we generate
positive and negative images by utilizing DINO embeddings. The fusion of these
images with the corresponding audio from the anchor image yields audio-
enhanced image features. An efficient learning process by the proposed audio-
visual adapter weights is facilitated through the establishment of a
contrastive training objective, utilizing our pixel matching aggregation
strategy. (Right) In inference, we extract audio-enhanced image features and
employ $k$-means clustering to form clusters. Optionally, we enhance object
boundaries by matching the clustered feature map with mask proposals from a
pre-trained SAM model. PMA refers to pixel matching aggregation. All vision-
audio (ViA) model weights are shared and frozen.

In unsupervised AVS, detailed annotations for pixel-level masks are not
available. To address this challenge, we propose an unsupervised approach
named MoCA.

###  3.1 MoCA

As depicted in Figure 2, our MoCA leverages the existing capabilities of
foundational models, including DINO [3], ImageBind [11], and SAM [25], within
a contrastive learning framework. Initially, we generate positive and negative
image pairs using DINO embeddings. To seamlessly integrate these embeddings
for unsupervised AVS, we propose an audio-visual adapter design with a minimal
number of extra parameters. This design is integrated into a frozen multimodal
foundation model, e.g., ImageBind, to produce audio-enhanced image features.

For accurate pixel segmentation association with the audio signal, we
introduce a novel region matching aggregation strategy. This strategy
facilitates the identification of object-level correspondences under the
guidance of coarse image-level pairing supervision.

##### Audio-Visual Adapter (AdaAV)

The objective of our audio-visual adapter proposal is to integrate audio
event-specific knowledge into frozen visual feature representations. This
integration results in audio-enhanced image features that capture pixel-level
correlations with audio signal. This forms the basis for exploring the
inherent association between audio and visual objects.

The architecture for this design is illustrated in our supplementary. Although
our subsequent discussion is grounded in the context of ImageBind [11], it’s
important to note that our design is versatile, and integration with other
multimodal Vision-Audio (ViA) models can be similarly achieved.

Each image frame, $I_{t}$, is individually inputted into ImageBind’s image
encoder. Within the encoder, $I_{t}$ undergoes decomposition into $n$ non-
overlapping patches, which are then flattened into visual embeddings
$\mathbf{X}^{(0)}_{v}\in\mathbb{R}^{n\times d}$. Simultaneously, $A_{t}$ is
projected into Mel-spectrograms, patched, and fed into the ImageBind audio
encoder, revealing audio embeddings denoted as
$\mathbf{X}^{(0)}_{a}\in\mathbb{R}^{k\times d}$ (where $0$ signifies
layer-$0$, i.e., the input layer). With $\mathbf{X}^{(\ell)}_{a}$ and
$\mathbf{X}^{(\ell)}_{v}$ representing audio and visual inputs, respectively,
at layer $l$, both audio and image trunks employ a multi-headed attention
(MHA) layer. The output from the MHA layer is then passed through a multi-
layer perceptron (MLP) layer.

Our audio-visual adapter module, AdaAV, is introduced between the intermediate
layers of the image and audio trunks (please refer to Figure 3). For a given
intermediate layer-$l$, AdaAV generates an audio-enhanced image feature
representation,

$\displaystyle\mathbf{F}_{v}^{(\ell)}=\text{AdaAV}(\mathbf{X}_{a}^{(\ell)},\mathbf{X}_{v}^{(\ell)})$

The updated MHA and MLP operations in each layer of the image trunk are as
follows:

$\displaystyle\mathbf{Y}_{v}^{(\ell)}=\mathbf{X}^{(\ell)}_{v}+\mathrm{MHA}(\mathbf{X}^{(\ell)}_{v})+\mathrm{AdaAV}(\mathbf{X}_{a}^{(\ell)},\mathbf{X}_{v}^{(\ell)})$
$\displaystyle\mathbf{X}_{v}^{(\ell+1)}=\mathbf{Y}^{(\ell)}_{v}+\mathrm{MLP}(\mathbf{Y}^{(\ell)}_{v})+\mathrm{AdaAV}(\mathbf{Y}_{a}^{(\ell)},\mathbf{Y}_{v}^{(\ell)})$

AdaAV employs a limited number ($m$) of latent audio tokens, denoted as
$\mathbf{L}^{(l)}_{a}\in\mathbb{R}^{m\times d}$, to effectively integrate
audio-specific knowledge into the visual representation. Here, the value of
$m$ is significantly smaller than the overall number of audio tokens. The
primary purpose of these latent tokens is to succinctly encapsulate
information from the audio tokens, facilitating efficient information transfer
into the intermediate layer visual representations.

Distinct sets of latent tokens are utilized at each layer. A cross-modal
attention block is employed to compress all tokens from the audio modality
into these latent tokens. Subsequently, another cross-modal attention block is
utilized to fuse information between the compressed latent tokens of the audio
modality and all tokens of the image modality.

In alignment with previous research on adapters [20, 24, 36], a bottleneck
module is incorporated. This module comprises a learnable down-projection
layer, a non-linear activation function, and a learnable up-projection layer.

Figure 3:  Fusing frozen ViA encoders using AdaAV: We augment frozen visual-
audio (ViA) encoders using a lightweight audio-visual adapter, AdaAV, which,
given an input image frame ($I_{t}$) and corresponding audio ($A_{t}$),
generates an audio-enhanced image feature ($f^{a}$).

####  3.1.1 Pixel matching aggregation (PMA)

To achieve AVS, it is necessary to establish cross-modal associations at the
pixel level. In pursuit of this goal, we delve into pixel-level correlations
involved in audio-enhanced image features. Drawing inspiration from the stereo
matching literature [39, 19], we propose a pixel matching strategy that gauges
the similarity between audio-enhanced image features generated at the pixel
level. Formally, given two audio-enhanced feature tensors $f^{a}$ and $g^{a}$,
a matching cost is computed for each spatial location in $f^{a}$, considering
a set of pixels along the corresponding scanline of $g^{a}$. This cost
signifies the matching likelihood of a pixel at a specific disparity in
$f^{a}$ with a pixel in $g^{a}$. We further refine this process to identify
optimal correspondences by pinpointing disparities with the lowest costs,
signifying regions containing potential sounding source objects.

We compute a combined matching cost as:

$C=\lambda_{ssd}\cdot C_{ssd}+\lambda_{ncc}\cdot C_{ncc}$ (1)

where $\lambda_{ssd/ncc}$ is the weight hyper-parameter, and the first term is
the sum of squared differences (SSD) [44, 45]:

$C_{ssd}(f^{a},g^{a})=\sum_{i,j}\left(f^{a}(i,j)-g^{a}(i,j)\right)^{2}$ (2)

and the second term is the normalized cross-correlation (NCC) [38]:

$C_{ncc}(f^{a},g^{a})=\frac{\sum_{i,j}\left(f^{a}(i,j)\cdot
g^{a}(i,j)\right)}{\sqrt{\sum_{i,j}\left(f^{a}(i,j)^{2}\right)\cdot\sum_{i,j}\left(g^{a}(i,j)^{2}\right)}}$
(3)

Associating the pixels between two images in such a fashion permits a
versatile matching of their objects, accommodating variations in imaging
conditions such as object translation and rotation. We posit that this dual-
metrics cost brings about a complementary advantage, where SSD captures
absolute intensity differences, and NCC considers both the similarity in shape
and the scale of the signals. The positive impact of this fusion is
empirically substantiated through experiments, as illustrated in Table 5.
Given $f^{a}$, $g^{a}$ as the two audio-enhanced features whose similarity is
to be computed, SSD captures absolute intensity by summing the squared
discrepancies in pixel values between $f^{a}$ and $g^{a}$. NCC alternatively
computes the similarity in the pattern or structure (shape) by normalizing the
data, and hence less sensitive to variations in amplitude (scale).

####  3.1.2 Model training

In our design, the AdaAV is the sole learnable module, while the audio and
image encoder weights remain fixed. The learning of AdaAV weights is achieved
through a contrastive training objective, which is outlined as follows:

$\displaystyle
L_{c}=\max\left(0,C(f^{a},f^{a}_{pos})-C(f^{a},f^{a}_{neg})+\alpha,0\right)$
(4)

where $\alpha$ is the margin parameter and $C$ is defined in Eq. (1). Positive
($f^{a}_{pos}$) and negative ($f^{a}_{neg}$) sets are formed by leveraging the
pre-trained DINO [3] as an off-the-shelf image featurizer. Specifically,
global image features are extracted using DINO embeddings through global
average pooling. Subsequently, we create a lookup table containing each
image’s K-Nearest Neighbors (KNNs) based on cosine similarity in the DINO’s
feature space. This augments the original audio-image pairs by introducing
additional pairing information at the image level, providing the coarse one-
audio-to-multi-object-instances association information. For any images $x$
and their corresponding random nearest neighbors, we form the set of positive
images, denoted as $x_{pos}$. To constitute the set of negative images
$x_{neg}$ relative to $x$, we randomly sample images by shuffling $x$ while
ensuring that no image matches with itself or its top KNNs. The audio
embedding $A$ corresponding to $x$ is then employed to augment both the
positive set ($f^{a}_{pos}$) and negative set ($f^{a}_{neg}$), as illustrated
in Figure 2 for further details.

####  3.1.3 Inference

During the inference phase, when presented with an image and accompanying
audio, we employ the AdaAV module to extract enhanced features from the audio-
imbued image. Subsequently, we refine the learned pixel-level associations by
applying a cosine distance-based k-means algorithm [18]. Additionally, we
observe that overlaying the generated clusters with mask proposals derived
from a pre-trained SAM model contributes to obtaining more precise object
boundaries.

To generate mask proposals, we initiate an open-world object discovery on the
RGB image frame using a pre-trained Open World Object Detector (OWOD) model
[31] (refer to Section 4.1 (II) for object discovery details). The bounding
boxes produced by OWOD serve as input for the pre-trained SAM, resulting in
proposal segmentation masks. These masks delineate visual objects within the
given image frame, as depicted in Figure 4. By overlaying the mask generated
from the audio-enhanced image feature onto these mask proposals, we
effectively filter out potential auditory objects among the visual
counterparts. Please note, the term overlay refers to choosing the proposal
mask having IoU$>$0.5 with the mask generated from the fused ViA encoder
(_i.e_. from the audio-enhanced feature).

It is important to note that while mask proposal matching is not a primary
focus of our work, we conduct an ablative study in Section 4.4 to underscore
the significance of our pixel matching strategy discussed earlier.

Figure 4: Generating mask proposals using a cascade of a pre-trained Open-
world Object Detector (OWOD) [31] and Segment anything model (SAM) [25]. Mask
proposals represent the existing visual objects in a given frame.

##  4 Experiments

Training Data For training our AdaAV weights, we employ VGGSound dataset [5],
a large scale audio-visual dataset collected from YouTube. The dataset
consists of over 200k clips for 300 different sound classes, spanning a large
number of challenging acoustic environments and noise characteristics of real
applications.

Benchmark settings To assess the proposed problem setting and method, we use
the public AVSBench dataset [52]. It contains videos from YouTube, split into
5 distinct clips. Each clip is paired with a spatial segmentation mask
indicating the audible object. AVSBench includes two subsets: the Semi-
supervised Single Source Segmentation (S4) and the fully supervised Multiple
Sound Source Segmentation (MS3), differing in the number of audible objects.

Additionally, we also report our scores on the AVS-Semantic (AVSS) dataset
[51]. We evaluate our proposed method on the S4, MS3 and the AVSS test split
without using any audio-mask pairs from the train and validation split
(_i.e_., unsupervised AVS).

Performance metrics We use average Intersection Over Union ($M_{IoU}$) and
$F_{score}$ as metrics. A higher $M_{IOU}$ implies better region similarity,
and an elevated $F_{score}$ indicates improved contour accuracy.

###  4.1 Baselines

The existing supervised training framework for AVS demands resource-intensive
audio-mask pairs annotations, not applicable for benchmarking unsupervised
AVS. To address the challenges, we present an alternative approach that
eliminates the need for audio-mask pairs. This method leverages an untrained
methodology informed by insights from sophisticated established foundational
models.

Our proposal involves three meticulously designed integration mechanisms. They
bootstrap foundation models related to unimodalities and associated tasks,
such as segmentation, audio tagging, and visual object grounding.

(I) AT-GDINO-SAM In the first baseline, we adopt the Audio Spectrogram
Transformer (AST) [13] to generate audio tags for the AVS task, utilizing its
capacity to capture global context in audio sequences with multiple events.
Trained on the large Audioset dataset [9], AST detects diverse polyphonic
audio events across 521 categories like animal sounds, instruments, and human
activity. Given an audio sequence, $A_{t}$, we pad it to a maximum of 960 msec
and obtain its corresponding audio tags $\\{AT_{i}\\}_{i=1}^{C_{a}}$ using the
pre-trained AST model. The tags are ranked based on their probability scores
across 521 generic classes from the Audioset ontology. Relevant audio tags are
filtered using an empirically determined threshold, $\tau_{AT}$ and forwarded
to a pre-trained GroundingDINO model [28], generating bounding boxes in image
frame $I_{t}$. These boxes serve as visual prompts for Segment Anything model
(SAM) [25], to produce producing binary masks.

(II) OWOD-BIND We use a pre-trained Open World Object Detector (OWOD) [31]
model to generate class-agnostic object proposals that are further processed
by a segmentation pipeline. Unlike traditional object detectors which consider
unfamiliar objects as background, OWOD models have been designed to handle
unknown objects during training and inference. Given an image frame, $I_{t}$,
we use OWOD to generate $C_{v}$ proposal bounding box proposals,
$\\{BB_{i}\\}_{i=1}^{C_{v}}$. These are filtered by objectness score [31]
using a threshold $\tau_{BB}$. To link boxes and acoustic cues, both
modalities need a shared latent space embedding semantics. Towards this end,
we utilize ImageBind’s [11] latents, and extract image and audio embeddings
which (post global average pooling) are used to rank the proposals by cosine
similarity with the audio embedding. Bounding boxes above $\tau_{BIND}$ form
the final mask.

(III) SAM-BIND Alternatively, instead of relying on the OWOD model to generate
bounding box proposals, we can randomly position single-point input prompts in
a grid across the image. From each point, SAM can predict multiple masks.
These masks are refined and filtered for quality, employing non-maximal
suppression (NMS) [12] to remove duplicates.

###  4.2 Implementation Details

Baselines We resize all image frames to 224 $\times$ 224\. For all our
evaluation results we use $\tau_{AT}$ as 0.5 for AT-GDINO-SAM and $\tau_{BB}$
and $\tau_{BIND}$ as 0.5 and 0.7 respectively, for OWOD-BIND. For refining the
mask outputs from SAM-BIND, we use IoU threshold of 0.5 for NMS.

MoCA For the frozen ViA we choose the publicly released ImageBind-Huge model
with 6 and 12 transformer encoders in the audio and image trunks respectively.
The resolution of the input images are 224 $\times$ 224\. To optimize the
model parameters, we employ the Adam [29] optimizer with an initial learning
rate of 1e-4 with cosine decay. We use $\lambda_{SSD}$=$\lambda_{NCC}$=1 and
$\alpha$=0.3. We train the models for a maximum of 10000 iterations on a
single NVIDIA RTX A5500 GPU. The batch size is set to 8. For the mask proposal
matching we consider only the proposal masks which have an IoU higher than 0.5
when compared with the mask generated from the fused ViA encoder (post
$k$-means).

###  4.3 Benchmark Results

We first provide a quantitative comparison in terms of $M_{IoU}$ and
$F_{score}$, followed by a more qualitative comparison in terms of the quality
of the generated masks.

(a) Quantitative Comparison: We present our primary AVSBench results on the
test set for both S4 and MS3, and the AVSS dataset in Table 1.

Comparing with salient object detection (SOD) and sound source localization
(SSL) based approaches, show a substantial gap between these and our MoCA in
both S4 and MS3 subset. It is important to acknowledge that SOD operates
solely on visual data without accounting for sound, prioritizing the dataset
it was trained on. Consequently, it struggles with scenarios like MS3, where
the auditory elements change while the visual components remain constant. On
the other hand, SSL provides a more equitable comparison to our MoCA, as it
incorporates audio signals to guide object segmentation. However, a
significant disparity is apparent between SSL and both our MoCA and the
proposed training-free baselines (AT-GDINO-SAM, OWOD-BIND, SAM-BIND).

Comparing with unsupervised baseline approaches, we can observe that both,
OWOD-BIND and SAM-BIND, outperform AT-GDINO-SAM in terms of both $M_{IoU}$ and
$F_{score}$ by a significant margin. This is because AST itself achieves a
mean average precision (mAP) of 0.485 when evaluated on the audio tagging task
of Audioset, and hence the generated audio tags are prone to errors.
Additionally, we believe despite AST’s training data _i.e_., Audioset follows
a generic ontology, many rare events (_e.g_. “lawn mover”, “tabla” etc.) are
under-represented and hence are unable to cope with an open-set inference.

We can observe that OWOD-BIND achieves an absolute improvement of 0.06 and
0.08 over SAM-BIND in terms of $M_{IoU}$ and $F_{score}$, respectively under
the MS3 setting. Similar trend is observed under the S4 setting, with 0.16
improvement in both $M_{IoU}$ and $F_{score}$ respectively.

It can be clearly noted that MoCA surpasses all the baseline approaches by
more than 10% in terms of $M_{IoU}$ and more than 13% in terms of the
$F_{score}$ on both the S4 and MS3 settings. In contrast, to our best
performing baseline OWOD-BIND, MoCA differs primarily in the latent space
matching generated by the ImageBind encoders. This testifies our novel pixel-
level association and matching strategy.

Comparing with supervised AVS approaches, our unsupervised MoCA only falls
short of the supervised AVS-Benchmark by 0.20 in terms of both $M_{IoU}$ and
$F_{score}$ under the MS3 setting, where the latter benefits from the
effective audio-visual contrastive loss design and trainable parameters. This
suggests both the promising potential of unsupervised AVS and the impressive
performance of our model design.

Table 1: Performance comparison on the AVSBench test split under the S4, MS3 and the AVSS setting. Grayed: supervised learning methods. | Approach | Mask-free | # learnable | S4 | MS3 | AVSS
---|---|---|---|---|---|---
|  | training | params $\downarrow$ | $\rm{M_{IoU}}\uparrow$ | $\rm{F_{score}}\uparrow$ | $\rm{M_{IoU}}\uparrow$ | $\rm{F_{score}}\uparrow$ | $\rm{M_{IoU}}\uparrow$ | $\rm{F_{score}}\uparrow$
Supervised | AV-SAM [35] | ✗ | 1191M | $0.40$ | $0.56$ | - | - | - | -
AVSBench [52] | ✗ | 101M | $0.78$ | $0.87$ | $0.54$ | $0.64$ | 0.29 | 0.35
SAMA-AVS [27] | ✗ | 75M | $0.81$ | $0.90$ | 0.63 | 0.69 | - | -
DG-SCT [8] | ✗ | 342M | $0.80$ | $0.89$ | 0.53 | 0.64 | - | -
SOD | iGAN [32] | ✓ | 86M | $0.61$ | $0.77$ | $0.42$ | $0.54$ | - | -
LGVT [49] | ✓ | 90M | $0.74$ | 0.87 | $0.40$ | $0.59$ | - | -
SSL | LVS [4] | ✓ | 20M | $0.37$ | $0.51$ | $0.29$ | $0.33$ | - | -
MSSL [40] | ✓ | 21M | $0.44$ | $0.66$ | $0.26$ | $0.36$ | - | -
EZ-VSL [34] | ✓ | 18M | $0.45$ | $0.68$ | $0.28$ | $0.34$ | - | -
Mix-Localize [21] | ✓ | 19M | $0.44$ | $0.69$ | $0.32$ | $0.36$ | - | -
Baseline | AT-GDINO-SAM | ✓ | - | $0.38$ | $0.46$ | $0.25$ | $0.29$ | 0.24 | 0.25
SAM-BIND | ✓ | - | $0.42$ | $0.51$ | $0.28$ | $0.36$ | 0.24 | 0.26
OWOD-BIND | ✓ | - | 0.58 | 0.67 | 0.34 | 0.44 | 0.26 | 0.29
Our | MoCA | ✓ | 1.3M | 0.68 | 0.79 | 0.57 | 0.62 | 0.31 | 0.33

(b) Qualitative Comparison: We show example segmentation from MoCA and the AVS
supervised benchmark approach in Figure 5. We note that MoCA is significantly
better at resolving fine-grained details within the images such as object
boundaries (guitar in the left example and the guitarist in the right
example). Although AVSBench uses a PVTv2 [47] architecture to output a high
resolution segmentation, the network misses fine-grained details and generates
discontinuous object segmentations. In contrast, MoCA despite not using any
audio-mask pairs during training, can capture overlapping objects and fine
details. It is evident that the baseline method, although is able to segment
visual objects, but finds it difficult to filter out potential sounding
objects among them. In the circumstances when there are more than one objects
of the same object category (_e.g_. multiple humans) in the same frame, MoCA,
unlike the baseline, is able to appropriately highlight the "sounding human".
Also, we provide qualitative comparison of MoCA and AVSBench on the AVSS split
in the supplementary.

Figure 5: Qualitative comparisons(left: S4, right: MS3): (a) RGB frame, (b)
ground truth mask, (c) AVSBench [52](supervised), (d) OWOD-BIND (baseline),
(e) MoCA (ours). MoCA produces more precise segmentation of overlapping
objects without utilizing any audio-visual masks during training.

###  4.4 Ablation Study

Effect of pixel matching aggregation: As discussed in Section 3.1.3, we align
the clustered predictions from MoCA with proposal masks, to obtain finer
object segmentation boundaries. From Table 3 it is evident that matching the
proposals yields an overall improved segmentation performance, with maximum
contribution of this performance owed to the audio-enhanced feature map
learned by MoCA. For the scores in first row, we select random points as input
to the SAM model111github.com/facebookresearch/segment-
anything/blob/main/notebooks/automatic_mask_generator_example and match all
the segmentation maps with the closest ground truth in the MS3 test set. It
can be noted that, this deprived of the mapping with MoCA generated enhanced
feature, drops the performance significantly, with more than 0.2 drop compared
to MoCA(row 3). Additionally, we make it more concrete to particularly
establish the higher contribution of PMA over MPM by combining existing SSL
approaches with MPM. As can be seen in Table 3, combining MPM with Mix-
Localize [21] and EZ-VSL [34] fails to surpass MoCA and even MoCA without MPM,
due to the sparse nature of SSL outputs. To further understand this, we
pictorially highlight the under-segmentation (Fig. 6(d)) and over-segmentation
(Fig. 6(f)) problems that occur when converting SSL output (heatmaps) to
binary masks by employing MPM. Using this we reiterate the efficacy of pixel-
level association learned by our proposed PMA strategy leading to fine-grained
localization, while the MPM gives further (less significant than PMA)
advantage by refining the object edges.

Figure 6: Comparing SSL methods equipped with MPM: (a) input frame, (b) ground
truth mask, (c) Mix-Localize [21], (d) Mix-Localize + MPM, (e) EZ-VSL [34],
(f) EZ-VSL + MPM, (g) MoCA (ours) – producing more precise segmentation of all
overlapping sounding objects (Guitarist and guitar in this example).

Table 2: Effect of pixel matching aggregation (PMA) and mask proposal matching (MPM) (Sec 3.1.3), on the performance of AVSBench MS3 test split. PMA | MPM | MS3
---|---|---
|  | $\rm{M_{IoU}}\uparrow$ | $\rm{F_{score}}\uparrow$
✓ | ✗ | 0.52 | 0.59
✗ | ✓ | 0.34 | 0.44
✓ | ✓ | 0.57 | 0.62

Table 3: Comparing MoCA and SSL methods equipped with mask proposal matching (MPM) on the AVSBench MS3 test split. Method | $M_{IoU}$
---|---
Mix-Loc[21]+MPM | 0.45
EZ-VSL[34]+MPM | 0.43
MoCA W/o MPM | 0.52
MoCA | 0.57

Loss function: We ablate the impact of SSD and NCC components in calculating
the loss function in Table 5. The results indicate that: (1) Using only NCC
compared to only SSD results in a higher $M_{IoU}$ and $F_{score}$ (+6.55%).
This is primarily because SSD is sensitive to the intensity of the pixels when
computing feature correspondences. (2) Combining SSD and NCC for computing the
correlation leads to the best segmentation performance, confirming our design
consideration. The NCC formulation for computing the correlation is invariant
to the shape of the potential sounding region and hence agnostic to the scale
of the sounding object among the anchor and positive images.

Table 4: Effect of SSD and NCC on the performance of AVSBench S4 and MS3 test split.  Loss | S4 | MS3
---|---|---
| $\rm{M_{IoU}}\uparrow$ | $\rm{F_{score}}\uparrow$ | $\rm{M_{IoU}}\uparrow$ | $\rm{F_{score}}\uparrow$
SSD | 0.61 | 0.72 | 0.40 | 0.57
NCC | 0.65 | 0.76 | 0.44 | 0.59
SSD+NCC | 0.68 | 0.79 | 0.57 | 0.62

Table 5: Effect of the number of AdaAV blocks on the performance of AVSBench S4 and MS3 test split.  #(AdaAV) | S4 | MS3
---|---|---
| $\rm{M_{IoU}}\uparrow$ | $\rm{F_{score}}\uparrow$ | $\rm{M_{IoU}}\uparrow$ | $\rm{F_{score}}\uparrow$
2 | 0.60 | 0.69 | 0.39 | 0.47
4 | 0.65 | 0.73 | 0.43 | 0.51
6 | 0.68 | 0.79 | 0.57 | 0.62

Number of adapter blocks: Our proposed audio-visual adapters decide the amount
of audio specific information that is added into the frozen image encoders. To
examine the effect of this component design, we vary the number of AdaAV
blocks from 2 to 6 (the maximum number of adapters we can add since the
ImageBind audio trunk consists of 6 transformer encoder blocks). From Table 5,
we observe that increasing the number of adapter blocks improves both
$M_{IoU}$ and $F_{score}$ by a huge margin albeit, non-uniformly with lesser
positive impact when increasing from 4 blocks to 6. Using two AdaAV blocks
yields the least performance indicating minimal audio-enhancement and hence
poor audio-guided segmentation.

Placement of AdaAV blocks: We experiment with the optimal placement of the
AdaAV blocks relative to the ImageBind audio and image encoders blocks in
Table. 6. We fix number of AdaAV blocks to 6. Please note that the frozen
ImageBind image encoder consists of 12 transformer encoder blocks. Notably,
the 6 AdaAV blocks can be placed either along the first 6 (counting from the
input layer) image encoder blocks, along the last 6 image encoder blocks, or
interleaved such that every alternate encoder block in the image trunk is
fused with one AdaAV block. Our observations are as follows: (1) Fusing every
alternate encoder _i.e_. interleaving, and fusing only blocks closer to the
output _i.e_., last 6, yields almost similar performance with a max deviation
of 0.01 in terms of $M_{IoU}$ on the MS3 setting. (2) Fusing blocks closer to
the output yields a maximum of 3% improvement in $M_{IoU}$ on the MS3 setting,
indicating a higher adaptability is favoured closer to the latter layers.

Table 6: Effect of positioning the AdaAV blocks w.r.t to the ImageBind image and audio trunk, on the performance of AVSBench S4 and MS3 test split. (k=6) AdaAV | S4 | MS3
---|---|---
Positioning | $\rm{M_{IoU}}\uparrow$ | $\rm{F_{score}}\uparrow$ | $\rm{M_{IoU}}\uparrow$ | $\rm{F_{score}}\uparrow$
Interleaved | 0.68 | 0.77 | 0.56 | 0.60
first-k | 0.67 | 0.78 | 0.54 | 0.60
last-k | 0.68 | 0.79 | 0.57 | 0.62

##  5 Conclusion

We introduce a more challenging unsupervised Audio-Visual Segmentation (AVS)
problem, aiming to scaling its applicability. For performance benchmarks, we
show that harnessing self-supervised audio-visual models leads is effective.
Our proposed method, Modality Correspondence Alignment (MoCA), features a
novel pixel matching aggregation strategy for accurate pixel-audio
associations. Extensive experiments demonstrate that MoCA outperforms well-
designed methods and closely approaches fully supervised counterparts.
Notably, MoCA achieves substantial mIoU improvements in both the AVSBench (S4:
+17.24%; MS3: +67.64%) and AVSS (+19.23%) audio-visual segmentation settings.

## References

  * [1] Afouras, T., Owens, A., Chung, J.S., Zisserman, A.: Self-supervised learning of audio-visual objects from video. In: ECCV (2020)
  * [2] Alwassel, H., Mahajan, D., Korbar, B., Torresani, L., Ghanem, B., Tran, D.: Self-supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems 33, 9758–9770 (2020)
  * [3] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104.14294 (2021)
  * [4] Chen, H., Xie, W., Afouras, T., Nagrani, A., Vedaldi, A., Zisserman, A.: Localizing visual sounds the hard way. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16867–16876 (2021)
  * [5] Chen, H., Xie, W., Vedaldi, A., Zisserman, A.: Vggsound: A large-scale audio-visual dataset. In: IEEE ICASSP (2020)
  * [6] Chen, Y., Xian, Y., Koepke, A., Shan, Y., Akata, Z.: Distilling audio-visual knowledge by compositional contrastive learning. In: CVPR (2021)
  * [7] Cheng, Y., Wang, R., Pan, Z., Feng, R., Zhang, Y.: Look, listen, and attend: Co-attention network for self-supervised audio-visual representation learning. In: Proceedings of the 28th ACM International Conference on Multimedia. pp. 3884–3892 (2020)
  * [8] Duan, H., Xia, Y., Mingze, Z., Tang, L., Zhu, J., Zhao, Z.: Cross-modal prompts: Adapting large pre-trained models for audio-visual downstream tasks. Advances in Neural Information Processing Systems 36 (2024)
  * [9] Gemmeke, J.F., Ellis, D.P., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M., Ritter, M.: Audio set: An ontology and human-labeled dataset for audio events. In: IEEE ICASSP (2017)
  * [10] Georgescu, M.I., Fonseca, E., Ionescu, R.T., Lucic, M., Schmid, C., Arnab, A.: Audiovisual masked autoencoders. In: ICCV (2023)
  * [11] Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, I.: Imagebind: One embedding space to bind them all. In: CVPR (2023)
  * [12] Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR (2014)
  * [13] Gong, Y., Chung, Y.A., Glass, J.: Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778 (2021)
  * [14] Gong, Y., Rouditchenko, A., Liu, A.H., Harwath, D., Karlinsky, L., Kuehne, H., Glass, J.: Contrastive audio-visual masked autoencoder. arXiv preprint arXiv:2210.07839 (2022)
  * [15] Guzhov, A., Raue, F., Hees, J., Dengel, A.: Audioclip: Extending clip to image, text and audio. In: IEEE ICASSP (2022)
  * [16] Hamilton, M., Zhang, Z., Hariharan, B., Snavely, N., Freeman, W.T.: Unsupervised semantic segmentation by distilling feature correspondences. ICLR (2022)
  * [17] Hao, D., Mao, Y., He, B., Han, X., Dai, Y., Zhong, Y.: Improving audio-visual segmentation with bidirectional generation. arXiv preprint arXiv:2308.08288 (2023)
  * [18] Hartigan, J.A., Wong, M.A.: Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series c (applied statistics) 28(1), 100–108 (1979)
  * [19] Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge university press (2003)
  * [20] Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for nlp. In: ICML (2019)
  * [21] Hu, X., Chen, Z., Owens, A.: Mix and localize: Localizing sound sources in mixtures. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10483–10492 (2022)
  * [22] Huang, P.Y., Sharma, V., Xu, H., Ryali, C., Fan, H., Li, Y., Li, S.W., Ghosh, G., Malik, J., Feichtenhofer, C.: Mavil: Masked audio-video learners. arXiv preprint arXiv:2212.08071 (2022)
  * [23] Hwang, J., Yu, S.X., Shi, J., Collins, M.D., Yang, T., Zhang, X., Chen, L.: Segsort: Segmentation by discriminative sorting of segments. CoRR abs/1910.06962 (2019), http://arxiv.org/abs/1910.06962
  * [24] Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., Carreira, J.: Perceiver: General perception with iterative attention. In: ICML (2021)
  * [25] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [26] Liu, C., Li, P., Zhang, H., Li, L., Huang, Z., Wang, D., Yu, X.: Bavs: Bootstrapping audio-visual segmentation by integrating foundation knowledge. arXiv preprint arXiv:2308.10175 (2023)
  * [27] Liu, J., Wang, Y., Ju, C., Zhang, Y., Xie, W.: Annotation-free audio-visual segmentation. arXiv preprint arXiv:2305.11019 (2023)
  * [28] Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023)
  * [29] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [30] Ma, S., Zeng, Z., McDuff, D., Song, Y.: Active contrastive learning of audio-visual video representations. arXiv preprint arXiv:2009.09805 (2020)
  * [31] Maaz, M., Rasheed, H., Khan, S., Khan, F.S., Anwer, R.M., Yang, M.H.: Class-agnostic object detection with multi-modal transformer. In: ECCV (2022)
  * [32] Mao, Y., Zhang, J., Wan, Z., Dai, Y., Li, A., Lv, Y., Tian, X., Fan, D.P., Barnes, N.: Transformer transforms salient object detection and camouflaged object detection. arXiv preprint arXiv:2104.10127 1(2), 5 (2021)
  * [33] Mao, Y., Zhang, J., Xiang, M., Lv, Y., Zhong, Y., Dai, Y.: Contrastive conditional latent diffusion for audio-visual segmentation. arXiv preprint arXiv:2307.16579 (2023)
  * [34] Mo, S., Morgado, P.: Localizing visual sounds the easy way. In: European Conference on Computer Vision. pp. 218–234. Springer (2022)
  * [35] Mo, S., Tian, Y.: Av-sam: Segment anything model meets audio-visual localization and segmentation. arXiv preprint arXiv:2305.01836 (2023)
  * [36] Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., Sun, C.: Attention bottlenecks for multimodal fusion. In: NeurIPS (2021)
  * [37] Nunez, E., Jin, Y., Rastegari, M., Mehta, S., Horton, M.: Diffusion models as masked audio-video learners. arXiv preprint arXiv:2310.03937 (2023)
  * [38] Pratt, W.K.: Digital image processing: PIKS Scientific inside, vol. 4. Wiley Online Library (2007)
  * [39] Prince, S.J.: Computer vision: models, learning, and inference. Cambridge University Press (2012)
  * [40] Qian, R., Hu, D., Dinkel, H., Wu, M., Xu, N., Lin, W.: Multiple sound sources localization from coarse to fine. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16. pp. 292–308. Springer (2020)
  * [41] Rouditchenko, A., Zhao, H., Gan, C., McDermott, J., Torralba, A.: Self-supervised audio-visual co-segmentation. In: IEEE ICASSP (2019)
  * [42] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
  * [43] Shi, Z., Wu, Q., Li, H., Meng, F., Xu, L.: Cross-modal cognitive consensus guided audio-visual segmentation. arXiv preprint arXiv:2310.06259 (2023)
  * [44] Szeliski, R.: Computer vision: algorithms and applications. Springer Nature (2022)
  * [45] Trucco, E., Verri, A.: Introductory techniques for 3-D computer vision, vol. 201. Prentice Hall Englewood Cliffs (1998)
  * [46] Van Gansbeke, W., Vandenhende, S., Georgoulis, S., Van Gool, L.: Unsupervised semantic segmentation by contrasting object mask proposals. arxiv preprint arxiv:2102.06191 (2021)
  * [47] Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In: ICCV (2021)
  * [48] Wu, H.H., Seetharaman, P., Kumar, K., Bello, J.P.: Wav2clip: Learning robust audio representations from clip. In: IEEE ICASSP (2022)
  * [49] Zhang, J., Xie, J., Barnes, N., Li, P.: Learning generative vision transformer with energy-based latent space for saliency prediction. Advances in Neural Information Processing Systems 34, 15448–15463 (2021)
  * [50] Zhou, J., Guo, D., Wang, M.: Contrastive positive sample propagation along the audio-visual event line. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
  * [51] Zhou, J., Shen, X., Wang, J., Zhang, J., Sun, W., Zhang, J., Birchfield, S., Guo, D., Kong, L., Wang, M., et al.: Audio-visual segmentation with semantics. arXiv preprint arXiv:2301.13190 (2023)
  * [52] Zhou, J., Wang, J., Zhang, J., Sun, W., Zhang, J., Birchfield, S., Guo, D., Kong, L., Wang, M., Zhong, Y.: Audio–visual segmentation. In: ECCV (2022)
  * [53] Zlateski, A., Jaroensri, R., Sharma, P., Durand, F.: On the importance of label quality for semantic segmentation. In: CVPR (2018)

##  6 Exploiting Audio-Visual Feature Correspondences from frozen ViA

Recent progress in self-supervised audio-visual feature learning [1, 2, 7, 14,
10, 11] has yielded methods with powerful and semantically relevant features
that improve a variety of downstream tasks. Though most works aim to generate
vector representations for an image and a corresponding audio, there haven’t
been many works showing the semantic relevance among intermediate dense
features. Towards this direction, we examine the correlation between the dense
feature maps, by fusing the audio and visual representations frozen ViA
models. More formally, let $f\in\mathbb{R}^{EIJ}$ be the feature tensor for an
image where $E$ represents the intermediate layer embedding dimension and
$(I,J)$ represent the spatial dimensions. Additionally, let
$A\in\mathbb{R}^{E}$ be the audio embedding for the corresponding audio
signal. We compute the audio-enhanced image feature tensor by taking the
cosine similarity of $A$ with every spatial (or pixel) level feature
$f_{ij}\forall i\in I$ and $\forall j\in J$ in the image tensor $f$, _i.e_.
$f^{a}_{ij}=cosine\\_sim(f_{ij},A)$. Notably, we notice that for two image
features $f$ and $g$, the introduction of the same audio embedding $A$ results
in correlated tensors $f^{a}$ and $g^{a}$, provided that images corresponding
to $f$ and $g$, consist a potential sounding object described by the audio
representation in $A$. For instance, Figure 7 shows how the audio-enhanced
features for three different images when enhanced with the same audio are in
correspondence with relevant semantic areas particularly around the potential
sounding object. It is evident that, in contrast to $f^{a}$ and $g_{a}$,
$h_{a}$ does not effectively emphasize a distinct potential sounding area that
might serve as the likely source for the sound of a helicopter, especially
considering the absence of a helicopter in the image.

Figure 7: Audio-Visual Feature Correspondence: $A$ is the audio embedding of
the sound of helicopter (AVSBenchS4/train/6M_JfKJxNVA.wav). @ represents
cosine similarity, computed at spatial (pixel) level.

##  7 Related tasks to AVS - SSL and SOD

Sound source localization (SSL), saliency object detection (SOD), and audio-
visual segmentation are all related tasks that involve processing audio and
visual information to understand and analyze a scene. AVS aims to identify and
separate audio and visual elements within a scene, such as identifying which
sound sources correspond to which visual objects or events. SSL is limited to
identifying the location of different sound sources in the scene, wherein the
location need not be particularly as fine-grained as the output of an AVS
task. SOD models are trained with the objective to identify visually salient
objects or regions in a visual scene. Tt is important to be noted that there
is no audio cue involved in SOD, or in other words, the salient object may or
may not be a sounding object. To conclude, while all three tasks, localize the
objects in a visual scene, SSL is a more closely related task to AVS, than
SOD. This is also testified from our observations made in Section 4.3 (main
paper) and Table 1 (main paper).

###  7.1 Additional comparison with Sound Source Localization (SSL) methods

In Table 7, we report the results of two state-of-the-art SSL methods, _i.e_.,
LVS [4] and MSSL [40] on the AVSBench MS3 test set. LVS uses the background
and the most confident regions of sounding objects to design a contrastive
loss, followed by obtaining the localization map by computing the audio-visual
similarity. MSSL is a two-stage method for multiple SSL and the localization
map is obtained by Grad-CAM [42].

Table 7: Comparing MoCA with SoTA SSL approaches on AVSBench MS3 test split.  Method | MS3
---|---
| $\rm{M_{IoU}}\uparrow$ | $\rm{F_{score}}\uparrow$
LVS [4] | 0.29 | 0.33
MSSL [40] | 0.26 | 0.36
MoCA(w o MPM) | 0.52 | 0.59
MoCA | 0.57 | 0.62

There is a substantial gap between the results of SSL methods and our MoCA. We
hypothesize this is primarily because, unlike MoCA, the SOTA SSL methods lack
a pixel-level aggregation strategy. This is even straightforward from our
comparison in row 3, _i.e_., MoCA without the mask proposal matching (MPM),
which essentially rules out any contribution of the mask matching from the
pre-trained SAM model. This strengthens our claim regarding the importance of
the pixel matching aggregation (PMA) in MoCA.

###  7.2 How does MoCA perform in a SSL test setup?

Following the test setup adopted in [21], we present our scores if MoCA was to
be utilized for the SSL task. Similar to [21], we present our scores on the
VGGSound-Single dataset [5] (Table 8). Owing to the fine-grained localization,
MoCA surpasses SSL methods in both<EMAIL_ADDRESS>as well as AUC.

Method |<EMAIL_ADDRESS>| AUC
---|---|---
Mix-Loc[21] | 36.3 | 38.9
EZ-VSL[34] | 38.9 | 39.5
MoCA | 40.6 | 43.5
Table 8: SSL on VGGSound-Single dataset [5]

##  8 Qualitative analysis on the AVSS (semantic segmentation) split.

Figure 8: Qualitatively comparing MoCA with AVSBench (supervised) method, on
the AVSS split.

For obtaining semantic labels for AVSS using MoCA, we utilize our baseline,
SAM-BIND. The use of MPM stage is same as the inference of S4 and MS3 setups,
but, we distinguish between the mapped MPM masks if their IoU$<$0.3. As shown
depicted in Fig. 8 RGB frame, (b) GT, (c) AVSBench-Supervised[51], (d) MoCA.
We encircle the regions where [51] over-segments or under-segments objects. In
frames$\\{1,2\\}$, compared to MoCA, [51] over-segments the silent person in
the background. In frames$\\{3,4\\}$, [51] fails to segment the gun owing to
its small size. Again, in frames$\\{5,6\\}$, [51] fails to segment both, the
lady in the audience (bottom-left) and the instrument (top-right). Overall,
the efficacy of our pixel matching aggregation (PMA), weighing both the audio
and visual modality appropriately, is shown in the last row.

##  9 Ablation study on varying the margin in triplet loss

The margin parameter defines a minimum difference that should be maintained
between the distances of the anchor-positive pair and the anchor-negative
pair. We ablate this hyper-parameter and report the performance of MoCA on the
AVSBench MS3 test split in Table 9. We observe that optimal performance in
terms of both the metrics, is obtained with a margin of 0.3, we argue that
choosing a smaller margin makes the loss function that is too easy to satisfy
and the model might converge to a solution where the positive and negative
pairs are not well separated. Moreover, increasing the margin further up to
0.5, enforces a stricter condition on the distances, making the optimization
problem more challenging.

Table 9: Effect of varying the margin in triplet loss, on the performance of AVSBench MS3 test split. Margin | MS3
---|---
($\alpha$) | $\rm{M_{IoU}}\uparrow$ | $\rm{F_{score}}\uparrow$
0.1 | 0.54 | 0.56
0.3 | 0.57 | 0.62
0.5 | 0.56 | 0.58

##  10 Baseline approaches

For tackling this newly proposed problem, Unsupervised AVS, towards
establishing strong baselines, we formulate a novel Cross-Modality Semantic
Filtering (CMSF) approach to accurately associate the underlying audio-mask
pairs by leveraging the off-the-shelf multi-modal foundation models (_e.g_.,
detection [13], open-world segmentation [25] and multi-modal alignment [11]).
Guiding the proposal generation by either audio or visual cues, we design two
training-free variants: AT-GDINO-SAM and OWOD-BIND, respectively. Figure 9
shows a pictorial representation of our baseline approaches.

Figure 9: Overview of baseline models: Bounding box proposals are generated
using a cascade of two pre-trained models: Audio Tagging and GroundingDINO.
Favourable boxes are passed as input to SAM to yield segmentation masks.

##  11 Ablation study on varying the IOU match for mask proposal matching
(MPM)

In the MPM stage, we match the audio-enhanced clustered features (generated as
an outcome of the PMA) with the proposal masks from the pre-trained SAM model.
The proposal masks represent all visual objects present in the scene, whereas
the PMA clusters represent the sounding object. In Table 10, we vary the IOU
threshold that is set to choose a proposal mask as the part of the final AVS
output. In simpler words, when matching the PMA cluster with the proposal
mask, if the IOU is above a certain threshold only then we consider the
proposal mask as a part of the final AVS output. A lower threshold yields
"white" masks, since almost every proposal object mask is considered into the
final output. Conversely, a higher threshold, yields "black" masks, since
almost no object is shown in the final output due to poor match with the PMA
clusters. We observe that generally a higher value of threshold is preferred.

Table 10: Effect of varying the IOU match for mask proposal matching (MPM), on the performance of AVSBench MS3 test split. IOU | MS3
---|---
threshold | $\rm{M_{IoU}}\uparrow$ | $\rm{F_{score}}\uparrow$
0.1 | 0.35 | 0.39
0.3 | 0.48 | 0.48
0.5 | 0.57 | 0.62
0.7 | 0.52 | 0.54
0.9 | 0.54 | 0.54