# Unveiling and Mitigating Bias in Audio Visual Segmentation

Peiwen Sun  Beijing University of Posts and TelecommunicationsBeijingChina
<EMAIL_ADDRESS>,  Honggang Zhang  Beijing University of Posts and
TelecommunicationsBeijingChina<EMAIL_ADDRESS>and  Di Hu  Renmin University
of ChinaBeijingChina<EMAIL_ADDRESS>
(2024)

###### Abstract.

Community researchers have developed a range of advanced audio-visual
segmentation models aimed at improving the quality of sounding objects’ masks.
While masks created by these models may initially appear plausible, they
occasionally exhibit anomalies with incorrect grounding logic. We attribute
this to real-world inherent preferences and distributions as a simpler signal
for learning than the complex audio-visual grounding, which leads to the
disregard of important modality information. Generally, the anomalous
phenomena are often complex and cannot be directly observed systematically. In
this study, we made a pioneering effort with the proper synthetic data to
categorize and analyze phenomena as two types “audio priming bias” and “visual
prior” according to the source of anomalies. For audio priming bias, to
enhance audio sensitivity to different intensities and semantics, a perception
module specifically for audio perceives the latent semantic information and
incorporates information into a limited set of queries, namely active queries.
Moreover, the interaction mechanism related to such active queries in the
transformer decoder is customized to adapt to the need for interaction
regulating among audio semantics. For visual prior, multiple contrastive
training strategies are explored to optimize the model by incorporating a
biased branch, without even changing the structure of the model. During
experiments, observation demonstrates the presence and the impact that has
been produced by the biases of the existing model. Finally, through
experimental evaluation of AVS benchmarks, we demonstrate the effectiveness of
our methods in handling both types of biases, achieving competitive
performance across all three subsets. Page: https://gewu-
lab.github.io/bias_in_AVS/

Audio-Visual Segmentation, Multimodal Bias, Multimodal Learning

††journalyear: 2024††copyright: acmlicensed††conference: Proceedings of the
32nd ACM International Conference on Multimedia; October 28-November 1, 2024;
Melbourne, VIC, Australia††booktitle: Proceedings of the 32nd ACM
International Conference on Multimedia (MM ’24), October 28-November 1, 2024,
Melbourne, VIC, Australia††doi: 10.1145/3664647.3680802††isbn:
979-8-4007-0686-8/24/10††ccs: Computing methodologies Scene understanding

##  1\. Introduction

Figure 1. The illogical anomalies caused by biases. (a) Even if each audio
yields a satisfactory mask separately, the dominance still exists when they
overlap. (b) In training data, the piano generally sounds. During testing,
regardless of the sound presented, the model still tends to prioritize the
piano. These two phenomena of the illogical anomalies can be categorized into
“audio priming bias”, and “visual prior”, which typically are simultaneously
observed as a general impediment in the AVS model.

In the real world, audio and visual are two closely related modalities that
provide fundamental perception in regular life, often applied in forms such as
videos. One classic task within the field of audio-visual understanding is
Audio-Visual Localization (AVL) (Senocak et al., 2018; Hu et al., 2021), which
enables the unsupervised localization of sounding objects by utilizing sound
as guidance. With the increasing demand for stronger perceptual capabilities
in autonomous driving (Van der Heiden et al., 2018; Zürn and Burgard, 2022),
and embodied intelligence (Gupta et al., 2021; Nilsson et al., 2021), it is
urgent to establish finer grounding beyond bounding boxes and heat maps
between audio and visual elements. Therefore, a new task called Audio-Visual
Segmentation (AVS) has emerged. It introduces a pixel-level, fine-grained
scene understanding, bringing novel challenges to the field of audio-visual
understanding. Furthermore, the performance of AVS showcases the current
ability of machines to understand and integrate modalities in complex
scenarios.

The existing works on AVS can be broadly categorized into fusion-based (Zhou
et al., 2022; Liu et al., 2023a; Ling et al., 2023; Huang et al., 2023; Li et
al., 2023b) and prompt-based methods (Wang et al., 2023; Gao et al., 2023; Mo
and Tian, 2023; Liu et al., 2023d) according to Wang et al. (Wang et al.,
2023). The former primarily integrates audio and visual information at
different stages to localize sounding objects, while the latter focuses on
generating effective audio prompts and further finetuning the visual
foundation model pre-trained on large segmentation datasets (Zhou et al.,
2019; Kirillov et al., 2023). Currently, prompt-based methods are the cutting-
edge models in this field, demonstrating exceptional performance. However,
researchers have observed illogical anomalies in these approaches (Yuanhong et
al., 2023; Wang et al., 2023). As illustrated in Fig. 1, even when the mask of
the object seems plausible, two anomalous but interesting phenomena can be
observed. 1) The segmentation results after overlaying audio do not align with
the superimposition of masks guided by separate audios, even when each audio
yields a satisfactory mask. Instead, dominance occurs. 2) Regardless of
whether the piano produces sound, the model usually tends to segment the
piano. Phenomena like Fig. 1 have a wide impact on the current model, which
makes analyzing and addressing these phenomena an urgent work to improve the
grounding behavior.

Generally, the phenomena depicted on the left and right sides of Fig. 1 are
often coupled and cannot be directly observed individually. We manage to
control the influencing factor with artificial data, including volume, etc.,
and reasonably categorize phenomena into two types of biases: “audio priming
bias” and ”visual prior”. Specifically, “audio priming bias” refers to the
phenomenon that the model tends to focus on audio salient content but not
whole content. This bias is characterized by the model’s insensitivity to
audio of specific intensity or semantics and consequently the dominance of
certain semantic audio in multi-source scenarios. As for “visual prior”, it
refers to the phenomenon that the model may directly segment the common-
sounding objects. The most common observed form of such bias is that
regardless of the audio guidance employed, the model consistently segments the
whole or part of certain objects. It is evident that both of these biases
significantly impact the behavior of the current AVS models.

To provide the community with a comprehensive solution to address these
biases, we adopt the divide-and-conquer thought. For audio priming bias, we
first introduce semantic-aware active queries. These active queries contain
rich latent semantic information gathered by the perception module, enhancing
the sensitivity to audio cues. Furthermore, we employ a customized interaction
mechanism in the transformer decoder specially designed to enhance the active
queries and suppress the dormant ones for better collaboration of specific
audio semantics. As for visual priors, we explore three types of contrastive
debias strategies without changing the structure of the model. Our findings
indicate that the soft and gradual debias strategy based on uncertainty yields
superior results across various AVS methods including ours.

Finally, through experimental evaluation of AVS benchmarks, we demonstrate the
effectiveness of our methods in handling both types of biases, achieving
highly competitive performance across all three subsets. In summary, our
contributions are threefold:

  * •

We make a pioneering attempt to categorize the complex phenomena into two
types of bias: “audio priming bias” and “visual prior” with necessary
observation and analysis.

  * •

For audio priming bias, we propose semantic-aware active queries and
customized interaction mechanisms to improve the sensitivity and cooperation
of complex audio scenarios. Then, for visual prior, the novel debias
strategies are utilized to contrastively reorganize the distribution of
logits.

  * •

Experimental results reveal that the proposed method and strategy effectively
mitigate biases while achieving versatility and comparable performance.

##  2\. Related Works

###  2.1. Audio-Visual Segmentation

Before the emergence of AVS, traditional AVL tasks (Senocak et al., 2018; Hu
et al., 2021; Park et al., 2023) relied on unsupervised learning that used
bounding boxes or coarse heat maps for predicting the positions of sounding
objects. However, for applications such as autonomous driving (Van der Heiden
et al., 2018; Zürn and Burgard, 2022), and embodied intelligence (Gupta et
al., 2021; Nilsson et al., 2021), a more precisely multimodal grounding task
to localize sounding objects at a pixel level is required. This is where AVS
comes into the stage. Before getting into the supervised methods, both self-
supervise (Rouditchenko et al., 2019) and weakly-supervise (Mo and Raj, 2024)
methods certainly alleviate the problem of the huge and unnecessary annotation
cost of AVS. The existing supervised AVS approaches can be broadly categorized
into fusion-based (Zhou et al., 2022; Liu et al., 2023a; Ling et al., 2023;
Huang et al., 2023; Li et al., 2023b; Yuanhong et al., 2023) and prompt-based
(Gao et al., 2023; Wang et al., 2023; Liu et al., 2023d; Yan et al., 2023;
Wang et al., 2024a, b; Ma et al., 2024). The pioneering fusion-based work
(Zhou et al., 2022) employs a multi-stage strategy to integrate audio with
multi-scale visual features. Building upon this, CATR (Li et al., 2023b)
further proposes a segmentation paradigm that combines both temporal and
fusion information. Taking the thought of bidirectional generation, Hao el al.
(Hao et al., 2023) achieve a further boosting on the performance. In contrast,
prompt-based methods like AVSegFormer (Gao et al., 2023) and GAVS (Wang et
al., 2023) directly decode fused features using projected audio queries.
Notably, GAVS leverages rich visual information to achieve a generalizable
model in zero-shot and few-shot scenarios. Moreover, taking a holistic
perspective, Yan et al. (Yan et al., 2023) present a general framework for AVS
and Ref-VOS (Wu et al., 2022) (Referring Video Object Segmentation), which
both are cross-model guided video segmentation. However, previous AVS studies
have primarily focused on achieving better multimodal fusion and plausible
masks from the framework perspective, without delving into the challenges
underlying the establishment of reasonable grounding.

###  2.2. Bias in Cross-Model Guided Tasks

Figure 2. The illustration of audio priming bias. We have discovered that
higher audio intensity results in stronger guiding capabilities in the green
block. Also, the audio with distinct semantic attributes is easier to learn
and possesses stronger guiding capabilities in the grey block. For instance,
music has a greater guiding capability compared to human sound. Consequently,
the diverse guiding capabilities cause the phenomenon of dominance in the red
block.

Due to the widespread interest in the Visual Question Answering (VQA) task
(Antol et al., 2015; Anderson et al., 2018), researchers have tackled two
types of bias, namely “visual priming bias” and “language prior”, as discussed
in previous works (Ramakrishnan et al., 2018; Yang et al., 2021). “Visual
priming bias” was initially identified by Antol et al. (Antol et al., 2015),
where models tend to prioritize visually salient content while disregarding
the other context (Goyal et al., 2017; Yang et al., 2021). Subsequently, the
issue of ”language prior” has gained attention from researchers (Ramakrishnan
et al., 2018; Agrawal et al., 2018; Yang et al., 2021), referring to the
tendency of models to generate statistically common answers (e.g. the problem
of green and yellow banana). The approaches for addressing the aforementioned
bias can be broadly categorized into non-extra-based methods (Ramakrishnan et
al., 2018; Cadene et al., 2019; Clark et al., 2019), and extra-based methods,
involving additional human supervision (Wu and Mooney, 2019; Selvaraju et al.,
2019). Different from extra-based methods, non-extra-based methods introduce a
branch of the question-only model and conduct joint training with the target
VQA model by specific learning strategies. Researchers have proposed several
customized learning strategies including sharing the same question
encoder(Ramakrishnan et al., 2018), blocking the backpropagation of the
question-only model (Cadene et al., 2019), pretraining question-only model
(Clark et al., 2019). Moreover, it has been discovered recently that there is
also a certain similarity between bias in VQA and hallucination (Leng et al.,
2023; Zhu et al., 2024) in the popular Multimodal Large Language Model (MLLM).

However, the bias in AVS has never been explored by researchers before. Here,
we hold the holistic opinion that VQA and AVS, as cross-model guided tasks,
share certain similarities. VQA involves obtaining answers to questions based
on image cues, while AVS involves obtaining masks from images based on audio
cues. Therefore, in simple terms, VQA can be understood as a symmetrical
problem to AVS in terms of modalities. So, the anomalous phenomena observed in
AVS can potentially be explained by the bias theory present in VQA.

##  3\. Problem and Analysis

Based on the same thought of bias categories in VQA (Yang et al., 2021), we
similarly classify the phenomena of anomalies into two types, “audio priming
bias”, and “visual prior” in Fig. 1. The segmentation failure cases observed
in the previous model often tend to be a combination of both biases.

###  3.1. Audio Priming Bias

The phenomenon that the model tends to focus on audio salient content but not
whole content is called “audio priming bias”. Firstly, through observation
experiments involving manual intervention in audio (detailed in the appendix),
we have observed the phenomenon of audio priming bias as illustrated in Fig.
2. Hence, the following observations can be discerned. 1) Audio with different
intensities demonstrates varying guiding capability, as shown in Fig. 2 green
block. 2) When controlling other variables including volume, we can observe a
clear variance of the guiding capability by different semantics through the
box plot. 3) In cases where multiple audios are simultaneously present, the
overlaying of audio does not always lead to separate related masks being
superimposed. Instead, the dominant of the audio appears as shown in Fig. 2
red block. Therefore, we contend that the primary determinants of audio
guiding abilities are the intensity and semantic attributes of the audio.
Consequently, the diverse guiding capabilities give rise to the phenomenon of
dominance. Therefore, a well-designed mechanism is demanded to alleviate audio
priming bias.

###  3.2. Visual Prior

The phenomenon that the model may directly segment the common-sounding objects
is called “visual prior”. Figure 4 reveals that, regardless of the audio
provided, the AVS model consistently segments the partial or entire region of
the particular object. Since the piano generally appears with a high sounding
probability in the training data, the model tends to segment once sees the
piano. Statistically, the occurrence frequency in (Zhou et al., 2023) and the
sounding probability in Fig. 4 between different semantics are imbalanced in
the dataset. According to previous works (Li et al., 2023a; Ramakrishnan et
al., 2018; Leng et al., 2023), such preference and distribution provide strong
prior information and make the model inclined to obtain statistically
plausible results, rather than achieving the desired challenging grounding
behavior. Moreover, due to the imbalance of inherent preference and
distribution in the real world, addressing visual prior becomes more
significant.

##  4\. Methods

The overall pipeline for the AVS task is normally constructed by the encoder-
decoder structure in Fig. 3 involving an image encoder, a pixel decoder, and a
transformer decoder. Generally, the multimodal information is fused and
perceived in the transformer decoder (Cheng et al., 2021; Gao et al., 2023; Li
et al., 2023b; Jain et al., 2023). Therefore, the mitigation of audio priming
bias requires an enhancing mechanism in the transformer decoder, while the
mitigation of visual prior requires distribution reorganization after
acquiring the logits. Methodologically, on one hand, to enhance the audio
sensitivity and cooperation of different intensity and semantic attributes, we
introduce the semantic-aware active queries utilizing a perception module and
interaction enhancement mechanism. On the other hand, multiple training
strategies are explored to contrastively optimize the debias model and
reorganize the logits without modifying the structure.

###  4.1. Preliminaries

Given an audio-visual video sequence, we first divide it into $t$ non-
overlapping audio and visual segment pairs with a length of $1s$.

Visual.  After the division, we extract visual representations
$F_{V}\in\mathbb{R}^{t\times d_{V}\times H\times W}$ using a pre-trained Swin-
base model (Dosovitskiy et al., 2020). The visual encoder is frozen and only
tuned with two-layer MLP adapters.

Audio. Similarly, audio representations $F_{A}\in\mathbb{R}^{t\times d_{A}}$
is encoded from VGGish (Gemmeke et al., 2017; Hershey et al., 2017), where $t$
represents the duration of the audio in seconds and matches the number of
video frames. The audio representations are extracted in advance by the frozen
encoder.

Task. Given a dataset
$\mathcal{D}=\\{F_{V}^{i},F_{A}^{i},\mathcal{M}_{gt}^{i}\\}^{n}_{i=1}$
consisting of triplets of images $F_{V}^{i}\in F_{V}$, audio $F_{A}^{i}\in
F_{A}$and masks $\mathcal{M}_{gt}^{i}\in\mathcal{M}_{gt}$, the AVS task is to
learn the mapping $\mathcal{G}(F_{V},F_{A})\rightarrow\mathcal{M}_{gt}$.

###  4.2. Semantic-Aware Active Queries

In transformer decoder of conventional semantic segmentation, each learnable
query ($\in\mathbb{R}^{1\times d_{A}}$ among initialized learnable queries
$q\in\mathbb{R}^{t\times N\times d_{A}}$, where $N$ is the number of learnable
queries in $1s$ interval) functions like a region proposal network (Ren et
al., 2015) and has the ability to generate mask proposals. These queries are
simply inherited by adding the original, projected or attended audio features
$F_{A}$ and propose region without extra semantic regulation, in the AVS model
(Gao et al., 2023; Li et al., 2023b; Wang et al., 2023) as shown in the upper
block in Fig. 3. However, in an ideal AVS model, the learnable queries need to
not only generate regions but also perceive and regulate the interactions
between audio semantics adaptively.

To achieve the basic perception of audio semantics, we first introduce the
following perception module aiming to enhance sensitivity to audio of various
intensities and semantics. The logical and straightforward approach is to
employ a separately trained projection $\Phi(\cdot)$ module with an
optimization objective for audio alone to perform a coarse perception of
latent audio semantics and to finally project to a latent semantic mark
$\Phi(F_{A})\in\mathbb{R}^{t\times N}$ with padding of 0, as

(1) $\displaystyle\Phi(F_{A})[\tau]$
$\displaystyle=\\{1,x_{1},x_{2},...\\},\text{while }\tau\in t$ (2)
$\displaystyle x_{i}=$ $\displaystyle\begin{cases}1&\text{if }F_{A}\in
i^{th}\text{ class or cluster},\\\ 0&\text{otherwise}.\end{cases}$

Here, we use K-means, Gaussian Mixture Models (GMMs), or BEATs (Chen et al.,
2022) to perform projection in both unsupervised and supervised ways. So each
audio segment will be assigned with $1\leq\mathbf{C}\leq 8$ classes or
$\mathbf{C}=1$ clusters. The overall goal of a latent semantic mark is a
coarse filter to have a coarse and basic perception of the audio semantics.
The items encoded by $1$ in $\Phi(F_{A})$ are termed active marks, whereas
those encoded by $0$ in $\Phi(F_{A})$ are termed dormant marks. The active or
dormant states are relative and vary from samples. Afterward, each audio
segment will receive the latent semantic marks
$\Phi(F_{A})\in\mathbb{R}^{t\times N}$ based on the projection mentioned
above. Then the semantic-enhanced audio features are obtained by element-wise
product $\odot$ with broadcast

(3) $F_{A}^{\prime}=\text{repeat}(F_{A},N)\odot\Phi(F_{A}),$

where $F_{A}^{\prime}\in\mathbb{R}^{t\times N\times d_{A}}$ is semantic-
enhanced audio features. Furthermore, we consistently encode the $0^{th}$ mark
out of $N$ as an active mark shown in Eq. 1 to capture global audio semantic
information.

Building upon the semantic-enhanced audio features, we carry out self-
attention $\text{SA}(\cdot)$ with the residual of initialized learnable
queries $q$ and obtain the semantic-aware queries $q^{\prime}$ as

(4) $\displaystyle
q^{\prime}=\text{SA}{(F_{A}^{\prime},F_{A}^{\prime},F_{A}^{\prime})}+q=\text{softmax}\left(F_{A}^{\prime}{F_{A}^{\prime}}^{T}\right)F_{A}^{\prime}+q,$

where $q^{\prime}\in\mathbb{R}^{t\times N\times d_{A}}$ is the semantic-aware
queries. Here, to enhance conciseness, the equations in this paper for
attention have excluded projection and scaling factors. The semantic-aware
queries obtained are utilized as the learnable query inputs for the
transformer decoder layers. Similarly, the queries in the same position
corresponding to latent semantic active marks are active queries
$q^{\prime}_{a}$, while those corresponding to dormant marks are dormant
queries $q^{\prime}_{d}$, where
$\\{q^{\prime}\\}=\\{q^{\prime}_{a}\\}\cup\\{q^{\prime}_{d}\\}$. Moreover, the
adequate number of active queries $q^{\prime}_{a}$ per segment is ensured in
multiple ways and please refer to the appendix for more details.

Figure 3.  In conventional semantic segmentation, the learnable queries are
solely responsible for generating regions. However, in an ideal AVS model with
a similar structure, the learnable queries need to not only generate regions
but also perceive and regulate the interaction between audio semantics. So,
this mechanism is designed to enhance the understanding of latent semantics
and improve the interaction to only occur between semantic-aware active
queries.

###  4.3. Enhancing Interaction within Active Queries

Figure 4. Illustration of visual prior. (a) The model always tends to learn
statistically plausible results, rather than achieve the harder desired
grounding behavior. Note: the bar chart uses blue to represent the proportion
of the object being present in the image and emitting sound, while the orange
color represents the proportion of the object being present but not emitting
sound. As an example, since the piano generally appears with a high sounding
probability in the training data, the model will segment once sees the piano.
(b) To deal with the visual prior, we introduce debias strategies through the
idea of contrasting the audio-visual model with the biased branch. (c) The
ideal result is the mask without visual prior.

After introducing active and dormant queries, we observed that regions
proposed by dormant queries are sometimes unnecessarily optimized and
backpropagated. It is clearly detrimental since dormant queries do not contain
adequate audio semantic information. The widely used transformer decoder layer
in per-mask segmentation consists of three modules to process learnable query
features $q\in\mathbb{R}^{N\times t\times d_{A}}$ in the following order: a
cross-attention $\text{CA}(\cdot)$, a self-attention module
$\text{SA}(\cdot)$, and a feed-forward network $FFN(\cdot)$. For the original
transformer decoder, the interactions occur between all learnable query
features, and the same query can propose regions of different semantics.
Original cross-attention and self-attention in $l^{th}$ layer can be denoted
as

$\displaystyle q_{(l)}^{ca}$
$\displaystyle=\text{CA}(q_{(l-1)},F_{V_{(l)}},F_{V_{(l)}})+q_{(l-1)}$ (5)
$\displaystyle=\text{softmax}(\mathbf{m}_{(l-1)}+q_{(l-1)}F_{V_{(l)}})F_{V_{(l)}}+q_{(l-1)},$
$\displaystyle q_{(l)}^{sa}$
$\displaystyle=\text{SA}(q_{(l)}^{ca},q_{(l)}^{ca},q_{(l)}^{ca})+q_{(l)}^{ca}$
(6)
$\displaystyle=\text{softmax}(q_{(l)}^{ca}{q_{(l)}^{ca}}^{T})+q_{(l)}^{ca},$

where $l$ is the layer index, $q_{(l)}$ refers to the query features
corresponding to the learnable queries $q$. $q_{(l)}^{ca}$ and $q_{(l)}^{sa}$
is the temporary state after the cross-attention $\text{CA}(\cdot)$ and after
self-attention $\text{SA}(\cdot)$ respectively. Note, $F_{V_{(l)}}$ are the
image features under transformation, $\mathbf{m}_{(l)}$ is the masked
attention introduced by Chen et al. (Cheng et al., 2021).

However, we argue that audio interactions should only occur between active
queries related to perceived latent semantics. This ensures that a specific
set of active queries should only propose the region of the same semantics.

(7) $\displaystyle{q_{a}^{ca}}_{(l)}$
$\displaystyle=\text{CA}({{q^{\prime}_{a}}_{(l-1)}},F_{V_{(l)}},F_{V_{(l)}})+{q^{\prime}_{a}}_{(l-1)},$
(8) $\displaystyle{q^{sa}_{a}}_{(l)}$
$\displaystyle=\text{SA}({q_{a}^{ca}}_{(l)},{q_{a}^{ca}}_{(l)},{q_{a}^{ca}}_{(l)})+{q_{a}^{ca}}_{(l)},$

where only active query features ${q^{\prime}_{a}}_{(l-1)}$ are updated to
${q^{\prime}_{a}}_{(l)}$ by $FFN({q^{sa}_{a}}_{(l)})$, and dormant query
features ${q^{\prime}_{d}}_{(l)}$, are masked during backpropagation. Finally,
based on the ${q^{\prime}_{a}}_{(l)}$ in each layer, the mask logits
$\in\mathbb{R}^{T\times N\times H\times W}$ and class logits
$\in\mathbb{R}^{T\times N\times\mathcal{K}}$ are obtained, where $\mathcal{K}$
is the number classes of semantics ($\mathcal{K}$=$1$ for binary).

Following each transformer decoder layer, query matching aims to determine
which of the predicted regions fits the referred objects. Therefore, we match
the predicted regions from the active queries and the ground truth regions by
minimizing the matching cost

(9)
$\begin{gathered}(\mathcal{M},\mathcal{M}_{gt})=\underset{\mathcal{M}\in\mathcal{M}_{q^{\prime}_{a}}}{\arg\min}C_{\text{match}}\left(\mathcal{M},\mathcal{M}_{gt}\right),\end{gathered}$

where $\mathcal{M}_{q^{\prime}_{a}}$ are the regions proposed by
$q^{\prime}_{a}$, and $\mathcal{M}_{gt}$ is the ground truth of original
image. $\mathcal{M}_{q^{\prime}_{a}}$ is calculated by multiplying mask logits
and class logits, so the cost $C_{\text{match}}$ evaluates the predicted
region in both mask and semantic level. In practice, we separately compute the
costs of mask and class logits and jointly optimize the summation as
$C_{\text{match}}$. Then, the losses $\mathcal{L}$ are calculated in every
intermediate transformer decoder layer, and only the one in the final layer is
given a higher weight and used for inference.

Based on the method above, interactions related to active queries are retained
and enhanced, while interactions of dormant queries are suppressed with a
reduction of the computational workload. Although this method slightly reduces
the number of proposed valid regions $\mathcal{M}_{q^{\prime}_{a}}$, this
trade-off is proven advantageous for better cooperation between audio
semantics in further experiments.

###  4.4. Contrastive Debias Strategy

To recognize bias and then avoid the biased region appearing along with the
referred region, it is a natural idea to introduce a new biased branch and
conduct joint training to the current AVS model without modifying their
structures. We first briefly bring up two simple and direct methods and carry
out analysis and experiment for comparison. Subsequently, a well-designed
uncertainty-based strategy is proposed with competitive performance and
versatility. To maintain brevity, operations on $logits$ in Sec. 4.4 are
applied to both mask logits and class logits separately.

To begin with, the simplest approach to reorganize the distribution is the
direct logits ensemble strategy of separately trained biased model
$\mathcal{G}_{b}(F_{V})\rightarrow\mathcal{M}_{gt}$ and vanilla AVS model
$\mathcal{G}(F_{V},F_{A})\rightarrow\mathcal{M}_{gt}$, and subsequently merge
their output logits using simple operations such as weighted subtraction and
multiplication. As the second strategy, the bias-only strategy requires a new
objective of training by involving a bias-only branch with the mapping of
$\mathcal{G}_{b}(F_{V})\rightarrow\mathcal{M}_{gt}$ and jointly optimize the
overall model with a bias-only branch. To obtain the debias logits, we simply
compute an element-wise product $\odot$ of the logits of $\mathcal{G}$ and
$\mathcal{G}_{b}$

(10)
$logits(\bar{\mathcal{G}}(F_{V},F_{A}))=logits(\mathcal{G}(F_{V},F_{A}))\odot\sigma(logits(\mathcal{G}_{b}(F_{V}))),$

where $\bar{\mathcal{G}}$ is the debias projection of the triplets
$\mathcal{D}$ and $\sigma$ is the sigmoid function. The final optimization
goal can be calculated from the logits as

(11)
$\mathcal{L}_{\text{bias}}(\mathcal{G},F_{V},F_{A},\mathcal{G}_{b})=\mathcal{L}(\bar{\mathcal{G}},F_{V},F_{A})+\mathcal{L}(\mathcal{G}_{b},F_{V}),$

Based on the updated logits, the region of the man in Fig. 4 will be given
fewer logits and yield a greater loss than the original.

Although plausibly reorganizing the logits distribution, these two methods
demonstrate fluctuations in experiments and do not consistently improve
performance in popular methods. These issues can be attributed to the hard and
direct introduction of the biased branch, which forces the network to learn
from the large distribution gap, resulting in fluctuations in gradient
direction and magnitude. To address this, we propose a soft and gradual
strategy by introducing audio uncertainty through Gaussian noise.

Our proposed uncertainty-based strategy provides a gradual and soft way to
estimate the biased output distribution. It does not require multitask-like
loss but involves contrast $\mathcal{G}_{n}$ output distributions derived from
image and distorted audio inputs. We first follow the idea of the forward
diffusion process $r(F_{A(T)}\mid F_{A(0)})$ in image generation (Ho et al.,
2020) to construct the distorted audio

(12) $\displaystyle r\left(F_{A(\tau)}\mid
F_{A(\tau-1)}\right)=\mathcal{N}\left(F_{A(\tau)};\sqrt{1-\beta}F_{A(\tau-1)},\beta\mathbf{I}\right),$
$\displaystyle r\left(F_{A(T)}\mid
F_{A(0)}\right)=\prod_{\tau=1}^{T}r\left(F_{A(\tau)}\mid
F_{A(\tau-1)}\right),$

where $F_{A(0)}$ denotes the original audio features without any noise and $I$
refers to an identity matrix. We incrementally add a small amount of Gaussian
noise for $T$ steps. As step $\tau$ goes larger, the amount of noise added in
each step is controlled by $\beta$. Finally, if $T\rightarrow\infty$,
$F_{A(T)}$ will be completely distorted. Then, the final logits are updated as

(13) $\displaystyle logits(\bar{\mathcal{G}}(F_{V},F_{A}))=$
$\displaystyle(\alpha+1)logits(\mathcal{G}(F_{V},F_{A}))$
$\displaystyle-\alpha logits(\mathcal{G}_{n}(F_{V},F_{A(T)})),$

where larger $\alpha$ values indicate a stronger amplification of differences
between the two distributions ($\alpha$ = 0 reduces to regular logits). During
inference, $T\rightarrow\infty$, so $F_{A(T)}$ only contains Gaussian noise
for the biased branch. Based on such logits, the overall loss

(14)
$\mathcal{L}_{\text{uncertainty}}=\mathcal{L}(\bar{\mathcal{G}},F_{V},F_{A}),$

is calculated by updated logits. Finally, the loss and mask result will be
calculated from the debias $logits(\bar{\mathcal{G}}(F_{V},F_{A}))$. In
summary, replacing the hard bias-only branch with a gradually distorted audio
branch provides a soft and learnable transition for the model to reorganize
the distribution caused by visual prior.

###  4.5. Loss and Inference

Considering both masks and semantics, the overall loss $\mathcal{L}$ mentioned
in Sec.4.4 is obtained within every intermediate transformer decoder layer by
mask loss $\mathcal{L}_{mask}$ (Lin et al., 2017) and class loss
$\mathcal{L}_{cls}$ based on the updated mask logits and class logits by
debias strategy as

(15) $\displaystyle\mathcal{L}$
$\displaystyle=\mathcal{L}_{mask}+\lambda_{cls}\cdot\mathcal{L}_{cls},$ (16)
$\displaystyle\mathcal{L}_{mask}$
$\displaystyle=\lambda_{focal}\cdot\mathcal{L}_{focal}+\lambda_{dice}\cdot\mathcal{L}_{dice},$

where $\mathcal{L}_{mask}$ is implemented by a combination of the dice loss
$\mathcal{L}_{dice}$ [29] and the focal loss $\mathcal{L}_{focal}$ functions,
and $\mathcal{L}_{cls}$ is implemented by the cross-entropy loss.After
adopting the active queries, both losses
$\mathcal{L}_{mask},\mathcal{L}_{cls}$ are calculated and backpropagated
within the matched pairs of masks $\mathcal{M}$ from active queries and ground
truth $\mathcal{M}_{gt}$. During inference, the overall logits
$\in\mathbb{R}^{(\mathcal{K}+1)\times H\times W}$ are obtained by the
multiplication of mask logits and class logits in the last layer.

To address the shortcut effect resulting from easily learned regions proposed
by specific queries, we drop the relatively well-learned pairs
$(\mathcal{M},\mathcal{M}_{gt})$ under the probability of $p$ in the
calculation of $\mathcal{L}$ and empirical results show slight improvements.

Figure 5. (a) The performance comparison of different methods on V1M under
certain intensity conditions. Our method brings more sensitivity to low-
intensity scenarios. (b) The performance comparison of different methods on
our Co-AVS subsets. Our method has advantages in both performance and
robustness. Table 1. Comparison of performance on all three subsets of AVS-
Benchmarks. The bold score is utilized to represent optimal results. “-”
refers to the methods that do not support the corresponding subsets. Except
for specific references, the “strategy” mentioned in the comparison refers to
the uncertainty-based strategy that achieves the best outcomes.

Method | Audio-backbone | Visual-backbone | V1S | V1M | AVSS
---|---|---|---|---|---
mIoU(%) | F-score | mIoU(%) | F-score | mIoU(%) | F-score
AVSBench (Zhou et al., 2022) | VGGish | PVT-v2 | 78.70 | 0.879 | 54.00 | 0.645 | 29.77 | 0.352
AVSC (Liu et al., 2023b) | VGGish | PVT-v2 | 81.29 | 0.886 | 59.50 | 0.657 | - | -
AVS-BG (Hao et al., 2023) | VGGish | PVT-v2 | 81.71 | 0.904 | 55.10 | 0.668 | - | -
AQFormer (Huang et al., 2023) | VGGish | PVT-v2 | 81.60 | 0.894 | 61.10 | 0.721 | - | -
AuTR (Liu et al., 2023a) | VGGish | Swin-Base | 80.40 | 0.891 | 56.20 | 0.672 | - | -
BAVS (Liu et al., 2023c) | BEATs | Swin-Base | 82.68 | 0.898 | 59.63 | 0.659 | - | -
SAMA (Liu et al., 2023d) | VGGish | ViT-Huge | 81.53 | 0.886 | 63.14 | 0.691 | - | -
†CATR (Li et al., 2023b) | VGGish | PVT-v2 | 81.40 | 0.896 | 59.00 | 0.700 | 36.66 | 0.420
AV-SAM (Mo and Tian, 2023) | ResNet18 | ViT-Base | 40.47 | 0.566 | - | - | - | -
AVSBG (Hao et al., 2023) | VGGish | PVT-v2 | 81.71 | 0.904 | 55.10 | 0.668 | - | -
GAVS (Wang et al., 2023) | VGGish | ViT-Base | 80.06 | 0.902 | 63.70 | 0.774 | - | -
AVSegFormer (Gao et al., 2023) | VGGish | PVT-v2 | 82.06 | 0.899 | 58.36 | 0.693 | 32.80 | 0.385
MUTR (Yan et al., 2023) | VGGish | Video-Swin-Base | 81.60 | 0.897 | 64.00 | 0.735 | - | -
Ours (w/o. strategy) | VGGish | Swin-base | 82.92 | 0.928 | 66.12 | 0.792 | 41.93 | 0.476
Ours (w/. strategy) | VGGish | Swin-base | 83.31 | 0.930 | 67.22 | 0.808 | 44.42 | 0.499

  * †

CATR: To make a fair comparison, the results of CATR here are without
supplemented annotation of the training set.

##  5\. Experiments

###  5.1. Implementation Details

Dataset. Our proposed method is evaluated on the AVS Benchmarks (Zhou et al.,
2022), which contains three subsets. Firstly, the single-source subset V1S
(S4) contains 4932 videos over 23 categories. In this subset, only the first
sampled frame is annotated. Secondly, the multi-source subset V1M (MS3)
contains 424 videos that include two or more sound sources and all sounding
objects are visible in the frames. Finally, the AVSBench-semantic (AVSS)
subset, an extension of V1S and V1M, contains 12,356 videos of $10s$,
differing from the $5s$ videos in V1S and V1M. Additionally, to validate the
cooperation performance in complex audio scenarios, a small multi-source test
set of AVS called Co-AVS is created, containing 450 videos of $5s$. Original
audio clips are replaced with synthesized audio clips of 6 different levels
intensities of each semantics, like the red block in Fig. 2.

Setting. We conduct training and evaluation using the VGGish backbone
pretrained on Youtube-8M (Abu-El-Haija et al., 2016) and Swin-base Transformer
backbone pretrained on semantic-ADE20K (Zhou et al., 2019) by Mask2Former
(Cheng et al., 2021). The parameters
$\lambda_{focal},\lambda_{dice},\lambda_{cls}$ in the loss are set to
$5,5,2,1$. The AdamW optimizer is adopted with a learning rate of 1e-4 for the
visual encoder adapters and 1e-3 for other learnable parameters, with 60
training epochs. $N$ and $k$ in Fig. 3 is set to 100 and 9. Drop possibility
$p$ in Sec. 4.5 and $\alpha$ in Eq. 13 are set to $0.4$ and $1$ without
elaborate searching. $T$ in Eq.12 are set to 1000.

Metrics. To conduct a comprehensive evaluation of our model, we carry out
tests using mean Intersection over Union (mIoU) and F-score as the performance
metrics (Zhou et al., 2022; Gao et al., 2023; Wang et al., 2023).

Table 2. Ablation study of the methods and strategies proposed in this study
on V1M.

| Contrastive
---
debias
strategy
| Dropping
---
dominant
queries
| Interaction
---
enhancement
| Active
---
query
of $0^{th}$
| Active
---
queries
mIoU(%) | F-score
✓ | ✓ | ✓ | ✓ | ✓ | 67.22 | 0.808
✗ | ✓ | ✓ | ✓ | ✓ | 66.12 | 0.792
✗ | ✗ | ✓ | ✓ | ✓ | 65.95 | 0.790
✗ | ✗ | ✗ | ✓ | ✓ | 62.57 | 0.744
✗ | ✗ | ✗ | ✗ | ✓ | 62.36 | 0.743
✗ | ✗ | ✗ | ✗ | ✗ | 62.16 | 0.734

###  5.2. Results Comparison

When compared to methods that do not incorporate the designed debias method,
our method, which leverages methods for both biases, demonstrates strong
competitiveness. As illustrated in Tab. 1, our method achieves comparable
results across all V1S, V1M, and AVSS subsets, showcasing performance
improvements of up to 0.63%, 3.22%, 7.76% in mIoU, respectively. Even when
considering slight variations in the backbones used by different methods, our
approach consistently outperforms others. The limited improvement observed in
V1S can primarily be attributed to the constrained manifestation of biases
stemming from the singular and simplistic nature of the data. However, we
still believe that this does not undermine our ability to demonstrate the
effectiveness of our bias handling techniques.

Table 3. The analysis of projection strategy to obtain semantic-aware queries
by class or cluster. The ablation of $topK$ can also be regarded as the
ablation of a fixed number of active queries.

method |  filtering |  mIoU(%) | F-score
---|---|---|---
Classification |  $\theta$ = 0.8 |  64.55 | 0.757
$\theta$ = 0.6 |  66.36 | 0.783
$\theta$ = 0.4 |  67.22 | 0.808
$\theta$ = 0.2 |  64.52 | 0.740
$topK$ = 1 |  61.79 | 0.695
$topK$ = 3 |  64.49 | 0.742
$topK$ = 5 |  63.81 | 0.719
Clustering |  Kmeans |  64.18 | 0.746
GMMs |  64.57 | 0.742
None |  None |  63.59 | 0.702

Furthermore, both quantitative performance in Fig. 5 and qualitative analysis
in Fig. 6 indicate that our model successfully alleviates both biases
mentioned before and our predicted masks exhibit superior quality.
Furthermore, due to the increased complexity of synthesized audio scenarios in
Co-AVS subset, achieving a lower performance drop on synthesized audio
necessitates better semantic collaboration. As shown in Fig.5, our
demonstrates stronger robustness on complex audio scenarios.

Figure 6. Our model successfully alleviates both biases and our predicted
masks exhibit superior quality than before. Table 4. Ablation analysis and the
versatility experiments of image-biased strategy on V1M.
“${\color[rgb]{1,0,0}\uparrow}$” signifies a positive effect achieved by
employing the contrastive debiased strategy compared to the vanilla method,
while “${\color[rgb]{0.078125,0.546875,0.39453125}\downarrow}$” indicates a
negative effect.

Method | Backbone | Vanilla | Logits ensemble | Bias only | Uncertainty based
---|---|---|---|---|---
mIoU (%) | Fscore | mIoU (%) | Fscore | mIoU (%) | Fscore | mIoU (%) | Fscore
AVSbench (Zhou et al., 2022) | PVT-v2 | 54.00 | 0.645 | 54.12${\color[rgb]{1,0,0}\uparrow}$ | 0.659${\color[rgb]{1,0,0}\uparrow}$ | 53.43${\color[rgb]{0.078125,0.546875,0.39453125}\downarrow}$ | 0.640${\color[rgb]{0.078125,0.546875,0.39453125}\downarrow}$ | 54.81${\color[rgb]{1,0,0}\uparrow}$ | 0.682${\color[rgb]{1,0,0}\uparrow}$
AVSegformer (Gao et al., 2023) | PVT-v2 | 58.36 | 0.693 | 56.85${\color[rgb]{0.078125,0.546875,0.39453125}\downarrow}$ | 0.662${\color[rgb]{0.078125,0.546875,0.39453125}\downarrow}$ | 58.86${\color[rgb]{1,0,0}\uparrow}$ | 0.687${\color[rgb]{0.078125,0.546875,0.39453125}\downarrow}$ | 59.63${\color[rgb]{1,0,0}\uparrow}$ | 0.718${\color[rgb]{1,0,0}\uparrow}$
GAVS (Wang et al., 2023) | ViT-base | 63.70 | 0.766 | 51.21${\color[rgb]{0.078125,0.546875,0.39453125}\downarrow}$ | 0.738${\color[rgb]{0.078125,0.546875,0.39453125}\downarrow}$ | 64.11${\color[rgb]{1,0,0}\uparrow}$ | 0.788${\color[rgb]{1,0,0}\uparrow}$ | 64.80${\color[rgb]{1,0,0}\uparrow}$ | 0.784${\color[rgb]{1,0,0}\uparrow}$
Ours | Swin-base | 66.12 | 0.792 | 64.42${\color[rgb]{0.078125,0.546875,0.39453125}\downarrow}$ | 0.795${\color[rgb]{1,0,0}\uparrow}$ | 66.50${\color[rgb]{1,0,0}\uparrow}$ | 0.797${\color[rgb]{1,0,0}\uparrow}$ | 67.22${\color[rgb]{1,0,0}\uparrow}$ | 0.808${\color[rgb]{1,0,0}\uparrow}$

###  5.3. Ablation Study

Overall ablation. We conduct an ablation experiment on the module and the
strategy designed for biases. In this experiment, we progressively remove the
modifications of each module in the set of V1M. Through this process, we aimed
to assess the impact of these modifications on the segmentation results. As
demonstrated in Tab. 2, the experiments indicate that the methods we designed
for audio priming bias and visual prior respectively achieved an increase of
3.96% and 1.10% in mIoU. Different modules have all achieved an increase in
both mIoU and F-score, reaching the design objectives. The ablation study
further indicates that the inclusion of active queries alone does not directly
result in a significant increase in performance. It necessitates the
implementation of enhancement of interaction, ultimately leading to a combined
improvement of 3.76%. This finding emphasizes the importance of enhancing
interaction in conjunction with active queries, as they aid the transformer
encoder in establishing audio-visual grounding.

Ablation on projection of Equation 2.  The objective of projection is to map
each audio clip to several classes or a cluster with latent semantic
information. Therefore, we naturally conducted experiments using both
supervised and unsupervised approaches. In the supervised experiments, we
employed Beats (Chen et al., 2022) for supervised multi-class classification
and used threshold $\theta$ or top$K$ strategies to roughly filter the samples
with abstract semantic information in Tab. 3. The results showed that the
different settings in the multi-classification do give an improvement of mIoU
on V1M up to 67.22%. In the unsupervised experiments, we utilized the K-means
and Gaussian Mixture Model (GMM) with Expectation Maximization for clustering,
with the same number of clusters. Moreover, an adequate number of active
queries per sample is ensured here for fairness. The results demonstrated
that, regardless of supervised or unsupervised approaches, introducing latent
semantic information along with active queries consistently improved
performance, with a maximum improvement of 3.63% in Tab. 3.

###  5.4. Versatility of Contrastive Debias Strategies

As general strategies, all three contrastive debias strategies are able to
reorganize the distribution of logits. To validate the versatility, we not
only applied the debias strategies in our framework but also explored its
application in other popular models, as shown in Tab. 4. The performance of
the logits ensemble and bias-only strategy fluctuates, achieving performance
improvements only in certain methods. When considering all four methods, the
uncertainty-based approach emerges as a superior solution due to its general
improvement. Within our framework, it attained mIoU of 67.22% and exhibited
commendable training stability.

##  6\. Conclusion

This study presents a novel and systematic exploration of the bias issues
existing in the previous AVS model. By defining and analyzing these biases as
“audio priming bias” and “visual prior”, we propose targeted solutions for
each bias. For audio priming bias, we introduce semantic-aware active queries
that enhance the audio sensitivity of intensity and semantics with a designed
perception module. Then further interaction of active queries is forced to
collaborate on semantic understanding. For the visual prior, we employ a
contrastive debias strategy that improves the overall performance of the model
without modifying its structure. The meticulously designed methods and
comprehensive experiment across all three subsets, demonstrate the superiority
and versatility of our approach.

## Acknowledgements

This research was supported by National Natural Science Foundation of China
(NO.62106272), and Public Computing Cloud, Renmin University of China.

## References

  * (1)
  * Abu-El-Haija et al. (2016) Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. 2016.  Youtube-8m: A large-scale video classification benchmark.  _arXiv preprint arXiv:1609.08675_ (2016).
  * Agrawal et al. (2018) Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018.  Don’t just assume; look and answer: Overcoming priors for visual question answering. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4971–4980.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6077–6086.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  Vqa: Visual question answering. In _Proceedings of the IEEE international conference on computer vision_. 2425–2433.
  * Cadene et al. (2019) Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. 2019\.  Rubi: Reducing unimodal biases for visual question answering.  _Advances in neural information processing systems_ 32 (2019).
  * Chen et al. (2022) Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. 2022.  Beats: Audio pre-training with acoustic tokenizers.  _arXiv preprint arXiv:2212.09058_ (2022).
  * Cheng et al. (2021) Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G. Schwing. 2021.  Mask2Former for Video Instance Segmentation.  arXiv:2112.10764 [cs.CV]
  * Clark et al. (2019) Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019.  Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases.  _arXiv preprint arXiv:1909.03683_ (2019).
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).
  * Gao et al. (2023) Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. 2023.  AVSegFormer: Audio-Visual Segmentation with Transformer.  arXiv:2307.01146 [cs.CV]
  * Gemmeke et al. (2017) Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017.  Audio set: An ontology and human-labeled dataset for audio events. In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 776–780.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6904–6913.
  * Gupta et al. (2021) Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. 2021.  Embodied intelligence via learning and evolution.  _Nature communications_ 12, 1 (2021), 5721.
  * Hao et al. (2023) Dawei Hao, Yuxin Mao, Bowen He, Xiaodong Han, Yuchao Dai, and Yiran Zhong. 2023.  Improving audio-visual segmentation with bidirectional generation.  _arXiv preprint arXiv:2308.08288_ (2023).
  * Hershey et al. (2017) Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017\.  CNN architectures for large-scale audio classification. In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 131–135.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ 33 (2020), 6840–6851.
  * Hu et al. (2021) Di Hu, Yake Wei, Rui Qian, Weiyao Lin, Ruihua Song, and Ji-Rong Wen. 2021.  Class-aware sounding objects localization via audiovisual correspondence.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 44, 12 (2021), 9844–9859.
  * Huang et al. (2023) Shaofei Huang, Han Li, Yuqing Wang, Hongji Zhu, Jiao Dai, Jizhong Han, Wenge Rong, and Si Liu. 2023.  Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation.  arXiv:2309.09501 [cs.CV]
  * Jain et al. (2023) Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. 2023.  Oneformer: One transformer to rule universal image segmentation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2989–2998.
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023.  Segment Anything.  _arXiv:2304.02643_ (2023).
  * Leng et al. (2023) Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2023.  Mitigating object hallucinations in large vision-language models through visual contrastive decoding.  _arXiv preprint arXiv:2311.16922_ (2023).
  * Li et al. (2023b) Kexin Li, Zongxin Yang, Lei Chen, Yi Yang, and Jun Xun. 2023b.  Catr: Combinatorial-dependence audio-queried transformer for audio-visual video segmentation.  _arXiv preprint arXiv:2309.09709_ (2023).
  * Li et al. (2023a) Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023a.  Evaluating object hallucination in large vision-language models.  _arXiv preprint arXiv:2305.10355_ (2023).
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017.  Focal loss for dense object detection. In _Proceedings of the IEEE international conference on computer vision_. 2980–2988.
  * Ling et al. (2023) Yuhang Ling, Yuxi Li, Zhenye Gan, Jiangning Zhang, Mingmin Chi, and Yabiao Wang. 2023.  Hear to Segment: Unmixing the Audio to Guide the Semantic Segmentation.  arXiv:2305.07223 [cs.SD]
  * Liu et al. (2023b) Chen Liu, Peike Li, Xingqun Qi, Hu Zhang, Lincheng Li, Dadong Wang, and Xin Yu. 2023b.  Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics.  arXiv:2307.16620 [cs.SD]
  * Liu et al. (2023c) Chen Liu, Peike Li, Hu Zhang, Lincheng Li, Zi Huang, Dadong Wang, and Xin Yu. 2023c.  BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledge.  _arXiv preprint arXiv:2308.10175_ (2023).
  * Liu et al. (2023a) Jinxiang Liu, Chen Ju, Chaofan Ma, Yanfeng Wang, Yu Wang, and Ya Zhang. 2023a.  Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation.  arXiv:2307.13236 [cs.SD]
  * Liu et al. (2023d) Jinxiang Liu, Yu Wang, Chen Ju, Ya Zhang, and Weidi Xie. 2023d.  Annotation-free Audio-Visual Segmentation.  _arXiv preprint arXiv:2305.11019_ (2023).
  * Ma et al. (2024) Juncheng Ma, Peiwen Sun, Yaoting Wang, and Di Hu. 2024.  Stepping stones: A progressive training strategy for audio-visual semantic segmentation.  _arXiv preprint arXiv:2407.11820_ (2024).
  * Mo and Raj (2024) Shentong Mo and Bhiksha Raj. 2024.  Weakly-supervised audio-visual segmentation.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Mo and Tian (2023) Shentong Mo and Yapeng Tian. 2023.  AV-SAM: Segment anything model meets audio-visual localization and segmentation.  _arXiv preprint arXiv:2305.01836_ (2023).
  * Nilsson et al. (2021) David Nilsson, Aleksis Pirinen, Erik Gärtner, and Cristian Sminchisescu. 2021.  Embodied visual active learning for semantic segmentation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 2373–2383.
  * Park et al. (2023) Sooyoung Park, Arda Senocak, and Joon Son Chung. 2023.  Marginnce: Robust sound localization with a negative margin. In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 1–5.
  * Ramakrishnan et al. (2018) Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. 2018.  Overcoming language priors in visual question answering with adversarial regularization.  _Advances in Neural Information Processing Systems_ 31 (2018).
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ 28 (2015).
  * Rouditchenko et al. (2019) Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh McDermott, and Antonio Torralba. 2019.  Self-supervised audio-visual co-segmentation. In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2357–2361.
  * Selvaraju et al. (2019) Ramprasaath R Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. 2019.  Taking a hint: Leveraging explanations to make vision and language models more grounded. In _Proceedings of the IEEE/CVF international conference on computer vision_. 2591–2600.
  * Senocak et al. (2018) Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. 2018.  Learning to localize sound source in visual scenes. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4358–4366.
  * Van der Heiden et al. (2018) Remo MA Van der Heiden, Christian P Janssen, Stella F Donker, Lotte ES Hardeman, Keri Mans, and J Leon Kenemans. 2018.  Susceptibility to audio signals during autonomous driving.  _PloS one_ 13, 8 (2018), e0201963.
  * Wang et al. (2023) Yaoting Wang, Weisong Liu, Guangyao Li, Jian Ding, Di Hu, and Xi Li. 2023.  Prompting Segmentation with Sound is Generalizable Audio-Visual Source Localizer.  _arXiv preprint arXiv:2309.07929_ (2023).
  * Wang et al. (2024a) Yaoting Wang, Peiwen Sun, Yuanchao Li, Honggang Zhang, and Di Hu. 2024a.  Can Textual Semantics Mitigate Sounding Object Segmentation Preference?  _arXiv preprint arXiv:2407.10947_ (2024).
  * Wang et al. (2024b) Yaoting Wang, Peiwen Sun, Dongzhan Zhou, Guangyao Li, Honggang Zhang, and Di Hu. 2024b.  Ref-avs: Refer and segment objects in audio-visual scenes.  _arXiv preprint arXiv:2407.10957_ (2024).
  * Wu et al. (2022) Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. 2022.  Language as Queries for Referring Video Object Segmentation.  arXiv:2201.00487 [cs.CV]
  * Wu and Mooney (2019) Jialin Wu and Raymond Mooney. 2019.  Self-critical reasoning for robust visual question answering.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Yan et al. (2023) Shilin Yan, Renrui Zhang, Ziyu Guo, Wenchao Chen, Wei Zhang, Hongyang Li, Yu Qiao, Zhongjiang He, and Peng Gao. 2023.  Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation.  arXiv:2305.16318 [cs.CV]
  * Yang et al. (2021) Chao Yang, Su Feng, Dongsheng Li, Huawei Shen, Guoqing Wang, and Bin Jiang. 2021.  Learning content and context with language bias for visual question answering. In _2021 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 1–6.
  * Yuanhong et al. (2023) Chen Yuanhong, Liu Yuyuan, Wang Hu, Liu Fengbei, Wang Chong, and Carneiro Gustavo. 2023.  A Closer Look at Audio-Visual Semantic Segmentation.  _arXiv preprint arXiv:2304.02970_ (2023).
  * Zhou et al. (2019) Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2019.  Semantic understanding of scenes through the ade20k dataset.  _International Journal of Computer Vision_ 127, 3 (2019), 302–321.
  * Zhou et al. (2023) Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. 2023.  Audio-Visual Segmentation with Semantics.  _arXiv preprint arXiv:2301.13190_ (2023).
  * Zhou et al. (2022) Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. 2022.  Audio-Visual Segmentation. In _European Conference on Computer Vision_.
  * Zhu et al. (2024) Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. 2024.  IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding.  _arXiv preprint arXiv:2402.18476_ (2024).
  * Zürn and Burgard (2022) Jannik Zürn and Wolfram Burgard. 2022.  Self-Supervised Moving Vehicle Detection from Audio-Visual Cues.  arXiv:2201.12771 [cs.CV]