# Metric Spaces in Which Many Triangles Are Degenerate

Vašek Chvátal  Noé de Rancourt  Guillermo Gamboa Quintero  Ida Kantor  Péter
G.N. Szabó

###### Abstract

Richmond and Richmond (American Mathematical Monthly 104 (1997), 713–719)
proved the following theorem: If, in a metric space with at least five points,
all triangles are degenerate, then the space is isometric to a subset of the
real line. We prove that the hypothesis is unnecessarily strong: In a metric
space on $n$ points, fewer than $7n^{2}/6$ suitably placed degenerate
triangles suffice. However, fewer than $n(n-1)/2$ degenerate triangles, no
matter how cleverly placed, never suffice.

fn1fn1footnotetext: Department of Computer Science and Software Engineering,
Concordia University, Montreal (Emeritus) and Department of Applied
Mathematics, Charles University, Prague (Visiting). Supported by NSERC grant
RGPIN/5599-2014 and by H2020-MSCA-RISE project CoSP-GA No. 823748.
E-mail address<EMAIL_ADDRESS> Univ. Lille,
CNRS, UMR 8524 – Laboratoire Paul Painlevé, F-59000 Lille, France. Supported
by the Labex CEMPI (ANR-11-LABX-0007-01).
E-mail address<EMAIL_ADDRESS>fn2fn2footnotetext:  Computer Science
Institute of Charles University, Prague. Supported by GAČR grant 2-17398S.
E-mail address<EMAIL_ADDRESS> Computer Science
Institute of Charles University, Prague. Supported by GAČR grant 22-19073S and
by Charles University project UNCE/SCI/004.
E-mail address<EMAIL_ADDRESS>Department of
Engineering Mathematics, Faculty of Engineering and Information Technology,
University of Pécs, Pécs, Hungary and Department of Computer Science and
Information Theory, Budapest University of Technology and Economics, Hungary
E-mail address<EMAIL_ADDRESS>
Given a metric space $(V,\mbox{\rm dist\/})$, we follow [1] in writing $[rst]$
to signify that $r,s,t$ are pairwise distinct points of $V$ and $\mbox{\rm
dist\/}(r,s)+\mbox{\rm dist\/}(s,t)=\mbox{\rm dist\/}(r,t)$. Following [5], we
refer to three-point subsets of $V$ as triangles; if $[rst]$, then the
triangle $\\{r,s,t\\}$ is called degenerate. If a metric space is isometric to
a subset of the real line with the usual Euclidean metric, then all its
triangles are degenerate. Richmond and Richmond [5] proved the converse under
a mild lower bound on the number of points in the metric space:

######  Theorem 1 (Richmond and Richmond).

If, in a metric space with at least five points, all triangles are degenerate,
then this metric space is isometric to a subset of the real line with the
usual Euclidean metric.

Here, the lower bound on the number of points cannot be relaxed: consider the
distances defined on $\\{a,b,c,d\\}$ by $\mbox{\rm
dist\/}(a,b)\\!=\\!dist(b,c)\\!=\\!\mbox{\rm dist\/}(c,d)\\!=\\!\mbox{\rm
dist\/}(d,a)\\!=\\!1$ and $\mbox{\rm dist\/}(a,c)\\!=\\!\mbox{\rm
dist\/}(b,d)\\!=\\!2$.

In the finite case, Szabó [6, Theorem 1] proved that the hypothesis of Theorem
1 can be relaxed as soon as the space has six points and that it can be
relaxed further and further as the number of points gets larger and larger.

######  Theorem 2.

If, in a metric space with $n$ points such that $n\geq 6$, at most $n-4$
triangles are nondegenerate, then either this metric space is isometric to a
subset of the real line with the usual Euclidean metric or precisely $n-4$
triangles are nondegenerate and either their intersection consists of two
points or $n=6$ and the two triangles are disjoint.

Actually, Theorem 1 of [6], in addition to dealing also with smaller $n$,
provides a complete description of metric spaces with $n$ points and precisely
$n-4$ nondegenerate triangles. This description is irrelevant to our
discussion except for its special case $n=5$. In making note of its corollary
(which is our Lemma 3), we shall call a four-point metric space linear if it
is isometric to a subset of the real line with the usual Euclidean metric.
(Readers beware: our meaning of the term “linear” is different from that of
[6, Definition 1].) If all four triangles in a four-point metric space are
degenerate and yet the space is not linear, then we shall call it circular
(since its points can be labeled as $0,1,2,3$ in such a way that $[012]$,
$[123]$, $[230]$, $[301]$).

######  Lemma 3.

If, in a metric space with five points, precisely one triangle is
nondegenerate, then this metric space has a four-point circular subspace.

Let us call a set $A$ of three-point subsets of a set $V$ an anchor in $V$ if,
for every metric space $(V,\mbox{\rm dist\/})$, the assumption that all
triangles in $A$ are nondegenerate implies that $(V,\mbox{\rm dist\/})$ is
isometric to a subset of the real line with the usual Euclidean metric. In
this terminology, Theorems 1 and 2 show that every set of $\binom{n}{3}-n+5$
three-point subsets of an $n$-point set $V$ is an anchor in $V$. The purpose
of this note is to point out that there are far smaller anchors:

######  Theorem 4.

In every $n$-point set where $n\neq 4$, there is an anchor of fewer than
$\frac{7}{6}n^{2}$ sets.

(The circular metric space shows that there are no anchors in four-point
sets.)

###### Proof.

Given a positive integer $n$ other than $4$, consider a set $V$ of size $n$.
Since the set of all three-point subsets of $V$ is an anchor in $V$, we may
assume that $n\geq 10$: otherwise $\binom{n}{3}<\frac{7}{6}n^{2}$. We shall
rely on the following theorem of Fort and Hedlund [4, Theorem 1]:

> For every integer $m$ such that $m\geq 3$ and for every set $X$ of size $m$
> there is a family $F$ of
>
> $\left\lceil\frac{m}{3}\left\lceil\frac{m-1}{2}\right\rceil\right\rceil$
>
> three-point subsets of $X$ such that every two-point subset of $X$ is
> contained in a member of $F$.

(Actually, Fort and Hedlund also proved that the size of $F$ in their theorem
cannot be reduced.) We shall apply this theorem to a subset $X$ of $V$ such
that $\lvert X\rvert=n-2$. Set $Y=V-X$ and let $A$ denote the union of $F$ and
all triangles in $V$ that intersect $Y$. We have

$\lvert
A\rvert=\left\lceil\frac{n-2}{3}\left\lceil\frac{n-3}{2}\right\rceil\right\rceil+(n-2)+2\binom{n-2}{2}=\left\lceil\frac{n-2}{3}\left\lceil\frac{n-3}{2}\right\rceil\right\rceil+(n-2)^{2}.$

It is a routine matter to verify that

$\left\lceil\frac{m}{3}\left\lceil\frac{m-1}{2}\right\rceil\right\rceil\leq\left\lceil\frac{m^{2}}{6}\right\rceil\leq\frac{m^{2}+2}{6}$

for all $m$. It follows that

$\lvert A\rvert\leq\frac{(n-2)^{2}+2}{6}+(n-2)^{2}<\frac{7}{6}n^{2}.$

We are going to prove that $A$ is an anchor in $V$. For this purpose, consider
an arbitrary metric space $(V,\mbox{\rm dist\/})$ such that all triangles in
$A$ are degenerate. Our task is to show that every triangle outside $A$ is
also degenerate. Since the triangle is outside $A$, it is disjoint from $Y$.
Label its three points as $x,y,z$ and label the two points of $Y$ as $a,b$. By
definition of $F$, there are (not necessarily distinct) points $u,v,w$ in $X$
such that $\\{u,y,z\\}\in F$, $\\{x,v,z\\}\in F$, $\\{x,y,w\\}\in F$.
Theorem 1 applied to $(\\{a,b,u,y,z\\},\mbox{\rm dist\/})$ shows that
$(\\{a,b,y,z\\},\mbox{\rm dist\/})$ is linear;
Theorem 1 applied to $(\\{a,b,x,v,z\\},\mbox{\rm dist\/})$ shows that
$(\\{a,b,x,z\\},\mbox{\rm dist\/})$ is linear;
Theorem 1 applied to $(\\{a,b,x,y,w\\},\mbox{\rm dist\/})$ shows that
$(\\{a,b,x,y\\},\mbox{\rm dist\/})$ is linear.
To summarize, each of the four-point subspaces of $(\\{a,b,x,y,z\\},\mbox{\rm
dist\/})$ that do not contain the triangle $\\{x,y,z\\}$ is linear. This,
combined with Lemma 3, implies that $\\{x,y,z\\}$ is degenerate. ∎

We do not know whether or not the coefficient $\frac{7}{6}$ in Theorem 4 can
be reduced. Nevertheless, we do know that it cannot be reduced below
$\frac{1}{2}$:

######  Theorem 5.

Every anchor in a set of size $n$, with $n\geq 5$, includes at least
$n(n-1)/2$ triangles.

###### Proof.

Given a set $V$ of size $n$ and a set $C$ of fewer than $n(n-1)/2$ triangles
in $V$, we shall find a metric space on ground set $V$ where all triangles in
$C$ are degenerate, but at least one triangle is nondegenerate. For this
purpose, note that there are fewer than $3n(n-1)/2$ pairs $(\\{x,y\\},z)$ such
that $\\{x,y,z\\}\in C$, and so some $\\{x,y\\}$ belongs to fewer than $3$ of
these pairs. To put it differently, there are pairwise distinct points
$x,y,z_{1},z_{2}$ in $V$ such that $\\{x,y,z\\}\in C$ only if $z=z_{1}$ or
$z=z_{2}$. Having enumerated the remaining points of $V$ as
$z_{3},z_{4},\ldots z_{n-2}$, consider the metric space induced by the graph
with vertex set $V$ and edges
$z_{1}x,\;xz_{2},\;z_{1}y,\;yz_{2},\;z_{2}z_{3},\;z_{3}z_{4},\ldots
z_{n-3}z_{n-2}$: there, a triangle is nondegenerate if and only if it is one
of $\\{x,y,z_{3}\\}$, $\\{x,y,z_{4}\\}$, …$\\{x,y,z_{n-2}\\}$. ∎

All our theorems extend to the context of pseudometric betweenness defined in
[2, p. 643], which is more general than the context of metric spaces. An early
draft of this note appeared as [3] with a discussion of weakly saturated
hypergraphs and their relation to anchors.

## References

  * [1] Aboulker, P., Chen, X., Huzhang, G., Kapadia, R., Supko, C. (2016). Lines, betweenness and metric spaces. Discrete Comput. Geom. 56 (2): 427–448.
  * [2] Beaudou, L., Chvátal, V., Bondy, A., Chen, X., Chiniforooshan, E., Chudnovsky, M., Fraiman, N., Zwols, Y. (2013). Lines in hypergraphs. Combinatorica 33 (6): 633–654.
  * [3] Chvátal, V., Kantor, I. (2022). Metric spaces in which many triangles are degenerate. arXiv:2209.14361 [math.MG].
  * [4] Fort Jr, M.K. and Hedlund, G.A.(1958). Minimal coverings of pairs by triples. Pacific J. Mathematics 8 (4): 709–719.
  * [5] Richmond, B., Richmond, T. (1997). Metric spaces in which all triangles are degenerate. Amer. Math. Monthly 104 (8): 713–719.
  * [6] Szabó, P. G. N. (2022). Betweenness structures of small linear co-size. Discrete Appl. Math. 322: 404–424.


# CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual
Video Segmentation

Kexin Li  Zhejiang UniversityHangzhouChina<EMAIL_ADDRESS>,  Zongxin Yang
0000-0001-8783-8313 Zhejiang UniversityHangzhouChina<EMAIL_ADDRESS>,
Lei Chen  Finvolution GroupShanghaiChina<EMAIL_ADDRESS>,  Yi Yang
Zhejiang UniversityHangzhouChina<EMAIL_ADDRESS>and  Jun Xiao  Zhejiang
UniversityHangzhouChina<EMAIL_ADDRESS>
(2023)

###### Abstract.

Audio-visual video segmentation (AVVS) aims to generate pixel-level maps of
sound-producing objects within image frames and ensure the maps faithfully
adhere to the given audio, such as identifying and segmenting a singing person
in a video. However, existing methods exhibit two limitations: 1) they address
video temporal features and audio-visual interactive features separately,
disregarding the inherent spatial-temporal dependence of combined audio and
video, and 2) they inadequately introduce audio constraints and object-level
information during the decoding stage, resulting in segmentation outcomes that
fail to comply with audio directives. To tackle these issues, we propose a
decoupled audio-video transformer that combines audio and video features from
their respective temporal and spatial dimensions, capturing their combined
dependence. To optimize memory consumption, we design a block, which, when
stacked, enables capturing audio-visual fine-grained combinatorial-dependence
in a memory-efficient manner. Additionally, we introduce audio-constrained
queries during the decoding phase. These queries contain rich object-level
information, ensuring the decoded mask adheres to the sounds. Experimental
results confirm our approach’s effectiveness, with our framework achieving a
new SOTA performance on all three datasets using two backbones. The code is
available at https://github.com/aspirinone/CATR.github.io.

Combinatorial-Dependence; Audio-Constrained Queries; Blockwise-Encoded Gate

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
31st ACM International Conference on Multimedia; October 29–November 3, 2023;
Ottawa, ON, Canada.††booktitle: Proceedings of the 31st ACM International
Conference on Multimedia (MM ’23), October 29–November 3, 2023, Ottawa, ON,
Canada††price: 15.00††isbn: 979-8-4007-0108-5/23/10††doi:
10.1145/3581783.3611724††ccs: Computing methodologies Video segmentation
Figure 1. AVVS Task Description and CATR Contributions. The objective of the
Audio-Visual Video Segmentation (AVVS) task is to generate pixel-level maps
identifying sound-producing objects within image frames (a). Previous
approaches separately addressed the temporal dependencies of video and the
audio-video interaction information (b), neglecting the unique spatial-
temporal dependencies inherent to audio and video as a combination. CATR
initially merges audio and video features, subsequently capturing the spatial-
temporal dependencies of this combination. Note that the red arrow symbolizes
the Video-to-Audio (V-to-A) information, while the blue arrow denotes the
Audio-to-Video (A-to-V) information. Additionally, we introduce innovative
audio-constrained learnable queries to enhance object-aware segmentation (c).

##  1\. Introduction

Audio-visual video segmentation (AVVS) aims to generate pixel-level maps of
sound-producing objects within image frames and ensure the maps faithfully
adhere to the given audio. For instance, when a person sings, AVVS enables the
identification and segmentation of individuals in the video (see Figure 1
(a)). This capability has significant implications for various applications,
such as video editing and surveillance. Despite the successful integration of
multi-modal guidance approaches using point, box, scribble, text, and verbal
cues for segmentation in recent studies (Cheng et al., 2023; Kirillov et al.,
2023; Zou et al., 2023), audio guidance has not yet been incorporated. This
gap can be attributed to the inherent challenges of AVVS, including the
ambiguous semantic information embedded in sounds and establishing
correspondence between sounds and pixel-level predictions. Therefore, future
research needs to investigate the integration of multi-knowledge
representations (Yang et al., 2021c), including audio, video, segmentation,
etc.

In the domain of referring video object segmentation (Wu et al., 2022; Zou et
al., 2023; Liang et al., 2023) and audio-visual understanding (Lin et al.,
2019; Lin and Wang, 2020; Tian et al., 2018; Wu et al., 2019; Xu et al., 2020;
Xuan et al., 2020; Lin et al., 2019; Ramaswamy and Das, 2020; Duan et al.,
2021; Zhou et al., 2021; Tian et al., 2020; Wu and Yang, 2021; Lin et al.,
2021; Yu et al., 2022), substantial efforts have been devoted to investigating
multi-modal segmentation techniques. Zhou et al. (Zhou et al., 2022), for
example, introduced a framework incorporating the TPAVI module, which
facilitates audio-visual pixel-level segmentation. However, these methods
still encounter two primary limitations in the realms of audio fusion and
audio-guided video decoding:

Firstly, Seperate-Dependence Fusion. The challenges in utilizing audio
features arise from the ambiguous semantic information embedded within sounds,
such as differentiating a child’s cry from a cat’s meow, in contrast to the
clear linguistic references to ”child” and ”cat”. As a result, establishing
precise pixel-level associations under these indeterminate auditory cues is
difficult. However, we found that audio has a unique advantage: its temporal
properties align with those of video features, capturing distinct but
complementary aspects of the same event. Existing methods do not fully exploit
this property, addressing video temporal information and audio-visual
interactions separately, which constrains their effectiveness. Various
combinations of audio and video exhibit unique spatial-temporal dependencies,
contributing to more accurate and robust results. Thus, a method that captures
the spatial-temporal characteristics of audio and video in combination is
essential.

Secondly, Object-Limited Queryless Decoding. Previous methods typically derive
the final mask directly after decoding video features, as exemplified by the
use of an FCN decoder (Zhou et al., 2022). This approach neglects audio
guidance information and omits object-level information during the decoding
stage, potentially leading to segmentation errors in complex environments. For
instance, in Figure 1 (b), the second frame of the video contains a violin, a
piano, and people simultaneously. With audio containing only violin sounds and
human singing, the target segmentation objects should be the person and the
violin. However, due to the absence of audio constraints during the decoding
phase, previous methods may erroneously segment the piano, influenced by the
video’s focus on the front and back frames. Consequently, it is essential to
introduce audio restrictions and provide object-level guidance information
during the decoding phase.

To address the above limitations, we designed targeted modules:

(1) Combinatorial-Dependence Fusion. To comprehensively assess audio-visual
combinatorial-dependence, we design to combine audio and video features from
their respective temporal and spatial dimensions, followed by capturing this
combination’s spatial-temporal dependence. Commonly, transformers are used to
capture temporal dependencies; assuming a video frame dimension of $H\times
W$, the merged feature dimension becomes $(H\times W+T)$. However, due to the
substantial memory consumption associated with this encoder, we propose an
innovative decoupling transformer that considerably reduces memory usage while
allowing the extraction of spatial-temporal interaction information between
audio-audio, video-video, audio-video, and video-audio combinations.

(2) Object-Aware Audio-Queried Decoding. To enable attention focus on the
object of interest, we propose an audio-queried decoder. Specifically, we
apply an audio constraint to all object queries, allowing the model to
leverage audio information to direct attention toward the desired object.
These conditional queries serve as inputs for the model, which produces
object-aware dynamic kernels to filter segmentation masks from feature maps.

On the whole, we propose a Combinatorial-Dependence Audio-Queried Transformer
Network (CATR; Figure. 2), which contains two main components: Decoupled
Audio-Visual Transformer Encoding Module (DAVT; detailed in Section. 4) and
Audio-Queired Decoding Module (detailed in Section. 4.1). In encoding, we
design an innovative decoupling block, which consists of two steps: initially,
we merge audio and video features of corresponding frames while concurrently
capturing their temporal information. Subsequently, we facilitate interaction
between video features containing temporal information and audio features. By
stacking decoupling blocks, we can efficiently capture audio-visual spatial-
temporal correlations in a memory-efficient manner. In addition, Audio-Queried
Decoder Module innovatively employ an audio constraint to all object queries
to produce object-aware dynamic kernels to filter the segmentation of desired
object. Moreover, we design a Blockwise-Encoded Gate to utilize all the
features extracted from each encoder block. This Blockwise-Encoded Gate
enables modeling of the overall distribution of all encoder blocks from a
global perspective, thereby balancing the contributions of different encoder
blocks.

We conduct extensive experiments on three popular benchmarks and achieve new
state-of-the-art performance on all datasets with two backbones (On S4, CATR
84.4% $\mathcal{J}$ / 91.3% $\mathcal{F}$ vs. TPAVI 78.7% $\mathcal{J}$ /
87.9% $\mathcal{F}$; On M3, CATR 61.8% $\mathcal{J}$ / 71% $\mathcal{F}$ vs.
TPAVI 54% $\mathcal{J}$ / 64.5% $\mathcal{F}$). Our code and benchmark will be
released.

Overall, our contributions are summarized as follows:

  * •

We introduce an encoding-decoding framework CATR that presents a novel
spatial-temporal audio-video fusion block to fully consider the audio-visual
combinatorial dependence in a decoupled and memory-efficient manner.

  * •

We propose the audio-constrained learnable queries to incorporate audio
information comprehensively during decoding. These audio-constrained queries
contain abundant object-level information that can select which object is
being referred to segment. In addition, we introduce a Blockwise-Encoded Gate
that allows for the selective fusion of features from different encoder
blocks.

  * •

We conduct extensive experiments on three popular benchmarks, and achieve new
superior state-of-the-art performance on all three datasets with two
backbones.

Figure 2. CATR employs an encoder-decoder structure. (a) In encoding, we merge
audio and video features and capture their spatial-temporal combinatorial-
dependencies. To conserve memory, we devise decoupling methods, utilizing
temporal A-V and temporal V-A to fusion audio and video features. (b) To
balance the contributions of multiple encoder blocks, we implement a blockwise
gating method for utilizing all video features from each block. (c) In
decoding, we introduce audio-constrained learnable queries, which harness
audio features to extract object-level information, guiding target object
segmentation.

##  2\. Related Work

###  2.1. Video Object Segmentation (VOS)

The VOS task (Wang et al., 2021b; Yang et al., 2021b) aims to segment the
object of interest throughout the entire video sequence. It is divided into
two settings: semi-supervised and unsupervised. For semi-supervised VOS (Yang
et al., 2021a; Yang and Yang, 2022; Zhang et al., 2023; Lu et al., 2020), the
target object is decided given a one-shot mask of the first video frame. As
for unsupervised VOS (Li et al., 2023a), it needs to segment all the primary
objects automatically. Many excellent works are proposed and proven to achieve
impressive segmentation performance. However, these fancy designs are limited
to a single visual modality.

###  2.2. Audio-Visual Video Segmentation (AVVS)

The human ability to identify objects is not solely reliant on visual cues but
also on auditory signals. For instance, the distinct sounds of a dog barking
or a bird chirping are easily recognizable. This observation underscores the
complementarity of audio and visual information. However, while speech-guided
video segmentation is a more reliable means of distinguishing instance-level
objects, sound can only provide information about object categories, making it
a challenging task to locate and segment the object producing the sound. Zhou
et al. (Zhou et al., 2022) pioneered the audio-visual segmentation (AVVS) task
and proposed a framework incorporating the TPAVI module, a groundbreaking
approach for achieving pixel-level segmentation using audio information.
Nonetheless, their framework’s handling of multi-modal feature fusion and
audio guidance was inadequate. Thus, we present a novel framework that
addresses these limitations.

###  2.3. Vision Transformers

Transformer (Vaswani et al., 2017a) was first introduced for sequence-to-
sequence translation in natural language processing community and has achieved
marvelous success in most computer vision tasks (Dosovitskiy et al., 2021;
Kamath et al., 2021; Liu et al., 2021; Li et al., 2023b) such as object
detection (Carion et al., 2020; Zhu et al., 2021), tracking (Chen et al.,
2021b; Meinhardt et al., 2022; Sun et al., 2020; Yan et al., 2021) and
segmentation (Yang and Yang, 2022; Cheng et al., 2021; Zheng et al., 2021;
Liang et al., 2022). The Transformer employs an attention mechanism to
facilitate the transformation of input into output representations. Building
upon this foundation, the DETR (Carion et al., 2020) has advanced the field by
introducing a learnable query mechanism, which serves to expand the range of
output possibilities. By employing an intelligent query and output matching
mechanism, DETR is capable of determining the most optimal association between
input and output elements. Furthermore, the VisTR (Wang et al., 2021a) extends
the capabilities of DETR to the domain of video segmentation, achieving
notable advancements. DeAOT (Yang and Yang, 2022) decouples the visual and
identification features in hierarchical propagation (Yang et al., 2021a) and
achieves state-of-the-art performance in semi-supervised VOS. Inspired by
these works, our work also relies on the query mechanism of Transformer but
considers an additional modality, i.e., audio, as the object reference.
Moreover, we propose an effective spatial-temporal fusion module to realize
audio-guided video segmentation.

##  3\. Method

###  3.1. Overview

Our pipeline for AVVS task can be formulated as encoding-decoding (depicted in
Figure. 2). To address limitations in previous methods, such as inadequate
correlation and vague reference, we carefully design two modules: the
Decoupled Audio-Visual Transformer Encoding Module (DAVT; detailed in
Section.4) and the Audio-Queired Decoding Module (detailed in Section.4.1).
These modules enable effective audio-visual spatial-temporal connection and
capture the object-level information to achieve more explicit reference,
respectively. Moreover, we design a Blockwise-Encoded Gate to enable modeling
of the overall distribution of all encoder blocks. In addition, CATR aims to
output a pixel-level map of the object(s) that produce sound at the time of
the image frame,

(1) $\\{M_{t}\\}_{t=1}^{T}=CATR({S_{t,v}},{S_{t,a}}),$

where we denote the video sequence as $S=\\{S_{t,v},S_{t,a}\\}_{t=1}^{T}$.
Moreover, $S_{v}$ denotes the visual sequence and $S_{a}$ denotes the audio
sequence. The predictions are denoted as $\\{M_{t}\\}_{t=1}^{T}$,
${M_{t}}\in\mathbb{R}^{H\times W}$.

##  4\. Decoupled Audio-Visual Transformer

In contrast to existing methods that account for the video temporal
information and audio-visual interaction separately, we propose a method that
obtains the spatial-temporal combinatorial-dependence between audio-audio,
video-video, audio-video, and video-audio in a novel decoupling memory-
efficient manner.

Stack DAVT Blocks. To conserve memory, we designed the Decoupled Audio-Visual
Transformer (DAVT). The DAVT block involves two steps. Initially, we combine
the audio and video features of corresponding frames and capture their
temporal information simultaneously. Subsequently, we interact processed video
features with audio features, respectively.

For a video sequence $S_{v}$, we extract visual features after popular
backbones and atrous spatial pyramid pooling (Chen et al., 2017) and obtain
hierarchical visual feature maps. We denote the video features as
${F_{v}}\in\mathbb{R}^{T\times H\times W\times C}$, where $T$, $H$, $W$ and
$C$ signifying the number of frames, height, width, and channel, respectively.
Given an audio sequence $S_{a}$, we employ a convolutional neural network
VGGish (Hershey et al., 2017) pre-trained on AudioSet (Gemmeke et al., 2017)
as backbone to extract audio features ${F_{a}}\in\mathbb{R}^{T\times d}$,
where $d=128$.

(2) ${{F}_{v}^{l+1},{F}_{a}^{l+1}}=DAVT({F}_{v}^{l},{F}_{a}^{l}),$

where $DAVT(\cdot)$ denotes the Decoupled Audio-Visual Transformer block, $l$
denotes the $l$-th block. By stacking multiple DAVT blocks, we can effectively
capture the spatial-temporal correlation between audio-audio, video-video,
audio-video and video-audio in a memory-efficient manner.

Spatial Audio-Visual Fusion. To obtain the audio-visual overall dependence,
the visual features $F_{v}^{l}$ and audio features $F_{a}^{l}$ are linearly
projected to a shared dimension $D$. The video features for each frame are
flattened and individually merged with the audio embeddings, yielding a set of
$T$ multi-modal sequences, each of shape $(H\times W+1)\times D$.

(3)
${\tilde{{F}}_{v}^{l},\tilde{{F}}_{a}^{l}}=SF(Concat({F}_{v}^{l},{F}_{a}^{l})),$

where $Concat(\cdot)$ denotes the concatenate operation, and $SF(\cdot)$
denotes the spatial audio-visual fusion function, which is employed as self-
attention. Then we obtain the processed video feature $\tilde{{F}}_{v}^{l}$
that contains the corresponding frame audio information. Similarly, the audio
feature $\tilde{{F}}_{a}^{l}$ contains the corresponding frame video
information.

Temporal A-to-V Fusion. Employing a transformer-based encoder will consume a
large amount of memory, so we use the decoupling method to carry out Audio-to-
Video (A-to-V) interaction and Video-to-Audio (V-to-A) interaction
respectively.

(4)
${\hat{{F}}_{v}^{l},\hat{{F}}_{a}^{l}}=TAV(\tilde{{F}}_{v}^{l},\tilde{{F}}_{a}^{l})=\operatorname{Softmax}\left(\frac{\tilde{{F}}_{v}^{l}W^{Q}\cdot\left(\tilde{{F}}_{a}^{l}W^{K}\right)^{\mathrm{T}}}{\sqrt{d_{\text{head
}}}}\right)\tilde{{F}}_{a}^{l}W^{V}$

where the $TAV(\cdot)$ denotes the Temporal Audio-to-Video Fusion, which is
employed as multi-head attention (Vaswani et al., 2017b). In $TAV(\cdot)$, the
query is the processed video feature $\tilde{{F}}_{v}^{l}$, and key is the
audio feature $\tilde{{F}}_{a}^{l}$. Moreover,
$W^{Q},W^{K},W^{V}\in\mathbb{R}^{C\times d_{\text{head }}}$ are learnable
parameters.

Temporal V-to-A Fusion. Correspondingly, we also design a Temporal Video-to-
Audio Fusion function $TVA(\cdot)$,

(5)
${\check{{F}}_{v}^{l},\check{{F}}_{a}^{l}}=TVA(\tilde{{F}}_{v}^{l},\tilde{{F}}_{a}^{l})=\operatorname{Softmax}\left(\frac{\tilde{{F}}_{a}^{l}W^{Q}\cdot\left(\tilde{{F}}_{v}^{l}W^{K}\right)^{\mathrm{T}}}{\sqrt{d_{\text{head
}}}}\right)\tilde{{F}}_{v}^{l}W^{V},$

where $TVA(\cdot)$ denotes the Temporal Video-to-Audio Fusion that is also
employed as multi-head attention. In $TVA(\cdot)$, the query is the processed
audio feature $\tilde{{F}}_{a}^{l}$ and key is the video feature
$\tilde{{F}}_{v}^{l}$.

After we obtain the video feature $\check{{F}}_{v}^{l}$ that from $TVA(\cdot)$
and $\hat{{F}}_{v}^{l}$ that from $TAV(\cdot)$, we merge the
$\check{{F}}_{v}^{l}$ and $\hat{{F}}_{v}^{l}$ by element-wise adding and
obtain the fully interacted video feature $\bar{{F}}_{v}^{l}$

Blockwise-Encoded Gate. The existing method typically employs the features of
the last encoder block alone as the decoder input, which is insufficient
because the features of each encoder block contain varying degrees of multi-
modal interaction information (see Figure 4). Thus, we design gate mechanisms
to utilize all the features extracted from each encoder block and balance the
contributions of different encoder blocks.

Figure 3. Attention maps generated from spatial fusion & temporal A-V/V-A
fusions. Sample 1: target is person & piano; spatial fusion focuses on person,
neglects piano; with temporal A-V/V-A fusions, the attention map accurately
highlights both. Sample 2: target is the piano; spatial fusion wrongly
emphasizes person, but temporal A-V/V-A maps correctly focus on piano.
Consequently, we draw the conclusion that spatial fusion provides an initial
integration of audio information, whereas temporal A-V and V-A fusions further
consolidate this information to accurately identify the target object.

Suppose we have two video features $\bar{{F}}_{v}^{l}$ and
$\bar{{F}}_{v}^{l+1}$ from different Spatial-Temporal Encoding blocks, we
design a gate unit and ${G}^{l+1}$ denotes the $(l+1)$-th output vector,

(6)
$\displaystyle\begin{split}{G}^{l+1}&=Pool(Sigmoid(Conv(Concat(\bar{{F}}_{v}^{l},\bar{{F}}_{v}^{l+1})))),\\\
{{F}^{final}_{v}}&=Conv({G^{l}}\cdot\bar{{F}}_{v}^{l}+{G^{l+1}}\cdot\bar{{F}}_{v}^{l+1}),\end{split}$

where $Concat(\cdot)$ denotes the concatenate operation, $Conv(\cdot)$ denotes
the convolution layer, $Sigmoid(\cdot)$ denotes the sigmoid function and
$Pool(\cdot)$ denotes the global average pooling. The output channel of
$Conv(\cdot)$ is $C$, which means the resulted gate vector ${G}^{l+1}$ has $C$
different elements which correspond to $C$ gate values (we set $C=256$ here).

The gate values ${G}^{l+1}$ is applied for weighting the different-blocks
video features $\bar{{F}}_{v}^{l}$ and $\bar{{F}}_{v}^{l+1}$. To obtain the
final video encoding feature ${F}^{final}_{v}$, we fuse all the re-weighted
features by element-wise addition and convolutional layers.

###  4.1. Audio-Queired Decoding

The existing methods fall short in effectively capturing object-level details
and offering explicit information for cross-modal reasoning. To overcome this
limitation, we propose audio-constrained queries, which impose an audio
constraint on all object queries and generate object-aware dynamic kernels
that filter target object segmentation masks from feature maps. Our approach
aims to provide a comprehensive solution that enhances object recognition by
incorporating audio signals into the process.

Audio-Constrained Query. We hierarchically fuse the final video feature
${F}^{final}_{v}$ and the multi-layer features from backbone with an FPN-like
(Lin et al., 2017a) decoder, then we obtain the semantically-rich video
feature maps ${F_{seg}}=\\{f_{seg,t}\\}_{t=1}^{T}$, where
$f_{seg,t}\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times C}$.

To capture the object-level information comprehensively, we devised a set of
$N$ learnable queries. These queries, along with the audio feature, were fed
into the decoder embedding and position embedding in the transformer,
resulting in queries with abundant object-level information. Next, we use two-
layer dynamic kernels $\mathcal{G}_{\mathrm{kernel}}$ to generate a
corresponding sequence segmentation for each query. Finally, the binary masks
are generated by dynamic convolution:

(7) ${M_{i}}=\\{{F_{seg}}*\omega_{i}\\}_{i=1}^{N},$

where $M_{i}\in\mathbb{R}^{N\times\frac{H}{8}\times\frac{W}{8}}$ denotes the
segmentation mask with $N$ queries. $\omega_{i}$ and $F_{seg}$ denote the
$i$-th dynamic kernel weights and its exclusive feature map, respectively.

Query Matching. The aim of query matching is to determine which of the
predicted sequences best fits the referred object. Here, we denote each
ground-truth sequence as
$y=(M,R)=(\\{M_{t}\\}_{t=1}^{T},\\{R_{t}\\}_{t=1}^{T})$, where $M$ denotes the
ground-truth mask and $R$ denotes a probability scalar indicating whether the
instance corresponds to the referenced object and ascertains the visibility of
this object within the current frame. In addition, we denote the prediction
set as $\hat{y}=\\{\hat{y}_{i}\\}_{i=1}^{N}$,where
$\hat{y}_{i}=(\\{\hat{M}_{i,t}\\}_{t=1}^{T},\\{\hat{R}_{i,t}\\}_{t=1}^{T})$.

To find the best prediction from $N$ conditional queries, we use a reference
head $\mathcal{G}_{\mathrm{Ref}}$, which consists of a single linear layer
followed by a softmax layer. Then we obtain the positive sample by minimizing
the matching cost:

(8)
$\displaystyle\begin{split}\hat{y}_{\mathrm{pos}}&=\underset{\hat{y}_{i}\in\hat{y}}{\arg\min}\;\mathcal{C}_{\text{match
}}\left(y,\hat{y}_{i}\right),\\\ \mathcal{C}_{\text{match
}}\left(y,\hat{y}_{i}\right)&=\mathcal{C}_{\text{dice
}}(M,\hat{M}_{i})+\mathcal{C}_{\text{ref }}(R,\hat{R}_{i})\end{split}$

where $\hat{y}_{\mathrm{pos}}$ denotes the permutation in $N$ conditional
queries with the lowest total cost. $\mathcal{C}_{\text{dice }}$ takes on the
role of overseeing and evaluating the predicted mask sequence in direct
comparison with the ground-truth mask sequence, with this evaluation process
being conducted by the Dice coefficients (Milletari et al., 2016), and
$\mathcal{C}_{\text{ref }}$ utilizes cross-entropy to guide the reference
predictions, aligning them with the corresponding ground-truth reference
identity.

###  4.2. Loss and Inference

We consider both mask and reference identity, and we define our loss function
as follows:

(9)
$\displaystyle\begin{split}&\mathcal{L}(y,\hat{y}_{i})=\mathcal{L}_{\mathrm{Mask}}\left(M_{i},\hat{M}_{i}\right)+\mathcal{L}_{\mathrm{Ref}}\left(R_{i},\hat{R}_{i}\right)\\\
&=\lambda_{d}\mathcal{L}_{\text{Dice
}}(M_{i},\hat{M}_{i})+\lambda_{f}\mathcal{L}_{\text{Focal
}}(M_{i},\hat{M}_{i})+\lambda_{r}\mathcal{L}_{\mathrm{Ref}}(R_{i},\hat{R}_{i})\end{split}$

where $\mathcal{{L}}_{\mathrm{Mask}}$ ensures mask alignment between the
predicted and ground-truth, and $\mathcal{L}_{\mathrm{Ref}}$ supervises the
reference identity predictions. In addition, $\mathcal{L}_{\mathrm{Mask}}$ is
implemented by a combination of the Dice (Milletari et al., 2016) and the per-
pixel Focal (Lin et al., 2017b) loss functions, and
$\mathcal{L}_{\mathrm{Ref}}$ is implemented by a cross-entropy term.

For inference, CATR will predict $N$ object sequences. For each sequence, we
obtain the predicted reference probabilities and the reference score set
$P=\\{p_{i}\\}_{i=1}^{N}$, We select the object sequence with the highest
score and its index is denoted as $R_{pos}$,

(10) $R_{pos}=\underset{i\in\\{1,2,…,N\\}}{\arg\max}\;p_{i}$

Finally, we return the final mask $M_{pos}$ that corresponds to $R_{pos}$.

|  |  | S4 | M3
---|---|---|---|---
Task | Method | Backbone | $\mathcal{M_{J}}$ | $\mathcal{M_{F}}$ | $\mathcal{M_{J}}$ | $\mathcal{M_{F}}$
| LVS (Chen et al., 2021a) | resnet18 | 37.9 | 51 | 29.5 | 33
SSL | MSSL (Qian et al., 2020) | resnet18 | 44.9 | 66.3 | 26.1 | 36.3
| 3DC (Mahadevan et al., 2020) | resnet152 | 57.1 | 75.9 | 36.9 | 50.3
VOS | SST (Duke et al., 2021) | resnet101 | 66.3 | 80.1 | 42.6 | 57.2
| iGAN (Mao et al., 2021) | resnet50 | 61.6 | 77.8 | 42.9 | 54.4
SOD | LGVT (Zhang et al., 2021) | swin | 74.9 | 87.3 | 40.7 | 59.3
|  | resnet50 | 72.8 | 84.8 | 47.9 | 57.8
| TPAVI (Zhou et al., 2022) | PVT-v2 | 78.7 | 87.9 | 54.0 | 64.5
|  | resnet50 | 74.8 | 86.6 | 52.8 | 65.3
| CATR | PVT-v2 | 81.4 | 89.6 | 59.0 | 70.0
|  | resnet50 | 74.9 | 87.1 | 53.1 | 65.6
AVVS | CATR* | PVT-v2 | 84.4 | 91.3 | 62.7 | 74.5
Table 1. Quantitative comparisons on AVSBench-object datasets (single-source,S4; multi-source,M3). Blue indicates the best performance with resnet backbone, while red indicates the best performance among all settings. * denotes that the training datasets are supplemented annotation with AOT. Task | Method | Backbone | AVSS
---|---|---|---
$\mathcal{M_{J}}$ | $\mathcal{M_{F}}$
VOS | 3DC (Mahadevan et al., 2020) | resnet18 | 17.3 | 21.6
AOT (Yang et al., 2021a) | resnet50 | 25.4 | 31.0
AVSS | TPAVI (Zhou et al., 2023) | PVT-v2 | 29.8 | 35.2
CATR | PVT-v2 | 32.8 | 38.5
Table 2. Quantitative comparisons on AVSBench-semantic datasets (AVSS). Red
indicates the best performance.

##  5\. Experiment

###  5.1. Implementation Details

Datasets. We train and validate our model on three datasets: Semi-supervised
Single-sound Source Segmentation (S4), Fully-supervised Multiple-sound Source
Segmentation (M3), and Fully-supervised Audio-Visual Semantic Segmentation
(AVSS). S4 and M3 datasets provide binary segmentation maps identifying the
pixels of sounding objects, while the AVSS dataset offers semantic
segmentation maps as labels. The S4 dataset contains audio samples with a
single target object, supplying ground-truth solely for the initial frame
during training. Evaluation necessitates predictions for all video frames in
the test set. In contrast, both M3 and AVSS datasets contain audio samples
with multiple target objects and furnish ground-truth data for all frames
throughout the training phase.

Training Details. We conduct training and evaluation on S4, M3 and AVSS
datasets, with the backbone ResNet-50 and Pyramid Vision Transformer (PVT-v2).
The channel size of the spatial-temporal encoding module is set to $C=256$. We
use the VGGish model to extract audio features and use the Adam optimizer with
a learning rate of 1e-5 for the fully-supervised MS3 settings, 1e-4 for the
semi-supervised S4 and the fully-supervised AVSS settings. The batch size is
set to 4 and the number of audio-constrained queries is set as 50. On S4, we
set the training epoch as 25, and On M3 and AVSS, we set the training epoch as
100.

Evaluation Metrics. We use standard metrics Jaccard index (Everingham et al.,
2010) $\mathcal{J}$ and F-score $\mathcal{F}$ as the evaluation metrics, where
$\mathcal{J}$ and $\mathcal{F}$ measure the region similarity and contour
accuracy, respectively. In our experiment, we use $\mathcal{M_{J}}$ and
$\mathcal{M_{F}}$ to denote the mean metric values over the whole dataset.

PVT-v2 | M3 | S4
---|---|---
$\mathcal{M_{J}}$ | $\mathcal{M_{F}}$ | $\mathcal{M_{J}}$ | $\mathcal{M_{F}}$
Baseline | 51.2 | 63.3 | 77.9 | 87.6
(+) Annotation | 53.1 | 64.7 | 79.1 | 88.5
(+) Decoupled A-V Transformer | 57.7 | 68.6 | 82.6 | 90.1
(+) Blockwise-Encoded Gate | 60.8 | 70.3 | 83.8 | 91.1
(+) Audio-queried Decoding | 62.7 | 74.5 | 84.4 | 91.3
Table 3. Ablation analysis on the M3 and S4 dataset with PVT-v2 backbone.

###  5.2. Comparison

Compare with Other Tasks. Audio-visual video segmentation (AVVS) is a
relatively new and emerging task, first introduced by (Zhou et al., 2022),
which aims to segment target objects in videos based on corresponding sounds.
Although some well-established tasks, such as sound source localization (SSL),
video object segmentation (VOS) and salient object detection (SOD) can perform
video object segmentation, we utilize state-of-the-art methods from these
related tasks as a comparative benchmark for our experiments. As evident in
Table 1, there exists a significant performance gap between SSL-based methods
and our CATR, primarily due to the lack of pixel-level results in SSL.
Furthermore, our model demonstrates a clear advantage over video object
segmentation (VOS) and salient object detection (SOD) methods on both S4 and
M3 datasets. This superior performance can be attributed to the fact that VOS
and SOD are single-mode tasks and do not utilize sound information. In
summary, the comparison with SOTA methods from related tasks substantiates the
exceptional performance of our model in AVVS.

Compare with SOTA TPAVI. Our proposed CATR outperforms the previous SOTA TPAVI
on all datasets (S4, M3 and AVSS) with two backbones (see Figure 1 and 2).
This improvement is due to the integration of the decoupled audio-visual
transformer encoding module (DAVT) and the object-aware audio-queried decoding
module. The DAVT block captures the combinatorial dependence of audio and
video, combining audio and video in the space dimension to capture the
temporal characteristics of this multi-modal combination. Compared to the
previous model that considered the audio-visual temporal and interactive
features independently, the combinatorial dependence we obtained is better
equipped to locate the referred object. Additionally, our object-aware audio-
queried decoder utilizes multiple queries containing rich audio cues and
object-level information, providing more accurate object segmentation and
target location compared to the previous model’s decoding directly. By
considering both object-level and audio-constrained decoding, our model
achieves more precise results.

Compare with the Processed Data. Limited datasets exist for audio-visual video
segmentation, leading (Zhou et al., 2022) to introduce AVSBench-object
datasets first. Among these, S4 represents a semi-supervised learning task,
providing ground-truth for the first frame in the training set. To maximize
dataset utility without incurring additional labor, we devised a complementary
approach for S4 and M3 datasets. Specifically, during M3 training, we employed
AOT (Yang et al., 2021a) to predict unlabeled frames of the S4 training set,
using these predictions as ground-truth for the AVVS task. Concurrently, we
preserved the same setting as TPAVI, implementing semi-supervised training for
the S4 dataset and fully supervised training for the M3 dataset.

Table 1’s experimental results demonstrate that our model’s performance on the
original dataset (CATR) surpasses the previous state-of-the-art TPAVI, and the
supplementary labeling method (CATR*) further enhances the model’s
effectiveness.

Resnet50 | M3 | S4
---|---|---
$\mathcal{M_{J}}$ | $\mathcal{M_{F}}$ | $\mathcal{M_{J}}$ | $\mathcal{M_{F}}$
TPAVI w audio | 47.9 | 57.8 | 72.8 | 84.8
CATR w/o audio | 36.4 | 51.4 | 73.2 | 84.6
CATR w audio | 52.1 | 64.6 | 74.1 | 86.1
PVT-v2 | M3 | S4
$\mathcal{M_{J}}$ | $\mathcal{M_{F}}$ | $\mathcal{M_{J}}$ | $\mathcal{M_{F}}$
TPAVI w audio | 54.0 | 64.5 | 78.7 | 87.9
CATR w/o audio | 43.9 | 57.6 | 80.7 | 89.1
CATR w audio | 62.7 | 74.5 | 84.4 | 91.3
Table 4. Comparison between TPAVI with audio information and CATR without audio information. PVT-v2 | M3
---|---
$\mathcal{M_{J}}$ | $\mathcal{M_{F}}$
CATR | 62.7 | 74.5
w/o spatial fusion | 59.7 | 70.7
w/o temporal A-V fusion | 58.4 | 69.8
w/o temporal V-A fusion | 61.8 | 71.0
Table 5. Ablation analysis of spatial-temporal encoding. Gate Channel | M3 | S4
---|---|---
$\mathcal{M_{J}}$ | $\mathcal{M_{F}}$ | $\mathcal{M_{J}}$ | $\mathcal{M_{F}}$
2 | 58.4 | 69.8 | 83.5 | 90.8
64 | 61.8 | 70.9 | 83.9 | 91.1
128 | 62.1 | 72.9 | 84.2 | 91.2
256 | 62.7 | 74.5 | 84.4 | 91.3
Table 6. Analysis of the number of channels in Blockwise-Encoded Gate with
PVT-v2 backbone.

###  5.3. Contribution of The Core Components

Table 3 demonstrates the contributions of each proposed module to the overall
performance enhancement in CATR, utilizing PVT-v2 and ASPP as encoding and
expanded fused feature maps as decoding in the baseline. Given the limited
original training samples, it is essential to maximize the use of available
data. We augment the two training sets, respecting their semi-supervised and
fully supervised configurations, due to the similarity of their segmentation
objectives. Specifically, we incorporated M3 video data into the S4 training
set and supplemented the S4 training set with AOT-generated ground-truth when
training the M3 dataset. The second row in Table 3 indicates that our
additional annotation improves the performance of both M3 and S4 datasets.

Furthermore, our experiments reveal that the decoupled audio-visual
transformer encoding, blockwise-encoded gate, and audio-queried decoding
modules significantly enhance the model’s performance. Notably, the audio-
queried decoding module exhibits a more substantial improvement in the M3
dataset than in the S4 dataset ($\mathcal{M_{J}}$ is up 6.5 vs. 0.6). This is
attributable to the multiple objectives in M3 dataset videos, which complicate
segmentation target identification. The audio-constrained query contains rich
object-level information and guides the segmentation effectively.

Figure 4. Visualization of video features after processing at each stage. We
observed that the initial features, generated by the backbone network,
appeared indistinct. However, the video features progressively aligned with
the desired segmentation object after the spatial-temporal encoding module.
Figure 5. Comparative analysis of the TPAVI method and our proposed CATR. We
present two qualitative examples from the M3 and S4 datasets. The M3 dataset
example (left) demonstrates TPAVI’s inability to detect the transition of
auditory objects, such as from a violin to a guitar, whereas CATR accurately
predicts these changes in alignment with the audio signal. In S4 example
(right), CATR exhibits better performance on pixel-level segmentation in the
presence of a complex background.

The Impact of Decoupled A-V Transformer Encoding. We developed a decoupled
spatial-temporal encoding block consisting of three components. Initially, we
integrated audio and video features in the spatial domain using a spatial
fusion method, capturing the temporal dependence of this combination.
Subsequently, the spatially fused features were processed through both the
temporal A-V and temporal V-A modules. Table 5 demonstrates the contributions
of each component to overall performance, highlighting the critical role
played by the temporal A-V module. We attribute its significance to the
predominant use of video features in the final decoding process, where video
features serve as the key and value within the temporal A-V module, preserving
crucial video information. To further illustrate this, we examined the
attention maps of features processed by the spatial-temporal encoding block,
depicted in Figure 3. The initial spatial fusion attention map appears
scattered, particularly in the second example, indicating insufficient
integration of audio guidance information. In contrast, attention maps for
both temporal A-V and temporal V-A modules are more precise and focused, with
the temporal A-V map in the second example almost exclusively centered on the
piano, underscoring its importance.

The Impact of Blockwise-Encoded Gate. To optimize the contributions from each
encoder block, the Blockwise-Encoded Gate was devised to fully harness the
potential of the individual encoders’ features. Table 3 demonstrates the
enhancement in model performance when incorporating the Blockwise-Encoded
Gate. Table 6 examines the influence of varying the number of channels within
the Blockwise-Encoded Gate, where channels denote the quantity obtained after
passing through a convolutional layer, reflecting the designated number of
weights. The experimental findings indicate that optimal performance is
achieved with 256 channels, corresponding to our feature dimension. This
suggests that assigning a weight to each feature channel enables the model to
effectively account for the proportional contribution of each feature.

The Impact of Audio-Queried Decoding. In the decoding phase, we developed $N$
learnable queries incorporating auditory cues and comprehensive object-level
information. We employed the $\mathcal{C}_{\text{match }}$ function to select
the query optimally aligned with the audio features, which served as the final
mask. As demonstrated in Table 3, our audio-constrained query decoding
approach substantially enhances the model’s performance. Previous models
neglect audio information in their decoding stages, resulting in segmentation
outcomes predominantly influenced by adjacent video frames. Consequently, by
emphasizing audio features during the decoding process, we effectively improve
overall performance.

###  5.4. The Impact of Audio Signals

The enhanced performance of CATR prompts an inquiry: Does this improvement
stem from a superior comprehension of pixel-level video features or more
effective utilization of audio features? To investigate, we conducted an
experiment that removed audio features from the Spatial-Temporal Encoding
Module and applied self-attention to video features. Additionally, we replaced
learnable queries, originally constrained by audio, with video features in the
decoding module. Table 4 presents the results of CATR without audio. These
findings reveal that (1) our model effectively leverages audio features, as
evidenced by the significant improvement in the M3 dataset when comparing CATR
with and without audio ($\mathcal{M_{J}}$ is 0.627 vs. 0.439 with PVT-v2); and
(2) our model demonstrates a more advanced understanding of pixel-level video
features, as shown by the superior performance in the S4 dataset, even without
employing audio information, surpassing the previous state-of-the-art TPAVI
($\mathcal{M_{J}}$ is 0.807 vs. 0.787 with PVT-v2).

##  6\. Conclusion

We introduce a novel Combinatorial-Dependence Audio-Queried Transformer (CATR)
framework that achieves state-of-the-art performance on all three datasets
using two backbones. Unlike previous methods that treated temporal video
information and audio-visual interaction separately, our proposed
combinatorial dependence fusion approach comprehensively accounts for the
spatial-temporal dependencies of audio-visual combination. Additionally, we
propose the audio-constrained learnable queries to incorporate audio
information comprehensively during decoding. These queries contain object-
level information that can select which object is being referred to segment.
To further enhance performance, we introduce a blockwise-encoded gate that
balances contributions from multiple encoder blocks. Our experimental results
demonstrate the significant impact of these novel components on overall
performance.

Limitations: Objects with similar auditory characteristics can confound video
segmentation outcomes when they coexist within a single frame. To address this
challenge, we plan to explore the refinement of audio feature pre-processing
in future research. Broader Impact: The exceptional performance of CATR
enables its practical implementation in audio-guided video segmentation
applications. These applications include utilizing auditory cues to accentuate
objects in augmented and virtual reality environments and generating pixel-
level object maps for surveillance inspections. We expect that our research
will contribute to practical applications of audio-guided video segmentation.

###### Acknowledgements.

This work was supported by the Fundamental Research Funds for the Central
Universities (No. 226-2023-00048), the National Key Research & Development
Project of China (2021ZD0110700), and the National Natural Science Foundation
of China (U19B2043, 61976185).

## References

  * (1)
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020.  End-to-End Object Detection with Transformers. In _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I_ _(Lecture Notes in Computer Science, Vol. 12346)_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer, 213–229.  https://doi.org/10.1007/978-3-030-58452-8_13
  * Chen et al. (2021a) Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. 2021a.  Localizing visual sounds the hard way. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16867–16876.
  * Chen et al. (2017) Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. 2017\.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  _IEEE transactions on pattern analysis and machine intelligence_ 40, 4 (2017), 834–848.
  * Chen et al. (2021b) Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. 2021b.  Transformer Tracking. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 8126–8135.  https://doi.org/10.1109/CVPR46437.2021.00803
  * Cheng et al. (2021) Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. 2021\.  Per-Pixel Classification is Not All You Need for Semantic Segmentation. In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 17864–17875.  https://proceedings.neurips.cc/paper/2021/hash/950a4152c2b4aa3ad78bdd6b366cc179-Abstract.html
  * Cheng et al. (2023) Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang. 2023.  Segment and track anything.  _arXiv preprint arXiv:2305.06558_ (2023).
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.  https://openreview.net/forum?id=YicbFdNTTy
  * Duan et al. (2021) Bin Duan, Hao Tang, Wei Wang, Ziliang Zong, Guowei Yang, and Yan Yan. 2021\.  Audio-visual event localization via recursive fusion by joint co-attention. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 4013–4022.
  * Duke et al. (2021) Brendan Duke, Abdalla Ahmed, Christian Wolf, Parham Aarabi, and Graham W Taylor. 2021.  Sstvos: Sparse spatiotemporal transformers for video object segmentation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5912–5921.
  * Everingham et al. (2010) Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2010\.  The pascal visual object classes (voc) challenge.  _International journal of computer vision_ 88 (2010), 303–338.
  * Gemmeke et al. (2017) Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017\.  Audio set: An ontology and human-labeled dataset for audio events. In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 776–780.
  * Hershey et al. (2017) Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin W. Wilson. 2017.  CNN architectures for large-scale audio classification. In _2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017_. IEEE, 131–135.  https://doi.org/10.1109/ICASSP.2017.7952132
  * Kamath et al. (2021) Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. 2021\.  MDETR - Modulated Detection for End-to-End Multi-Modal Understanding. In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 1760–1770.  https://doi.org/10.1109/ICCV48922.2021.00180
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023\.  Segment anything.  _arXiv preprint arXiv:2304.02643_ (2023).
  * Li et al. (2023a) Liulei Li, Wenguan Wang, Tianfei Zhou, Jianwu Li, and Yi Yang. 2023a.  Unified Mask Embedding and Correspondence Learning for Self-Supervised Video Segmentation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18706–18716.
  * Li et al. (2023b) Wenhui Li, Song Yang, Qiang Li, Xuanya Li, and An-An Liu. 2023b.  Commonsense-Guided Semantic and Relational Consistencies for Image-Text Retrieval.  _IEEE Transactions on Multimedia_ (2023).
  * Liang et al. (2022) Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang. 2022\.  Gmmseg: Gaussian mixture based generative semantic segmentation models. In _Advances in Neural Information Processing Systems_ , Vol. 35. 31360–31375.
  * Liang et al. (2023) Chen Liang, Wenguan Wang, Tianfei Zhou, Jiaxu Miao, Yawei Luo, and Yi Yang. 2023\.  Local-Global Context Aware Transformer for Language-Guided Video Segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 45, 8 (2023), 10055–10069.  https://doi.org/10.1109/TPAMI.2023.3262578
  * Lin et al. (2017a) Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. 2017a.  Feature Pyramid Networks for Object Detection. In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society, 936–944.  https://doi.org/10.1109/CVPR.2017.106
  * Lin et al. (2017b) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017b.  Focal loss for dense object detection. In _Proceedings of the IEEE international conference on computer vision_. 2980–2988.
  * Lin et al. (2019) Yan-Bo Lin, Yu-Jhe Li, and Yu-Chiang Frank Wang. 2019\.  Dual-modality seq2seq network for audio-visual event localization. In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2002–2006.
  * Lin et al. (2021) Yan-Bo Lin, Hung-Yu Tseng, Hsin-Ying Lee, Yen-Yu Lin, and Ming-Hsuan Yang. 2021.  Exploring cross-video and cross-modality signals for weakly-supervised audio-visual video parsing.  _Advances in Neural Information Processing Systems_ 34 (2021), 11449–11461.
  * Lin and Wang (2020) Yan-Bo Lin and Yu-Chiang Frank Wang. 2020.  Audiovisual transformer with instance attention for audio-visual event localization. In _Proceedings of the Asian Conference on Computer Vision_.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 9992–10002.  https://doi.org/10.1109/ICCV48922.2021.00986
  * Lu et al. (2020) Xiankai Lu, Wenguan Wang, Martin Danelljan, Tianfei Zhou, Jianbing Shen, and Luc Van Gool. 2020\.  Video object segmentation with episodic graph memory networks. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_. Springer, 661–679.
  * Mahadevan et al. (2020) Sabarinath Mahadevan, Ali Athar, Aljoša Ošep, Sebastian Hennen, Laura Leal-Taixé, and Bastian Leibe. 2020.  Making a case for 3d convolutions for object segmentation in videos.  _arXiv preprint arXiv:2008.11516_ (2020).
  * Mao et al. (2021) Yuxin Mao, Jing Zhang, Zhexiong Wan, Yuchao Dai, Aixuan Li, Yunqiu Lv, Xinyu Tian, Deng-Ping Fan, and Nick Barnes. 2021.  Transformer transforms salient object detection and camouflaged object detection.  _arXiv preprint arXiv:2104.10127_ (2021).
  * Meinhardt et al. (2022) Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixé, and Christoph Feichtenhofer. 2022.  TrackFormer: Multi-Object Tracking with Transformers. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_. IEEE, 8834–8844.  https://doi.org/10.1109/CVPR52688.2022.00864
  * Milletari et al. (2016) Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. 2016\.  V-net: Fully convolutional neural networks for volumetric medical image segmentation. In _2016 fourth international conference on 3D vision (3DV)_. Ieee, 565–571.
  * Qian et al. (2020) Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. 2020\.  Multiple sound sources localization from coarse to fine. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16_. Springer, 292–308.
  * Ramaswamy and Das (2020) Janani Ramaswamy and Sukhendu Das. 2020.  See the sound, hear the pixels. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 2970–2979.
  * Sun et al. (2020) Peize Sun, Yi Jiang, Rufeng Zhang, Enze Xie, Jinkun Cao, Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, and Ping Luo. 2020\.  TransTrack: Multiple-Object Tracking with Transformer.  _CoRR_ abs/2012.15460 (2020).  arXiv:2012.15460 https://arxiv.org/abs/2012.15460
  * Tian et al. (2020) Yapeng Tian, Dingzeyu Li, and Chenliang Xu. 2020.  Unified multisensory perception: Weakly-supervised audio-visual video parsing. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_. Springer, 436–454.
  * Tian et al. (2018) Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. 2018.  Audio-visual event localization in unconstrained videos. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 247–263.
  * Vaswani et al. (2017a) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017a.  Attention is All you Need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 5998–6008.  https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  * Vaswani et al. (2017b) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017b.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2021b) Wenguan Wang, Tianfei Zhou, Fatih Porikli, David Crandall, and Luc Van Gool. 2021b.  A survey on deep learning technique for video segmentation.  _arXiv e-prints_ (2021), arXiv–2107.
  * Wang et al. (2021a) Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. 2021a.  End-to-End Video Instance Segmentation With Transformers. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 8741–8750.  https://doi.org/10.1109/CVPR46437.2021.00863
  * Wu et al. (2022) Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. 2022.  Language as queries for referring video object segmentation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4974–4984.
  * Wu and Yang (2021) Yu Wu and Yi Yang. 2021\.  Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1326–1335.
  * Wu et al. (2019) Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. 2019\.  Dual attention matching for audio-visual event localization. In _Proceedings of the IEEE/CVF international conference on computer vision_. 6292–6300.
  * Xu et al. (2020) Haoming Xu, Runhao Zeng, Qingyao Wu, Mingkui Tan, and Chuang Gan. 2020.  Cross-modal relation-aware networks for audio-visual event localization. In _Proceedings of the 28th ACM International Conference on Multimedia_. 3893–3901.
  * Xuan et al. (2020) Hanyu Xuan, Zhenyu Zhang, Shuo Chen, Jian Yang, and Yan Yan. 2020.  Cross-modal attention network for temporal inconsistent audio-visual event localization. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 279–286.
  * Yan et al. (2021) Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. 2021.  Learning Spatio-Temporal Transformer for Visual Tracking. In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 10428–10437.  https://doi.org/10.1109/ICCV48922.2021.01028
  * Yang et al. (2021c) Yi Yang, Yueting Zhuang, and Yunhe Pan. 2021c.  Multiple knowledge representation for big data artificial intelligence: framework, applications, and case studies.  _Frontiers of Information Technology & Electronic Engineering_ 22, 12 (2021), 1551–1558.
  * Yang et al. (2021a) Zongxin Yang, Yunchao Wei, and Yi Yang. 2021a.  Associating objects with transformers for video object segmentation.  _Advances in Neural Information Processing Systems_ 34 (2021), 2491–2502.
  * Yang et al. (2021b) Zongxin Yang, Yunchao Wei, and Yi Yang  2021b.  Collaborative video object segmentation by multi-scale foreground-background integration.  _TPAMI_ 44, 9 (2021), 4701–4712.
  * Yang and Yang (2022) Zongxin Yang and Yi Yang. 2022.  Decoupling Features in Hierarchical Propagation for Video Object Segmentation. In _NeurIPS_.
  * Yu et al. (2022) Jiashuo Yu, Ying Cheng, Rui-Wei Zhao, Rui Feng, and Yuejie Zhang. 2022.  Mm-pyramid: Multimodal pyramid attentional network for audio-visual event localization and video parsing. In _Proceedings of the 30th ACM International Conference on Multimedia_. 6241–6249.
  * Zhang et al. (2021) Jing Zhang, Jianwen Xie, Nick Barnes, and Ping Li. 2021\.  Learning generative vision transformer with energy-based latent space for saliency prediction.  _Advances in Neural Information Processing Systems_ 34 (2021), 15448–15463.
  * Zhang et al. (2023) Yurong Zhang, Liulei Li, Wenguan Wang, Rong Xie, Li Song, and Wenjun Zhang. 2023\.  Boosting Video Object Segmentation via Space-time Correspondence Learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2246–2256.
  * Zheng et al. (2021) Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, and Li Zhang. 2021.  Rethinking Semantic Segmentation From a Sequence-to-Sequence Perspective With Transformers. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 6881–6890.  https://doi.org/10.1109/CVPR46437.2021.00681
  * Zhou et al. (2023) Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, et al. 2023\.  Audio-Visual Segmentation with Semantics.  _arXiv preprint arXiv:2301.13190_ (2023).
  * Zhou et al. (2022) Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. 2022\.  Audio-Visual Segmentation. In _Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII_ _(Lecture Notes in Computer Science, Vol. 13697)_ , Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer, 386–403.  https://doi.org/10.1007/978-3-031-19836-6_22
  * Zhou et al. (2021) Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, and Meng Wang. 2021.  Positive sample propagation along the audio-visual event line. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8436–8444.
  * Zhu et al. (2021) Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2021\.  Deformable DETR: Deformable Transformers for End-to-End Object Detection. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.  https://openreview.net/forum?id=gZ9hCDWe6ke
  * Zou et al. (2023) Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. 2023.  Segment everything everywhere all at once.  _arXiv preprint arXiv:2304.06718_ (2023).