beforefloat afterfloat [table]capposition=top capbtabboxtable[][]

11institutetext: Centre for Vision, Speech and Signal Processing (CVSSP),
University of Surrey, UK
11email<EMAIL_ADDRESS>
# CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised
Audio-Visual Video Parsing

Faegheh Sardari 11 0000-0002-9134-0427 Armin Mustafa 11 0000-0002-1779-2775
Philip J. B. Jackson 11 0000-0001-7933-5935
Adrian Hilton 11 0000-0003-4223-238X

###### Abstract

Weakly supervised audio-visual video parsing (AVVP) methods aim to detect
audible-only, visible-only, and audible-visible events using only video-level
labels. Existing approaches tackle this by leveraging unimodal and cross-modal
contexts. However, we argue that while cross-modal learning is beneficial for
detecting audible-visible events, in the weakly supervised scenario, it
negatively impacts unaligned audible or visible events by introducing
irrelevant modality information. In this paper, we propose CoLeaF, a novel
learning framework that optimizes the integration of cross-modal context in
the embedding space such that the network explicitly learns to combine cross-
modal information for audible-visible events while filtering them out for
unaligned events. Additionally, as videos often involve complex class
relationships, modelling them improves performance. However, this introduces
extra computational costs into the network. Our framework is designed to
leverage cross-class relationships during training without incurring
additional computations at inference. Furthermore, we propose new metrics to
better evaluate a method’s capabilities in performing AVVP. Our extensive
experiments demonstrate that CoLeaF significantly improves the state-of-the-
art results by an average of 1.9% and 2.4% F-score on the LLP and UnAV-100
datasets, respectively. Code is available at:
https://github.com/faeghehsardari/coleaf.

###### Keywords:

Unaligned audio-visual learning Audio-visual video parsing Weakly supervised
learning Event detection

##  1 Introduction

Audio-visual learning has emerged as a crucial field within computer vision,
enabling AI systems to develop a more nuanced understanding of the world. Most
audio-visual tasks (e.g., audio-visual event localization [18, 23, 4] and
audio-visual question answering [25, 10, 14]) assume that audio and visual
data are temporally aligned and share a common concept. However, in practice,
this assumption is not always valid. For instance, a baby might cry out of the
camera’s view. The Audio-Visual Video Parsing (AVVP) task [17] instead
introduces an unaligned setting of audio-visual learning to recognize and
localize audible-only, visible-only, and audible-visible events in a video.
Since training a fully supervised AVVP model requires modality-specific
densely annotated data, making it a costly endeavour, the AVVP task is tackled
in a weakly supervised scenario where only video-level labels are available,
see Fig.2.

Figure 1: An example of weakly supervised AVVP task. The method learns to
temporally detect audible-only, visible-only, and audible-visible events in a
video, using only video-level labels during training.

Figure 2: Performance of CMPAE [3] when using only unimodal (U) information
versus exploiting both unimodal and cross-modal contexts (U+C) in detecting
audible-only (A), visible-only (V) and audible-visible (AV) event types, in
terms of true positive (TP), true negative (TN), false positive (FP), and
false negative (FN) rates.

Metric |  | A |  | V |  | AV
---|---|---|---|---|---|---
| U/(U+C) |  | U/(U+C) |  | U/(U+C)
TP |  |  43.5 / 42.7 |  |  67.8 / 66.4 |  | 80.8 / 84.6
TN |  |  97.6 / 97.4 |  |  97.3 / 97.0 |  | 95.2 / 95.6
FP |  |  0.4 / 0.6 |  | 0.9 / 0.7 |  |  3.3 / 3.8
FN |  |  94.7 / 95.5 |  |  97.7 / 99.2 |  | 33.2 / 29.9

To address the weakly supervised AVVP task, all existing approaches, such as
[17, 24, 3], incorporate unimodal and cross-modal contexts. However, we argue
that while extracting cross-modal information is beneficial for multi-modal
learning tasks, in the weakly supervised scenario, it can negatively impact
the detection of audible-only and visible-only events (i.e., unaligned
audible-visible events) by introducing irrelevant modality information. We
investigated this in Table 2 by comparing the performance of CMPAE [3] with
and without using cross-modal contexts. The results show that leveraging
cross-modal contexts improves the performance on average in detecting audible-
visible events, while in detecting unaligned events, a network focusing solely
on unimodal information yields better results. These findings motivated us to
raise this question:  Can we optimize the integration of cross-modal context
in the embedding space such that the network explicitly learns to leverage
cross-modal information for audible-visible events, while filtering them out
for unaligned events? To achieve this, we introduce CoLeaF, a novel learning
framework for the weakly supervised AVVP task. CoLeaF relies on two network
branches: Reference and Anchor. Reference focuses on unimodal information,
while Anchor incorporates unimodal and cross-modal contexts. Both branches
process the same audio and visual input signals and are trained simultaneously
using the video-level labels. Additionally, during training, Reference is
leveraged by a novel contrastive objective to optimize cross-modal learning in
Anchor based on the event types present in the input video. The proposed
objective encourages Anchor to learn embedding representations similar to
those learned in Reference, but the strength of the encouragement adapts to
the degree of unaligned audible-visible events present in the input video.
More unaligned events lead to a stronger encouragement, while fewer unaligned
events lead to a weaker encouragement. The unalignment degree is computed from
the Reference’s output predictions. Since the optimization process relies on
the performance of Reference, its effectiveness may diminish if Reference is
solely trained on ‘modality-agnostic’ video-level labels. To address this
without requiring additional annotations, Anchor collaboratively distils free
modality-aware pseudo-labels for Reference. After training, Anchor is deployed
for inference.

To effectively perform the AVVP task, we should exploit unimodal and cross-
modal contexts while aware of class dependencies. For example, this is often
seen when musical instruments are played alongside people singing. Authors in
[12, 13] design networks to explicitly learn cross-class relationships.
However, their approaches are computationally expensive. For example, in [12],
to capture cross-class dependencies, they extend their model to map each
temporal segment of the video into the maximum number of event classes, and
apply attention layers within the event classes for each segment individually.
For a video with a length of T segments and C class categories, this led to a
significant computational overhead with a ratio of $T\times C\times C$. This
key challenge motivated us to raise this novel question: Can we take advantage
of explicitly modelling co-occurrence class relationships without imposing the
overhead of heavy computational costs? To achieve this, in CoLeaF, Reference
that is employed only during training is designed to explicitly learn co-
occurrence class relationships and distil this knowledge to Anchor. This novel
design allows us to benefit from cross-class relationships during training
without introducing computational overhead on Anchor that is employed at
inference.

For a deeper understanding of how our proposed framework models unaligned
events, we introduce novel metrics for AVVP. Current metrics evaluate the
method’s performance in detecting audible-only and visible-only events by
comparing the estimated audio and visual event proposals individually with
audio and visual ground-truth labels. In our proposed metrics, we consider
both audio and visual modalities, as comparing them individually can lead to
the mistaken interpretation of an audible-visible event as both audible-only
and visible-only. Our key contributions are summarized as: (i) we pioneer the
first exploration in explicitly optimizing the integration of cross-modal
context for the weakly supervised AVVP task, (ii) we propose an approach that
allows the network to explicitly learn cross-class relationships without
adding computational overhead during inference. This can benefit various
computer vision tasks that demand a deeper understanding of interactions
between classes, (iii) we introduce novel metrics for effective evaluation of
AVVP which are beneficial for future development of this task, (iv) we
evaluate CoLeaF on two challenging publicly available datasets and improve the
state-of-the-art results on average by 1.9% and 2.4% F-score on LLP and
UnAV-100 datasets, respectively, and (v) we perform extensive ablation studies
to evaluate our proposed framework’s design.

##  2 Related Works

Tian et al. [17] defined the AVVP task and also introduced the LLP dataset for
this purpose. Due to the laborious nature of the labelling process for this
task, for the training set, they obtained only video-level labels, so the
learning process is performed in a weakly supervised setting. To perform the
task, they design the Hybrid Attention Network (HAN), where they extract and
combine unimodal and cross-modal contexts from audio and visual data by
applying self-attention and cross-attention mechanisms and introduce
Multimodal Multiple Instance Learning (MMIL) to facilitate network training
under weakly supervised setting. Following HAN, the proposed approaches can be
divided into the two categories:

Architectural Improvements –  To better explore unimodal and cross-modal
contexts, the authors in [24, 8] expand HAN to a multi-scale pyramid network.
Alternatively, Gao et al. [3] attempt to decrease the uncertainty of HAN’s
predictions under weakly supervised learning by introducing a modality
presence-absence confidence term in the cross-entropy loss function based on
Subjective Logic Theory [16]. Similar to the network architecture, their loss
confidence term is also explicitly derived from the integration of unimodal
and cross-modal information. However, as stated in Section 1, the cross-modal
learning in these architectures negatively affects the detection of unaligned
events. In contrast, in our proposed framework, the network explicitly learns
to filter out cross-modal contexts for unaligned events. On the other hand,
[12, 13] additionally learn cross-class dependencies. Although they improve
HAN’s performance, they suffer from computational complexity. In contrast, our
framework leverages the class relationships during training without incurring
additional costs in inference.

Label Refinement –  The core goal of these approaches [21, 8, 2, 3, 26] is to
enhance the performance of existing network architectures by obtaining
modality-specific knowledge for training. For example, the authors in [3]
devised a two-stage approach in which they initially extract offline modality-
specific annotations by applying large-scale pre-trained models such as CLAP
[22] and CLIP [15] on the audio and visual data. Subsequently, they train a
network backbone from scratch using the additional labels and a cross-entropy
loss function. However, the training process of such approaches is complicated
and inefficient. Alternatively, [8] propose an online method where the
estimated probabilities of a network, focusing on individual modalities, guide
the final estimation of the AVVP task through the mean square error loss.
These approaches may optimize the integration of cross-modal information as
they obtain hard/soft modality-aware labels for training. However, their
optimization process is implicit and suboptimal. In contrast, our proposed
approach explicitly optimizes cross-modal learning in the embedding space.
Furthermore, in these approaches, the pseudo labels are obtained from
‘modality-agnostic’ video-level labels, making them unreliable. However, our
online collaborative solution mitigates the impact of weak labels on our
optimization process.

##  3 Methodology

In this section, we first define the weakly supervised audio-visual video
parsing task, followed by a brief review of the HAN network [17]. Then, we
elaborate on our proposed framework CoLeaF.

###  3.1 Preliminaries

Problem Formulation –  In weakly supervised AVVP, our goal is to detect
audible-only, visible-only, and audible-visible events in a given video by
training from video-level labels, as defined in [17]. For the video sequence
$S=\\{A_{t},V_{t}\\}^{T}_{t=1}$ with a length of T segments, $A_{t}$ and
$V_{t}$ represent the audio and visual signals at the $t^{th}$ segment,
respectively. At the inference, the network task is to provide the multi-class
event labels $Y_{t}=\\{y^{a}_{t,i},y^{v}_{t,i},y^{av}_{t,i}\\}^{C}_{i=1}$ for
each time segment of $S$, where
$y^{a}_{t,i},y^{v}_{t,i},y^{av}_{t,i}\in\\{0,1\\}$ indicate audible-only,
visible-only, and audible-visible events, respectively, and $C$ represents the
maximum number of event classes in the dataset. However, during training, we
lack access to the ‘modality-specific segment-level’ labels $Y_{t}$, having
only the ‘video-level’ labels that are ‘modality-agnostic’
${Y}=\\{{y}_{i}\\}^{C}_{i=1}$, where $y_{i}\in\\{0,1\\}$.

Hybrid Attention Network (HAN) –  HAN [17] is a popular baseline for the
weakly supervised AVVP task. In HAN, first, two pre-trained models are applied
on the audio and visual signals of the input video $S$ to extract segment-
level audio and visual input tokens, $F^{a}=\\{f^{a}_{t}\\}^{T}_{t=1}$ and
$F^{v}=\\{f^{v}_{t}\\}^{T}_{t=1}$, respectively. Then, it aggregates unimodal
and cross-modal information through applying and combining self-attention and
cross-attention mechanisms [20] on the input tokens

${\hat{f}}^{a}_{t}={f}^{a}_{t}+\text{Self-
attn}({f}^{a}_{t},{F}^{a})+\text{Cross-attn}({f}^{a}_{t},{F}^{v}),$ (1)
${\hat{f}}^{v}_{t}={f}^{v}_{t}+\text{Self-
attn}({f}^{v}_{t},{F}^{v})+\text{Cross-attn}({f}^{v}_{t},{F}^{a}),$ (2)

Finally, HAN uses a share fully connected layer FC to predict multi-class
segment-level audio and visual event probabilities from the audio and visual
output tokens as ${P}^{\phi}=Sigmoid(FC(\hat{F}^{\phi}))$, where
$P^{\phi}=\\{{p}^{\phi}_{t}\in\rm{I\\!R}^{1\times C}\\}^{T}_{t=1}$ and
$\phi\in\\{a,v\\}$. During the inference $P^{\phi}$ is utilized to predict
segment-level audible-only, visible-only, and audible-visible events. However,
during the training, as we only have video-level labels, HAN applies a
Multimodal Multiple Instance Learning (MMIL) module to provide video-level
probabilities as

$\small W^{\vartriangle}=\textit{Soft-
max}_{t}(FC_{t}([\hat{F}^{a};\hat{F}^{v}]),~{}W^{\blacktriangle}=\textit{Soft-
max}_{av}(FC_{av}([\hat{F}^{a};\hat{F}^{v}]),$ (3)
$\small\mathbb{P}=\sum_{t=1}^{T}\sum_{m=1}^{M}(W^{\vartriangle}_{t,m}\odot
W^{\blacktriangle}_{t,m}\odot{P}_{t,m}),~{}\text{where }{P}=[P^{a};P^{v}],$
(4)

and $M=2$ refers to audio and visual modalities,
${P},W^{\vartriangle},W^{\blacktriangle}\in\rm{I\\!R}^{T\times 2\times C}$,
$\mathbb{P}\in\rm{I\\!R}^{1\times C}$, and $[;]$ and $\odot$ indicates the
concatenation and Hadamard product operations, respectively. $\textit{Soft-
max}_{t}$ and $\textit{Soft-max}_{av}$ represent the Softmax layers along the
temporal and audio-visual dimensions, respectively. With the video-level
probabilities $\mathbb{P}$, and the video-level labels $Y$, HAN is optimized
through Binary Cross Entropy (BCE) loss
$\mathcal{{L}}_{video}=BCE(Y,\mathbb{P})$.

Figure 3: The overall scheme of CoLeaF for the weakly supervised AVVP task.
CoLeaF has two network branches, Reference and Anchor. The Reference branch
focuses on unimodal information and cross-class relationships, while the
Anchor branch exploits both unimodal and cross-modal contexts. The branches
are trained simultaneously through the conventional video-level losses
$\\{\mathcal{{L}}^{Ref}_{video},\mathcal{{L}}^{Anchr}_{video}\\}$ and our
novel contrastive and knowledge distillation losses
$\\{\mathcal{L}^{Anch}_{Evt},\mathcal{L}^{Ref}_{SelfMo},\mathcal{L}^{Anchr}_{CoCls}\\}$.
In inference, Anchor is deployed for the AVVP task.

###  3.2 CoLeaF: Contrastive-Collaborative Learning Framework

In this section, we introduce CoLeaF, a novel learning framework designed to
address two key challenges in the weakly supervised AVVP task: (i) effectively
integrating the cross-modal contexts with the lack of modality-specific
annotations, and (ii) efficiently learning cross-class relationships. CoLeaF
relies on two network branches, Reference and Anchor, as shown in Fig. 3. The
Reference branch, used only during training, is dedicated to exploiting
unimodal and co-occurrence class relationships. The Anchor branch exploits
unimodal and cross-modal contexts and serves as the main network branch
deployed during inference. The branches receive the same audio and visual
input data and are trained simultaneously using two types of losses: (a) the
conventional video-level weakly supervised loss, as in HAN, and (b) our
proposed collaborative losses, including event-aware contrastive loss, co-
occurrence class knowledge distillation loss, and self-modality-aware
knowledge distillation loss. In the following, we explain the details of each
branch and the losses.

Reference Branch –  This branch addresses the weakly supervised AVVP task by
primarily focusing on unimodal information. This can be achieved by applying
two self-attention layers on the audio and visual input tokens individually.
However, as videos with multi-label events contain complex relationships
between the event classes, we simultaneously leverage this information to
improve the method’s performance. To model explicitly the cross-class
relationships, inspired by [13], we first concatenate the audio and visual
input tokens, $F^{a}=\\{f^{a}_{t}\\}^{T}_{t=1}$ and
$F^{v}=\\{f^{v}_{t}\\}^{T}_{t=1}$, with learnable audio and visual class
tokens, $C^{a}=\\{c^{a}_{i}\\}^{C}_{i=1}$ and
$C^{v}=\\{c^{v}_{i}\\}^{C}_{i=1}$,

${X}^{\phi}=\\{{x}^{\phi}_{n}\\}^{T+C}_{n=1}=[\\{f^{\phi}_{t}\\}^{T}_{t=1};\\{c^{\phi}_{i}\\}^{C}_{i=1}],$
(5)

where $\phi\in\\{a,v\\}$ and $[;]$ indicates the concatenation operations. We
then forward them into our self-attention backbones to exploit both intra-
modality and class relationships,

$\\{\ddot{x}^{\phi}\\}^{T+C}_{n=1}=\\{\text{Self-
attn}_{\phi}({x}^{\phi}_{n},{X}^{\phi})\\}^{T+C}_{n=1}.$ (6)

We take the outputs of self-attention layers which correspond to the audio and
visual input tokens, $\ddot{F}^{\phi}=\\{\ddot{x}^{\phi}_{n}\\}^{T}_{n=1}$, to
predict segment-level audio and visual event probabilities through a fully
connected layer as
$\\{\ddot{p}^{\phi}_{t}\\}^{T}_{t=1}=Sigmoid(FC_{\phi}(\ddot{F}^{\phi}))$,
where $\ddot{p}^{\phi}_{t}\in\rm{I\\!R}^{1\times C}$. Subsequently, the audio
and visual event probabilities are computed as
${\ddot{\mathcal{P}}}^{\phi}=\sum_{t=1}^{T}(\ddot{w}^{\phi}_{t}\odot\ddot{p}^{\phi}_{t})$

$\small{\ddot{\mathcal{P}}}^{\phi}=\sum_{t=1}^{T}(\ddot{w}^{\phi}_{t}\odot\ddot{p}^{\phi}_{t}),$
(7)

where ${\ddot{\mathcal{P}}}^{\phi},\ddot{w}_{t}\in\rm{I\\!R}^{1\times C}$, and
$\ddot{w}_{t}$ are temporal learnable weight. With the audio and visual event
probabilities and video-level annotations $Y\in\rm{I\\!R}^{1\times C}$, the
Reference branch is optimized through the video-level loss
$\mathcal{{L}}^{Ref}_{video}=\sum_{\phi\in\\{a,v\\}}BCE(Y,\ddot{\mathcal{P}}^{\phi})$.

Optimizing the Reference branch by relying only on the $\ddot{F}^{\phi}$
output tokens does not guarantee that the input class tokens learn to model
the class events. To ensure that they represent the classes effectively, we
also utilize the output tokens of self-attention layers which correspond to
the input class tokens,
$\\{\ddot{c}^{\phi}\\}^{C}_{i=1}=\\{\ddot{x}^{\phi}_{n}\\}^{T+C}_{n=T+1}$, for
the network optimization. To achieve this, we apply a global average pooling
layer on these tokens along their embedding dimension to convert them into a
1-D dimensional vector, followed by a sigmoid activation function to convert
them into range 0 and 1 as
$CLS^{\phi}=\\{Sigmoid(AvgPool(\ddot{c}^{\phi}_{i}))\\}^{C}_{i=1}$. Then, they
are supervised by video-level annotations and cross entropy loss. Therefore,
the video-level loss is extended as
$\mathcal{{L}}^{Ref}_{video}=\sum_{\phi\in\\{a,v\\}}BCE(Y,\ddot{\mathcal{P}}^{\phi})+BCE(Y,{CLS}^{\phi})$.

Anchor Branch –  This branch tackles the weakly supervised AVVP task by
exploiting both unimodal and cross-modal contexts. To achieve this, a backbone
that performs the AVVP task using self-attention and cross-attention
mechanisms (e.g., HAN) can serve as Anchor to predict segment-level and video-
level probabilities $P^{\phi}=\\{p^{\phi}_{t}\in\rm{I\\!R}^{1\times
C}\\}^{T}_{t=1}$ and $\mathbb{P}\in\rm{I\\!R}^{1\times C}$ (Eqs 3-4),
respectively, from the audio and visual input tokens $F^{\phi}$. The segment-
level labels are employed for inference phase, as well as later in this
section for our proposed self-modality-aware knowledge distillation loss,
while the video-level probabilities are utilized to compute the video-level
loss $\mathcal{{L}}^{Anch}_{video}=BCE(Y,\mathbb{P})$, as in HAN network.

Event-Aware Contrastive Learning – We argue that while leveraging cross-modal
information is beneficial in the detection of audible-visible events, in the
weakly supervised scenario, it can negatively impact the detection of audible-
only and visible-only events by introducing irrelevant modality information.
Our findings in Table 2 support this argument. To tackle this challenge, we
introduce contrastive learning to encourage the learned representations in the
Anchor branch to become similar to those of the Reference branch for unaligned
audio-visible event types. To achieve this, for each audio and visual output
tokens of the Anchor branch ${\hat{f}}^{\phi}_{t}$, the output tokens of the
corresponding segment from the Reference branch $\ddot{x}^{\phi}_{t}$ can be
chosen as positive samples, and the output tokens of the rest of its segments
can be considered as negative set. Subsequently, the Anchor branch can be
optimized through Noise Contrastive Estimation (NCE) loss [5]. In this
formulation, regardless of the event types in a video, Anchor is encouraged to
align with Reference. However, a video can contain multiple events spanning
audible-only, visible-only, and audible-visible, and our goal is to get close
to the tokens’ representations of the Reference branch for only unaligned
audio-visible event types. We address this issue by introducing a novel event-
aware NCE loss that adjusts the strength of contrastive learning encouragement
based on the degree of audible-only and visible-only events present in the
input video

$\small\mathcal{L}^{Anch}_{Evt}=-\frac{1}{T}*\sum_{\phi\in\\{a,v\\}}\vartheta^{\phi}*\sum_{t=1}^{T}\log{\frac{\exp{({{\hat{f}}^{\phi^{\intercal}}_{t}}.{\ddot{x}^{\phi}}_{t}/\tau)}}{\sum_{n=1,n\neq
t}^{T}\exp{({{\hat{f}}^{\phi^{\intercal}}_{t}}.{\ddot{x}^{\phi}}_{n}/\tau)}}},$
(8)

where weight $\vartheta^{\phi}$ is computed from a set of pseudo-labels
$\ddot{G}^{\phi}=\\{\ddot{g}^{\phi}_{i}\\}^{C}_{i=1}$ distilled from the
Reference branch

$\displaystyle\small\ddot{g}^{\phi}_{i}=\begin{cases}1&\text{if
$\ddot{\mathcal{P}}^{\phi}_{i}>\theta$}\\\ 0&\text{else}\end{cases},$ (9)
$\displaystyle
N^{a}=\sum_{c=1}^{C}\ddot{g}^{a}_{i}\odot(1-\ddot{g}^{v}_{i}),N^{v}=\sum_{c=1}^{C}\ddot{g}^{v}_{i}\odot(1-\ddot{g}^{a}_{i}),N^{av}=\sum_{c=1}^{C}\ddot{g}^{a}_{i}\odot\ddot{g}^{v}_{i},$
(10) $\displaystyle\vartheta^{a}=\frac{N^{a}}{N^{a}+N^{av}},\text{ and
}\vartheta^{v}=\frac{N^{v}}{N^{v}+N^{av}}.$ (11)

$N^{a}$, $N^{v}$, and $N^{av}$ indicate the number of audible-only, visible-
only, and audible-visible events in the video, respectively.
$\vartheta^{\phi}$ reflects the degree of encouragement in the contrastive
learning and takes a value between 0 and 1 where 0 indicates there is no event
or all the events are audible-visible, resulting in no encouragement, and 1
indicates all the events are unaligned, resulting in maximum encouragement.

Self-Modality-Aware Knowledge Distillation –  If the Reference branch relies
solely on ‘modality-agnostic’ video-level labels for training, we face two key
limitations: (i) Reference cannot acquire rich representations necessary to
teach Anchor, and (ii) the proposed event-aware contrastive loss may not
effectively optimize cross-modal learning since it relies on the predicted
event types by Reference. To overcome these without additional annotations, we
propose to leverage free modality-aware knowledge from the Anchor branch to
improve the performance of the Reference branch. To perform this, we first
distil pseudo labels ${G}^{\phi}=\\{{g}^{\phi}_{i}\\}^{C}_{i=1}$ from the
estimated audio and visual event probabilities in the Anchor branch,

${g}^{\phi}_{i}=\begin{cases}1&\text{if ${\mathcal{P}}^{\phi}_{i}>\theta$}\\\
0&\text{else}\end{cases}$ (12)

Then, ${G}^{\phi}$ are used to compute our proposed self-modality-aware
knowledge distillation loss for the Reference branch as
$\mathcal{{L}}^{Ref}_{SelfMo}=\sum_{\phi\in\\{a,v\\}}BCE({G}^{\phi},\mathcal{\ddot{P}}^{\phi})$.

Co-occurrence Class Knowledge Distillation –  The primary goal of modeling the
co-occurrence class relationships explicitly in the Reference branch is to
transfer them to the Anchor branch so that we can benefit from this knowledge
during inference without suffering from the associated computation overhead.
Optimizing the Anchor branch through our proposed contrastive learning may
implicitly transfer class relationships from the Reference branch to the
Anchor branch, but cannot fully transfer them because it is not designed for
this purpose. To explicitly transfer the cross-class relationships, we
introduce the co-occurrence class knowledge distillation loss, where we first
obtain the audio and visual class correlation matrices,
${\ddot{M}^{\phi}}\in\rm{I\\!R}^{C\times C}$ and
${{M}^{\phi}}\in\rm{I\\!R}^{C\times C}$, for the Reference and Anchor branches
as

$\small{\ddot{M}^{\phi}}_{i,j}=\ddot{\mathcal{P}}^{\phi}_{i}*\ddot{\mathcal{P}}^{\phi}_{j}\text{,
and }{{M}^{\phi}}_{i,j}={\mathcal{P}}^{\phi}_{i}*{\mathcal{P}}^{\phi}_{j}.$
(13)

By computing the correlations, we quantify the relationships between each pair
of classes. Then, we encourage the Anchor branch to mimic the audio and visual
cross-class correlations of the Reference branch through the mean square error
(MSE) as
$\mathcal{{L}}^{Anch}_{CoCls}=\sum_{\phi\in\\{a,v\\}}MSE(\ddot{M}^{\phi},{M}^{\phi})$.

##  4 New Metrics for AVVP

Current approaches utilize 5 metrics, under segment-level and event-level, to
evaluate the method’s performance for the AVVP task. These metrics include
F-score for audible-only (A), visible-only (V), and audible-visible (AV)
events separately, as well as the average F-score of the A, V, and AV
(Type@AV), and the overall F-score for all A and V events within each sample
(Event@AV). To compute these metrics, they first employ a threshold on the
estimated audio and visual event probabilities to identify audible and visible
events, $\hat{y}^{a}_{t}$ and $\hat{y}^{v}_{t}$, respectively. Then, the
F-score for A and V are computed by comparing $\hat{y}^{a}_{t}$ and
$\hat{y}^{v}_{t}$ with their corresponding audio and visual ground-truth
labels, ${y}^{a}_{t}$ and ${y}^{v}_{t}$, respectively. The F-score for AV
instead is computed by simultaneously considering both audible and visible
events and their ground-truth labels.

We argue that the A and V metrics fail to accurately evaluate the method’s
performance in detecting audible-only and visible-only events because they
consider only their corresponding modalities. In contrast, we believe that
both modalities should be considered when computing A and V, similar to AV.
For instance, consider a scenario where the network predicts
$\hat{y}^{a}_{t}=1$ and $\hat{y}^{v}_{t}=1$ for an audible-only event with
ground-truth labels $y^{a}_{t}=1$ and $y^{v}_{t}=0$. According to the A
metric, this prediction is considered a true positive. However, when both
modalities are considered, it becomes evident that the network’s prediction is
incorrect, as it predicted an audible-visible event instead of an audible-only
event. Thus, this should be considered a false positive for the audible-only
metric. Consequently, a network that excels at detecting audible-visible
events may falsely exhibit high true positive rates for the A and V metrics.
This issue becomes particularly prominent when using cross-modal context in a
weakly supervised scenario, as methods under these settings inherently tend to
detect events as audible-visible. To overcome this, we introduce ‘audible-
only’ and ‘visible-only’ event proposals, $\hat{y}^{ao}_{t}$ and
$\hat{y}^{vo}_{t}$, as well as ‘audible-only’ and ‘visible-only’ ground-truth
labels, ${y}^{ao}_{t}$ and ${y}^{vo}_{t}$, that are derived by considering
both modalities as

$\displaystyle\small\hat{y}^{ao}_{t}=\hat{y}^{a}_{t}\odot(1-\hat{y}^{v}_{t}),\text{
and }\hat{y}^{vo}_{t}=\hat{y}^{v}_{t}\odot(1-\hat{y}^{a}_{t}),$ (14)
$\displaystyle{y}^{ao}_{t}={y}^{a}_{t}\odot(1-{y}^{v}_{t}),\text{ and
}{y}^{vo}_{t,i}={y}^{v}_{t}\odot(1-{y}^{a}_{t}).$ (15)

These new event proposals and their corresponding ground-truth labels are used
to compute the F-score for audible-only (Ao) and visible-only (Vo) events.
Subsequently, we have Type@AVo, which computes the average F-score of Ao, Vo,
and AV, and Event@AVo, which computes the overall F-score for all Ao and Vo
events within each sample.

##  5 Experimental Results

Dataset – LLP [17] is the only benchmark dataset for the AVVP task. It
comprises 11,849 10-second YouTube video clips spanning 25 event classes,
offering a diverse range of video events. Notably, 7,202 of these videos
contain multiple event categories. In the LLP dataset, videos are partitioned
into training, validation, and testing sets. While the training set includes
only video-level annotations, the validation and test sets feature annotations
for individual audio and visual events with second-wise temporal boundaries.

Implementation Details – We implement the Reference branch by employing two
self-attention layers, _i.e._ , one for each modality, followed by two fully
connected layers that obtain the temporal weights and estimate class
probabilities, respectively. For the Anchor branch, we employ CMPAE [3] as the
backbone. Following previous works [17, 3], audio input tokens are extracted
through pre-trained VGGish [7], and visual tokens are obtained through the
pre-trained models ResNet152 [6] and R(2+1)D [19]. The feature dimension for
all input tokens, including the class tokens, is set to 512, as in previous
approaches. To train CoLeaF, we utilized the Adam optimizer with an initial
learning rate of $5\times 10^{-4}$ and a batch size of 128 for 15 epochs. The
learning rate was decayed by a factor of 0.25 every 6 epochs. Our experiments
were performed using PyTorch on an NVIDIA GeForce RTX 3090 GPU.

###  5.1 Ablation Studies

In this section, we examine the impact of key components of CoLeaF, using the
Ao, Vo, and AV metrics on LLP.

Impact of Event-Aware Contrastive Learning –  To evaluate the effect of our
proposed contrastive loss, $\mathcal{L}^{Anch}_{Evt}$, we compare the
performance of our framework against three variants: one without
$\mathcal{L}^{Anch}_{Evt}$, and two where it is replaced with the standard NCE
loss $\mathcal{L}_{NCE}$ and BCE loss $\mathcal{L}_{BCE}$, the latter using
the pseudo-labels employed by $\mathcal{L}^{Anch}_{Evt}$. Table 2 demonstrates
that standard contrastive learning is ineffective for the AVVP task. While
$\mathcal{L}_{BCE}$ yields average improvement, its effectiveness is not
consistent across all event types, i.e., it deteriorates performance in
detecting the visible-only events in both segment and event levels. Our
contrastive loss tailored for this task increases the method’s performance
across all event types (see blue numbers in Table 2). Specifically, it leads
to a significant improvement (over 2% F-score) in detecting audible-only
events in both segment and event levels.

Impact of Self-Modality-Aware Knowldge –  In Table 2, we ablate our proposed
self-modality-aware knowledge distillation loss $\mathcal{L}^{Anch}_{SelfMo}$.
The results demonstrate that the additional free modality-aware knowledge
obtained by Anchor for Reference during training eventually enhances the
performance of Anchor itself across all event types, specifically resulting in
over a 1% improvement in F-score for detecting unaligned audible-visible event
types.

Impact of Learning Co-occurrence Class Relationships –  In CoLeaF, the class
tokens and $\mathcal{{L}}^{Anch}_{CoCls}$ are introduced to explicitly model
and transfer the cross-class relationships from Reference to Anchor. In Table
4, we conduct ablations on these two components. The results demonstrate that
leveraging both components increases the method’s performance over a 1%
F-score across all event types (see blue numbers in Table 4). Additionally,
the results reveal that (i) the network can benefit from
$\mathcal{{L}}^{Anch}_{CoCls}$ with or without the use of the class tokens and
(i) adding the class tokens does not improve the performance on average unless
the network leverages $\mathcal{{L}}^{Anch}_{CoCls}$. In these experiments, we
also observed that using the class tokens increased the computational
complexity of Reference by 41.7% Flops (i.e., from 12.7 to 18.0 GFlops) for 25
classes of LLP. However, our novel design in CoLeaF allows us to benefit from
the knowledge learned from them without suffering from their computational
overhead in inference. Notably, this novel design can benefit other video
understanding tasks where exploiting class dependencies is crucial.

captionskip=4pt [2]  Loss Segment-level Event-level Ao Vo AV Ao Vo AV
$\mathcal{L}_{NCE}$ 46.1 60.8 56.3 40.5 60.6 50.1 $\mathcal{L}_{BCE}$ 48.5
61.2 57.4 43.5 61.1 51.2 - 47.3 61.8 56.7 41.6 61.7 50.3
$\mathcal{L}^{Anch}_{Evt}$ 49.3 62.4 58.6 44.1 62.2 52.1 (+2.1) (+0.6) (+0.9)
(+2.5) (+0.5) (+0.8)

$\mathcal{{L}}^{Anch}_{SelfMo}$ | Segment-level |  | Event-level
---|---|---|---
Ao | Vo | AV |  | Ao | Vo | AV
✗ | 48.1 | 60.9 | 57.9 |  | 42.8 | 60.6 | 51.4
✓ | 49.3 | 62.4 | 58.6 |  | 44.1 | 62.2 | 52.1
| (+1.2) | (+1.5) | (+0.7) |  | (+1.3) | (+1.6) | (+0.7)

Table 1: Ablation studies on $\mathcal{L}^{Anch}_{Evt}$. Table 2: Ablation
studies on $\mathcal{L}^{Ref}_{SelfMo}$.

captionskip=4pt [2]  Tkn $\mathcal{{L}}^{Anch}_{CoCls}$ Segment-level Event-
level Ao Vo AV Ao Vo AV ✓ ✗ 47.5 60.3 57.0 41.9 60.2 50.4 ✗ ✓ 48.2 62.7 57.3
42.5 62.5 52.0 ✗ ✗ 47.8 60.8 56.6 42.2 60.6 49.8 ✓ ✓ 49.3 62.4 58.6 44.1 62.2
52.1 (+1.1) (+1.6) (+2.0) (+1.9) (+1.6) (+2.3)

| Segment-level |  | Event-level
---|---|---|---
| Ao | Vo | AV |  | Ao | Vo | AV
HAN [18] | 33.1 | 50.7 | 48.9 |  | 31.0 | 50.1 | 43.0
HAN$\circledast$ | 36.3 | 54.1 | 49.4 |  | 34.2 | 53.5 | 43.7
| (+3.2) | (+3.4) | (+0.5) |  | (+3.2) | (+3.4) | (+0.7)
CMPAE [3] | 48.2 | 57.9 | 57.5 |  | 43.6 | 57.5 | 49.6
CMPAE$\circledast$ | 49.3 | 62.4 | 58.6 |  | 44.1 | 62.2 | 52.1
| (+1.1) | (+4.5) | (+1.1) |  | (+0.5) | (+4.7) | (+2.5)

Table 3: Ablation studies on class tokens (Tkn) and
$\mathcal{{L}}^{Anch}_{CoCls}$. Table 4: Framework generalization on LLP.
$\circledast$ indicates that the network is embedded in CoLeaF as the Anchor
branch.

Framework Generalization – CoLeaF is a general learning framework, which means
that any existing AVVP method can be embedded in our framework to (i)
explicitly optimize the integration of cross-modal information and (ii)
benefit from modeling co-occurrence class relationships in an end-to-end
manner. In Table 4, we present two examples: HAN [17], a common architecture
for AVVP, and CMPAE [3]. The results demonstrate that our proposed framework
significantly enhances the performance of HAN and CMPAE, i.e., on average, the
F-score is increased by 2.4% in both networks. This improvement is achieved in
an end-to-end manner without requiring additional annotations or altering
their network architecture.

###  5.2 State-of-the-Art Comparison

In this section, we compare the performance of CoLeaF with current state-of-
the-art approaches using both traditional and our proposed metrics. The
results of the previous methods under the new metrics were reproduced using
their available code whenever accessible. Note that to obtain the event
proposals from the estimated output probabilities, CMPAE [3] utilizes a set of
‘selected thresholds’ for different event categories of LLP, while in the
standard setting used by other state-of-the-art approaches, a fixed threshold
is applied to all event classes. For a fair comparison, we also present the
CMPAE’s results under the standard setting.

Table 5 presents comparative results on the LLP dataset when only video-level
labels are used for training. The results demonstrate that our proposed
approach CoLeaF achieves new state-of-the-art results on all new metrics, as
well as AV under both standard and ‘selected thresholds’ settings. CoLeaF
exhibits over 1.1% F-score improvement on most new metrics. Specifically, it
significantly increases Vo and Event@Avo by 3.6% and 2.0% F-score at the
segment level, and Vo, AV, and Type@Avo by 2.2%, 2.7%, and 2.4% F-score at the
event level (under the standard setting), and Type@Avo by 2.1% F-score at the
event level (under the ‘selected thresholds’ setting). On average across new
metrics and AV, our proposed method improves the state-of-the-art results by
1.9% and 1.2% F-score under standard and selected threshold settings,
respectively.

Analysis of Metrics – In Section 4, we stated that the traditional A and V
metrics do not adequately reflect the method’s ability to detect audible-only
and visible-only events, respectively. As in the A and V metrics, apart from
counting audible-only and visible-only events, respectively, they mistakenly
include audible-visible events as true positive. This can have a detrimental
impact on the analysis of the method’s performance. For example, using the
traditional A metric, HAN and MA [21] appear to show nearly identical
performance in detecting audible-only events at 60.1% and 60.3% F-score (at
segment-level), respectively. However, when we evaluate them using our
proposed Ao metric that counts correctly audible-only events, we observe that
MA outperforms HAN with a margin of 10.4%. Similarly, based on the V metric
(at event-level), CMPAE [3] outperforms JoMoLD [2] in detecting visible-only
events, while our proposed Vo metric reveals that JoMoLD achieves a better
performance than CMPAE. Consequently, the traditional Type@AV and Event@AV
have similar issues.

Table 5: Weakly supervised AVVP results on the LLP dataset when utilizing
video-level labels for training. All methods use VGGish, ResNet152, and
R(2+1)D to generate the input tokens. ‘T@’ and ‘E@’ refers ‘Type@AV’ and
‘Event@AV’, respectively. ${\dagger}$ indicates the results are under
‘selected thresholds’ setting. The best and the second-best results are in
Bold and underlined, respectively.

Method |  | Segment-level (%) |  | Event-level (%)
---|---|---|---|---
| A | Ao | V | Vo | AV | T@ | T@o | E@ | E@o |  | A | Ao | V | Vo | AV | T@ | T@o | E@ | E@o
AVE [18] |  | 47.2 | – | 37.1 | – | 35.4 | 39.9 |  – | 41.6 | – |  | 40.4 | – | 34.7 | – | 31.6 | 35.5 |  – | 36.5 | –
AVSDN [11] |  | 47.8 | – | 52.0 | – | 37.1 | 45.7 | – | 50.8 | – |  | 34.1 | – | 46.3 | – | 26.5 | 35.6 | – | 37.7 | –
CVCMS [12] |  | 59.2 | – | 59.9 | – | 53.4 | 57.5 | – | 58.1 | – |  | 51.3 | – | 55.5 | – | 46.2 | 51.0 | – | 49.7 | –
MM-Pyr [24] |  | 60.9 | – | 54.4 | – | 50.0 | 55.1 | – | 57.6 | – |  | 52.7 | – | 51.8 | – | 44.4 | 49.9 | – | 50.5 | –
DHHN [8] |  | 61.3 | – | 58.3 | – | 52.9 | 57.5 | – | 58.1 | – |  | 54.0 | – | 55.1 | – | 47.3 | 51.5 | – | 51.5 | –
MGN [13] |  | 60.8 | – | 55.4 | – | 50.0 | 55.5 | – | 57.2 | – |  | 51.1 | – | 52.4 | – | 44.4 | 49.3 | – | 49.1 | –
HAN [17] |  | 60.1 | 33.1 | 52.9 | 50.7 | 48.9 | 54.0 | 44.3 | 55.4 | 24.1 |  | 51.3 | 31.0 | 48.9 | 50.1 | 43.0 | 47.7 | 41.4 | 48.0 | 22.0
MA [21] |  | 60.3 | 43.5 | 60.0 | 53.6 | 55.1 | 58.9 | 50.8 | 57.9 | 32.0 |  | 53.6 | 38.8 | 56.4 | 52.4 | 49.0 | 53.0 | 46.8 | 50.6 | 27.8
JoMoLD [2] |  | 61.3 | 46.2 | 63.8 | 58.8 | 57.2 | 60.8 | 54.4 | 59.9 | 35.4 |  | 53.9 | 40.9 | 59.9 | 59.0 | 49.6 | 54.5 | 50.3 | 52.5 | 31.1
CMPAE [3] |  | 63.9 | 48.2 | 63.3 | 57.9 | 57.5 | 61.2 | 54.2 | 62.1 | 35.7 |  | 57.1 | 43.6 | 60.6 | 57.5 | 49.6 | 55.8 | 50.3 | 55.6 | 32.0
CoLeaF |  | 63.5 | 49.3 | 64.5 | 62.4 | 58.6 | 62.1 | 56.7 | 61.8 | 37.7 |  | 56.2 | 44.1 | 62.0 | 62.2 | 52.1 | 56.7 | 52.7 | 54.5 | 33.4
|  |  | (+1.1) |  | (+3.6) | (+1.1) |  | (+1.4) |  | (+2.0) |  |  | (+0.5) |  | (+2.2) | (+2.7) |  | (+2.4) |  | (+1.4)
CMPAE${\dagger}$ [3] |  | 64.2 | 48.9 | 66.4 | 61.7 | 59.2 | 63.3 | 55.5 | 62.8 | 37.6 |  | 56.6 | 43.8 | 63.7 | 61.1 | 51.8 | 51.9 | 50.3 | 55.7 | 34.0
CoLeaF${\dagger}$ |  | 64.2 | 49.6 | 67.1 | 63.1 | 59.8 | 63.8 | 57.0 | 61.9 | 38.4 |  | 57.1 | 45.0 | 64.8 | 62.7 | 52.8 | 58.2 | 52.4 | 55.5 | 34.6
|  |  | (+0.7) |  | (+1.4) | (+0.6) |  | (+1.5) |  | (+0.8) |  |  | (+1.2) |  | (+1.6) | (+1.0) |  | (+2.1) |  | (+0.6)

Table 6: Weakly supervised AVVP results on the LLP dataset when utilizing
video-level labels and CLIP and CLAP annotations for training. All methods use
VGGish, ResNet152, and R(2+1)D to generate the input tokens (see Table 5 for
more definitions and terms).

Method |  | Segment-level (%) |  | Event-level (%)
---|---|---|---|---
| A | Ao | V | Vo | AV | T@ | T@o | E@ | E@o |  | A | Ao | V | Vo | AV | T@ | T@o | E@ | E@o
JoMoLD [2] |  | 61.5 | 46.3 | 64.7 | 64.8 | 58.3 | 61.5 | 56.5 | 60.3 | 38.0 |  | 54.7 | 41.5 | 61.5 | 63.5 | 52.4 | 56.2 | 52.5 | 53.1 | 33.6
CMPAE [3] |  | 64.1 | 48.5 | 64.3 | 66.9 | 58.9 | 62.4 | 58.1 | 62.0 | 40.0 |  | 57.0 | 42.7 | 62.1 | 66.0 | 53.5 | 57.6 | 54.1 | 55.0 | 35.1
VALOR [9] |  | 61.8 | 49.0 | 65.9 | 66.7 | 58.4 | 62.0 | 57.9 | 61.5 | 40.8 |  | 55.4 | 44.2 | 62.6 | 65.0 | 52.2 | 56.7 | 53.8 | 54.2 | 35.8
CoLeaF |  | 64.2 | 51.2 | 64.4 | 67.7 | 59.3 | 62.6 | 59.4 | 62.5 | 41.3 |  | 57.6 | 45.5 | 63.2 | 67.0 | 54.2 | 57.9 | 55.6 | 55.6 | 36.4
|  |  | (+2.2) |  | (+0.8) | (+0.4) |  | (+1.3) |  | (+0.5) |  |  | (+1.3) |  | (+1.0) | (+0.7) |  | (+1.5) |  | (+0.6)

Qualitative Results – In Fig. 5, we qualitatively compare CoLeaF with the
state-of-the-art JoMoLD [2] and CMPAE [3] on a test video sample of LLP. For
the ‘Speech’ event, while JoMoLD and CMPAE cannot detect it, CoLeaF recognizes
it correctly as audible-only. While all methods detect the audible-visible
event ‘Singing’, CoLeaF’s event predictions have a better overlap with the
ground-truth annotations.

Figure 4: Qualitative comparison to previous AVVP approaches (JoMoLD [2] and
CMPAE [3]) training with weak labels.

Method | AV (Seg) | AV (Evn)
---|---|---
HAN [17] | 35.0 | 41.4
MA [21] | 37.9 | 44.8
JoMoLD [2] | 36.4 | 41.2
CMPAE [3] | 39.7 | 43.8
CoLeaF |  41.5 (+1.8) |  47.8 (+3.0)

Figure 5: Weakly supervised DAVE results on the UnAV-100 dataset. All methods
use pre-trained VGGish [7] and two-stream I3D [1] to generate the input
tokens. The best and the second-best results are in Bold and underlined,
respectively.

CLIP and CLAP Annotations – Table 5 addresses the weakly supervised AVVP task
without the need for additional annotations. The authors in [9] generate
offline dense modality-specific annotations for the LLP dataset with large-
scale pretrained models, CLIP [15], and CLAP [22]. In Table 6, we present
comparative results with recent state-of-the-art approaches in which the
networks are additionally trained on these annotations. The results of JoMoLD
[2] and CMPAE [3] were obtained by adapting their available code. The results
demonstrate that in this scenario with the modality-specific pseudo labels,
CoLeaF still improves the state-of-the-art approaches consistently across all
new metrics and AV F-score.

###  5.3 Generalization on the Large-Scale UnAV-100 Dataset

To demonstrate the superior generalization of our proposed framework on a
different, yet challenging dataset, we present its results to the large-scale
UnAV-100 dataset, a benchmark for the Dense Audio-Visual Event Localization
(DAVE) task [4]. UnAV-100 includes videos with multiple events and encompasses
a diverse range of 100 distinct audio-visible action types, four times more
than the LLP dataset. Furthermore, UnAV-100 features untrimmed videos ranging
from 0.2 to 60 seconds, significantly increasing the dataset’s complexity. To
obtain weakly supervised results, we derived video-level labels from the
videos’ temporal annotations (see supp. file for the implementation details
for this task). The results of other state-of-the-art approaches were provided
through their available code. Comparative results on UnAV-100 using AV F-score
are available in Table 5. It is shown that CoLeaF outperforms the other
approaches by 1.8% and 3.0% AV F-score at the segment and event levels,
respectively, and achieves state-of-the-art results on this dataset.

##  6 Conclusion

We introduce CoLeaF, a novel learning framework designed to: (i) explicitly
optimize integrating cross-modal contexts in the embedding space for the
weakly supervised AVVP task; and (ii) benefit from explicitly modelling cross-
class relationships during training without adding computational overhead at
inference. Having introduced novel metrics to better understand an AVVP
method’s capability in detecting audible-only and visible-only events, we
evaluate CoLeaF on LLP, the only benchmark dataset for the weakly supervised
AVVP task, and UnAV-100, a challenging benchmark dataset for the weakly
supervised DAVE task. Our method achieves new state-of-the-art results on both
datasets. Ablation studies demonstrate the effectiveness of CoLeaF’s key
components. Future work will adapt our end-to-end framework to benefit from
language models.

## Acknowledgement

This research is supported by UKRI EPSRC Platform Grant EP/P022529/1, and
EPSRC BBC Prosperity Partnership AI4ME: Future Personalised Object-Based Media
Experiences Delivered at Scale Anywhere EP/V038087/1.

## References

  * [1] Carreira, J., Zisserman, A.: Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6299–6308 (2017)
  * [2] Cheng, H., Liu, Z., Zhou, H., Qian, C., Wu, W., Wang, L.: Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing. In: European Conference on Computer Vision. pp. 431–448. Springer (2022)
  * [3] Gao, J., Chen, M., Xu, C.: Collecting Cross-Modal Presence-Absence Evidence for Weakly-Supervised Audio-Visual Event Perception. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18827–18836 (2023)
  * [4] Geng, T., Wang, T., Duan, J., Cong, R., Zheng, F.: Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22942–22951 (2023)
  * [5] Gutmann, M., Hyvärinen, A.: Noise-Contrastive Estimation: A New Estimation Principle for Unnormalized Statistical Models. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. pp. 297–304. JMLR Workshop and Conference Proceedings (2010)
  * [6] He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [7] Hershey, S., Chaudhuri, S., Ellis, D.P., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B., et al.: CNN Architectures for Large-Scale Audio Classification. In: IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 131–135. IEEE (2017)
  * [8] Jiang, X., Xu, X., Chen, Z., Zhang, J., Song, J., Shen, F., Lu, H., Shen, H.T.: DHHN: Dual Hierarchical Hybrid Network for Weakly-Supervised Audio-Visual Video Parsing. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 719–727 (2022)
  * [9] Lai, Y.H., Chen, Y.C., Wang, Y.C.F.: Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser. Advances in Neural Information Processing systems (2023)
  * [10] Li, G., Wei, Y., Tian, Y., Xu, C., Wen, J.R., Hu, D.: Learning to Answer Questions in Dynamic Audio-Visual Scenarios. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19108–19118 (2022)
  * [11] Lin, Y.B., Li, Y.J., Wang, Y.C.F.: Dual-Modality Seq2Seq Network for Audio-Visual Event Localization. In: IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 2002–2006. IEEE (2019)
  * [12] Lin, Y.B., Tseng, H.Y., Lee, H.Y., Lin, Y.Y., Yang, M.H.: Exploring Cross-Video and Cross-Modality Signals for Weakly-Supervised Audio-Visual Video Parsing. Advances in Neural Information Processing Systems 34, 11449–11461 (2021)
  * [13] Mo, S., Tian, Y.: Multi-Modal Grouping Network for Weakly-Supervised Audio-Visual Video Parsing. Advances in Neural Information Processing Systems 35, 34722–34733 (2022)
  * [14] Nadeem, A., Hilton, A., Dawes, R., Thomas, G., Mustafa, A.: CAD–Contextual Multi-modal Alignment for Dynamic AVQA. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2023)
  * [15] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Quantify Classificationl Models from Natural Language Supervision. In: International Conference on Machine Learning. pp. 8748–8763 (2021)
  * [16] Sensoy, M., Kaplan, L., Kandemir, M.: Evidential Deep Learning to Quantify Classification Uncertainty. Advances in Neural Information Processing Systems 31 (2018)
  * [17] Tian, Y., Li, D., Xu, C.: Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing. In: European Conference in Computer Vision. pp. 436–454. Springer (2020)
  * [18] Tian, Y., Shi, J., Li, B., Duan, Z., Xu, C.: Audio-Visual Event Localization In Unconstrained Videos. In: European Conference on Computer Vision. pp. 247–263 (2018)
  * [19] Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A Closer Look at Spatiotemporal Convolutions for Action Recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6450–6459 (2018)
  * [20] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention Is All You Need. Advances in Neural Information Processing systems 30 (2017)
  * [21] Wu, Y., Yang, Y.: Exploring Heterogeneous Clues for Weakly-Supervised Audio-Visual Video Parsing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1326–1335 (2021)
  * [22] Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., Dubnov, S.: Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. In: IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 1–5 (2023)
  * [23] Xia, Y., Zhao, Z.: Cross-Modal Background Suppression for Audio-Visual Event Localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19989–19998 (June 2022)
  * [24] Yu, J., Cheng, Y., Zhao, R.W., Feng, R., Zhang, Y.: Mm-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsing. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 6241–6249 (2022)
  * [25] Yun, H., Yu, Y., Yang, W., Lee, K., Kim, G.: Pano-AVQA: Grounded Audio-Visual Question Answering on 360deg Videos. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2031–2041 (October 2021)
  * [26] Zhou, J., Guo, D., Zhong, Y., Wang, M.: Improving Audio-Visual Video Parsing with Pseudo Visual Labels. arXiv preprint arXiv:2303.02344 (2023)


# On Geodesic Distances and Contextual Embedding Compression for Text
Classification

Rishi Jha Equal contribution Kai Mihata11footnotemark: 1
Paul G. Allen School of Computer Science & Engineering,
University of Washington, Seattle, WA, USA
{rjha01<EMAIL_ADDRESS>
###### Abstract

In some memory-constrained settings like IoT devices and over-the-network data
pipelines, it can be advantageous to have smaller contextual embeddings. We
investigate the efficacy of projecting contextual embedding data (BERT) onto a
manifold, and using nonlinear dimensionality reduction techniques to compress
these embeddings. In particular, we propose a novel post-processing approach,
applying a combination of Isomap and PCA. We find that the geodesic distance
estimations, estimates of the shortest path on a Riemannian manifold, from
Isomap’s k-Nearest Neighbors graph bolstered the performance of the compressed
embeddings to be comparable to the original BERT embeddings. On one dataset,
we find that despite a 12-fold dimensionality reduction, the compressed
embeddings performed within 0.1% of the original BERT embeddings on a
downstream classification task. In addition, we find that this approach works
particularly well on tasks reliant on syntactic data, when compared with
linear dimensionality reduction. These results show promise for a novel
geometric approach to achieve lower dimensional text embeddings from existing
transformers and pave the way for data-specific and application-specific
embedding compressions.

Figure 1: Visualization of two-dimensional PCA and Isomap compressions based
on BERT embeddings for the SMS-SPAM dataset (Almeida et al., 2013). Spam is
represented by a blue dot and ham by an orange x. We see that for this dataset
in two dimensions, the Isomap compression appears more linearly separable than
the PCA compression, making classification easier for the former.

##  1 Introduction

Contextual embeddings, like those BERT (Devlin et al., 2019) generates,
improve on non-contextual word embeddings by providing contextual semantics to
the real-valued representation of a text. Although these models have been
shown to achieve state-of-the-art performance on most NLP tasks, they are
notably expensive to train. To help combat this, as mentioned by May et al.
(2019), model compression techniques like data quantization (Gong et al.,
2014), model pruning (Han et al., 2016), and knowledge distillation (Sanh et
al., 2019, Hinton et al., 2015) have been developed. However, at 768
dimensions, the embeddings themselves can be prohibitively large for some
tasks and settings.

Smaller embeddings both enable more compact data sizes in storage-constrained
settings and over-the-air data pipelines, and help lower the requisite memory
for using the embeddings for downstream tasks. For non-contextual word
embeddings, Ling et al. (2016) note that loading matrices can take multiple
gigabytes of memory, a prohibitively large amount for some phones and IoT
devices. While contextual embeddings are smaller, downstream models will face
similar headwind for large corpora.

Although there has been more extensive study in the efficacy of compressing
non-contextual word embeddings (Raunak et al., 2019, Mu and Viswanath, 2018),
to the best of our knowledge few contextual embedding compression post-
processing approaches have been proposed (Li and Eisner, 2019). In their work,
Li and Eisner (2019) propose the Variational Information Bottleneck, an
autoencoder to create smaller, task specific embeddings for different
languages. While effective, the computational expense of additional training
loops is not appropriate for some memory constrained applications.

Our approach more closely mirrors the work of Raunak et al. (2019) who propose
a Principal Component Analysis (PCA)-based post-processing algorithm to lower
the dimensionality of non-contextual word embeddings. They find that they can
replicate, or, in some cases, increase the performance of the original
embeddings. One limitation to this approach is the lack of support for
nonlinear data patterns. Nonlinear dimensionality reductions, like the Isomap
shown in Figure 1, can pick up on latent textual features that evade linear
algorithms like PCA. To achieve this nonlinearity, we extend this approach to
contextual embeddings, adding in additional geodesic distance information via
the Isomap algorithm (Tenenbaum et al., 2000). To the best of our knowledge,
the application of graph-based techniques to reduce the dimensionality of
contextual embeddings is novel.

The goal of this paper is not to compete with state-of-the-art models, but,
rather, (1) to show that 12-fold dimensionality reductions of contextual
embeddings can, in some settings, conserve much of the original performance,
(2) to illustrate the efficacy of geodesic similarity metrics in improving the
downstream performance of contextual embedding compressions, and (3) propose
the creation of more efficient, geodesic-distance-based transformer
architectures. In particular, our main result is showing that a 64-dimensional
concatenation of compressed PCA and Isomap embeddings are comparable to the
original BERT embeddings and outperform our PCA baseline. We attribute this
success to the locality data preserved by the k-Nearest Neighbors (k-NN) graph
generated by the Isomap algorithm.

Figure 2: PCA baseline performance on CoLA (Warstadt et al., 2019) and SST-2
(Socher et al., 2013). PCA embedding performance by dimension is represented
by the solid blue line. Regression at 768 dimensions is represented by an
orange dashed line. On these two datasets, even at much smaller
dimensionality, we see that PCA has comparable performance.

##  2 Related Work

As best we know, there is very little literature regarding the intersection of
contextual embedding compression and geodesic distances. Most of the existing
work in related spaces deals with non-contextual word embeddings. Despite the
rapid growth in the popularity of transformers, these embeddings still retain
popularity.

For non-contextual word embeddings, Mu and Viswanath (2018) propose a post-
processing algorithm that projects embedded data away from the dominant
principal components, in order to greater differentiate the data. Raunak et
al. (2019) expand on this algorithm by combining it with PCA reductions. Both
approaches are effective, but, are limited to linear dimensionality
reductions.

Some nonlinear approaches include Andrews (2016) and Li and Eisner (2019) who
both use autoencoder-based compressions. Notably, the former only addresses
non-contextual embeddings.

Meanwhile the usage of graphs in NLP is well established, but their usage in
the compression of contextual embeddings is not well documented. Wiedemann et
al. (2019) use a k-NN classification to achieve state-of-the-art word sense
disambiguation. Their work makes clear the effectiveness of the k-NN approach
in finding distinctions in hyper-localized data.

##  3 Method

With the goal of reducing the contextual embedding dimensionality, we first
processed our data using a pre-trained, uncased BERT Base model. Then, we
compressed the data to a lower dimension using both PCA and Isomap as
described in Section 3.2. This method aims to capture as much information as
possible from the original BERT embeddings while preserving graphical locality
information and nonlinearities in the final contextual embeddings.

###  3.1 Isomap and Geodesic Distances

For this paper, to blend geodesic distance information and dimensionality
reduction, we use Tenenbaum’s Isomap (Tenenbaum et al., 2000). Isomap relies
on a weighted neighborhood graph that allows for the inclusion of complex,
nonlinear patterns in the data, unlike a linear algorithm like PCA. In
specific, this graph is constructed so that the edges between each vertex
(datapoint) and its k-nearest neighbors have weight corresponding to the
pairwise Euclidean distance on a Riemannian manifold. Dijkstra’s shortest path
between two points then estimates their true geodesic distance. These
geodesics are particularly useful for delineating points that are close in
Euclidean space but not on a manifold i.e. similar BERT embeddings with
different meanings.

If we assume the data follows a manifold, Isomap can exploit the Riemannian
locality of these complex contextual embeddings. As Figure 1 shows, in some
cases this is a good assumption to make since we are then able to dissect
complex embeddings into near-linearly separable clusters. Notably, there are
some limitations to this approach. If the manifold is sparse, i.e. there are
few data points on certain regions of the manifold, or k is too small, the
shortest path estimation of the geodesic distance can be unrepresentative of
the true distance. On the contrary, if k is too large, Isomap overgeneralizes
and loses its fine-grained estimation of the Riemannian surface.

Nonetheless, we hypothesize that these global geodesic distance approximations
explain the empirical advantage Isomap has in our setting over other popular
nonlinear dimensionality reduction techniques. Many alternatives, like Locally
Linear Embeddings (Roweis and Saul, 2000) focus, instead, on preserving intra-
neighborhood distance information that may not encompass inter-neighborhood
relationships as Isomap does.

###  3.2 Our Approach

We applied our post-processing method to the BERT embeddings through three
different dimensionality reductions. We used (1) PCA, (2) Isomap, and (3) a
concatenation of embeddings from the two before training a small regression
model on the embeddings. This approach aims to use linear and nonlinear
dimensionality reduction techniques to best capture the data’s geodesic
locality information.

#### PCA.

To compute linearly-reduced dimensionality embeddings, we used PCA to reduce
the 768-dimensional BERT embeddings down to a number of components ranging
from 16 to 256. While there are other linear dimensionality reduction
techniques, PCA is a standard benchmark and empirically performed the best.
These serve as a linear baseline for reduced dimension embeddings.

#### Isomap.

To compute geodesic locality information, we post-processed our BERT
embeddings with Isomap. The final Isomap embeddings ranged from 16 to 96
dimensions, all computed with 96 neighbors and Euclidean distance.

#### Concatenated Embeddings.

To include features from both of these reductions, we combined an Isomap
embedding with a PCA embedding to form concatenations of several dimensions.
We experimented with ratios of PCA embedding size to Isomap embedding size
from 0 to $\frac{1}{2}$ at $\frac{1}{8}$ intervals. We found that this ratio
was the main determinant of relative accuracy, so for analysis we fixed the
total dimension to 64.

Embedding | Isomap Dim | PCA Dim | CoLA | SST-2
---|---|---|---|---
PCA | N/A | 64 | 0.339 | 0.842
Concatenation* | 16 | 48 | 0.421 | 0.846
Concatenation | 32 | 32 | 0.384 | 0.822
Concatenation | 48 | 16 | 0.357 | 0.817
Isomap | 64 | N/A | 0.332 | 0.814
BERT | N/A | N/A | 0.455 | 0.847
Table 1: 64-dimensional embedding performance on CoLA (Warstadt et al., 2019)
and SST-2 (Socher et al., 2013). CoLA is measured by Matthews correlation and
SST-2 by accuracy. While Isomap did not perform the best outright, on these
datasets we found that some inclusion of locality data proved meaningful. This
shows the trade-off between locality information and performance mentioned in
Section 4.4. The best 12-fold compression performance is asterisked.

##  4 Experiments and Results

We assess the results of these compression techniques on two text
classification datasets. We provide the code for our
experiments111https://github.com/kaimihata/geo-bert.

###  4.1 Data

We evaluate our method on two text classification tasks: the Corpus of
Linguistic Acceptability (CoLA) (Warstadt et al., 2019), a 10657 sentence
binary classification dataset on grammatical correctness and the Stanford
Sentiment Treebank v2 (SST-2) (Socher et al., 2013), a 70042 sentence binary
(positive / negative) sentiment classification dataset.

For CoLA, we used the predefined, 9594 datapoint train set and for SST-2, we
used the first 8000 samples of their training set to construct ours due to
computational limitations. For testing and evaluation, we used the
corresponding datasets defined by GLUE (Wang et al., 2018). In addition, for
all of our evaluations, we used the same pre-trained BERT embeddings for
consistency.

###  4.2 Training and Evaluation

All of these post-processed embeddings, as well as the BERT embeddings, were
trained on a downstream regression model consisting of one hidden layer (64
dim) with ReLU activation, a learning rate of $0.0001$, and were optimized via
ADAM (Kingma and Ba, 2015). The BERT embeddings are used as a baseline for
comparison.

To evaluate our embeddings on CoLA and SST-2, we used their GLUE-defined
metrics of Matthews correlation and validation accuracy, respectively. For
each embedding experiment, our procedure consisted of running our post-
processing method on the BERT embeddings then training the downstream model.
Each reported metric is the average of three of these procedures.

###  4.3 Baseline Comparison

Agnostic of post-processing algorithm, we found reduced-dimensionality
embeddings were competitive with the original embeddings. Although smaller
reduction factors, understandably, performed better, we found that even when
reduced by a factor as large as 12, our PCA embeddings experienced small
losses in performance on both datasets (Figure 2). To demonstrate the effect
of the inclusion of locality data, we picked an embedding size of 64
dimensions (a reduction factor of 12) to balance embedding size and
performance for our main experiment.

In comparison to our 768-dimensional baseline, at 64 dimensions, the best
reduction results were within 7.5% and 0.1% for CoLA and SST-2, respectively
(Table 1). These results show that with or without the presence of locality
data, compressed embeddings can perform comparably to the original embeddings.

###  4.4 Locality Information Trade-off

As shown in Table 1, on neither dataset did the fully PCA or Isomap embeddings
perform the best. The best performer was, instead, a combination of these two
approaches. This indicates that there must exist a trade-off on the
effectiveness of locality data. While without locality data, the embedding
obviously misses out on geodesic relationships, too much locality information
may replace more useful features that the PCA embeddings extract. Just as the
quality of the geodesic distance estimations rely on how well the data fits
the underlying manifold, as discussed in Section 3, so, too, does its
effectiveness. To explain this phenomenon, we hypothesize that the addition of
small amounts of locality data bolsters performance by describing the geodesic
relationships without drowning out important syntactic and semantic
information provided by PCA.

###  4.5 Task-Specific Locality

While the best reduction consisted of a concatenation of 16-dimensional Isomap
and 48-dimensional PCA embeddings, whether the other concatenations performed
better than our PCA baseline was dependent on the task. For CoLA, we found
that all three concatenated embeddings performed better than PCA, whereas for
SST-2, only the top performing concatenated embedding beat out our baseline.
To describe this disparity we look towards the nature of the datasets and
tasks. Notably, CoLA requires models to identify proper grammar, a syntactic
task, while SST-2 requires models to understand the sentiment of sentences, a
semantic task. Syntactic data often has some intrinsic structure to it, and
perhaps our manifold approach encompasses this information well. Based on this
result, exploring this distinction could be an exciting avenue for further
study.

##  5 Conclusions and Future Work

We present a novel approach for compressing BERT embeddings into effective
lower dimension representations. Our method shows promise for the inclusion of
geodesic locality information in transformers and future compression methods.
We hope our results lead to more work investigating the geometric structure of
transformer embeddings and developing more computationally efficient NLP
training pipelines. To further this work, we plan to investigate the efficacy
of (1) other graph dimensionality reduction techniques, (2) non-Euclidean
distance metrics, and (3) our approach on different transformers. In addition,
we would like to investigate whether datasets for other tasks can be
effectively projected onto a manifold.

## Acknowledgments

We would like to thank Raunak Kumar, Rohan Jha, and our three reviewers for
their thoughtful and thorough comments on improving our paper. In addition, we
would like to thank our deep learning professor Joseph Redmon for inspiring
this project.

## References

  * Almeida et al. (2013) Tiago Almeida, Jose Gomez Hidalgo, and Tiago Silva. 2013.  Towards sms spam filtering: Results under a new dataset.  _International Journal of Information Security Science_ , 2:1–18.
  * Andrews (2016) Martin Andrews. 2016.  Compressing word embeddings.  In _Neural Information Processing_ , pages 413–422, Cham. Springer International Publishing.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gong et al. (2014) Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. 2014.  Compressing Deep Convolutional Networks using Vector Quantization.  _arXiv e-prints_ , page arXiv:1412.6115.
  * Han et al. (2016) Song Han, Huizi Mao, and William J Dally. 2016.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  _International Conference on Learning Representations (ICLR)_.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015.  Distilling the knowledge in a neural network.  In _NIPS Deep Learning and Representation Learning Workshop_.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Li and Eisner (2019) Xiang Lisa Li and Jason Eisner. 2019.  Specializing word embeddings (for parsing) by information bottleneck.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2744–2754, Hong Kong, China. Association for Computational Linguistics.
  * Ling et al. (2016) Shaoshi Ling, Yangqiu Song, and Dan Roth. 2016.  Word embeddings with limited memory.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 387–392, Berlin, Germany. Association for Computational Linguistics.
  * May et al. (2019) Avner May, Jian Zhang, Tri Dao, and Christopher Ré. 2019.  On the Downstream Performance of Compressed Word Embeddings.  _arXiv e-prints_ , page arXiv:1909.01264.
  * Mu and Viswanath (2018) Jiaqi Mu and Pramod Viswanath. 2018.  All-but-the-top: Simple and effective postprocessing for word representations.  In _International Conference on Learning Representations_.
  * Raunak et al. (2019) Vikas Raunak, Vivek Gupta, and Florian Metze. 2019.  Effective dimensionality reduction for word embeddings.  In _Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)_ , pages 235–243, Florence, Italy. Association for Computational Linguistics.
  * Roweis and Saul (2000) Sam T. Roweis and Lawrence K. Saul. 2000.  Nonlinear Dimensionality Reduction by Locally Linear Embedding.  _Science_ , 290(5500):2323–2326.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.01108.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Tenenbaum et al. (2000) Joshua B. Tenenbaum, Vin de Silva, and John C. Langford. 2000.  A global geometric framework for nonlinear dimensionality reduction.  _Science_ , 290(5500):2319–2323.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Wiedemann et al. (2019) Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019.  Does bert make any sense? interpretable word sense disambiguation with contextualized embeddings.  In _Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers_ , pages 161–170, Erlangen, Germany. German Society for Computational Linguistics & Language Technology.