# A-JEPA: Joint-Embedding Predictive Architecture Can Listen

Zhengcong Fei, Mingyuan Fan, Junshi Huang Kunlun Inc<EMAIL_ADDRESS>The corresponding author.

###### Abstract

This paper presents that the masked-modeling principle driving the success of
large foundational vision models can be effectively applied to audio by making
predictions in a latent space. We introduce Audio-based Joint-Embedding
Predictive Architecture (A-JEPA), a simple extension method for self-
supervised learning from the audio spectrum. Following the design of I-JEPA,
our A-JEPA encodes visible audio spectrogram patches with a curriculum masking
strategy via context encoder, and predicts the representations of regions
sampled at well-designed locations. The target representations of those
regions are extracted by the exponential moving average of context encoder,
_i.e._ , target encoder, on the whole spectrogram. We find it beneficial to
transfer random block masking into time-frequency aware masking in a
curriculum manner, considering the complexity of highly correlated in local
time and frequency in audio spectrograms. To enhance contextual semantic
understanding and robustness, we fine-tune the encoder with a regularized
masking on target datasets, instead of input dropping or zero. Empirically,
when built with Vision Transformers structure, we find A-JEPA to be highly
scalable and sets new state-of-the-art performance on multiple audio and
speech classification tasks, outperforming other recent models that use
externally supervised pre-training.

##  1 Introduction

Figure 1: Overview of A-JEPA. The audio-based joint-embedding predictive
architecture uses a context encoder to predict and align the representations
of various target blocks in a latent space, originating from the same audio
spectrogram.

Numerous cognitive theories state that the process of adapting an internal
model to predict the absence of input information is a crucial mechanism for
learning in biological systems Rao and Ballard (1999). The recent
accomplishments of substantial foundational language models have been
propelled by the utilization of the self-supervised mask-denoising paradigm,
which involves learning through the process of filling in missing information
Devlin et al. (2018); Brown et al. (2020); Touvron et al. (2023). Moreover,
masked pretraining tasks also dominate the performance of representation
learning in computer vision Bao et al. (2021); He et al. (2022); Zhou et al.
(2021); Fei et al. (2023a).

The recently proposed image-based Joint-Embedding Predictive Architecture
(JEPA) LeCun (2022); Assran et al. (2023) demonstrates that one could learn
well-performed image representations by predicting the masked image regions in
a latent space. Compared to other methods for masked image modeling He et al.
(2022); Fei (2019); Tong et al. (2022); Fei et al. (2022); Fei (2020); Yan et
al. (2021); Feichtenhofer et al. (2022), which predict low-level visual tokens
or pixels, JEPAs make predictions in a high-level representation space, where
unnecessary pixel-level details can be eliminated, thereby leading the model
to concentrate on more semantic features Assran et al. (2023). However, the
implementation of this learning principle on sensory data, _e.g._ , audio,
continues to be a promising endeavor.

In this work, we study the problem of self-supervised representation learning
on audio data and extend the JEPA-based learning principle to the audio
spectrogram, which we refer to as A-JEPA: Audio-based Joint-Embedding
Predictive Architecture. Different from JEPA for images, it is believed that
the random block masking of audio spectrogram may be easy due to the
correlation of information along the time and frequency axis Tong et al.
(2022). Therefore, the region masking strategy of A-JEPA is designed in a
curriculum manner for the spectrogram of audio signal, _i.e._ , gradually from
random block to time-frequency aware masking in a schedule. After that, we
train the context-encoder by restoring the missing regions of masked
spectrogram with the guidance from exponential moving averaged target-encoder
in learned representation space. We minimize the patch-normalized mean square
error to optimize the networks. At the fine-tuning stage, we discard the
decoder and fine-tune the context encoder with regularized patch-masking,
where the attention weights for the masked token are solely reliant on others
rather than directly dropped patches or set to zero. We target to manipulate
the connections between audio patches via masking, where the networks are
forced to exploit partial neighbors’ information to produce a meaningful
representation.

Empirically, the A-JEPA model exhibits exceptional performance in multiple
audio and speech classification tasks, thereby establishing a new state-of-
the-art benchmark. Notably, as a self-supervised audio-only model, it
surpasses other contemporary competitors focused on masking and reconstruction
by a significant margin of +1.3 points in terms of mAP on the AudioSet-2M
dataset. We further provide the visualization and audible examples to
qualitatively ascertain the efficacy of the A-JEPA. More encouragingly, our
findings indicate that increasing the pre-training audio dataset size yields
continued performance enhancements, even under fixed computational
constraints. This observation suggests a promising avenue for further
advancements in audio foundation models. Through an extensive empirical
evaluation, we demonstrate that:

  * •

This paper delves into a simple extension of JEPA to audio data, presenting a
unified and scalable framework for learning self-supervised audio
representations. To the best of our knowledge, it is the initial endeavor to
apply JEPA to the audio area, yielding remarkably favorable outcomes.

  * •

To cover the unique challenges of the audio domain, we introduce a curriculum
masking strategy for gradually time-frequency aware pattern during pre-
training, and regularized patch masking for robust information flow during
fine-tuning.

  * •

Experimental results substantiate the scalability and efficiency of the A-JEPA
framework. Moreover, A-JEPA exhibits superior performance compared to pixel-
reconstruction techniques like AudioMAE in AS-2M classification and other
downstream tasks. Finally, the code and models will be publicly available.

##  2 Related Works

#### Joint-embedding architectures for masked pre-training.

Masked and denoising autoencoders Vincent et al. (2008, 2010); Devlin et al.
(2019) have emerged as versatile methods for learning representations by
reconstructing the original source from masked or corrupted inputs Fei and
Huang (2023); Fei et al. (2023b). In CV, a set of approaches attempt to
integrate joint-embedding architectures with reconstruction-based approaches,
wherein they employ an invariance pretraining loss with a patch-level
reconstruction loss Zhou et al. (2021); Pathak et al. (2016); Chen et al.
(2020a); He et al. (2021); Wei et al. (2021). However, it is noteworthy that
view-invariance-based methods often exhibit a bias towards learning global
image representations. Adding local loss terms is proposed to improve
performance on other popular tasks in computer vision Bardes et al. (2022);
Chen et al. (2022b); Gidaris et al. (2020). The concept of contrastive
predictive coding Oord et al. (2018) is also closely connected to this
research direction on local loss terms. In the context of images Henaff
(2020), it uses a contrastive objective combined with a convolutional network
to discriminate between overlapping image patch representations. Furthermore,
I-JEPA Assran et al. (2023) aims to predict the representations of different
target blocks from a single context block, thereby constructing semantic
information. Our work extends the JEPA framework with unique designs for
representation learning with audio spectrograms.

#### Out-of-domain and in-domain pre-training for audio.

Pre-training for audio representation can generally be divided into two main
categories: (i) Transferring natural image supervised pre-trained ViT
Dosovitskiy et al. (2020) or ResNet He et al. (2015), _e.g._ , ImageNet Gong
et al. (2021a); Koutini et al. (2021); Nagrani et al. (2021); Chen et al.
(2022a); Gong et al. (2021b, 2022a). In this approach, the models operate over
audio spectrograms by deflating from three channels (RGB) into one channel
(spectrogram) in the pre-trained patch embedding in ViT and employing the rest
of the transformer blocks Gong et al. (2021a); Koutini et al. (2021). Chen et
al. (2022a) encodes spectrograms initialized from the Swin Transformer Liu et
al. (2021) and Nagrani et al. (2021) uses ImageNet-21K pre-trained ViT.
Instead, our A-JEPA focuses on audio-only self-supervised pre-training from
scratch. (ii) Audio-only self-supervised methods, which can be further split
by the input signal type, _e.g._ , raw waveform Schneider et al. (2019);
Baevski et al. (2020, 2022), frame-level features Hsu et al. (2021); Shi et
al. (2022); Srivastava et al. (2021), or spectrogram patches Gong et al.
(2021c); Baade et al. (2022)); and the objective used for self-supervision,
_e.g._ , contrastive van den Oord et al. (2018); Baevski et al. (2020);
Arandjelovic and Zisserman (2018); Patrick et al. (2021); Hsu et al. (2021) or
prediction and reconstruction Gong et al. (2021c); Baevski et al. (2022);
Srivastava et al. (2021); Shi et al. (2022). Baevski et al. (2020) takes raw
waveform as inputs and exploits contrastive learning to differentiate
contextualized representations across different time segments. Liu et al.
(2020) introduces a pretext task of masked acoustic model, aiming to
reconstruct frame-level Mel-features from masked time frames. Gong et al.
(2021c); Baade et al. (2022); Chong et al. (2022); Niizumi et al. (2022);
Huang et al. (2022) operates over spectrogram patches and employs joint
contrastive and reconstructive objectives on masked patches. Compared with
prediction at a low level, we build in a latent space and showcase its
superiority.

##  3 Approach

In this section, we review the standard joint-embedding predictive
architecture (JEPA) and describe how we instantiate it for the audio domain as
A-JEPA.

###  3.1 Joint-Embedding Predictive Architecture

The main idea of JEPA is to learn an encoder by predicting parts of input,
_i.e_ , target regions, based on the visible context in the latent space. The
basic architecture is made up of three networks: a context encoder
$E_{\theta}(\cdot)$ and target encoder $E_{\tilde{\theta}}(\cdot)$ extract the
representations of context and target regions, respectively. The predictor
$P_{\phi}(\cdot,\cdot)$ is used to predict the target representation condition
on the context representation from $E_{\theta}(\cdot)$. The parameters of
target encoder $E_{\tilde{\theta}}(\cdot)$ are updated via the exponential
moving average of context encoder weights at each iteration. The parameters of
context encoder and predictor are optimized jointly by minimizing the distance
of representations of target regions from predictor and target encoder.

###  3.2 Application to Audio

We instantiate the A-JEPA for audio by using a curriculum masking strategy in
pre-training and regularized masking in fine-tuning. Generally, the context
encoder extracts the representation of visible audio spectrogram pre-processed
by our masking strategy, based on which the predictor outputs the
representation of target regions in latent space. See Figure 1 for an
overview.

#### Input spectrogram.

In line with Gong et al. (2021a, 2022b); Huang et al. (2022), we transform
audio signals into Mel-spectrograms and segment them into non-overlapped grid
patches. These patches are then flattened and embedded by a linear projection.
We also add fixed sinusoidal positional encoding to the patch embedding.

Figure 2: Examples of our context and target-masking strategy on Mel-
spectrograms. Given an audio spectrogram, (i) random block: we randomly sample
4 target blocks with scale in the range (0.15, 0.2) and aspect ratio in the
range (0.75, 1.5); (ii) time and frequency: we randomly sample 3 target blocks
with scale in the range (0.05, 0.075) and _remove all the related time or
frequency_ in the total Mel-spectrograms. Next, we randomly sample a context
block with a scale in the range (0.85, 1.0) and _remove any overlapping target
blocks_. green patches are selected while gray patches are removed.  Figure 3:
Different curriculum function. With the increase of the pre-training step,
progressive function approx to hard (1) from easy (0) in different trends.

#### Curriculum masking strategy.

Vanilla JEPA randomly masks out a block of spectrogram patches for the
selection of context and target regions Assran et al. (2023). As the
spectrogram can be viewed as a 2D representation of audio along time and
frequency with implicit entangles, it is reasonable to explore treating time
and frequency differently during masking. Following Huang et al. (2022), we
consider both random block masking without any prior and masking a portion of
time and frequency of a spectrogram. It is believed that random block masking
is comparably easier than time-frequency aware masking, as the model can guess
the missing component by extrapolating nearby context, _e.g._ , formants in
vowels and frictional sounds in consonants around. That is, directly applying
block masking will retain enough information in the time domain and frequency
domain. It would be more difficult to completely obscure information from a
certain paradigm. To this end, we carefully design two masking strategies with
adjustable scale factors for specific masking ratios, as shown in Figure 2.

Based on the above two masking strategies, we further propose an annealing
dividing strategy with curriculum learning Bengio et al. (2009); Hacohen and
Weinshall (2019); Fei (2021). To be specific, we randomly decide whether to
use random block or time-frequency aware masking at each training step and
gradually anneal to the time-frequency aware masking method at the end of the
training. Figure 3 illustrate several popular progressive functions. Formally,
given the image $I$ and the mask scale factor $r$, we define block masking set
$\mathcal{M}$ as follows:

$\displaystyle p\sim\text{Bernoulli}(f(s))$ (1)
$\displaystyle\mathcal{M}=\begin{cases}\text{Time-frequency}(I,r)&\quad
p=1,\\\ \text{Block}(I,r)&\quad p=0\\\ \end{cases}$ (2)

where $f(s)$ is the progressing function Platanios et al. (2019), and $s$
denotes the training step. We set
$f(s)=\min(1,\sqrt{s\frac{1-c_{0}^{2}}{S}}+c_{0}^{2})$, $c_{0}>0$ is set to
0.01 by default, as prior experiments shown a slight advantage compared with
other schedule. Bernoulli($\cdot$) is the Bernoulli distribution with input
$f(s)$. The entire process can be seen in Algorithm 1. Intuitively, a smaller
value of $f(s)$ leads to easier reconstruction prediction. In this grade,
$f(s)$ gradually increases from 0 to 1 automatically during pre-training,
which results in a better representation performance.

#### Architecture.

The employed architecture is reminiscent of the setting in Huang et al.
(2022); He et al. (2021); Assran et al. (2023) and encompasses key components
as: (i) _Context-encoder and target-encoder._ A-JEPA uses a stack of standard
vision Transformers Dosovitskiy et al. (2020) as its encoder. The encoder
exclusively processes non-masked patches. Target encoder, shares an identical
structure and its weights are updated iteratively via an exponential moving
average of the context encoder weights. (ii) _Decoder_ is also composed of
standard Transformer blocks. The encoded patches from the encoder are padded
with trainable masked tokens. After restoring the original time-frequency
order in the audio spectrogram, we add fixed sinusoidal positional embeddings
and feed the restored sequence into the decoder. At the topmost of the
decoder, we add a linear head to predict and reconstruct the latent features.

#### Multi-mask objective.

The objective is also simply to average the L2 distance between the output
features of the predictor and the target encoder. To increase the efficiency
of A-JEPA, we utilize a multi-masking strategy Caron et al. (2020); Baevski et
al. (2023), which facilitates the amortization of the target computation
expenses.

#### Fine-tuning with regularized masking.

In the fine-tuning stage, we only keep and fine-tune the encoder and discard
the decoder. An average pooling layer is applied followed by a linear layer on
top in downstream tasks. On the other hand, certain works have attempted to
integrate masking operations into the fine-tuning stage Oneata and Cucu
(2022); Chen et al. (2020b). For example, SpecAug Park et al. (2019) takes
full-length input with the masked portion set to zero as data augmentation;
AudioMAE Huang et al. (2022) selectively encounters a subset of real-valued
input patches without the nullified ones with low masking ration. However,
these methods may still lead to a discrepancy between actual conditions.

In this study, we present regularized masking (RM), illustrated in Figure 4.
In a nutshell, we modify the computing of self-attention to control the
connections between different audio patch tokens, thereby influencing the
attention scores and contextual semantic representation. Specifically, in the
$l$-th layer, a certain percentage of audio patch tokens is first randomly
selected. These tokens are excluded from the calculation of attention weights
among the tokens, but entirely contributed by remaining others. In this
manner, the networks are required to utilize partial neighbors’ attention
information, and robust to acquire meaningful information. We utilize RM in
the fine-tuning phase while maintaining the attention calculation as the
vanilla during testing.

Figure 4: Regularize patch masking during fine-tuning. Masked patch is
forbidden to attend to attention computation where its attention score is
entirely contributed by others. It manipulates the connections between patch
tokens in self-attention via masking, where the networks are forced to exploit
partial neighbors’ information to produce a meaningful representation.
Algorithm 1 Curriculum Masking Strategy

Input: Image size ($w,h$), target scale $s_{t}$ and aspect ratio $r_{t}$,
context scale $s_{c}$ and aspect ratio $r_{c}$, target mask number $N_{t}$,
training step $s$, curriculum $f(\cdot)$;

1: Context size $\mathcal{S}_{c}$ = Sample block Size($w,h,s_{c},r_{c}$);

2: Target size $\mathcal{S}_{t}$ = Sample block Size($w,h,s_{t},r_{t}$);

3: $p\sim\text{Bernoulli}(f(s))$;

4: Target mask list $M_{t}$, compliment target mask list $C_{t}$;

5: for $i$ in range($N_{t}$) do

6: if $p$ =1 then

7: $m,c$ = Sample time-frequency mask($\mathcal{S}_{t}$);

8: else

9: $m,c$ = Sample block mask($\mathcal{S}_{t}$);

10: end if

11: $M_{t}$ append $m$, $C_{t}$ append $c$;

12: end for

13: Acceptable region $\mathcal{R}=C_{t}$;

14: Context mask $M_{c}$ = Sample context mask($\mathcal{S}_{c}$,
$\mathcal{R}$);

15: return $M_{t}$ and $M_{c}$

Model | Backbone | PT-Data | AS-20K | AS-2M | ESC-50 | SPC-2 | SPC-1 | SID
---|---|---|---|---|---|---|---|---
_No pre-training_ |  |
ERANN Verbitskiy et al. (2021) | CNN | - | - | 45.0 | 89.2 | - | - | -
PANN Kong et al. (2020) | CNN | - | 27.8 | 43.1 | 83.3 | 61.8 | - | -
_In-domain self-supervised pre-training_ |  |
wav2vec 2.0 Baevski et al. (2020) | Transformer | LS | - | - | - | - | 96.2* | 75.2*
HuBERT Hsu et al. (2021) | Transformer | LS | - | - | - | - | 96.3* | 81.4*
Conformer Srivastava et al. (2021) | Conformer | AS | - | 41.1 | 88.0 | - | - | -
SS-AST Gong et al. (2021c) | ViT-B | AS+LS | 31.0 | - | 88.8 | 98.0 | 96.0 | 64.3
Mask and reconstruction works |  |
MaskSpec Chong et al. (2022) | ViT-B | AS | 32.3 | 47.1 | 89.6 | 97.7 | - | -
MAE-AST Baade et al. (2022) | ViT-B | AS+LS | 30.6 | - | 90.0 | 97.9 | 95.8 | 63.3
Audio-MAE Huang et al. (2022) | ViT-B | AS | 37.0 | 47.3 | 94.1 | 98.3 | 96.9 | 94.8
A-JEPA | ViT-B | AS | 38.4 | 48.6 | 96.3 | 98.5 | 97.7 | 95.8
_Out-of-domain supervised pre-training_ |  |
PSLA Gong et al. (2021b) | EfficientNet | IN | 31.9 | 44.4 | - | 96.3 | - | -
AST Gong et al. (2021a) | DeiT-B | IN | 34.7 | 45.9 | 88.7 | 98.1 | 95.5 | 41.1
MBT Nagrani et al. (2021) | ViT-B | IN-21K | 31.3 | 44.3 | - | - | - | -
HTS-AT Chen et al. (2022a) | Swin-B | IN | - | 47.1 | 97.0 | 98.0 | - | -
PaSST Koutini et al. (2021) | DeiT-B | IN | - | 47.1 | 96.8 | - | - | -

Table 1:  Comparison with other state-of-the-art audio representation models
on audio and speech classification tasks. We use the mAP (%) metric for AS and
the accuracy metric for ESC/SPC/SID tasks. For pre-training (PT) dataset, we
simplify as AudioSet (AS), LibriSpeech (LS), and ImageNet (IN). $\dagger$
denotes Fine-tuning results with additional supervised training on AS-2M. We
gray-out models pre-trained with external non-audio datasets, e.g., ImageNet.
The best single models in AS-2M are compared in no ensembles. * represents
linear evaluation results from corresponding papers.

##  4 Experiments

We perform an extensive evaluation on six tasks, including audio
classification on AudioSet (AS-2M, AS-20K), Environmental Sound Classification
(ESC-50), and speech classification on Speech Commands (SPC-1 and SPC-2) and
VoxCeleb (SID). We use AudioSet for ablation studies. The implementation
details can be seen in Appendix A.

###  4.1 Experimental settings.

For a comparable comparison, similar to Huang et al. (2022), we utilize the
following tasks with datasets:

(i) AudioSet Gemmeke et al. (2017) (AS-2M, AS-20K) comprises approximately 2
million 10 seconds that are utilized for audio classification purposes. Each
clip in the dataset is weakly annotated for 527 types of audio events Li et
al. (2022); Hershey et al. (2017, 2021), with the possibility of multiple
events occurring within a single clip. The dataset consists of a full training
set, which is further divided into two subsets: a class-wise balanced set
containing 22,176 clips, and an unbalanced set containing 2,042,985 clips.
Additionally, an evaluation set with 20,383 clips is provided for testing
purposes. To conduct our experiments, we obtained and processed a subset of
the dataset, comprising 1.96M clips from the unbalanced training set, 21k
clips from the balanced training set, and 19k clips from the evaluation set.
We use the union of unbalanced and balanced training audio for pre-training
and fine-tuning for AS-2M; We use AS-2M for pre-training and the 20K balanced
set for fine-tuning in the AS-20K experiment. We report the testing mAP on the
19K eval set following AST Gong et al. (2021a).

(ii) Environmental Sound Classification (ESC-50) Piczak (2015) is a collection
of 2,000 environmental sound recordings for audio classification. Each lasts 5
seconds and there are 50 classes in ESC. We report accuracy under 5-fold
cross-validation with the same split used by Gong et al. (2021a).

(iii) Speech Commands (SPC-2, SPC-1) Warden (2018) pertain two keyword
spotting tasks. SPC-2 encompasses a set of 35 speech commands. The training
set is 84,843 and the testing and validation sets are with sizes 11,005, and
9,981 respectively. Each with a duration of 1 second. SPC-1 encompasses 10
classes of keywords, 1 silence class, and 1 unknown class that includes all
remaining 20 common speech commands. We employ the data and partitioning from
SUPERB wen Yang et al. (2021) benchmark to report the testing accuracy.

(iv) VoxCeleb (SID) Nagrani et al. (2020) comprises 150K utterances attributed
to 1,251 distinct speakers. The speaker identification task (SID) involves to
classify the utterances to identify their original speaker. We use the V1
version for training, validation, and testing sets with 128k, 6k, and 8k, and
report the testing accuracy metric.

Backbone | #Params | AS-20K | AS-2M
---|---|---|---
ViT-S | 22M | 33.7 | 46.1
ViT-B | 86M | 38.4 | 48.6
ViT-L | 304M | 38.8 | 48.8
Table 2: Ablating model size. Evaluating the impact of model size on
downstream tasks. A-JEPA benefits from larger models.

###  4.2 Comparison with the State-of-the-art

Table 1 presents a comprehensive comparison of A-JEPA against previous state-
of-the-art models for audio representation learning. The analysis is divided
into three distinct groups. To ensure a fair evaluation, our primary focus
lies on models within the middle group, which have undergone self-supervised
pre-training using in-domain audio-only datasets In addition, we include other
models without any pre-training at the top group and models with supervised
pre-training on out-of-domain ImageNet at the bottom group, where the latter
comprises the previous leading systems on the respective datasets. Among the
models with in-domain self-supervised pre-training, A-JEPA, pre-trained on
AudioSet, exhibits the highest performance across all tasks. Notably, its mAP
score of 38.4 on the AudioSet-20K dataset surpasses all alternative
approaches, including previous masking-reconstruction works Conformer
Srivastava et al. (2021), SS-AST Gong et al. (2021c) and AudioMAE Huang et al.
(2022). In the lowermost group of Table 1, A-JEPA also demonstrates superior
performance compared to previous state-of-the-art models that employed
ImageNet supervised pre-training. It is worth noting that A-JEPA, the proposed
approach, does not rely on external data or labels from unrelated domains.
More encouragingly, as indicated in the experiments conducted in Kong et al.
(2020), there is a potential for further mAP enhancement for A-JEPA if audio
data with a sampling rate of 32K becomes available. The advantage is still
maintained for the speech tasks, including SPC-1, SPC-2, and SID. In summary,
A-JEPA, which leverages audio-only pre-training from scratch using AudioSet,
achieves commendable performance in both audio and speech classification tasks
through the incorporation of feature alignment.

Figure 5: Visualization of A-JEPA predictor representations. The first column
presents the original audio spectrogram, while the second column displays the
context audio spectrogram, which is processed with a pre-trained A-JEPA ViT-B
encoder. red bounding boxes, in subsequent columns, showcase samples created
from a generative model. It decoded the output of the pre-trained A-JEPA
predictor, conditioned on positional mask tokens corresponding to the location
of the bounding box. It is worth noting that qualities shared among these
samples indicate the information contained in the A-JEPA prediction.

###  4.3 Model Analysis

#### Model scality.

We study the various design choices pertaining to encoder architectures in the
A-JEPA. Table 2 illustrates the trade-off between encoder model size and its
performance. It is observed that larger encoder models tend to exhibit
superior performance, albeit at the expense of increased computational
requirements and memory utilization. Moreover, it is noted that the accuracy
improvement of ViT-L over ViT-B/S is particularly pronounced when applied to
the smaller and more balanced AS-20K dataset. Additionally, the performance
disparity between ViT-S and ViT-B can be considerably diminished, e.g., from
4.7 to 2.5 mAP, by employing fine-tuning techniques with a greater volume of
in-domain data, e.g., from AS-20K to AS-2M, as observed in Huang et al.
(2022).

Mask | Backbone | Mask Ratio | mAP
---|---|---|---
Curriculum | ViT-B | 0.75 | 48.6
Random | ViT-B | 0.75 | 47.8
Inverse | ViT-B | 0.75 | 46.3
Table 3: Ablating masking strategy in pre-training. The proposed curriculum masking strategy is preferred. Model | Backbone | Masking Ratio | mAP
---|---|---|---
A-JEPA | ViT-B | 0.1 | 48.6
_w/o_ RM | ViT-B | 0.1 | 48.1
Table 4: Ablating regularized masking (RM) in fine-tuning. The proposed
regularized masking strategy in fine-tuning is helpful for guiding A-JEPA to
learn more robust audio representations.

#### Masking strategies in pre-training and fine-tuning.

We also present a comparison of different pre-training masking strategies for
A-JEPA in In Table 3. The results obtained from our experiments reveal that
curriculum masking outperforms random block masking. This finding suggests
that a guided approach to time-frequency aware masking, yields superior
performance. Moreover, inverse strategy denotes that first time and frequency
and then random maksing, which results in a more significant performance drop,
demonstrate the effectiveness of our method again. In general, we observe that
for task-agnostic pre-training, curriculum masking from easy to hard with a
high ratio is preferred. On the other hand, when it comes to fine-tuning, as
illustrated in Table 4, employing regularized patch masking with lower ratios
achieves better performance in downstream tasks.

#### Predictor depth and width.

The impact of decoder depth on mean average precision (mAP) is evaluated in
Table 5. A 16-layer decoder, being deeper than its shallower counterparts,
exhibits superior performance. Furthermore, Table 6 presents a comparison of
decoder widths, specifically the embedding dimension. The results indicate
that a 512-dimension decoder strikes a favorable balance between computational
requirements and performance, as increasing the width beyond this threshold
does not yield significant improvements.

Depth | Backbone | Epoch | mAP
---|---|---|---
8 | ViT-B | 24 | 48.3
16 | ViT-B | 24 | 48.6
Table 5: Ablating the predictor depth. Increasing the predictor depth leads to a prominent performance improvement of the pre-trained audio representations. Width | Backbone | Epoch | mAP
---|---|---|---
256 | ViT-B | 24 | 48.0
512 | ViT-B | 24 | 48.6
768 | ViT-B | 24 | 48.5
Table 6: Ablating the predictor width. Having a width bottleneck in the
predictor improves the downstream performances.

#### Pre-training data size and epochs.

Figure 5 depicted the influence of pre-training dataset size. A subset of data
is randomly sampled from originals for pre-training. It is observed that the
performance of the model consistently improves with an increase in the amount
of data used for pre-training. Meanwhile, as shown in Figure 7, we also plot
the mAP score at different training epochs. It is evident that prolonging the
training duration yields favorable outcomes; however, the performance reaches
a plateau after the 24th epoch.

Figure 6: Ablating pre-training data size. When increasing the size of the
pre-training dataset, we see a significant performance improvement in audio
representation for the ViT-B model.  Figure 7: Ablating pre-training epochs on
ViT-B backbone in AudioSet. Training for longer is beneficial for downstream
task performance.

###  4.4 Predictor Visualizations

The purpose of the predictor component in I-JEPA is to utilize the output of
the context-encoder and provide predictions for the representations of a
target object at a specific location, as specified by the positional mask
tokens. In this section, we aim to analyze whether the predictor conditioned
on the positional mask tokens is learning to correctly capture positional
uncertainty in the target in the audio spectrogram. Specifically, we
qualitatively visualize the outputs of the predictor referring to Assran et
al. (2023). After pretraining, we freeze the context-encoder and predictor
weights, and train a decoder following the RCDM framework Bordes et al. (2021)
to map the average pool of the predictor outputs back to pixel space. The
outputs of the decoder for different random seeds are illustrated in Figure 5.
Shared characteristics observed across multiple samples indicate information
that is conveyed within the average-pooled predictor representation. It is
noteworthy that the A-JEPA predictor successfully captures positional
uncertainty and generates high-level audio components with an accurate pose.

##  5 Conclusion

We have conducted an investigation into an extension of JEPA to audio data.
Our A-JEPA is designed to reconstruct masked spectrogram patches from audio
recordings within a latent space and achieves superior performance on multiple
audio and speech classification tasks. We have made three noteworthy
observations: First, a straightforward application of JEPA yields remarkable
outcomes for audio spectrograms. Second, we find that it is possible to
enhance the quality of learned representations with a time-frequency aware
masking strategy progressing from easy to hard. Third, we show that
regularized masking, instead of directly dropping or setting to zero, can be
applied to fine-tuning, contributing to accuracy improvement. In the future,
we intend to explore multi-modal self-supervised learning with a joint latent
embedding, _e.g._ , video and text modals, to provide a better formulation of
audio representation guidance.

## References

  * Arandjelovic and Zisserman [2018] Relja Arandjelovic and Andrew Zisserman.  Objects that sound.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I, volume 11205 of Lecture Notes in Computer Science, pages 451–466. Springer, 2018.
  * Assran et al. [2023] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas.  Self-supervised learning from images with a joint-embedding predictive architecture.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619–15629, 2023.
  * Baade et al. [2022] Alan Baade, Puyuan Peng, and David Harwath.  Mae-ast: Masked autoencoding audio spectrogram transformer.  arXiv preprint arXiv:2203.16691, 2022.
  * Baevski et al. [2020] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * Baevski et al. [2022] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli.  data2vec: A general framework for self-supervised learning in speech, vision and language.  CoRR, abs/2202.03555, 2022.
  * Baevski et al. [2023] Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli.  Efficient self-supervised learning with contextualized target representations for vision, speech and language.  In International Conference on Machine Learning, pages 1416–1429. PMLR, 2023.
  * Bao et al. [2021] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  Beit: Bert pre-training of image transformers.  arXiv preprint arXiv:2106.08254, 2021.
  * Bardes et al. [2022] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicregl: Self-supervised learning of local visual features.  Advances in Neural Information Processing Systems, 35:8799–8810, 2022.
  * Bengio et al. [2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston.  Curriculum learning.  In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009.
  * Bordes et al. [2021] Florian Bordes, Randall Balestriero, and Pascal Vincent.  High fidelity visualization of what your self-supervised representation knows about.  arXiv preprint arXiv:2112.09164, 2021.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * Caron et al. [2020] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  Advances in neural information processing systems, 33:9912–9924, 2020.
  * Chen et al. [2020a] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1691–1703. PMLR, 2020.
  * Chen et al. [2020b] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia.  Gridmask data augmentation.  arXiv preprint arXiv:2001.04086, 2020.
  * Chen et al. [2022a] Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov.  Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection.  arXiv preprint arXiv:2202.00874, 2022.
  * Chen et al. [2022b] Yubei Chen, Adrien Bardes, ZENGYI LI, and Yann LeCun.  Intra-instance vicreg: Bag of self-supervised image patch embedding explains the performance.  2022\.
  * Chong et al. [2022] Dading Chong, Helin Wang, Peilin Zhou, and Qingcheng Zeng.  Masked spectrogram prediction for self-supervised audio pre-training, 2022.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * Fei and Huang [2023] Zhengcong Fei and Junshi Huang.  Incorporating unlikely negative cues for distinctive image captioning.  2023\.
  * Fei et al. [2022] Zhengcong Fei, Xu Yan, Shuhui Wang, and Qi Tian.  Deecap: Dynamic early exiting for efficient image captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12216–12226, 2022.
  * Fei et al. [2023a] Zhengcong Fei, Mingyuan Fan, Li Zhu, Junshi Huang, Xiaoming Wei, and Xiaolin Wei.  Masked auto-encoders meet generative adversarial networks and beyond.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24449–24459, 2023.
  * Fei et al. [2023b] Zhengcong Fei, Mingyuan Fan, Li Zhu, Junshi Huang, Xiaoming Wei, and Xiaolin Wei.  Uncertainty-aware image captioning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 614–622, 2023.
  * Fei [2019] Zheng-cong Fei.  Fast image caption generation with position alignment.  arXiv preprint arXiv:1912.06365, 2019.
  * Fei [2020] Zhengcong Fei.  Iterative back modification for faster image captioning.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 3182–3190, 2020.
  * Fei [2021] Zhengcong Fei.  Partially non-autoregressive image captioning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1309–1316, 2021.
  * Feichtenhofer et al. [2022] Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al.  Masked autoencoders as spatiotemporal learners.  Advances in neural information processing systems, 35:35946–35958, 2022.
  * Gemmeke et al. [2017] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776–780. IEEE, 2017.
  * Gidaris et al. [2020] Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord.  Learning representations by predicting bags of visual words.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6928–6938, 2020.
  * Gong et al. [2021a] Yuan Gong, Yu-An Chung, and James R. Glass.  AST: audio spectrogram transformer.  In Hynek Hermansky, Honza Cernocký, Lukás Burget, Lori Lamel, Odette Scharenborg, and Petr Motlícek, editors, Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August - 3 September 2021, pages 571–575. ISCA, 2021.
  * Gong et al. [2021b] Yuan Gong, YuAn Chung, and James Glass.  Psla: Improving audio tagging with pretraining, sampling, labeling, and aggregation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.
  * Gong et al. [2021c] Yuan Gong, Cheng-I Lai, Yu-An Chung, and James R. Glass.  Ssast: Self-supervised audio spectrogram transformer.  ArXiv, abs/2110.09784, 2021.
  * Gong et al. [2022a] Yuan Gong, Sameer Khurana, Andrew Rouditchenko, and James Glass.  Cmkd: Cnn/transformer-based cross-model knowledge distillation for audio classification, 2022.
  * Gong et al. [2022b] Yuan Gong, Cheng-I Lai, Yu-An Chung, and James Glass.  Ssast: Self-supervised audio spectrogram transformer.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10699–10709, 2022.
  * Hacohen and Weinshall [2019] Guy Hacohen and Daphna Weinshall.  On the power of curriculum learning in training deep networks.  In International conference on machine learning, pages 2535–2544. PMLR, 2019.
  * He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2015.
  * He et al. [2021] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  arXiv preprint arXiv:2111.06377, 2021.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022.
  * Henaff [2020] Olivier Henaff.  Data-efficient image recognition with contrastive predictive coding.  In International conference on machine learning, pages 4182–4192. PMLR, 2020.
  * Hershey et al. [2017] Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson.  Cnn architectures for large-scale audio classification.  In International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2017.
  * Hershey et al. [2021] Shawn Hershey, Daniel PW Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R Channing Moore, and Manoj Plakal.  The benefit of temporally-strong labels in audio event classification.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 366–370. IEEE, 2021.
  * Hsu et al. [2021] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021.
  * Huang et al. [2022] Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer.  Masked autoencoders that listen.  Advances in Neural Information Processing Systems, 35:28708–28720, 2022.
  * Kong et al. [2020] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley.  Panns: Large-scale pretrained audio neural networks for audio pattern recognition.  IEEE ACM Trans. Audio Speech Lang. Process., 28:2880–2894, 2020.
  * Koutini et al. [2021] Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer.  Efficient training of audio transformers with patchout.  CoRR, abs/2110.05069, 2021.
  * LeCun [2022] Yann LeCun.  A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27.  Open Review, 62, 2022.
  * Li et al. [2022] Juncheng B. Li, Shuhui Qu, Po-Yao Huang, and Florian Metze.  Audiotagging done right: 2nd comparison of deep learning methods for environmental sound classification.  CoRR, abs/2203.13448, 2022.
  * Liu et al. [2020] Andy T. Liu, Shu-Wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee.  Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders.  In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pages 6419–6423. IEEE, 2020.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9992–10002. IEEE, 2021.
  * Nagrani et al. [2020] Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman.  Voxceleb: Large-scale speaker verification in the wild.  Comput. Speech Lang., 60, 2020.
  * Nagrani et al. [2021] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun.  Attention bottlenecks for multimodal fusion.  In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 14200–14213, 2021.
  * Niizumi et al. [2022] Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino.  Masked spectrogram modeling using masked autoencoders for learning general-purpose audio representation.  arXiv:2204.12260, 2022.
  * Oneata and Cucu [2022] Dan Oneata and Horia Cucu.  Improving multimodal speech recognition by data augmentation and speech representations.  arXiv preprint arXiv:2204.13206, 2022.
  * Oord et al. [2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * Park et al. [2019] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le.  Specaugment: A simple data augmentation method for automatic speech recognition.  arXiv preprint arXiv:1904.08779, 2019.
  * Pathak et al. [2016] Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros.  Context encoders: Feature learning by inpainting.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2536–2544. IEEE Computer Society, 2016.
  * Patrick et al. [2021] Mandela Patrick, Po-Yao Huang, Ishan Misra, Florian Metze, Andrea Vedaldi, Yuki M. Asano, and João F. Henriques.  Space-time crop & attend: Improving cross-modal video representation learning.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 10540–10552. IEEE, 2021.
  * Piczak [2015] Karol J. Piczak.  ESC: Dataset for Environmental Sound Classification.  In Proceedings of the 23rd Annual ACM Conference on Multimedia, pages 1015–1018. ACM Press, 2015.
  * Platanios et al. [2019] Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom M Mitchell.  Competence-based curriculum learning for neural machine translation.  arXiv preprint arXiv:1903.09848, 2019.
  * Povey et al. [2011] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al.  The kaldi speech recognition toolkit.  In IEEE 2011 workshop on automatic speech recognition and understanding, number CONF. IEEE Signal Processing Society, 2011.
  * Rao and Ballard [1999] Rajesh PN Rao and Dana H Ballard.  Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.  Nature neuroscience, 2(1):79–87, 1999.
  * Schneider et al. [2019] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli.  wav2vec: Unsupervised pre-training for speech recognition.  In Gernot Kubin and Zdravko Kacic, editors, Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 3465–3469. ISCA, 2019.
  * Shi et al. [2022] Bowen Shi, Wei-Ning Hsu, and Abdelrahman Mohamed.  Robust self-supervised audio-visual speech recognition.  CoRR, abs/2201.01763, 2022.
  * Srivastava et al. [2021] Sangeeta Srivastava, Yun Wang, Andros Tjandra, Anurag Kumar, Chunxi Liu, Kritika Singh, and Yatharth Saraf.  Conformer-based self-supervised learning for non-speech audio tasks.  arXiv preprint arXiv:2110.07313, 2021.
  * Tong et al. [2022] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  Advances in neural information processing systems, 35:10078–10093, 2022.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * van den Oord et al. [2018] Aäron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  CoRR, abs/1807.03748, 2018.
  * Verbitskiy et al. [2021] Sergey Verbitskiy, Vladimir Berikov, and Viacheslav Vyshegorodtsev.  Eranns: Efficient residual audio neural networks for audio pattern recognition.  arXiv preprint arXiv:2106.01621, 2021.
  * Vincent et al. [2008] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and composing robust features with denoising autoencoders.  In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008, volume 307 of ACM International Conference Proceeding Series, pages 1096–1103. ACM, 2008.
  * Vincent et al. [2010] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  J. Mach. Learn. Res., 11:3371–3408, 2010.
  * Warden [2018] P. Warden.  Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition.  ArXiv e-prints, April 2018.
  * Wei et al. [2021] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan L. Yuille, and Christoph Feichtenhofer.  Masked feature prediction for self-supervised visual pre-training.  CoRR, abs/2112.09133, 2021.
  * wen Yang et al. [2021] Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung yi Lee.  SUPERB: Speech Processing Universal PERformance Benchmark.  In Proc. Interspeech 2021, pages 1194–1198, 2021.
  * Yan et al. [2021] Xu Yan, Zhengcong Fei, Zekang Li, Shuhui Wang, Qingming Huang, and Qi Tian.  Semi-autoregressive image captioning.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 2708–2716, 2021.
  * Zhou et al. [2021] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  ibot: Image bert pre-training with online tokenizer.  arXiv preprint arXiv:2111.07832, 2021.

##  Appendix A Implementation Details

For model structure, we use a vanilla 12-layer ViT-B by default as the
Transformer encoder and use a 16-layer vanilla Transformer as the decoder. The
design of other ViT model sizes is identical to Huang et al. [2022]. Following
Gong et al. [2021a]; Nagrani et al. [2021], we convert raw waveform, pre-
processed as mono channel under 16,000 sampling rate, into 128 Kaldi-
compatible Mel-frequency bands Povey et al. [2011] with a 25ms Hanning window
that shifts every 10 ms. In the case of a 10-second recording in AudioSet, the
resulting spectrogram is of 1$\times$1024$\times$128 dimension. For patch
embedding, we employ convolutional kernels with a size of $(16,16)$ and apply
a stride in both time and frequency, ensuring that the resulting patches are
non-overlapping. We sample 4 possibly overlapping target random blocks with a
scale in the range (0.15, 0.2) and aspect ratio in the range (0.75, 1.5);
sample time-frequency aware 3 target blocks with scale in the range (0.05,
0.075). We sample 1 context block mask with a random scale in the range (0.85,
1.0) and unit aspect ratio. We subsequently eliminate specific regions,
different for random and time-frequency, in the context block mask that
overlaps with any of the target block masks. These hyperparameters are tested
in prior experiments to balance a patch available number. The context-block
mask and target-block masks are sampled independently for each image in the
mini-batch.

The pre-training phase involves the utilization of AudioSet-2M, wherein we
conduct a random iteration over all audio recordings. We train for 24 epochs
with a batch size of 512, coupled with a learning rate of 2e-4. For each
audio, we randomly sample the starting time, cyclically extract a 10-second
audio segment, and randomly jitter its magnitude by up to $\pm$ 6dB. During
fine-tuning, we employ a 10% regularized patch masking ratio by default. For
the supervised fine-tuning on AudioSet-2M, to address the issue of imbalanced
training sample sizes across classes, we follow the common practice of using a
weighted sampling to balance the classes during training Huang et al. [2022].
In each epoch, we sample 200K instances, approximately 10% of AudioSet-2M,
without replacement. We fine-tune for 100 epochs, which aggregate to 10 full
epochs of AudioSet-2M. This sampling procedure ensures that the probability of
selecting an instance is inversely proportional to the occurrences of its
corresponding class within the dataset. For the smaller balanced AudioSet-20K,
we fine-tune for 60 epochs without weighted sampling. We use the Sqrt
progressive function based on prior observations.

##  Appendix B Pytorch Code for Time-frequency Aware Masking

We provide the pytorch implementation for time-frequency aware masking.

Listing 1 Pytorch code for time-frequency aware masking

⬇

1def sampe_time_frequency_mask(b_size, acceptable_regions=None,
min_ratio=0.35):

2 """b_size: sampled block size."""

3 h, w = b_size

4 def constrain_mask(mask, tries=0):

5 """ Helper to restrict given mask to acceptable regions """

6 N = max(int(len(acceptable_regions)-tries), 0)

7 for k in range(N):

8 mask *= acceptable_regions[k]

9

10 valid_mask = False

11 while not valid_mask:

12 # Sample block top-left corner

13 top = torch.randint(0, height - h, (1,))

14 left = torch.randint(0, width - w, (1,))

15 mask = torch.zeros((height, width), dtype=torch.int32)

16 mask[top:top+h, left:left+w] = 1

17 # Constrain mask to a set of acceptable regions

18 if acceptable_regions is not None:

19 constrain_mask(mask, tries)

20 mask = torch.nonzero(mask.flatten())

21 # If mask too small try again

22 valid_mask = len(mask) > min_ratio * h * w

23

24 mask_complement = torch.ones((height, width), dtype=torch.int32)

25 mask_complement[top:top+h, left:left+w] = 0

26

27 # Remove the time and frequency part

28 mask_complement_tf = torch.ones((height, width), dtype=torch.int32)

29 mask_complement_tf[top:top+h, :] = 0

30 mask_complement_tf[:, left:left+w] = 0

31 return mask, mask_complement, mask_complement_tf