# CLAP : Learning Audio Concepts From Natural Language Supervision

###### Abstract

Mainstream Audio Analytics models are trained to learn under the paradigm of
one class label to many recordings focusing on one task. Learning under such
restricted supervision limits the flexibility of models because they require
labeled audio for training and can only predict the predefined categories.
Instead, we propose to learn audio concepts from natural language supervision.
We call our approach Contrastive Language-Audio Pretraining (CLAP), which
learns to connect language and audio by using two encoders and a contrastive
learning to bring audio and text descriptions into a joint multimodal space.
We trained CLAP with 128k audio and text pairs and evaluated it on 16
downstream tasks across 8 domains, such as Sound Event Classification, Music
tasks, and Speech-related tasks. Although CLAP was trained with significantly
less pairs than similar computer vision models, it establishes SoTA for Zero-
Shot performance. Additionally, we evaluated CLAP in a supervised learning
setup and achieve SoTA in 5 tasks. Hence, CLAP’s Zero-Shot capability removes
the need of training with class labels, enables flexible class prediction at
inference time, and generalizes to multiple downstream tasks.

Index Terms—  contrastive learning, general purpose audio representation,
zero-shot, sound event classification, speech emotion recognition

##  1 Introduction

Fig. 1: CLAP jointly trains an audio and a text encoder to learn the
(dis)similarity of audio and text pairs in a batch using contrastive learning.
At testing time, the pretrained encoders are used to extract audio embeddings
from the testing audio and text embeddings from the class labels. Zero-Shot
linear classification is achieved by computing cosine similarity between the
embeddings.

The human auditory system can hear sounds and extract the kind of decisions or
meanings we need to interact with our surroundings [1]. For example, if we are
in a soccer game and suddenly hear the crowd cheering joyfully, we can assume
the local team scored! Computer models aim to understand audio cues by
automatically process audio signals and extract meaning [1]. Mainstream
Machine Learning models break the human hearing into tasks, such as the
classification of sound events and acoustic scenes. Such models are trained by
associating audio recordings to class labels of predefined categories for a
specific task and can only predict specific categories [2]. Learning under
such restricted supervision limits the flexibility to predict unseen classes.

Self-Supervised Learning (SSL) pretrains models with unlabeled audio, avoiding
the limited supervision of learning from class labels. However, SSL excludes
semantic knowledge from natural language. The pretrained model is then adapted
to a downstream task in a supervised setup learning under the class label
paradigm [3, 4, 5]. Mainstream and SSL models have static output layers that
can only predict the predefined categories. On the other hand, models that
enable Zero-Shot predictions can take an input audio and yield a prediction
score for any class typed by the user. Zero-shot requires no training stage so
there are no predefined categories. To enable such flexibility and
generalization, models need to learn the relationships between the acoustic
semantics and language semantics.

A middle path between both approaches is to learn audio concepts from natural
language supervision, which is under-explored. Computer Vision has
successfully developed models that learn image representations with natural
language supervision, achieving high performance across different downstream
tasks in Zero-Shot predictions and adapted (e.g. finetuned) to target datasets
in a supervised setup. Examples are Open AI’s CLIP [6], Florence [7], and [8].
In the audio domain, Wav2clip [9] and Audioclip [10] distill from CLIP and are
trained with audio and class labels from AudioSet [11] instead of audio and
natural language. Although the performance was promising, it is yet to be
explored how natural language can benefit flexibility and generalization to
new classes and tasks.

We call our approach Contrastive Language-Audio Pretraining (CLAP), which
learns to connect natural language and audio by using two encoders and
contrastive learning to bring audio and text descriptions into a joint
multimodal space. Our main contributions are first, to introduce our CLAP
model trained with 128k audio and text pairs. Second, our model enables Zero-
Shot predictions, thus it removes the need of training and forcing a
predefined set of categories and enables flexible class prediction at
inference time. Third, CLAP generalizes to 16 downstream tasks across 8
domains by establishing Zero-Shot SoTA performance. We also include insights
about the model in Section 4.

##  2 Method

CLAP is illustrated in Fig 1. The input is audio and text pairs passed to an
audio encoder and a text encoder. Both representations are connected in joint
multimodal space with linear projections. The space is learned with the
(dis)similarity of audio and text pairs in a batch using contrastive learning.
The pretrained encoders with their projection layers can be used to compute
audio and text embeddings and enable Zero-Shot Classification. Our method is
inspired by the CLIP model [6].

###  2.1 Contrastive Language-Audio Pretraining

Let the processed audio be $X_{a}$ s.t. $X_{a}\in\mathbb{R}^{F\times T}$ where
$F$ are the number of spectral components (e.g. Mel bins) and $T$ are the
number of time bins. Let the text be represented by $X_{t}$. Each audio-text
pair in a batch of $N$ is represented as $\\{X_{a},X_{t}\\}_{i}$ where
$i\in[0,N]$. For convenience, we dropped the $i$ notation, and henceforth
$\\{X_{a},X_{t}\\}$ will denote a batch of N.

From the pairs, the audio and text are passed through an audio encoder and a
text encoder respectively. Let $f_{a}(.)$ represent the audio encoder and
$f_{t}(.)$ represent the text encoder. For a batch of N:

$\hat{X}_{a}=f_{a}(X_{a});\hat{X}_{t}=f_{t}(X_{t})$ (1)

where $\hat{X}_{a}\in\mathbb{R}^{N\times V}$ are the audio representations of
dimensionality $V$, and $\hat{X}_{t}\in\mathbb{R}^{N\times U}$ are the text
representations of dimensionality $U$.

We brought audio and text representations, $\hat{X}_{a}$ and $\hat{X}_{t}$,
into a joint multimodal space of dimension $d$ by using a learnable linear
projection:

$E_{a}=L_{a}(X_{a});E_{t}=L_{t}(X_{t})$ (2)

where $E_{a}\in\mathbb{R}^{N\times d}$, $E_{t}\in\mathbb{R}^{N\times d}$,
$L_{a}$ and $L_{t}$ are the linear projections for audio and text
respectively.

Now that the audio and text embeddings ($E_{a}$, $E_{t}$) are comparable, we
can measure similarity:

$C=\tau*(E_{t}\cdot E_{a}^{\top})$ (3)

where $\tau$ is a temperature parameter to scale the range of logits. The
similarity matrix $C\in\mathbb{R}^{N\times N}$ has $N$ correct pairs in the
diagonal and $N^{2}-N$ incorrect pairs in the off-diagonal.

$\mathcal{L}=0.5*(\ell_{text}(C)+\ell_{audio}(C))$ (4)

where $\ell_{k}=\frac{1}{N}\sum_{i=0}^{N}\log diag(softmax(C))$ along text and
audio axis respectively. We used this symmetric cross-entropy loss
($\mathcal{L}$) over the similarity matrix to jointly train the audio encoder
and the text encoder along with their linear projections.

###  2.2 Zero-Shot Linear Classification

For Zero-Shot classification, we used CLAP’s ability to determine the
similarity between audio and text. Let’s consider a target dataset with $C$
class labels and $N$ test audios. First, we compute audio embeddings and text
embeddings for $N$ audios and $C$ classes using the pretrained encoders and
their projection layers. Second, because both the embeddings are in a common
space, we compute the cosine similarity between each testing audio and all the
class labels. Each audio will have as many logits as class labels. Third,
logits are turned into a probability distribution by applying softmax for
binary or multiclass and sigmoid for multilabel classification.

##  3 Experiments

Sound Event Classification Music Model ESC50 FSD50K US8K DCASE17 Task 4
AudioSet Music Speech Music Genres Mri. Stroke Mri. Tonic Random 0.02 $<$
0.005 0.1 0.05 $<$ 0.0018 0.5 0.1 0.1 0.1667 Benchmark (ZS) 0.6940[10]
0.0302[9] 0.6531[10] - - - - - - CLAP(ZS) 0.826 0.3024 0.7324 0.3 0.058 1.0
0.252 0.3447 0.1965 Benchmark (Best) 0.9715 [10] 0.641 [12] 0.90-7 [10] 0.646
[13] 0.471 [14] 0.992 [12] 0.883 [12] 0.975 [12] 0.942 [12] CLAP (Best) 0.9670
0.5859 0.8796 0.5938 - 1.0 0.9130 0.9794 0.9534

Instrument Classification Acoustic Scene Classification Emotion Recognition
Keyword Spotting Vocal Sound Classification Speaker Counting Model Beijing
Opera TUT2017 CRE MA-D RAV DESS Speech Comm. Vocal Sound Libri Count Random
0.25 0.06 0.1667 0.125 0.083 0.1667 0.090 CLAP (ZS) 0.4746 0.2963 0.1784
0.1599 0.1063 0.4945 0.1788 Benchmark (Best) 0.975 [12] 0.843 [15] 0.752 [12]
0.8182 [16] 0.987 [17] 0.905 [18] 0.785 [12] CLAP (Best) 0.9026 0.7463 0.6834
0.6436 0.9683 0.9795 0.7783

Table 1:  CLAP (ZS) Zero-Shot outperforms the literature. CLAP (Best) is the
best performance among our supervised setups. Higher is better for all
numbers, DCASE17 employs F1, FSD50K and AudioSet employs mAP, everything else
uses accuracy.

###  3.1 Datasets

Training. We used 128,010 audio and text pairs from 4 datasets to construct
the training dataset for CLAP. We extracted 36,796 pairs from FSD50k [19],
29,646 pairs from ClothoV2 [20], 44,292 from AudioCaps [21], 17,276 pairs from
MACS [22]. The dataset details are in appendix Section A and Table 4.
Downstream Tasks. We used 16 datasets from 8 different domains as downstream
tasks. Five tasks are Sound Event Classification. Five tasks are music
related, classification of music vs speech, music genres, strokes and tonics.
One task is Acoustic Scene Classification. Four are speech-based, Emotion
Recognition, Keyword Spotting, and Vocal Sound Classification (e.g. cough,
sneeze, laughter). One is counting speakers in a recording (0 to 10 speakers).
The datasets are in Table 1, and details are in appendix Section B and Table
5.

###  3.2 Experimental setup

Pre-processing. We used log Mel spectrogram representations of audio with a
sampling rate of 44.1 KHz, hop size of 320 secs, window size 1024 secs, and 64
Mel bins in the range of 50-8000 Hz. During training, each audio clip is
randomly truncated to a continuous segment of 5 secs, or padded if shorter.
The captions were not altered. The batches with audio and text pairs are
randomly sampled at training.
Encoders. We chose CNN14 [23] model as the audio encoder to provide a fair
comparison to previous SoTA models. The model has 80.8 million parameters, an
embedding size of 2048, and was pretrained with 2M audio clips from AudioSet.
The text encoder chosen is BERT [24]. We use HuggingFace [25] implementation
of BERT base uncased. The model has 110 million parameters. We limited the max
text sequence length to 100 chars for computational efficiency. The [CLS]
token from the final layer of BERT is used as the text embedding with a size
of 768. Both, the audio and text embeddings are projected into a multimodal
space with two learnable projection matrices resulting in an output dimension
of 1024. The temperature parameter $\tau$ is learnable and initialised to
0.007. To prevent training instability, the logits scaled by $\tau$ are
clipped to a maximum value of 100.
Training. We trained by unfreezing both encoders for 40 epochs. We use Adam
Optimiser [26] with an initial learning rate $10^{-3}$ and reduce the learning
rate on plateau by $10^{-1}$ with a patience of 10. The models are implemented
with PyTorch’s Distributed Data-Parallel and use 16GB V100 GPUs with scaling
from 8 to 24 GPUs.

###  3.3 Evaluation setups for CLAP

Zero-shot Evaluation studies the generalisation of CLAP to unseen classes and
audios. The setup is explained in Section 2.2. Instead of using the class
label, we constructed a natural language prompting, ‘This is a sound of [class
label]’. The prompt was kept the same for all the domains except three. For
Emotion Recognition we used ‘this person is feeling [class label]’, for
Keyword Spotting we only use the keyword, and for Speaker Counting we used
‘[number between 0 - 10] persons speaking’.
Supervised Feature Extraction Evaluation studies the quality of audio
representation learned by CLAP. Given a downstream task, we used CLAP as a
feature extractor followed by training a classifier of 1 or 3 fully-connected
layers (Freeze_L1 and Freeze_L3), similar to [23]. We used a learning rate of
$10^{-3}$ with Adam Optimizer for 30 epochs. We did not perform grid search
for tuning hyperparameters due to computation constraints.
Supervised Finetune Evaluation benchmarks CLAP against the best performance
for each task in the literature. Given a downstream task, we unfroze and
finetuned the audio encoder together with an attached 1 or 3 fully-connected
layers. We used a learning rate of $10^{-4}$ with Adam Optimizer for 30
epochs. We did not perform grid search for tuning hyperparameters due to
computation constraints.

##  4 Results and Discussion

For baseline comparisons we considered the best model performances in the
literature for Zero-Shot Learning ‘Benchmark (ZS)’ and for Supervised Learning
‘Benchmark (Best)’ and reported them in Table 1. We also discuss the effect of
freezing the encoders and the effect of prompts for Zero-Shot.

###  4.1 Zero-Shot (ZS) results

CLAP (ZS) achieved SoTA on established Sound Event Classification (SEC)
datasets like FSD50K, US8K and ESC50. For ESC50, CLAP achieved 82.6% accuracy
(acc) beating human performance of 81% and AudioCLIP (69%) by an absolute 12%.
In US8K, CLAP achieved 73% acc outperforming AudioCLIP (65%) by an absolute
8%. For the multi-label dataset FSD50K, CLAP beat Wav2CLIP (3%) by an absolute
27% mAP. On task GTZAN’s Music vs Speech Classification, CLAP even beat
supervised models achieving 100% acc. These results point to the possibility
of having reliable audio models with no training involved.

CLAP (ZS) performed better than random on all downstream tasks and achieved
good to slightly better than random on some music and speech-related tasks.
CLAP achieved 47% acc in Instrument Classification, an absolute 22% higher
than random. In the Vocal Sound dataset achieved 50% acc, an absolute 33%
improvement over random. In Emotion Recognition (ER) and Keyword Spotting
(KWS) CLAP outperformed random by up to an absolute 4% acc.

###  4.2 Supervised results

CLAP (Best) is the best performance among supervised setups and achieved SoTA
on 5 datasets. CLAP achieved in GTZAN Music vs Speech Classification 100% acc,
in GTZAN Music Genre Classification 91.3% acc, in Mri. Stroke Classification
97.94% acc, in Mri. Tonic Classification with 95.34% acc, and in Vocal Sounds
Classification 97.95% acc. In other tasks CLAP underperformed SoTA by at most
7%. The lowest performing task was ER’s RAVDESS with 64% acc vs a SoTA of 81%.

CLAP performs better in domains like SEC than in others like ER, which was
more evident in (ZS) than in (Best). We hypothesis that SEC tasks perform
better because CLAP’s training data consist of audio captioning datasets,
which mainly include the description of sound events, acoustic scenes,
actions, and objects. On the other hand, the training data is scarce on human
speech and the captions do not describe aspects of its content or context.
Therefore, CLAP underperforms on human speech tasks like KWS and ER. We posit
that as we increase training data and increase human speech-based captioning,
CLAP’s performance on speech datasets will increase.

###  4.3 Effect of freezing CLAP encoders

We studied how freezing the audio and/or text encoder during training affected
performance of the downstream tasks. We computed the average of the CLAP (ZS)
performance across the downstream tasks. The results are shown in 2. The best
Avg. CLAP (ZS) score is obtained by unfreezing both encoders and the worst
score by freezing both encoders. This is expected because unfreezing both
encoders allows them to learn the multimodal information from the pairs.
Surprisingly, unfreezing the text encoder performed better than unfreezing the
audio encoder. Our intuition was that unfreezing the audio encoder would
enable learning beyond the SEC coming from the pretrained AudioSet
information. However, unfreezing the text encoder was better for CLAP, and a
similar insight was found for CLIP models in Computer Vision [27]. This
valuable finding suggests that, under the CLAP learning paradigm, it is
possible to use an audio encoder of choice and turn it into a Zero-Shot
classifier.

Audio encoder (frozen) |  Text encoder (frozen) |  Avg. ZS score |  ESC50 (acc)
---|---|---|---
✓ | ✓ | 0.2809 | 0.5555
✗ | ✓ | 0.2818 | 0.6415
✓ | ✗ | 0.3109 | 0.7631
✗ | ✗ | 0.3265 | 0.826
Table 2:  Effect of freezing text and/or audio encoders on CLAP (ZS)
performance across all tasks and ESC50.

###  4.4 Changing prompts in Zero-Shot evaluation

The training data of CLAP consists of natural language captions containing one
or more sentences. However, the vast majority of datasets have class labels
defined by a few words– dog barking’ and ‘sneezing’. Using single words
instead of language description affects how Zero-Shot learning transfers. To
overcome this distribution difference, we used standard template prompts ‘This
is a sound of [class label]’. Experimentally, we found that using appropriate
prompts improved Zero-Shot performance of CLAP. For example, Table 3, shows
how changing prompts for ESC50 leads to a 5% acc increase in performance.

Prompt |  ESC50 (acc)
---|---
‘i can hear [class label]’ | 0.786
‘this is an audio of [class label]’ | 0.8005
‘[class label]’ | 0.812
‘this is [class label]’ | 0.8135
‘this is a sound of [class label]’ | 0.826
Table 3:  Effect of different prompts on ESC50 (ZS).

##  5 Conclusion

We introduced CLAPfor learning audio concepts from natural language
supervision. CLAP does not require gold standard class labels for training,
enables flexible class prediction, and generalizes to multiple downstream
tasks. The training data consists of 128k audio-text pairs which is at least
0.001% smaller than what similar Computer Vision models used. Nonetheless,
CLAP establishes SoTA in Zero-Shot performance and SoTA in supervised
performance for 5 tasks. Hence, CLAP shows potential for building an audio
foundation model that can learn by natural language supervision and can
generalize to a wide range of tasks while achieving SoTA performance.

## References

  * [1] Richard F Lyon,  “Machine hearing: An emerging field [exploratory dsp],”  IEEE signal processing magazine, vol. 27, pp. 131–139, 2010.
  * [2] Annamaria Mesaros, Aleksandr Diment, Benjamin Elizalde, Toni Heittola, Emmanuel Vincent, Bhiksha Raj, and Tuomas Virtanen,  “Sound event detection in the dcase 2017 challenge,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 6, pp. 992–1006, 2019.
  * [3] Yu Zhang, Daniel S Park, Wei Han, James Qin, Anmol Gulati, Joel Shor, Aren Jansen, Yuanzhong Xu, Yanping Huang, Shibo Wang, et al.,  “Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition,”  arXiv preprint arXiv:2109.13226, 2021.
  * [4] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.,  “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”  arXiv preprint arXiv:2110.13900, 2021.
  * [5] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  Advances in Neural Information Processing Systems, vol. 33, pp. 12449–12460, 2020.
  * [6] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.,  “Learning transferable visual models from natural language supervision,”  in International Conference on Machine Learning. PMLR, 2021, pp. 8748–8763.
  * [7] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.,  “Florence: A new foundation model for computer vision,”  arXiv preprint arXiv:2111.11432, 2021.
  * [8] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig,  “Scaling up visual and vision-language representation learning with noisy text supervision,”  in International Conference on Machine Learning. PMLR, 2021, pp. 4904–4916.
  * [9] Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello,  “Wav2clip: Learning robust audio representations from clip,”  in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 4563–4567.
  * [10] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel,  “Audioclip: Extending clip to image, text and audio,”  in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 976–980.
  * [11] Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter,  “Audio set: An ontology and human-labeled dataset for audio events,”  in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 776–780.
  * [12] Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W Schuller, Christian J Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, et al.,  “Hear 2021: Holistic evaluation of audio representations,”  arXiv preprint arXiv:2203.03022, 2022.
  * [13] Qiuqiang Kong, Yong Xu, Wenwu Wang, and Mark D Plumbley,  “Sound event detection of weakly labelled data with cnn-transformer and automatic threshold optimization,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2450–2460, 2020.
  * [14] Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer,  “Efficient training of audio transformers with patchout,”  arXiv preprint arXiv:2110.05069, 2021.
  * [15] Yoonchang Han, Jeongsoo Park, and Kyogu Lee,  “Convolutional neural networks with binaural representations and background subtraction for acoustic scene classification,”  the Detection and Classification of Acoustic Scenes and Events (DCASE), pp. 1–5, 2017.
  * [16] Cristina Luna-Jiménez, Ricardo Kleinlein, David Griol, Zoraida Callejas, Juan M. Montero, and Fernando Fernández-Martínez,  “A proposal for multimodal emotion recognition using aural transformers and action units on ravdess dataset,”  Applied Sciences, vol. 12, no. 1, 2022.
  * [17] Byeonggeun Kim, Simyung Chang, Jinkyu Lee, and Dooyong Sung,  “Broadcasted residual learning for efficient keyword spotting,”  arXiv preprint arXiv:2106.04140, 2021.
  * [18] Yuan Gong, Jin Yu, and James Glass,  “Vocalsound: A dataset for improving human vocal sounds recognition,”  in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 151–155.
  * [19] Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra,  “Fsd50k: An open dataset of human-labeled sound events,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 829–852, 2022.
  * [20] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen,  “Clotho: an audio captioning dataset,”  in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 736–740.
  * [21] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim,  “AudioCaps: Generating Captions for Audios in The Wild,”  in NAACL-HLT, 2019.
  * [22] Irene Martín-Morató and Annamaria Mesaros,  “What is the ground truth? reliability of multi-annotator data for audio tagging,”  in 2021 29th European Signal Processing Conference (EUSIPCO). IEEE, 2021, pp. 76–80.
  * [23] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley,  “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,”  IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 28, pp. 2880–2894, jan 2020.
  * [24] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “BERT: Pre-training of deep bidirectional transformers for language understanding,”  in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, June 2019, pp. 4171–4186, Association for Computational Linguistics.
  * [25] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.,  “Huggingface’s transformers: State-of-the-art natural language processing,”  arXiv preprint arXiv:1910.03771, 2019.
  * [26] Diederik P. Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  in ICLR (Poster), 2015.
  * [27] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer,  “Lit: Zero-shot transfer with locked-image text tuning,”  arXiv preprint arXiv:2111.07991, 2021.

##  Appendix A Training datasets

FSD50k [19] is a sound event classification dataset with audio clips from
freesound.org. The duration of the clips ranges from 0.3 to 30 seconds. We
used the 36k clips from training and validation. We constructed the caption
for each clip by concatenating the two sentences the associated title and
description in the metadata. We ignored the class label.
ClothoV2 [20] is an audio captioning dataset consisting of 7k audio clips. The
duration of the clips range from 15 to 30 seconds. Each clip has 5 captions
annotated by different participants. Thus, we created 5 pairs for each clip
extending the number of audio-text pairs by 5 times.
AudioCaps [21] is an audio captioning dataset consisting of 46k audio clips
from AudioSet. The duration of the clips is 10 seconds. Each clip has a
caption annotated via crowd-sourcing.
MACS [22] is an audio captioning dataset consisting of 4k audio clips. The
duration of the clips is 10 seconds. Each clip is captioned by multiple
participants. Similar to ClohtoV2, we paired the same audio with a each of
their associated captions to create a larger set of pairs consisting of 17k.
At the time of downloading the datasets, not all clips were available from the
web links.

Dataset | Pairs |  Unique audios |  Unique captions
---|---|---|---
FSD50k | 36,796 | 36,796 | 36,796
ClothoV2 | 29,646 | 5,929 | 29,646
AudioCaps | 44,292 | 44,292 | 44,292
MACS | 17,276 | 3,930 | 17,276
| 128,010 | 90,947 | 128,010
Table 4:  Training dataset statistics.  Domain | Dataset | Files | Dur. (secs) | Classes | Metric | Setup
---|---|---|---|---|---|---
Sound Event Classification (SEC) | ESC50 | 2k | 5 | 50 | ACC | 5 folds
FSD50K |  51k | 0.3 - 30 | 200 | mAP | train/val/test
UrbanSound8K |  8k |  $\leq$ 4 | 10 | ACC | 10 folds
DCASE2017 Task4 | 52k | 10 | 17 | ACC | train/val/test
AudioSet |  $\sim$2M | 10 | 527 | mAP | train/val/test
Music | GTZAN Music Speech | 120 | 30 | 2 | ACC | 10 folds
GTZAN Music Genre | 1k | 30 | 10 | ACC | 10 folds
Mridangam Stroke |  7k | 0.81 | 10 | ACC | 5 folds
Mridangam Tonic |  7k | 0.81 | 6 | ACC | 5 folds
Instrument Classification |  Beijing Opera Percussions | 236 | 4.77 | 4 | ACC | 5 folds
Acoustic Scene Classification | TUT 2017 | 6.3k | 10 | 15 | ACC | train/val/test
Emotion Recognition | CREMA-D |  7k | 5 | 6 | ACC | 5 folds
RAVDESS |  2.5k |  $\leq$ 5 | 8 | ACC | 5 folds
Keyword |  |  |  |  |  |
Spotting | Speech Commands | 100k | 1 | 12 | ACC | train/val/test
Vocal Sound Classification |  Vocal Sound |  21k | 5 | 6 | ACC | train/val/test
Speaker Counting | LibriCount 10 | 5k | 5 | 11 | ACC | 5 folds
Table 5:  Details from the 16 datasets used as Downstream Tasks.

##  Appendix B Downstream datasets

ESC50 is an environmental classification dataset comprising of 50 events. The
dataset consists of 2k files of 5 seconds each. The evaluation setup is 5 fold
cross validation and the evaluation metric is accuracy.
FSD50K is a sound event classification dataset comprising of 200 events. The
dataset consists of 51k files ranging from 0.3 to 30 seconds each. The
evaluation setup is train/val/test and the evaluation metric is mAP.
UrbanSound8K is urban sound classification dataset comprising of 10 sounds.
The dataset consists of 8k files of 4 seconds each. The evaluation setup is 10
fold cross validation and the evaluation metric is accuracy.
DCASE2017 Task4 is a sound event classification dataset comprising of 17
sounds recorded in domestic environment. The dataset consists of 30k files of
10 seconds each. The evaluation setup is train/val/test and the evaluation
metric is accuracy.
AudioSet is a sound event classification dataset comprising of 527 sounds from
YouTube videos. The dataset consists of 2M files of 10 seconds each. The
evaluation setup is train/val/test and the evaluation metric is accuracy.
TUT 2017 is an acoustic scene classification dataset comprising of 15 acoustic
scenes in both outdoor and indoor environment. The dataset consists of 52k
files of 10 seconds each. The evaluation setup is train/val/test and the
evaluation metric is accuracy.
GTZAN Music Speech is a binary classification dataset where the aim is to
distinguish between human speech and music. The dataset consists of 120 files
of 30 seconds each. The evaluation setup is 10 fold cross validation and the
evaluation metric is accuracy.
GTZAN Genres is music genre classification dataset comprising of 10 genres.
The dataset consists of 1k files of 30 seconds each. The evaluation setup is
10 fold cross validation and the evaluation metric is accuracy.
Mridangam Stroke is music stroke classification dataset comprising of 10
strokes from Mridangam (pitched percussion instrument). The dataset consists
of 1k files of 0.81 seconds each. The evaluation setup is 5 fold cross
validation and the evaluation metric is accuracy.
Mridangam Tonic is music tonic classification dataset comprising of 6 tonics
from Mridangam (pitched percussion instrument). The dataset consists of 1k
files of 0.81 seconds each. The evaluation setup is 5 fold cross validation
and the evaluation metric is accuracy.
Beijing Opera Percussions is an instrument classification dataset comprising
of 4 percussion instruments from Beijing Opera. The dataset consists of 236
files of 4.77 seconds each. The evaluation setup is 5 fold cross validation
and the evaluation metric is accuracy.
CREMA-D is an emotion recognition dataset comprising of 6 emotions. The
dataset consists of 7k files of 5 seconds each. The evaluation setup is 5 fold
cross validation and the evaluation metric is accuracy.
RAVDESS is an emotion recognition dataset comprising of 8 emotions. The
dataset consists of 2.5k files of 5 seconds each. The evaluation setup is 5
fold cross validation and the evaluation metric is accuracy.
Speech Commands V2 is an keyword spotting dataset comprising of 13 commands.
The dataset consists of 100k files of 1 seconds each. The evaluation setup is
train/val/test and the evaluation metric is accuracy.
Vocal Sound is a human vocal sound classification dataset comprising of 6
vocalizations. The dataset consists of 21k files of 5 seconds each. The
evaluation setup is train/val/test and the evaluation metric is accuracy.
LibriCount is a speaker count estimation dataset comprising of simulated
cocktail party environment audios consisting of 0 to 10 speakers. The dataset
consists of 5k files of 5 seconds each. The evaluation setup is 5 fold cross
validation and the evaluation metric is accuracy.

| Sound Event Classification | Music
---|---|---
Model | ESC50 | FSD50K | US8K |  DCASE17 Task 4 |  Music Speech |  Music Genres |  Mri. Stroke |  Mri. Tonic
YAMNet | 0.8375 | - | - | - | 0.969 | 0.847 | - | -
Open L3 | 0.7505 | 0.4470 | 0.7823 | - | 0.969 | 0.879 | 0.9666 | 0.9369
Wav2CLIP | 0.7589 | 0.3617 | - | - | 0.946 | 0.748 | 0.9471 | 0.8289
PaNN | 0.9085 | - | - | - | 0.992 | 0.860 | 0.9390 | 0.8244
Wav2Vec2 | 0.5610 | 0.1164 | - | - | 0.946 | 0.780 | 0.9432 | 0.8283
CLAP (S) | 0.9310 | 0.5905 | 0.8389 | 0.5330 | 1.0 | 0.7930 | 0.7754 | 0.6391
CLAP (F) | 0.9670 | 0.5859 | 0.8796 | 0.5938 | 1.0 | 0.9130 | 0.9794 | 0.9534

|  Instrument Classification |  Acoustic Scene Classification | Emotion Recognition |  Keyword Spotting |  Human Vocalization |  Speaker Estimation
---|---|---|---|---|---|---
Model |  Beijing Opera |  TUT 2017 |  CRE MA-D |  RAV DESS |  Speech Comm. |  Vocal Sounds |  Libri Count
YAMNet | 0.941 | - | 0.453 | 0.479 | 0.4104 | - | 0.6526
OpenL3 | 0.975 | - | 0.550 | 0.604 | 0.7634 | - | 0.6414
Wav2CLIP | 0.936 | - | 0.512 | 0.684 | 0.3466 | - | 0.5276
PaNN | 0.911 | - | 0.555 | 0.429 | 0.6182 | - | 0.6516
Wav2Vec2 | 0.907 | - | 0.6562 | - | 0.8785 | - | 0.6921
CLAP (S) | 0.7754 | 0.7099 | 0.2830 | 0.4515 | 0.3708 | 0.8411 | 0.5715
CLAP (F) | 0.9026 | 0.7463 | 0.6834 | 0.6436 | 0.9683 | 0.9795 | 0.7783

Table 6:  CLAP’s Supervised Feature Extraction (S) and Supervised Finetune (F) performance against SoTA models.  Dataset | Audio captions
---|---
ClothoV2 | A bow playing a stringed instrument in a one note tone repeatedly before violins join to create the melody
ClothoV2 | An insect buzzing in the foreground as birds chirp in the background
ClothoV2 | A camp fire crackles as the flames burn branches and leaves
AudioCaps | Several sirens are wailing and a horn is honked twice
AudioCaps | Church bells chime loudly and repeatedly
AudioCaps | Beating drum getting faster than children voices and clapping then adult male voice
FSD50K | Water dripping in a cave or underground temple
FSD50K | Canada geese flying down and landing near a lakeshore. Recorded with an Olympus LS-14.
FSD50K | Leaves falling in a forest near a pond. Recorded in October 2017 in a German forest using a Zoom H2n.
MACS | Two people having a conversation nearby while a lot of adults and a child talk far away
MACS | Birds are making a lot of noises and a distant child yells
MACS | Dog barks followed by adults talking and children voices
Table 7:  Randomly sampled raw captions from each dataset.

##  Appendix C Discussion

###  C.1 Batch size and CLAP performance

Appropriate batch size is a key contributor in performance of any contrastive
learning methods that uses positive-negative pairs. In computer vision
literature, larger batch size has shown to improve performance [6, 7, 27].

We use Tesla V100 16 GB GPUs for this analysis. The number of GPUs used vary
from 4 to 24 for batch size from 32 to 768. We measure the zero-shot
performance by computing average performance across N downstream tasks listed
in table 5. Our findings also indicate that the larger batch size does lead to
improved CLAP performance. However, we saw decreased performance with batch
size of 768. This might be an anamoly in an increasing zero-shot performance
trend. We leave the larger batch size investigation to future work.

Fig. 2: Effect of batch size on zero-shot performance

###  C.2 Training with AudioSet

AudioSet is a multilabled sound event dataset with 2M audio clips, but it does
not come with a text descriptions per clip, thus making it complicated to
extract audio and text pairs for training. We tried constructing the text
description with the title and the class label(s). However, adding about 1.7M
pairs to the existing 128k pairs resulted in a performance dropped in the
overall zero-shot numbers. For example, ESC50 performance dropped from 82.6%
to 67.15% acc and US8K dropped from 73.24% to 70.93% acc. Only Speech Commands
V2 (SCV2) performance improved from 10% to 15% acc. Perhaps due to the large
amount of audio containing speech in AudioSet. The quality of audio-text pairs
is key in training CLAP. In AudioSet, often the YouTube titles and
descriptions do not describe the acoustic content of the video segment under
consideration but instead describe the video as a whole. More intelligent ways
of generating descriptions can benefit CLAP training with AudioSet and other
many similar datasets. In general, finding helpful training data for CLAP
based on public datasets is difficult, thus relying on large-scale noisy pairs
is the only scalable approach.

##  Appendix D Acknowledgements

We thank Hamid Eghbalzadeh for early discussions and feedback.


# Robustness of a Network of Networks

Jianxi Gao,1,2 Sergey V. Buldyrev,3 Shlomo Havlin,4 and H. Eugene Stanley1
1Center for Polymer Studies and Department of Physics, Boston University,
Boston, MA 02215 USA
2Department of Automation, Shanghai Jiao Tong University, 800 Dongchuan Road,
Shanghai 200240, PR China
3Department of Physics, Yeshiva University, New York, NY 10033 USA
4Minerva Center and Department of Physics, Bar-Ilan University, 52900 Ramat-
Gan, Israel

(25 October 2010 — gbhs25oct.tex)

###### Abstract

Almost all network research has been focused on the properties of a single
network that does not interact and depends on other networks. In reality, many
real-world networks interact with other networks. Here we develop an
analytical framework for studying interacting networks and present an exact
percolation law for a network of $n$ interdependent networks. In particular,
we find that for $n$ Erdős-Rényi networks each of average degree $k$, the
giant component, $P_{\infty}$, is given by
$P_{\infty}=p[1-\exp(-kP_{\infty})]^{n}$ where $1-p$ is the initial fraction
of removed nodes. Our general result coincides for $n=1$ with the known Erdős-
Rényi second-order phase transition for a single network. For any $n\geq 2$
cascading failures occur and the transition becomes a first-order percolation
transition. The new law for $P_{\infty}$ shows that percolation theory that is
extensively studied in physics and mathematics is a limiting case ($n=1$) of a
more general general and different percolation law for interdependent
networks.

In recent years dramatic advances in the field of complex networks have
occurred Strogatz1998 ; bara2000 ; Callaway2000 ; Albert2002 ; Cohen2000 ;
Newman2003 ; Dorogovtsev2003 ; song2005 ; Pastor2006 ; Caldarelli2007 ;
Barrat2008 ; Shlomo2010 ; Neman2010 . The internet, airline routes, and
electric power grids are all examples of networks whose function relies
crucially on the connectivity between the network components. An important
property of such systems is their robustness to node failures. Almost all
research has been concentrated on the case of a single or isolated network
which does not interact with other networks. Recently, based on the motivation
that modern infrastructures are becoming significantly more dependent on each
other, a system of two coupled interdependent networks has been studied
Sergey2010 . A fundamental property of interdependent networks is that when
nodes in one network fail, they may lead to the failure of dependent nodes in
other networks which may cause further damage in the first network and so on,
leading to a global cascade of failures. Buldyrev et al. Sergey2010  developed
a framework for analyzing robustness of two interacting networks subject to
such cascading failures. They found that interdependent networks become
significantly more vulnerable compared to their noninteracting counterparts.

For many important examples, more than two networks depend on each other. For
example, diverse infrastructures are coupled together, such as water and food
supply, communications, fuel, financial transactions, and power stations
Peerenboom2001 ; Rinaldi2001 ; Rosato2008 ; Alessandro2010 . For further
examples see Section i@ in the Supplementary Information (SI). Understanding
the robustness due to such interdependencies is one of the major challenges
for designing resilient infrastructures.

We introduce here a model system, comprising a network of $n$ coupled
networks, where each network consists of $N$ nodes (See Fig. 1). The $N$ nodes
in each network are connected to nodes in neighboring networks by
bidirectional dependency links, thereby establishing a one-to-one
correspondence as illustrated in Fig. 2 in SI. We develop a mathematical
framework to study the robustness of this “network of networks” (NON). We find
an exact analytical law for percolation of a NON system composed of $n$
coupled randomly connected networks. Our result generalizes the known Erdős-
Rényi (ER) ER1959 ; ER1960 ; Bollob1985  result for the giant component of a
single network and the $n=2$ result found recently Sergey2010 , and shows that
while for $n=1$ the percolation transition is a second order transition, for
$n>1$ cascading failures occur and the transition becomes a first order
transition. Our results for $n$ interdependent networks suggest that the
classical percolation theory extensively studied in physics and mathematics is
a limiting case of a general theory of percolation in NONs, or networks with
multiple types of connectivity links. This general theory has many novel
features which are not present in classical percolation.

Additionally, we find:

(i) the robustness of NON significantly decreases with $n$, and

(ii) for a network of $n$ ER networks all with the same average degree $k$,
there exists a minimum degree $k_{\min}(n)$ increasing with $n$, below which
$p_{c}=1$, i.e., for $k<k_{\min}$ the NON will collapse once any finite number
of nodes fail. We find an analytical expression for $k_{min}(n)$, which
generalizes the known result $k_{\min}(1)=1$ for ER below which the network
collapses. We also discuss the critical effect of loops in the NON structure.

Real-world interacting networks (See SI for more details) are characterized by
complex correlations and a variety of organizational principles governing
their internal structures and interdependencies. Once these correlations are
quantified from the statistical analysis of actual data bases and the
organizational principles are specified from the engineering literature, real
world networks can be studied by computer simulations. These simulations will
have many parameters and therefore their outcome will also require complex
interpretation. It is therefore very important to develop simple analytically
tractable models for the robustness of interdependent networks against which
such simulations can be tested. Well-known examples of simplified models that
both demonstrate a fundamental phenomenon and significantly advance our
knowledge are the Ising model in statistical mechanics and the Erdős-Rényi
model in graph theory. This paper presents a simple model that can serve as a
benchmark for further studies of NONs.

We assume that a network $m$ $(m=1,2,...,n)$ in the NON is a randomly
connected network with a degree distribution $P_{m}(k)$. We call a pair of
networks A and B a fully interdependent pair if it satisfies the following
condition: each node $A_{i}$ of network A is connected to one and only one
node $B_{i}$ in network B by a bidirectional dependency link such that if node
$A_{i}$ fails, $B_{i}$ also fails and vice versa. Since the number of nodes in
each network is the same, these bidirectional links establish a one-to-one
correspondence between the nodes in the networks belonging to an
interdependent pair. Each node of the NON represents a network and each edge
represents a fully interdependent pair of networks. First, we will discuss the
case when the NON is a loopless tree of $n$ networks (Fig. 1). The dependency
edges in such a NON establish a unique one-to-one correspondence between the
nodes of any two networks not necessarily belonging to the same fully
interdependent pair. This one-to-one correspondence established by the
interdependency links between the nodes of different networks in the loopless
NON uniquely maps any set of nodes in one of the networks to a set of nodes
(which we will call an image of the original set) in any other network of the
NON (See SI for more details). In principle, the assumption of full
interdependence allows one to collapse all the networks of the NON onto a
single network with multiple types of links.

We assume that in order to remain functional a node must belong to a
sufficiently large mutually connected cluster Sergey2010  (See detailed
definition in SI). We will show that a large mutually connected cluster which
includes a finite fraction of the nodes in each network exists only in
networks of sufficiently high mean degree. We call this large mutually
connected cluster a mutual giant component, and we postulate that only nodes
in the mutual giant component remain functional.

We assume that due to an attack or random failure only a fraction of nodes $p$
in one particular network which we will call the root of the NON. We can now
observe a cascade of failures caused by the failure of the dependent nodes in
the networks connected directly to the root by the edges of the NON. The
damage will further spread to more distant networks. Moreover, fragmentation
of each network caused by the removal of certain nodes will cause malfunction
of other nodes which will now belong to small isolated clusters. This
malfunction will cause dependent nodes in neighboring networks to malfunction
as well. Depending on the time scales of these processes, the damage can
spread across the NON back and forth, which we can visualize as cascades of
failures, as shown in Fig. 3 of the SI section. At the end of this process
only the mutual giant component of the NON, if it exists, will remain
functional.

We now introduce generating functions Sergey2010 ; Newman2001 ; Newman2002PRE
; Shao2008 ; Shao2009  of each network, $G_{m0}(z)=\sum P_{m}(k)z^{k}$, and
the generating function of the associated branching process,
$G_{m1}(z)=G^{\prime}_{m0}(z)/G^{\prime}_{m0}(1)$. It is known Newman2001 ;
Newman2002PRE  that the generating functions of a randomly connected network
once a fraction $1-p$ of nodes has been randomly removed are the same as the
generating functions of the original network with the new argument $1-p(1-z)$.
Furthermore it is known Shao2008 ; Shao2009  that the fraction of nodes in the
giant component of a single randomly connected network is
$\mu_{\infty,1}=pg_{m}(p)$, where $g_{m}(p)=1-G_{m0}(1-p(1-f_{m}))$ and
$f_{m}$ satisfies a transcendental equation
$f_{m}(p)=G_{m1}(1-p(1-f_{m}(p)))$.

We next prove that the fraction of nodes, $\mu_{\infty,n}$, in the mutual
giant component of a NON composed of $n$ networks is the product:

$\mu_{\infty,n}=p\prod_{m=1}^{n}g_{m}(x_{m}),$ (1)

where each $x_{m}$ satisfies the equation

$x_{m}=\mu_{\infty,n}/g_{m}(x_{m}).$ (2)

The system of $n+1$ equations (1) and (2) defines $n+1$ unknowns:
$\mu_{\infty,n},x_{1},x_{2},...,x_{n}$ as functions of $p$ and the degree
distributions $P_{m}(k)$.

We derive Eqs. (1) and (2) by mathematical induction. (An alternative proof is
given in the SI). Indeed, for $n=1$, Eqs. (1) and (2) follow directly from the
definition of $\mu_{\infty,1}$. Assuming that the formulas are valid for a NON
of $n-1$ networks we will prove that they are valid also for a NON of $n$
networks. A loopless NON of $n$ networks can be always represented as one of
its networks connected by a single edge to the other $n-1$ networks in the
NON. All the nodes in the $n$-th network, which do not belong to the image of
the mutual giant component $\mu_{\infty,n-1}$ of the NON of $n-1$ networks
will stop to function. The fraction of the nodes in the image of this mutual
giant component onto the $n$-th network satisfies the equation
$x_{1,n}=\mu_{\infty,n-1}(p)$. The fraction of nodes belonging to the giant
component of this dependency image is $\mu_{1,n}=x_{1,n}g_{n}(x_{1,n})$. Only
the nodes in the NON of $n-1$ networks which belong to the dependency image of
the giant component of the $n$-th network will remain functional. Due to the
randomness of the connectivity links in different networks, this dependency
image can be represented as a random selection of the fraction
$g_{n}(x_{1,n})$ out of the originally survived nodes, or as random selection
of $p_{1}=pg_{n}(x_{1,n})$ fraction of nodes in one of the networks comprising
the NON of $n-1$ networks. The fraction of nodes in the new mutual giant
component of the NON of $n-1$ networks corresponding to this new random
selection will be $\mu_{\infty,n-1}(p_{1})$. The image of this mutual giant
component in the $n$-th network is equivalent to a random selection of
$x_{2,n}=\mu_{\infty,n-1}(p_{1})/g_{n}(x_{1,n})$ fraction of nodes out of the
entire $n$-th network. As we continue this process, the sequence of giant
components $\mu_{j,n}$ in the $n$-th network, randomly selected sets $x_{n}$
in the $n$-th network and randomly selected sets $p_{j}$ in the NON of $n-1$
networks will satisfy the recursion relations
$x_{j+1,n}=\mu_{\infty,n-1}(p_{j})/g_{n}(x_{j,n})$,
$\mu_{j+1,n}=x_{j+1,n}g_{n}(x_{j+1,n})$, $p_{j+1}=pg_{n}(x_{j+1,n})$.

In the limit $j\to\infty$, this process will converge, i.e. all the parameters
in the two successive steps will coincide: $x_{j+1,n}\to x_{j,n}\equiv x_{n}$,
$p_{j}\to pg_{n}(x_{n})$ and $\mu_{\infty,n-1}(p_{j})\to\mu_{\infty,n}$. Then
$x_{n}=\mu_{\infty,n}/g_{n}(x_{n})$ which is identical to the last equation in
Eqs. (2) and $\mu_{\infty,n-1}(p_{j})\to
pg_{n}(x_{n})\prod_{m=1}^{n-1}g_{m}(x_{m})=p\prod_{m=1}^{n}g_{m}(x_{m})\equiv\mu_{\infty,n}$
which is identical to Eq. (1). By the assumption of induction
$x_{m}=\mu_{\infty,n-1}(p_{j})/g_{m}(x_{m})=\mu_{\infty,n}/g_{m}(x_{m})$ which
completes the set of Eqs. (2). Finally $\mu_{j+1,n}\to
x_{n}g_{n}(x_{n})=\mu_{\infty,n}/g_{n}(x_{n})$ and thus the fraction of nodes
in the giant $n$-th network coincides with the mutual giant component in the
NON of $n-1$ networks. The SI presents an alternative analytical derivation of
Eqs. (1) and (2), which represent a certain type of cascading failures. The SI
also presents simulation results which agree well with the theory (Figs. 5 and
6 in SI).

For the case of a NON with loops, the closed path of fully interdependent
pairs starting form a network A and ending at the same network A will
establish a dependence of nodes $A_{i}$ on node $A_{j_{i}}$, where $j_{i}$ is
a transposition of $i$. Then the failure of single node $i$ will cause an
entire cycle in the transposition to fail. The average size of a cycle in the
transposition of $N$ elements grows as $N/\ln N$, so the initial failure of
$\ln N$ nodes will cause almost all the nodes of the NON to fail without
taking into account any connectivity links which will cause additional damage.
So the NON with loops is unstable against removal of an infinitely small
fraction of nodes unless the transposition $j_{i}$ is not random. In case when
the transposition $j_{i}$ is trivial, $j_{i}=i$, we have the same one-to-one
correspondence between the nodes as in the loopless NON and then Eq. (1) and
(2) are valid. This is since in our proof we did not use any other property of
a NON except the unique one-to-one correspondence of the nodes in different
networks.

The case of NON of $n$ Erdős-Rényi (ER) ER1959 ; ER1960 ; Bollob1985  networks
with average degrees $k_{1},k_{2},...k_{i},...,k_{n}$ can be solved
explicitly. In this case, we have $G_{1,i}(x)=G_{0,i}(x)=\exp[k_{i}(x-1)]$
Newman2002PRE . Accordingly $g_{i}(x_{i})=1-\exp[k_{i}x_{i}(f_{i}-1)]$, where
$f_{i}=\exp[k_{i}x_{i}(f_{i}-1)]$ and thus $g_{i}(x_{i})=1-f_{i}$. Using Eq.
(2) for $x_{i}$ we get

$f_{i}=\exp[-pk_{i}\prod_{j=1}^{n}(1-f_{j})],i=1,2,...,n.$ (3)

These equations can be solved analytically, as shown in detail in the SI
section. They have only a trivial solution ($f_{i}=1$) if $p<p_{c}$, where
$p_{c}$ is the mutual percolation threshold. When the $n$ networks have the
same average degree $k$, $k_{i}=k$ ($i=1,2,...,n$), we obtain from Eq. (3)
that $f_{c}\equiv f_{i}(p_{c})$ satisfies

$f_{c}=e^{\frac{f_{c}-1}{nf_{c}}}.$ (4)

where the solution can be expressed in term of the Lambert function $W(x)$
Lambert1758 ; Corless1996 , $f_{c}=-[nW(-\frac{1}{n}e^{-\frac{1}{n}})]^{-1}$.

Once $f_{c}$ is known, we obtain $p_{c}$ and $\mu_{\infty,n}\equiv P_{\infty}$
by substituting $k_{i}=k$ into Eq. (S10) of the SI section

$\begin{array}[]{lcl}p_{c}=[nkf_{c}(1-f_{c})^{(n-1)}]^{-1},&\mbox{}&\\\
P_{\infty}=\frac{1-f_{c}}{nkf_{c}}.&\mbox{}&\\\ \end{array}.$ (5)

For $n=1$ we obtain the known result $p_{c}=1/k$ of Erdős-Rényi ER1959 ;
ER1960 ; Bollob1985 . Substituting $n=2$ in Eqs. (4) and (5) one obtains the
exact results of Sergey2010 .

To analyze $p_{c}$ as a function of $n$, we find $f_{c}$ from Eq. (4) and
substitute it into Eq. (5), and we obtain $p_{c}$ as a function of $n$ for
different $k$ values, as shown in Fig. 2(a). It is seen that the NON becomes
more vulnerable with increasing $n$ or decreasing $k$ ($p_{c}$ increases when
$n$ increases or $k$ decreases). Furthermore, for a fixed $n$, when $k$ is
smaller than a critical number $k_{min}(n)$, $p_{c}\geq 1$ meaning that for
$k<k_{min}(n)$, the NON will collapse even if a single node fails. Fig. 2 (b)
shows the minimum average degree $k_{\min}$ as a function of the number of
networks $n$. From Eq. (5) we get the minimum of $k$ as a function of $n$

$k_{\min}(n)=[nf_{c}(1-f_{c})^{(n-1)}]^{-1},$ (6)

Note that Eq. (6) together with Eq. (4) yield the value of $k_{\min}(1)=1$,
reproducing the known ER result, that $\langle k\rangle=1$ is the minimum
average degree needed to have a giant component. For $n=2$, Eq. (6) yields the
result obtained in Sergey2010 , i.e., $k_{\min}=2.4554$.

From Eqs. (1)-(3) we obtain the percolation law for the order parameter, the
size of the mutual giant component for all $p$ values and for all $k$ and $n$,

$\mu_{\infty,n}\equiv P_{\infty}=p[1-\exp(-kP_{\infty})]^{n}.$ (7)

The solutions of equation (7) are shown in Fig. 3 for several values of $n$.
Results are in excellent agreement with simulations. The special case $n=1$ is
the known ER percolation law for a single network ER1959 ; ER1960 ; Bollob1985
. Note that Eqs. (4)–(7) are based on the assumption that all $n$ networks
have the same average degree $k$.

In summary, we have developed a framework, Eqs. (1) and (2), for studying
percolation of NON from which we derived an exact analytical law, Eq. (7), for
percolation in the case of a network of $n$ coupled ER networks. Equation (7)
represents a bound for the case of partially dependent networks parshani2010 ,
which will be more robust. In particular for any $n\geq 2$, cascades of
failures naturally appear and the phase transition becomes first order
transition compared to a second order transition in the classical percolation
of a single network. These findings show that the percolation theory of a
single network is a limiting case of a more general case of percolation of
interdependent networks. Due to cascading failures which increase with $n$,
vulnerability significantly increases with $n$. We also find that for any
loopless network of networks the critical percolation threshold and the mutual
giant component depend only on the number of networks and not on the topology
(see Fig. 1(a)). When the NON includes loops, and dependency links are random,
$p_{c}=1$ and no mutual giant component exists.

## References

  * (1) Watts D. J. & Strogatz S. H. Nature 393, 440-442 (1998).
  * (2) Albert R., Jeong H. & Barabási A. L. Nature 406, 378-382 (2000).
  * (3) Cohen R. et al. Phys. Rev. Lett. 85, 4626–4628 (2000).
  * (4) Callaway D. S. et al. Phys. Rev. Lett. 85, 5468-5471 (2000).
  * (5) Albert R. & Barabási A. L. Rev. Mod. Phys. 74, 47-97 (2002).
  * (6) Newman M. E. J. SIAM Review 45, 167-256 (2003).
  * (7) Dorogovtsev S. N. & Mendes J. F. F. Evolution of Networks: From Biological Nets to the Internet and WWW (Physics) (Oxford Univ. Press, New York, 2003).
  * (8) Song C. et al. Nature 433, 392-395 (2005).
  * (9) Satorras R. P. & Vespignani A. Evolution and Structure of the Internet: A Statistical Physics Approach (Cambridge Univ. Press, England, 2006).
  * (10) Caldarelli G. & Vespignani A. Large scale Structure and Dynamics of Complex Webs (World Scientific, 2007).
  * (11) Barrát A., Barthélemy M. & Vespignani A. Dynamical Processes on Complex Networks (Cambridge Univ. Press, England, 2008).
  * (12) Havlin S. & Cohen R. Complex Networks: Structure, Robustness and Function (Cambridge Univ. Press, England, 2010).
  * (13) Newman M. E. J. Networks: An Introduction (Oxford Univ. Press, New York, 2010).
  * (14) Buldyrev S. V. et al. Nature 464, 1025-1028 (2010).
  * (15) Peerenboom J., Fischer R. & Whitfield R. in Pro. CRIS/DRM/IIIT/NSF Workshop Mitigat. Vulnerab. Crit. Infrastruct. Catastr. Failures (2001).
  * (16) Rinaldi S., Peerenboom J. & Kelly T. IEEE Contr. Syst. Mag. 21, 11-25 (2001).
  * (17) Rosato V. et al. Int. J. Crit. Infrastruct. 4, 63-79 (2008).
  * (18) Vespignani A. Nature 464, 984-985 (2010).
  * (19) Erdős P. & Rényi A. I. Publ. Math. 6, 290-297 (1959).
  * (20) Erdős P. & Rényi A. Publ. Math. Inst. Hung. Acad. Sci. 5, 17-61 (1960).
  * (21) Bollobás B. Random Graphs (Academic, London, 1985).
  * (22) Newman M. E. J. Strogatz S. H. & Watts D. J., Phys. Rev. E 64, 026118 (2001).
  * (23) Newman M. E. J. Phys. Rev. E 66, 016128 (2002).
  * (24) Shao J. et al. Europhys. Lett. 84, 48004 (2008).
  * (25) Shao J. et al. Phys. Rev. E 80, 036105 (2009).
  * (26) Lambert J. H. Acta Helveticae physico mathematico anatomico botanico medica, Band III, 128-168, (1758).
  * (27) Corless R. M. et al. Adv. Computational Maths. 5, 329-359 (1996).
  * (28) Parshani R. et al. Phys. Rev. Lett. 105, 048701 (2010).

Figure 1: (color online) Three types of loopless networks of networks composed
of five coupled networks. All have same percolation threshold and same giant
component. The darker green node is the origin network on which failures
occur.

Figure 2: (a) The critical fraction $p_{c}$ for different $k$ and $n$ and (b)
minimum average degree $k_{\min}$ as a function of the number of networks $n$.
The results of (a) and (b) are obtained using Eqs. (5) and (6) respectively
and are in good agreement with simulations. In simulations $p_{c}$ was
calculated from the number of cascading failures which diverge at $p_{c}$
parshani2010  (see also Fig. 7 in SI). Figure 3:  For loopless NON,
$P_{\infty}$ as a function of $p$ for $k=5$ and several values of $n$. The
results obtained using Eq. (7) agree well with simulations.