Zuheng Kang, Jianzong Wang∗††thanks: ∗ Corresponding author: Jianzong Wang,
<EMAIL_ADDRESS>Junqing Peng, Jing Xiao

# SVVAD: Personal Voice Activity Detection for Speaker Verification

###### Abstract

Voice activity detection (VAD) improves the performance of speaker
verification (SV) by preserving speech segments and attenuating the effects of
non-speech. However, this scheme is not ideal: (1) it fails in noisy
environments or multi-speaker conversations; (2) it is trained based on
inaccurate non-SV sensitive labels. To address this, we propose a speaker
verification-based voice activity detection (SVVAD) framework that can adapt
the speech features according to which are most informative for SV. To achieve
this, we introduce a label-free training method with triplet-like losses that
completely avoids the performance degradation of SV due to incorrect labeling.
Extensive experiments show that SVVAD significantly outperforms the baseline
in terms of equal error rate (EER) under conditions where other speakers are
mixed at different ratios. Moreover, the decision boundaries reveal the
importance of the different parts of speech, which are largely consistent with
human judgments.

Index Terms: voice activity detection, personal VAD, speaker verification

##  1 Introduction

Voice activity detection (VAD) is a task that identifies whether human speech
is present or absent and is often used upstream of other speech components
such as automatic speech recognition (ASR), speaker verification (SV), and
speaker diarization (SD). It aims to reduce the impact of non-speech on
downstream speech tasks and indirectly improve their performance. However,
their goals are different. ASR and SD models need to efficiently and
accurately determine the boundary between speech and non-speech to avoid
missing content. SV is much more complicated because there are more factors
involved.

A typical VAD framework is considered to be a gating module that makes a
speech/non-speech decision for each frame. Early studies focused on signal and
statistical analysis and feature engineering [1, 2, 3, 4]. More recently, the
use of conventional deep learning methods, such as convolutional neural
networks (CNN) and recurrent neural networks (RNN), has shown significant
improvements in detection performance at low signal-to-noise ratios (SNR) [5,
6, 7]. Later, with the introduction of the attention mechanism, the model can
automatically compare the characteristics of speech and noise within audio to
derive more accurate judgments [8, 9]. Some authors use audiovisual
information for VAD detection [10, 11]. Although it improves the performance
of SV by VAD to some extent, these frameworks are insufficient because non-
target speakers are also identified and retained as speech labels. Since the
SV model uses only a single speaker for supervised training and does not
consider multiple speakers, this leads to a significant drop in verification
performance. Therefore, a target speaker-only VAD framework is required.

The personal VAD (PVAD) framework solves this problem by extending the
traditional VAD to recognize only the target speaker part and ignore the non-
target part. The feasibility of PVAD has been demonstrated in several studies.
The author of [12, 13] proposed and improved the concept of PVAD to solve the
problem of ``always running'' models on devices. [14, 15] extends the PVAD to
make it easier to use in ASR applications. Moreover, the concept of PVAD has
also been applied to the task of speech enhancement as a secondary task to
improve the performance of separation [16, 17, 18]. However, these methods
were not applicable to SV. Although the author of [19, 20, 21, 22, 23] tries
to employ SD to recognize the speech of different speakers and then find the
target speaker by some rules, this type of framework is inefficient for
downstream speech tasks due to the complexity of its process.

Traditional PVAD frameworks are inadequate for SV tasks for several reasons:
(1) Traditional PVAD models are trained with frame-by-frame supervision based
on human-assigned or ASR forced alignment labels. However, not every frame
predicted by the VAD model has a positive impact on the SV model. In practice,
segments identified as speech by traditional VAD are sometimes associated with
low SNR or multiple speakers talking simultaneously, which can severely
degrade the performance of SV; (2) These labels are usually set as hard
labels. However, soft labels are more suitable for SV scenarios because
different speech segments contribute differently; (3) In SV, the VAD model
faces a more complex situation and it will be more challenging to take all
factors into account.

To address these issues, we have made the following contributions: (1) We
propose a speaker verification-based voice activity detection (SVVAD)
framework, which manipulates speech features using FiLM [24] according to
their relevance to the SV model; (2) We propose a novel label-free training
method that uses triplet-like losses to avoid the performance degradation
caused by inaccurate human labeling; (3) Extensive experiments demonstrate
that SVVAD achieves significant improvements over the baseline model in terms
of equal error rate (EER) under various conditions where other speakers are
mixed at different ratios, and that the model-generated VAD decision boundary
is highly consistent with human judgment.

##  2 Methodology

###  2.1 Recap of Personal VAD Model and Motivation

Figure 1: The overview of PVAD framework.

To show the advancement of our proposed voice activity detection (VAD)
framework, we need to discuss previous approaches. In the conventional
personal VAD (PVAD) framework [12, 13, 19] (shown in Figure 1), first, a pre-
trained speaker verification (SV) model computes the speaker embedding of the
enrolled target user $\bm{v}^{\left(\mathrm{enr}\right)}$ from his/her
recordings. Then, the PVAD model takes this SV embedding as a priori and the
speech features of the audio to be tested
$\bm{x}_{t}^{\left(\mathrm{tst}\right)}$ as inputs to make predictions
$\bm{p}_{t}$ at frame $t$. It can be trained using binary cross entropy (BCE)
loss. Eventually, this model is capable of determining whether the frame
contains the target speaker.

To further discuss this approach, there is an Assumption of SV: (1) the
enrolled speech is restricted to contain only one speaker's voice, and usually
stored in the form of speaker embeddings; (2) the tested audio includes not
only the target speaker's speech, but also the non-target speaker's speech and
non-speech. Consequently, the PVAD model is required to identify whether the
tested speech contains the target speaker's speech (tss) or its opposite
(ntss) with its speaker embedding $\bm{v}^{\left(\mathrm{enr}\right)}$,
expressed as Equation 1. Where
$\bm{p}_{t}=\left[p_{t}^{\mathtt{tss}},p_{t}^{\mathtt{ntss}}\right]$ can be
represented as a posteriori for two categories.
$\bm{x}^{\left(\mathrm{tst}\right)}\in\mathbb{R}^{T\times F}$,
$\bm{v}^{\left(\mathrm{enr}\right)}\in\mathbb{R}^{E}$. $E$, $F$, $T$ denote
the embedding dimension, feature dimension and sequence length respectively.

$\bm{p}_{t}=\mathrm{PVAD}\left(\bm{x}_{t}^{\left(\mathrm{tst}\right)},\bm{v}^{\left(\mathrm{enr}\right)}\right)$
(1)

Although the vanilla PVAD model can identify the target speaker's speech, it
is originally designed for non-SV scenarios. This framework is not optimal for
SV because it is based on the hypothesis that retaining only the target
speaker's speech and ignoring other sounds can improve the performance of SV.
It is sub-optimal because not all speech of the target speaker contributes
positively to SV performance. Besides, this framework relies heavily on
accurate labels. Therefore, a more advanced VAD model for SV is needed that
can account for more complex situations.

###  2.2 Improving Personal VAD with Speaker Verification

In order to solve the issue described in $\S$ 2.1, we propose a speaker
verification-based voice activity detection (SVVAD) framework with two novel
approaches: (1) The FiLM layer [24] is used to construct soft VAD decisions to
modify the speech features to automatically identify the most informative
parts of the SV model; (2) A label-free training method is introduced to avoid
considering overly complex situations.

Network Architecture for SVVAD Backbone: Figure 2 outlines the network
architecture of the SVVAD backbone. It consists of a PVAD backbone and a FiLM
[24] backend. Seen from the input side, this architecture has two branches
that receive the speech features to be tested
$\bm{x}^{\left(\mathrm{tst}\right)}$ and the enrolled target speaker embedding
$\bm{v}^{\left(\mathrm{enr}\right)}$ inputs respectively. For the speech
feature branch, inspired by [13, 22, 25], we use Conformer [26] (with
$N_{\mathrm{conf}}$ encoder layers) as the speech feature extractor because of
its ability to distinguish the features of different speakers more
efficiently. To ensure consistency in the time dimension, we removed the
subsampling operation from the original Conformer header. In this way, both
the input $\bm{x}_{t}^{\left(\mathrm{tst}\right)}$ and output
$\bm{x}_{t}^{\left(\mathrm{conf}\right)}$ have the same dimension
$\mathbb{R}^{T\times F}$. For the speaker embedding branch, first, a pre-
trained SV model is needed to convert the enrolled speech feature
$\bm{x}^{\left(\mathrm{enr}\right)}$ into speaker embedding
$\bm{v}^{\left(\mathrm{enr}\right)}$ with dimension $\mathbb{R}^{E}$. Since
the embedding size is too large, it can then be reduced to
$\mathbb{R}^{E^{\prime}}$ by a fully connected layer. To merge with another
branch, this embedding will be replicated $T$ times to generate
$\bm{y}_{t}^{\left(\mathrm{enr}\right)}$ with dimension $\mathbb{R}^{T\times
E^{\prime}}$.

The information from these two branches is then combined by concatenation
$\oplus$, with dimension $\mathbb{R}^{T\times\left(E^{\prime}+F\right)}$.
Since the target speaker's information needs to affect every frame in the
tested speech, this problem can be solved by using a self-attentive mechanism,
i.e., through several layers of stacked Transformer encoder blocks (with
$N_{\mathrm{trans}}$ layers). The output $\bm{z}_{t}$ is shown in Equation 2.

Figure 2: The network architecture of SVVAD backbone.

$\bm{z}_{t}=\mathrm{Transformer}\left(\bm{y}_{t}^{\left(\mathrm{enr}\right)}\oplus\bm{x}_{t}^{\left(\mathrm{conf}\right)}\right)$
(2)

The fused information $\bm{z}_{t}$ is then sent to two separate fully
connected layers (FC) to create VAD decisions with
$\gamma\left(\bm{z}_{t}\right)$ and $\beta\left(\bm{z}_{t}\right)$ with
dimension $\mathbb{R}^{T\times 1}$. These decisions are then applied to the
original speech features $\bm{x}_{t}^{\left(\mathrm{tst}\right)}$ by the FiLM
layer [24] to create $\bm{x}_{t}^{\left(\mathrm{vad}\right)}$ with dimension
$\mathbb{R}^{T\times F}$, in Equation 3. Since they have different dimensions,
it is necessary to use one-to-many operations.

$\bm{x}_{t}^{\left(\mathrm{vad}\right)}=\mathrm{FiLM}\left(\bm{z}_{t}\right)=\gamma\left(\bm{z}_{t}\right)\cdot\bm{x}_{t}^{\left(\mathrm{tst}\right)}+\beta\left(\bm{z}_{t}\right)$
(3)

Figure 3: The overview of SVVAD framework.

Label-free Optimization with Triplet-like Loss: To completely avoid
constructing inaccurate strong VAD labels, it is effective to use only the
speaker's label for VAD-label-free training. Figure 3 summarizes the main
structure of the SVVAD framework: the modified speech features
$\bm{x}^{\left(\mathrm{vad}\right)}$ are directly sent into the SV model to
obtain the embedding $\bm{v}^{\left(\mathrm{vad}\right)}$. To achieve label-
free optimization during training, based on the assumptions of SV, we only
need to set different optimization objectives for different cases of the test
speech. Specifically, if the speech to be tested contains the voice of the
target speaker, it is considered a positive sample, otherwise it is a negative
sample. This idea is consistent with the triplet-loss (TL) mechanism [27],
shown in Equation 4. Where $\alpha$ is the margin, $B$ is the batch size.
$\bm{x}_{i}^{a}$, $\bm{x}_{i}^{p}$ and $\bm{x}_{i}^{n}$ is the anchor,
positive and negative sample respectively at sample $i$, and can then be
converted to speaker embeddings $\bm{v}_{i}^{a}$, $\bm{v}_{i}^{p}$ and
$\bm{v}_{i}^{n}$ by SV model. $\lVert\cdot\rVert_{2}$ is 2-norm.
$\left[\cdot\right]_{+}$ is equivalent to $\mathrm{max}\left(\cdot,0\right)$
or $\mathrm{ReLU}\left(\cdot\right)$.

$\mathcal{L}_{\mathrm{TL}}=\sum_{i}^{B}{\left[\lVert\bm{v}_{i}^{a}-\bm{v}_{i}^{p}\rVert_{2}^{2}-\lVert\bm{v}_{i}^{a}-\bm{v}_{i}^{n}\rVert_{2}^{2}+\alpha\right]_{+}}$
(4)

Since in SV, scoring uses the cosine distance instead of mean square error
(MSE) in Equation 4, the triplet loss can be rewritten in terms of the cosine
distance [28] (denoted as TLcos), in Equation 5. The cosine distance
$\mathrm{cos}\left(\cdot\right)$ is expressed as 1 minus its cosine
similarity.

$\mathcal{L}_{\mathrm{TLcos}}=\sum_{i}^{B}{\left[\cos\left(\bm{v}_{i}^{a},\bm{v}_{i}^{p}\right)-\cos\left(\bm{v}_{i}^{a},\bm{v}_{i}^{n}\right)+\alpha\right]_{+}}$
(5)

Since these losses only consider the distance between individual samples and
not their overall statistical properties, they may lead to suboptimal
performance. To this end, Lin's concordance correlation coefficient (CCC) [29]
loss is more reliable and can be used to stabilize the training process, so as
to improve the VAD performance. Then, the Equation 4 can be changed to
Equation 6 (denoted as TLccc), and Equation 5 can be modified to Equation 7, 8
(denoted as TLccos).

$\mathcal{L}_{\mathrm{TLccc}}=\left[\mathrm{CCC}\left(\bm{v}^{a},\bm{v}^{p}\right)-\mathrm{CCC}\left(\bm{v}^{a},\bm{v}^{n}\right)+\alpha\right]_{+}$
(6)

$\mathcal{L}_{\mathrm{TLccos}}=\left\\{\begin{matrix}\mathrm{CCC}\left(\begin{array}[]{c}\cos\left(\bm{v}^{a},\bm{v}^{p}\right)+\alpha,\\\
\cos\left(\bm{v}^{a},\bm{v}^{n}\right)\\\ \end{array}\right)&cond>0\\\
0&cond\leq 0\\\ \end{matrix}\right.$ (7)

$cond=\cos\left(\bm{v}^{a},\bm{v}^{p}\right)-\cos\left(\bm{v}^{a},\bm{v}^{n}\right)+\alpha$
(8)

###  2.3 Training and Inference

Let's start by discussing the inference phase, the model first accepts the
target speaker's embedding $\bm{v}^{\left(\mathrm{enr}\right)}$ and the speech
features to be tested $\bm{x}^{\left(\mathrm{tst}\right)}$ to construct a VAD
decision $\gamma\left(\bm{z}_{t}\right)$ and $\beta\left(\bm{z}_{t}\right)$.
These decisions modify the original speech feature
$\bm{x}^{\left(\mathrm{tst}\right)}$ by FiLM rules to form
$\bm{x}^{\left(\mathrm{vad}\right)}$. It can then be converted into a speaker
embedding $\bm{v}^{\left(\mathrm{vad}\right)}$ by a pre-trained SV model.
Finally, $\bm{v}^{\left(\mathrm{enr}\right)}$ and
$\bm{v}^{\left(\mathrm{vad}\right)}$ will be scored by cosine similarity.

In the training phase, except for the same operations as in the inference
phase, the data-loader will simultaneously construct the anchor, positive and
negative samples $\bm{x}^{a}$, $\bm{x}^{p}$ and $\bm{x}^{n}$ in real-time.
These samples are fed into the trainable VAD and non-trainable SV model to
obtain the speaker embeddings $\bm{v}^{a}$, $\bm{v}^{p}$ and $\bm{v}^{n}$,
which are then trained by the triplet-like losses through Equation 4 to 7.
However, since there is no existing dataset of multi-speaker speech, we will
adopt a special method to generate these samples.

###  2.4 Training Data Generation

Algorithm 1 Training Data Generation Policy

1:Dataset with single-speaker samples

2:Multi-speaker samples with anchor, positive, negative

3:for each iteration in training do

4: Randomly select the anchor speaker ID as $d_{a}$

5: Create anchor samples $S_{a}\leftarrow\mathrm{get}\left(d_{a}\right)$

6: Randomly select the number of speakers ($\leq 3$) for positive or negative,
and their corresponding speaker IDs as $D_{*}$, where $d_{a}\in D_{p}$ and
$d_{a}\notin D_{n}$

7: Create empty samples for positive and negative $S_{*}\leftarrow\phi$

8: while $\mathrm{length}\left(S_{*}\right)<$ a fixed duration do

9: /* the previous and current states are different */

10: if $\mathrm{random}\left(\right)<p_{\mathrm{spk}}$ then

11: $id\leftarrow\mathrm{select}\left(D_{*}\right)$

12: $s_{\mathrm{id}}\leftarrow\mathrm{get}\left(id\right)$

13: if $\mathrm{random}\left(\right)<p_{\mathrm{overlap}}$ then

14: $S_{*}\leftarrow\mathrm{overlapConcat}\left(S_{*},s_{\mathrm{id}}\right)$

15: else

16: $S_{*}\leftarrow\mathrm{concat}\left(S_{*},s_{\mathrm{id}}\right)$

17: end if

18: else

19: $s_{\phi}\leftarrow\mathrm{get}\left(\phi\right)$

20: $S_{*}\leftarrow\mathrm{concat}\left(S_{*},s_{\phi}\right)$

21: end if

22: end while

23: Audio augmentation for sample $S_{a}$ and $S_{*}$

24:end for

return $S_{a}$, $S_{p}$, $S_{n}$

Multi-speaker samples are generated by concatenating samples from the single-
speaker dataset, shown in pseudo-code of Algorithm 1. Where the sign $*$ is
either positive $p$ or negative $n$. $D_{*}$ is speaker IDs. $S_{*}$ is multi-
speaker speech samples. $p_{\mathrm{spk}}$ and $p_{\mathrm{overlap}}$ are the
probatility of speaking and the speech overlap. $\phi$ represents the empty
set or the silence. $\mathrm{length}\left(\cdot\right)$ is to obtain the
length of the sample. $\mathrm{random}\left(\right)$ is a value sampled
uniformly between 0 and 1. $\mathrm{get}\left(\cdot\right)$ is to get audio
samples of random duration by the given $id$ or silence $\phi$.
$\mathrm{select}\left(\cdot\right)$ is to randomly select speaker ids.
$\mathrm{concat}\left(\cdot,\cdot\right)$ is to concatenate the former and the
latter to form a longer sequence. $\mathrm{overlapConcat}\left(\cdot\right)$
is to operate concatenating but overlapped by a certain ratio. Audio
augmentation is performed as follows: all samples are augmented by
probabilistically adding noise with SNR of 10 dB to 30 dB, and RIR
reverberation. With this approach, the anchor, positive and negative samples
($S_{a}$, $S_{p}$ and $S_{n}$) are created. Since the SV model is well trained
and SVVAD requires an explicit learning target, the sample $\bm{x}^{a}$ is
used directly to construct $\bm{x}^{p}$.

##  3 Experiments

###  3.1 Experimental Setup

To validate our proposed SVVAD framework, we employ a pre-trained SV model111
https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb that shares the same
experimental setup with our framework. That is, the SV model uses ECAPA-TDNN
[30] from SpeechBrain [31] trained on the Voxceleb dataset [32]. The SVVAD has
trained with Voxceleb 1+2 dataset and tested on the test set constructed from
the Voxceleb1 (cleaned) verification set by concatenating audio segments from
different speakers with varying proportions $\mathcal{P}$ (percentage of the
duration of other speakers' voices). Considering the variability inherent in
the generation of synthetic data, each set of experimental data will be
constructed 3 times. The test metrics are similar to the SV frameworks and are
reported as the equal error rate (EER) in percentage (%) and minimum decision
cost function (minDCF) at $P_{\mathrm{target}}=0.01$ with
$C_{\mathrm{FA}}=C_{\mathrm{Miss}}=1$ [30].

Two baseline methods were added to the experiment for comparison to assess the
impact of the different methods on SV performance: (1) WebRTC-VAD [33] with
default settings; (2) The traditional PVAD model (in $\S$ 2.1). For the SVVAD
framework, the SV performance of the four variants of the triplet-like loss
(in Equation 4 to 8) will be measured.

###  3.2 Implementation Details

For speech features, the entire framework uses the same Log-Mel-Filterbanks
extractor as the pre-trained SV model [31]. In SVVAD, the size of the
Conformer encoder is $F=256$. The number of Conformer and Transformer layers
in SVVAD is $N_{\mathrm{conf}}=4$ and $N_{\mathrm{trans}}=3$ respectively, and
both have a feed-forward size of 256. The size of the speaker embedding and
the shrunken embedding are $E=192$ and $E^{\prime}=64$ respectively. The
parameter $\alpha$ in triplet-like loss is 0.9, 0.5, 0.55, 0.55 in TL, TLcos,
TLccc and TLccos respectively. The probability $p_{\mathrm{spk}}=0.9$ and
$p_{\mathrm{overlap}}=0.3$.

In the first stage, the model is optimized by SGD optimizer for fast
convergence, with the learning rate of 1e-2, the momentum of 0.9, and the
weight decay of 4e-4. The speech durations of training of anchor, positive and
negative are 6, 8, 8 seconds. The batch size is $B=8$. In the fine-tuning
stage, the optimizer is switched to AdamW [34], and the learning rate is
changed to 1e-4, and the weight decay is 2e-5. The speech durations of these
three are changed to 8, 12, 12 seconds. The batch size is $B=64$ with gradient
accumulation.

###  3.3 Evaluation Results

Table 1: Overall comparison of the proposed SVVAD with 4 triplet-like losses and the baseline models on EER (%) and minDCF (denoted as $C_{0.01}$). + denotes adding methods. $\mathcal{P}$ | 0% | 30% | 50% | 70%
---|---|---|---|---
| EER | $C_{0.01}$ | EER | $C_{0.01}$ | EER | $C_{0.01}$ | EER | $C_{0.01}$
Baselines
w/o VAD | 0.90 | 0.1104 | 2.27 | 0.2284 | 14.84 | 0.6950 | 34.70 | 0.9942
WebRTC | 1.14 | 0.1215 | 2.31 | 0.2332 | 14.50 | 0.6733 | 33.94 | 0.9873
PVAD | 6.22 | 0.5084 | 7.95 | 0.5621 | 9.89 | 0.6080 | 16.29 | 0.7893
SVVAD + Triplet-like Loss (ours)
+TL | 4.63 | 0.4412 | 6.39 | 0.4794 | 8.62 | 0.5893 | 18.32 | 0.8621
+TLcos | 5.48 | 0.5253 | 7.15 | 0.5336 | 9.73 | 0.6012 | 21.37 | 0.9157
+TLccc | 1.41 | 0.1201 | 1.89 | 0.1996 | 6.91 | 0.4912 | 13.42 | 0.7396
+TLccos | 2.06 | 0.1926 | 4.30 | 0.3729 | 7.26 | 0.5433 | 16.30 | 0.7605
Table 2: Ablation study of the effect of the combination of two types of losses on SV performance on EER (%). $\mathcal{L}$ / $\mathcal{P}$ | 0% | 30% | 50% | 70%
---|---|---|---|---
TL | 4.63 | 6.39 | 8.62 | 18.32
+TLcos | 4.17 | 5.76 | 7.75 | 17.29
TLccc | 1.41 | 1.89 | 6.91 | 13.42
black!10 +TLccos | 1.18 | 1.70 | 5.81 | 12.20

Overall Evaluation: Table 1 compares the EER and minDCF of various methods at
different $\mathcal{P}$. As for the baseline, the WebRTC-VAD does not enhance
the SV performance, sometimes even worse. The PVAD model can improve the SV
performance to some extent, but the improvement is limited. For SVVAD, among 4
triplet-like losses, the addition of CCC leads to a significant improvement in
the SV performance. But surprisingly, the cosine distance-based loss, which is
theoretically more suitable because it is used for scoring, perform worse than
the other losses. In our analysis, this may be due to the fact that the cosine
distance-based loss considers only one value as the learning target instead of
using the $E$-dimensional embedding for optimization, which may lead to a
large amount of loss of embedding information when training SVVAD, thus making
it difficult to converge. Furthermore, there is a noticeable drop in SV
performance when no one else is speaking ($\mathcal{P}=0\%$), which is
unsatisfactory.

Ablation Study: To solve the above problem, we combine two types of losses,
which can not only effectively reduce the ambiguity of the learning
objectives, but also optimize the cosine distance for scoring. The
experimental results are reported in Table 2. For faster training, the model
is trained on the basis of TL and TLccc. The total loss $\mathcal{L}$ consists
of two sub-losses, they are
$\left(\mathcal{L}_{\mathrm{TLcos}}+\xi\cdot\mathcal{L}_{\mathrm{TL}}\right)$
or
$\left(\mathcal{L}_{\mathrm{TLccos}}+\xi\cdot\mathcal{L}_{\mathrm{TLccc}}\right)$,
where $\xi$ is the hyperparameter determined by the experiment (set as 0.1).
The results demonstrate that by integrating these two sub-losses, the SV
performance is further improved when $\mathcal{P}$ is larger. In addition,
there is no major performance degradation at $\mathcal{P}=0\%$. The best-
performing model achieves relative EER reductions over PVAD of 78.6%, 41.3%
and 25.1% when $\mathcal{P}$ is 30%, 50% and 70% respectively.

Figure 4: The speech features of $\bm{x}^{\left(\mathrm{tst}\right)}$ and
$\bm{x}^{\left(\mathrm{vad}\right)}$ with their VAD decision boundaries (click
on the figure to hear the sound).

[poster=images/spect/xcat.pdf,mouse,repeat].449691audios/xcat.mp3

(a) Non-target Speaker

[poster=images/spect/xove.pdf,mouse,repeat].449691audios/xove.mp3

(b) Noise

Figure 5: The t-SNE plots of $\bm{v}^{\left(\mathrm{vad}\right)}$ for 10
example speakers in conditions of (1) with or without SVVAD; (2) the different
$\mathcal{P}$.

(a) w/o; 0%

(b) w/o; 30%

(c) w/o; 50%

(d) w/o; 70%

(e) w/; 0%

(f) w/; 30%

(g) w/; 50%

(h) w/; 70%

Generated Samples: According to Figure 5(h), SVVAD produces more aggregated
speaker embeddings when $\mathcal{P}$ is large, indicating that the model can
achieve a greater improvement in SV performance when the input is mixed with
more speech from non-target speakers. Conversely, when $\mathcal{P}$ is small,
the compactness of the cluster looks similar, suggesting that the SV
performance is nearly the same. Figure 4(b) illustrates the decision
boundaries generated by SVVAD, indicating that, similar to human perception,
the model attempts to mask out the speech of non-target speakers and reduce
strong noise, thereby suppressing their impact on SV performance degradation.
However, the embient noise may be negligible probably because the SV model is
inherently robust to it.

##  4 Conclusions

In this paper, we present SVVAD, a speaker verification-based voice activity
detection framework according to which speech features are most beneficial for
SV. We also introduce a label-free training method that optimizes with the
triplet-like loss approach, without relying on human labeling. Extensive
experiments on various scenarios show that SVVAD outperforms the baselines in
terms of EER in noisy and multi-speaker conditions. Furthermore, the decision
boundaries reveal the importance of different parts of speech for SV, which is
largely consistent with human perception. This work opens up new possibilities
for developing more robust and accurate VADs for SV systems in real-world
applications.

##  5 Acknowledgements

Supported by the Key Research and Development Program of Guangdong Province
(grant No. 2021B0101400003) and Corresponding author is Jianzong Wang
(jzwang@188.com).

## References

  * [1] J. Sohn, N. S. Kim, and W. Sung, ``A statistical model-based voice activity detection,'' _IEEE signal processing letters_ , vol. 6, no. 1, pp. 1–3, 1999\.
  * [2] Z.-H. Tan, N. Dehak _et al._ , ``rvad: An unsupervised segment-based robust voice activity detection method,'' _Computer speech & language_, vol. 59, pp. 1–21, 2020.
  * [3] T. Kinnunen and P. Rajan, ``A practical, self-adaptive voice activity detector for speaker verification with noisy telephone and microphone data,'' in _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2013, pp. 7229–7233.
  * [4] E. Sarkar, R. Prasad, and M. Magimai.-Doss, ``Unsupervised voice activity detection by modeling source and system information using zero frequency filtering,'' in _International Speech Communication Association (INTERSPEECH)_ , 2022.
  * [5] A. Vafeiadis, E. Fanioudakis, I. Potamitis, K. Votis, D. Giakoumis, D. Tzovaras, L. Chen, and R. Hamzaoui, ``Two-dimensional convolutional recurrent neural networks for speech activity detection,'' in _International Speech Communication Association (INTERSPEECH)_. IEEE, 2019.
  * [6] G.-B. Wang and W.-Q. Zhang, ``An rnn and crnn based approach to robust voice activity detection,'' in _2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2019, pp. 1347–1350.
  * [7] H. Dinkel, Y. Chen, M. Wu, and K. Yu, ``Voice Activity Detection in the Wild via Weakly Supervised Sound Event Detection,'' in _International Speech Communication Association (INTERSPEECH)_ , 2020, pp. 3665–3669.
  * [8] Z. Zheng, J. Wang, N. Cheng, J. Luo, and J. Xiao, ``Mlnet: An adaptive multiple receptive-field attention neural network for voice activity detection,'' in _International Speech Communication Association (INTERSPEECH)_ , 2020.
  * [9] J. Kim and M. Hahn, ``Voice activity detection using an adaptive context attention model,'' _IEEE Signal Processing Letters_ , vol. 25, no. 8, pp. 1181–1185, 2018.
  * [10] I. Ariav and I. Cohen, ``An end-to-end multimodal voice activity detection using wavenet encoder and residual networks,'' _Journal of Selected Topics in Signal Processing_ , vol. 13, no. 2, pp. 265–274, 2019.
  * [11] F. Tao and C. Busso, ``End-to-end audiovisual speech activity detection with bimodal recurrent neural models,'' _Speech Communication_ , vol. 113, pp. 25–35, 2019.
  * [12] S. Ding, Q. Wang, S.-Y. Chang, L. Wan, and I. Lopez Moreno, ``Personal VAD: Speaker-Conditioned Voice Activity Detection,'' in _Proc. Odyssey 2020 The Speaker and Language Recognition Workshop_ , 2020, pp. 433–439. [Online]. Available: http://dx.doi.org/10.21437/Odyssey.2020-62
  * [13] S. Ding, R. Rikhye, Q. Liang, Y. He, Q. Wang, A. Narayanan, T. O'Malley, and I. McGraw, ``Personal vad 2.0: Optimizing personal voice activity detection for on-device speech recognition,'' in _International Speech Communication Association (INTERSPEECH)_ , 2022.
  * [14] A. Jayasimha and P. Paramasivam, ``Personalizing speech start point and end point detection in asr systems from speaker embeddings,'' in _Spoken Language Technology Workshop (SLT)_. IEEE, 2021, pp. 771–777.
  * [15] Q. Wang, I. L. Moreno, M. Saglam, K. Wilson, A. Chiao, R. Liu, Y. He, W. Li, J. Pelecanos, M. Nika _et al._ , ``Voicefilter-lite: Streaming targeted voice separation for on-device speech recognition,'' _arXiv preprint arXiv:2009.04323_ , 2020.
  * [16] R. Giri, S. Venkataramani, J.-M. Valin, U. Isik, and A. Krishnaswamy, ``Personalized percepnet: Real-time, low-complexity target voice separation and enhancement,'' in _International Speech Communication Association (INTERSPEECH)_ , 2021.
  * [17] Y. Ju, W. Rao, X. Yan, Y. Fu, S. Lv, L. Cheng, Y. Wang, L. Xie, and S. Shang, ``Tea-pse: Tencent-ethereal-audio-lab personalized speech enhancement system for icassp 2022 dns challenge,'' in _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 9291–9295.
  * [18] S. E. Eskimez, T. Yoshioka, H. Wang, X. Wang, Z. Chen, and X. Huang, ``Personalized speech enhancement: New models and comprehensive evaluation,'' in _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 356–360.
  * [19] I. Medennikov, M. Korenevsky, T. Prisyach, Y. Khokhlov, M. Korenevskaya, I. Sorokin, T. Timofeeva, A. Mitrofanov, A. Andrusenko, I. Podluzhny _et al._ , ``Target-speaker voice activity detection: a novel approach for multi-speaker diarization in a dinner party scenario,'' in _International Speech Communication Association (INTERSPEECH)_ , 2020.
  * [20] X. Xiao, N. Kanda, Z. Chen, T. Zhou, T. Yoshioka, S. Chen, Y. Zhao, G. Liu, Y. Wu, J. Wu _et al._ , ``Microsoft speaker diarization system for the voxceleb speaker recognition challenge 2020,'' in _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 5824–5828.
  * [21] K. Wang, X. Mao, H. Wu, C. Ding, C. Shang, R. Xia, and Y. Wang, ``The bytedance speaker diarization system for the voxceleb speaker recognition challenge 2021,'' _arXiv preprint arXiv:2109.02047_ , 2021.
  * [22] M. Cheng, H. Wang, Z. Wang, Q. Fu, and L. Ming, ``The whu-alibaba audio-visual speaker diarization system for the misp challenge 2022,'' in _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023.
  * [23] M. He, D. Raj, Z. Huang, J. Du, Z. Chen, and S. Watanabe, ``Target-speaker voice activity detection with improved i-vector estimation for unknown number of speaker,'' in _International Speech Communication Association (INTERSPEECH)_ , August 2021.
  * [24] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, ``Film: Visual reasoning with a general conditioning layer,'' in _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , vol. 32, no. 1, 2018.
  * [25] Y. Liu, E. Han, C. Lee, and A. Stolcke, ``End-to-end neural diarization: From transformer to conformer,'' in _International Speech Communication Association (INTERSPEECH)_ , 2021.
  * [26] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, ``Conformer: Convolution-augmented Transformer for Speech Recognition,'' in _Proceedings of International Speech Communication Association (INTERSPEECH)_ , 2020, pp. 5036–5040.
  * [27] F. Schroff, D. Kalenichenko, and J. Philbin, ``Facenet: A unified embedding for face recognition and clustering,'' in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2015, pp. 815–823.
  * [28] C. Li, X. Ma, B. Jiang, X. Li, X. Zhang, X. Liu, Y. Cao, A. Kannan, and Z. Zhu, ``Deep speaker: an end-to-end neural speaker embedding system,'' _arXiv preprint arXiv:1705.02304_ , 2017.
  * [29] I. Lawrence and K. Lin, ``A concordance correlation coefficient to evaluate reproducibility,'' _Biometrics_ , pp. 255–268, 1989.
  * [30] B. Desplanques, J. Thienpondt, and K. Demuynck, ``ECAPA-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification,'' in _International Speech Communication Association (INTERSPEECH)_ , H. Meng, B. Xu, and T. F. Zheng, Eds. ISCA, 2020, pp. 3830–3834.
  * [31] M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, ``SpeechBrain: A general-purpose speech toolkit,'' 2021, arXiv:2106.04624.
  * [32] A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, ``Voxceleb: Large-scale speaker verification in the wild,'' _Computer Science and Language_ , 2019\.
  * [33] WebRTC. [Online]. Available: https://webrtc.org/
  * [34] I. Loshchilov and F. Hutter, ``Decoupled weight decay regularization,'' in _International Conference on Learning Representations_ , 2017.


# Second-order Co-occurrence Sensitivity of
Skip-Gram with Negative Sampling

Dominik Schlechtweg, Cennet Oguz, Sabine Schulte im Walde
Institute for Natural Language Processing, University of Stuttgart
<EMAIL_ADDRESS>
###### Abstract

We simulate first- and second-order context overlap and show that Skip-Gram
with Negative Sampling is similar to Singular Value Decomposition in capturing
second-order co-occurrence information, while Pointwise Mutual Information is
agnostic to it. We support the results with an empirical study finding that
the models react differently when provided with additional second-order
information. Our findings reveal a basic property of Skip-Gram with Negative
Sampling and point towards an explanation of its success on a variety of
tasks.

##  1 Introduction

The idea of second-order co-occurrence vectors was introduced by Schütze
(1998) for word sense discrimination and has since then been extended and
applied to a variety of tasks Lemaire and Denhiere (2006); Islam and Inkpen
(2006); Schulte im Walde (2010); Zhuang et al. (2018). The basic idea is to
represent a word $w$ not by a vector of the counts of context words it
directly co-occurs with, but instead by a count vector of the context words of
the context words, i.e., the second-order context words of $w$. These second-
order vectors are supposed to be less sparse and more robust than first-order
vectors (Schütze, 1998). Moreover, capturing second-order co-occurrence
information can be seen as a way of generalization. To see this, cf. examples
(1) and (1) inspired by Schütze and Pedersen (1993).

As far as the Soviet Communist Party and the Comintern were concerned …

…this is precisely the approach taken by the British Government. The nouns
Party and Government have similar meanings in these contexts, although they
have little contextual overlap: A frequent topic in the British corpus used by
Schütze and Pedersen is the Communist Party of the Soviet Union, but
governments are rarely qualified as communist in the corpus. Hence, there is
little overlap in first-order context words of Party and Government. However,
their context words Communist and British in turn have a greater overlap,
because they are frequently used to qualify the same nouns from the political
domain, as in Communist authorities and British authorities. Hence, although
Party and Government may have no first-order context overlap, they do have
second-order context overlap. According to Schütze and Pedersen, capturing
this information corresponds to the generalization “occurring with a political
adjective.”

While most traditional count-based vector learning techniques such as raw
count vectors or Point-wise Mutual Information (PPMI) do not capture second-
order co-occurrence information, truncated Singular Value Decomposition (SVD)
has been shown to do so. Regarding the more recently developed embeddings
based on shallow neural networks, such as Skip-Gram with Negative Sampling
(SGNS), it is presently unknown whether they capture higher-order co-
occurrence information. So far, this question has been neglected as a research
topic, although the answer is crucial to explain performance differences: Levy
et al. (2015) show that SGNS performs similarly to SVD and differently from
PPMI across semantic similarity data sets. If SGNS captures second-order co-
occurrence information, this provides a possible explanation for the observed
performance differences.

We examine this question in two experiments: (i) We create an artificial data
set with target words displaying context overlap in different orders of co-
occurrence and show that SGNS behaves similarly to SVD in capturing second-
order co-occurrence information. The experiment supplies additional and
striking evidence to prior work on SVD and introduces a method to further
investigate questions more precisely than done before. (ii) We transfer
second-order context information to the first-order level in a small corpus
and test the models’ reaction on a standard evaluation data set when provided
with the additional information. We find that SGNS and SVD, already capturing
second-order information, do not benefit, whereas PPMI benefits.

##  2 Related Work

An early connection between second-order context information and
generalization can be found in Schütze and Pedersen (1993). The authors claim
that SVD is able to generalize by using second-order context information as
described above. Later work supports this claim and indicates that SVD even
captures information from higher orders of co-occurrence Landauer et al.
(1998); Kontostathis and Pottenger (2002); Newo Kenmogne (2005); Kontostathis
and Pottenger (2006).

Since then, second-order co-occurrence information has mainly been exploited
for traditional count-based vector learning techniques with different aims.
Schütze (1998) had used second-order vectors for word sense clustering.
Various studies model synonymy or semantic similarity (Edmonds, 1997; Islam
and Inkpen, 2006; Lemaire and Denhiere, 2006) indicating that second-order co-
occurrence plays an important role for these tasks.

The only works we are aware of exploring second-order information for word
embeddings are Newman-Griffis and Fosler-Lussier (2017) learning embeddings
from nearest-neighbor graphs and Zhuang et al. (2018) indicating that a
specific type of word embeddings may benefit from second-order information.
However, no study investigated the question whether SGNS or other word
embeddings already capture higher-order co-occurrence information, which may
make the integration of second-order information superfluous.

##  3 Semantic Vector Spaces

We compare SGNS to two traditional count-based vector space learning
techniques: PPMI and SVD, where the former does not capture second-order
information while the latter does. All methods are based on the concept of
semantic vector spaces: A semantic vector space constructed from a corpus $C$
with vocabulary $V$ is a matrix $M$, where each row vector represents a word
$w$ in the vocabulary $V$ reflecting its co-occurrence statistics (Turney and
Pantel, 2010).

#### Positive Pointwise Mutual Information (PPMI).

For PPMI representations, we first construct a high-dimensional and sparse co-
occurrence matrix $M$. The value of each matrix cell $M_{i,j}$ represents the
number of co-occurrences of the word $w_{i}$ and the context $c_{j}$,
$\\#(w_{i},c_{j})$. Then, the co-occurrence counts in each matrix cell
$M_{i,j}$ are weighted by the smoothed and shifted positive mutual information
of target $w_{i}$ and context $c_{j}$ reflecting their degree of association.
The values of the transformed matrix are

${M}^{\textrm{PPMI}}_{i,j}=\max\left\\{\log\left(\frac{\\#(w_{i},c_{j})\sum_{c}\\#(c)^{\alpha}}{\\#(w_{i})\\#(c_{j})^{\alpha}}\right)-\log(k),0\right\\},$

where $k>1$ is a prior on the probability of observing an actual occurrence of
$(w_{i},c_{j})$ and $0<\alpha<1$ is a smoothing parameter reducing PPMI’s bias
towards rare words (Levy and Goldberg, 2014; Levy et al., 2015). To our
knowledge PPMI representations have never been claimed to capture higher-order
co-occurrence information.

#### Singular Value Decomposition (SVD).

Truncated Singular Value Decomposition is an algebraic algorithm finding the
optimal rank $d$ factorization of matrix $M$ with respect to L2 loss (Eckart
and Young, 1936).111We use ‘SVD’ to refer to the particular application of the
algebraic method described. It is used to obtain low-dimensional
approximations of the PPMI representations by factorizing
${M}^{\textrm{PPMI}}$ into the product of the three matrices
${U}{\Sigma}{V}^{\top}$. We keep only the top $d$ elements of $\Sigma$ and
obtain

${M}^{\textrm{SVD}}={U_{d}}{\Sigma^{p}_{d}},$

where $p$ is an eigenvalue weighting parameter Levy et al. (2015). Ignoring
$V$ for ${M}^{\textrm{SVD}}$ reduces dimensionality while preserving the dot-
products between rows. While it is not clear whether SVD generalizes better
than other models in general (Gamallo and Bordag, 2011), its sensitivity to
higher orders of co-occurrence has been shown empirically and mathematically
Kontostathis and Pottenger (2002); Newo Kenmogne (2005); Kontostathis and
Pottenger (2006). Kontostathis and Pottenger (2006) prove that the existence
of a non-zero value in a truncated term-to-term co-occurrence matrix follows
directly from the existence of a higher-order co-occurrence in the full
matrix. They also show that there is an empirical correlation between the
magnitude of the value and the number of higher-order co-occurrences found for
the particular term pair.

#### Skip-Gram with Negative Sampling (SGNS).

SGNS differs from the above techniques in that it directly represents each
word $w\in V$ and each context $c\in V$ as a $d$-dimensional vector by
implicitly factorizing $M=WC^{\top}$ when solving

$\arg\max_{\theta}\sum_{(w,c)\in D}\log\sigma(v_{c}\cdot v_{w})+\sum_{(w,c)\in
D^{\prime}}\log\sigma(-v_{c}\cdot v_{w}),$

where $\sigma(x)=\frac{1}{1+e^{-x}}$, $D$ is the set of all observed word-
context pairs and $D^{\prime}$ is the set of randomly generated negative
samples (Mikolov et al., 2013a, b; Goldberg and Levy, 2014). The optimized
parameters $\theta$ are $v_{c_{i}}=C_{i*}$ and $v_{w_{i}}=W_{i*}$ for $w,c\in
V$, $i\in 1,...,d$. $D^{\prime}$ is obtained by drawing $k$ contexts from the
empirical unigram distribution $P(c)=\frac{\\#(c)}{|D|}$ for each observation
of $(w,c)$, cf. Levy et al. (2015). The final SGNS matrix is given by

${M}^{\textrm{SGNS}}=W.$

Levy and Goldberg (2014) relate SGNS to SVD by showing that under specific
assumptions their learning objectives have the same global optimum. However,
it is unknown whether SGNS is also similar to SVD in capturing higher-order
co-occurrence information. The model architecture suggests that this is
possible: consider the two context vectors $\vec{c_{1}},\vec{c_{2}}$ in $C$ of
two words having large context overlap (e.g. the vectors for Communist and
British). $\vec{c_{1}},\vec{c_{2}}$ will be similar, because the dot product
with the same target vectors in $W$ will be maximized (as
$\vec{c_{1}},\vec{c_{2}}$ frequently occur as contexts of the same target
words). If $\vec{c_{1}},\vec{c_{2}}$ are then in turn each used to maximize
the dot product with two different new target vectors (e.g. the vectors for
Party and Government), these also tend to be similar.

#### Model Training.

For both experiments we use the implementation of Levy et al. (2015), allowing
us to train all models on extracted word-context pairs instead of the corpus
directly. We follow previous work in setting the hyper-parameters (Levy et
al., 2015). For PPMI we set $\alpha=.75$ and $k=5$. We set the number of
dimensions $d$ for SVD and SGNS to 300. SGNS is trained with 5 negative
samples, 5 epochs and without subsampling. For SVD we set $p=0$.

#### Similarity Measure.

For all methods we measure similarity between word vectors with Cosine
Distance (CD), where low CD means high similarity. CD is based on cosine
similarity, $cos(\vec{x},\vec{y})$, which measures the cosine of the angle
between two non-zero vectors $\vec{x},\vec{y}$ with equal magnitudes Salton
and McGill (1983). CD is then defined as

$CD(\vec{x},\vec{y})=1-cos(\vec{x},\vec{y}).$

##  4 Experiment 1: Simulating second-order context overlap

In order to see whether SGNS captures second-order co-occurrence information,
we artificially simulate context overlap for first- and second-order co-
occurrence separately. This allows us to simulate clear cases of overlap
controlling for confounding factors which are present in empirical data. We
generate target-context pairs in such a way that specific target words have
either context word overlap in first-order co-occurrence, or by contrast in
second order. We compare the behavior of PPMI, SVD and SGNS on three groups of
such target words (see Table 1):

first-order overlap (1st):

Target words $T$ occurring with the _same_ context words $C1$ in the first
order, while in the second order all context words from $C1$ have distinct
context words $C2$.

2nd-order overlap (2nd):

Target words $T$ occurring with _distinct_ context words $C1$ in the first
order, while all context words from $C1$ have the same context words $C2$.

no overlap (none):

Target words $T$ occurring with distinct context words $C1$ in the first order
and also all context words in $C1$ have distinct context words $C2$.

As an example, consider the column 2nd in Table 1. The target words $T$ are
$a$ and $b$. Each has distinct context words: $a$ occurs only with $c,d\in
C1$, while b occurs only with $e,f\in C1$. However, the first-order context
words $c,d,e,f\in C1$ do have context overlap: $c,d,e,f$ occur all with
$u,v\in C2$, i.e., they have the same second-order context words.

For each group we generate 10 target words ($T$). Per target word, each of
$C1$, $C2$ is constructed by first generating 1000 context words $C$,
assigning a sampling probability from a lognormal distribution to each context
word in $C$ and then sampling 1000 times from $C$.222By sampling from a
lognormal distribution we aim to approximate the empirical frequency
distribution of context words. Context words receive probabilities by randomly
sampling 1000 values from
$f(x)=\frac{1}{x\sqrt{2\pi}}\exp\left(-\frac{\log(x)^{2}}{2}\right)$ and
normalizing them to a probability distribution. For the 1st-group, the set of
context words $C$ will be shared across target words, meaning that they have a
context word overlap. For the target words in the 2nd-group this will not be
the case, but instead their first-order context words ($C1$) will have context
overlap (see Table 1). In this way, we simulate context overlap in first vs.
second order. For the target words in the none-group, $C$ will instead be
completely disjunct in both orders. Because co-occurrence is symmetric (if $a$
occurs with $c$, $c$ also occurs with $a$), for each pair ($a$,$c$) generated
by the above-described process, we also add the reverse pair ($c$,$a$). To
make sure that the pairs from the different groups (1st, 2nd, none) do not
interfere with each other, each string generated for a group is unique to the
group. Finally, we mix and randomly shuffle the pairs from all groups. In this
way, we generate 10 x 1000 x 1000 x 2 target-context pairs for each group: 10
target words occurring with 1000 context words in $C1$ where each in turn
occurs with 1000 context words in $C2$, plus each of these pairs
reversed.333Find the code generating the artificial pairs under:
https://github.com/Garrafao/SecondOrder.

Our main hypothesis is that SGNS and SVD will predict target words from the
2nd-group to be more similar on average than target words from the none-group
(although both groups have no first-order context overlap), while PPMI will
predict similar averages for both groups.

order 1st 2nd none $\mathbf{C1}$ a c a d b c b d a c a d b e b f a c a d b e b
f $\mathbf{C2}$ c u c v d w d x c u c v d u d v c u c v d w d x

Table 1: Artificial co-occurrence pairs with context overlap in different
orders of co-occurrence (1st, 2nd and none). $C1$ and $C2$ give co-occurrence
in first and second order respectively. For each pair (a,c) shown above we
also add the reverse pair (c,a).

#### Results.

Figure 1 shows the average cosine distance between the target words in each of
the three target word groups with context overlap in different orders (1st,
2nd and none). As expected, PPMI predicts the target words without contextual
overlap in any order (none) to be orthogonal to each other ($1.0$). Further,
PPMI is sensitive to first-order overlap, but not at all to second-order
overlap ($0.51$ vs. $1.0$). SVD also predicts orthogonality for the none-group
($1.0$) and shows sensitivity to first-order overlap ($0.34$), but is
extremely sensitive to second-order overlap: it predicts the target words in
2nd to be perfectly similar to each other ($0.0$), notwithstanding the fact
that they have no first-order context word overlap. SGNS shows a similar
behavior, although its vectors are distributed more densely: target words in
none are predicted to be least similar ($0.79$), while target words in 1st are
more similar ($0.11$) and in 2nd they are predicted to be completely similar
($0.0$).

We further hypothesize that the fact that for SGNS and SVD the average cosine
distance in 1st is higher than in 2nd is related to our choice to make the
context words $C1$ of the target words in 1st dissimilar to each other by
assigning completely distinct context words $C2$ (see Table 1). We test this
hypothesis by creating a second artificial set of target-context pairs
completely parallel to the above-described set with the only difference that
1st has additional context overlap in $C2$. On these targets words with
overlap in both orders we find that PPMI makes similar predictions ($0.56$) as
before, while for SVD and SGNS predictions drop to $0.0$, confirming our
hypothesis.

Figure 1: Results of simulation experiment. Values give average cosine
distances across target words with different levels of context overlap. Pair-
wise differences between group means (except PPMI 2nd vs. none) are
statistically significant according to a two-sample bootstrap test ($p<0.001$,
adjusted through Bonferroni correction for 9 tests, two-tailed).

#### Discussion.

SGNS and SVD capture second-order co-occurrence information. Notably, they are
more sensitive to the similarity of context words than to the words themselves
(2nd vs. 1st), which means that they abstract over mere co-occurrence with
specific context words and take into account the co-occurrence structure of
these words in the second order (and potentially higher orders). PPMI does not
have this property and only measures context overlap in first order.

##  5 Experiment 2: Propagating second-order co-occurrence information

We now propagate second-order information to the first-order level by
extracting second-order word-context pairs and adding them to the first-order
pairs. We hypothesize that the additional second-order information will impact
PPMI representations positively and stronger than SVD and SGNS, because we saw
that the latter already capture second-order information. We reckon that the
additional information is beneficial for PPMI in two ways: (i) it helps to
generalize as described in (1) and (1), and (ii) it overcomes data sparsity
for low-frequency words. Note that these two aspects are often highly related:
with only a limited amount of data available it is more likely that similar
words do not have exactly the same, but still similar context words.
Generalization then helps to overcome sparsity.

#### Corpus.

We use ukWaC Baroni et al. (2009), a ${>1}$B token web-crawled corpus of
English. The corpus provides part-of-speech tagging and lemmatization. We keep
only the lemmas which are tagged as nouns, adjectives or verbs. In order to
assure we have low-frequency words in the test data, we create a small corpus
by randomly choosing $1$M sentences and shuffling them. The final corpus
contains roughly $10$M tokens. Under these sparse conditions we expect to
observe strong effects on model performance highlighting the differences
between the models.

###  5.1 Pair Extraction

We first extract first-order word-context pairs by iterating over all
sentences and extracting one word context pair ($w$,$c$) for each token $w$
and each context word $c$ surrounding $w$ in a symmetric window of size 5
(base pairs). Then, we extract additional second-order pairs in the following
way: For each word type $t$ in the corpus, we build a second-order vector
$\vec{v}$ by summing over all of $t$’s first-order context token count vectors
Schütze (1998). Then we randomly sample $n$ second-order context tokens from
$\vec{v}$ (with replacement) where each context type $c_{i}$ has a sampling
probability of $\frac{\vec{v}_{i}}{\sum_{j=1}\vec{v}_{j}}$ and $\vec{v}_{i}$
is $\vec{v}$’s $i$th entry. We then exclude all sampled context tokens
$c=t$.444Find the code under: https://github.com/Garrafao/SecondOrder.

We extract second-order pairs only for words below a specific co-occurrence
frequency threshold $f$ to test the impact on sparse words separately. We
experiment with $f\in\\{$2k,20k,200k$\\}$. We set $n$ globally to $200\%$ of
$t$’s co-occurrence frequency to add a substantial amount of information. For
each of the second-order pairs we add the reverse pair ($c$,$w$). Finally, the
respective second-order pairs are combined with the base pairs and randomly
shuffled. In this way, we generate roughly 22/99/218M (2/20/200k) second-order
pairs from 57M base pairs. Then we train each model on each of the combined
pair files separately.

###  5.2 Results

Figure 2: Results of experiment 2. Values give correlation (Spearman’s $\rho$)
of model predictions with human similarity judgments.

We evaluate the obtained vector spaces on WordSim353 (Finkelstein et al.,
2002), a standard human similarity judgment data set, by measuring the
Spearman correlation of the cosine similarity for target word pairs with human
judgments. The results are shown in Figure 2. As we can see, the models show
different reactions to the additional second-order information: PPMI is the
only model benefiting (in one case), while SVD and SGNS never benefit from the
additional information and are always impacted negatively. The strongest
negative impact can be observed for SVD (-0.17). PPMI shows a clear
improvement with rather low-frequency words (20k). Adding second-order
information for high-frequency words (200k) has a strong negative impact for
PPMI and SVD.

#### Discussion.

The different reactions of PPMI vs. SVD and SGNS partly confirm our hypothesis
which was based on the findings in experiment 1: only PPMI benefits from
additional second-order information. However, we did not expect the observed
negative impacts, especially the strong performance drop for SVD. Moreover, it
is notable that SVD on the base pairs shows a much higher performance ($0.51$)
than SGNS and PPMI ($0.43$, $0.41$), which is not the case in less sparse
conditions Levy et al. (2015). This indicates that SVD makes much better use
of the available information and overcomes data sparsity in this way. It
remains for future research to determine how much the exploitation of higher-
order co-occurrence information contributes to this clear performance
advantage.

##  6 Conclusion

We showed that SGNS captures second-order co-occurrence information, a
property it shares with SVD and distinguishes it from PPMI. We further tested
the reaction of the models when provided with additional second-order
information, expecting that only PPMI would benefit. We find this confirmed,
but also observe unexpectedly strong negative impacts on SVD by the supposedly
redundant information. In general, SVD turns out to have strong performance
advantages over PPMI and SGNS in the sparse experimental conditions we
created.

Our findings are relevant for a variety of algorithms relying on the SGNS
architecture (i.a. Grover and Leskovec, 2016; Bamler and Mandt, 2017). Future
work will look into the relationship between the second-order sensitivity of
SVD and SGNS and their high performances across tasks. In addition, we aim to
use the introduced method of generating artificial context overlap to see
which higher orders of co-occurrence SVD, SGNS and other embedding types
Pennington et al. (2014); Peters et al. (2018); Athiwaratkun et al. (2018)
capture. Because the aim of the study was only to test the second-order
sensitivity of different models, we did not focus on finding the best way to
provide this information. Given the results for PPMI, however, developing a
smoother way to provide second-order information to models seems to be a
promising starting point for further research.

## Acknowledgments

The first author was supported by the Konrad Adenauer Foundation and the CRETA
center funded by the German Ministry for Education and Research (BMBF) during
the conduct of this study. We thank Agnieszka Falenska, Sascha Schlechtweg and
the anonymous reviewers for their valuable comments.

## References

  * Athiwaratkun et al. (2018) Ben Athiwaratkun, Andrew Gordon Wilson, and Anima Anandkumar. 2018.  Probabilistic fasttext for multi-sense word embeddings.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_ , pages 1–11, Melbourne, Australia.
  * Bamler and Mandt (2017) Robert Bamler and Stephan Mandt. 2017.  Dynamic word embeddings.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70, pages 380–389, Sydney, Australia.
  * Baroni et al. (2009) Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009.  The wacky wide web: A collection of very large linguistically processed web-crawled corpora.  _Language Resources and Evaluation_ , 43(3):209–226.
  * Eckart and Young (1936) Carl Eckart and Gale Young. 1936.  The approximation of one matrix by another of lower rank.  _Psychometrika_ , 1:211–218.
  * Edmonds (1997) Philip Edmonds. 1997.  Choosing the word most typical in context using a lexical co-occurrence network.  In _Proceedings of the 8th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 507–509.
  * Finkelstein et al. (2002) Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002.  Placing search in context: The concept revisited.  _ACM Transactions on Information Systems_ , 20(1):116–131.
  * Gamallo and Bordag (2011) Pablo Gamallo and Stefan Bordag. 2011.  Is singular value decomposition useful for word similarity extraction?  _Language Resources and Evaluation_ , 45(2):95–119.
  * Goldberg and Levy (2014) Yoav Goldberg and Omer Levy. 2014.  Word2vec Explained: Deriving Mikolov et al.’s negative-sampling word-embedding method.  _CoRR_ , abs/1402.3722.
  * Grover and Leskovec (2016) Aditya Grover and Jure Leskovec. 2016.  Node2vec: Scalable feature learning for networks.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 855–864, San Francisco, California, USA.
  * Islam and Inkpen (2006) Aminul Islam and Diana Inkpen. 2006.  Second order co-occurrence pmi for determining the semantic similarity of words.  In _Proceedings of the 5th International Conference on Language Resources and Evaluation_ , pages 1033–1038, Genoa, Italy.
  * Kontostathis and Pottenger (2002) April Kontostathis and William M. Pottenger. 2002.  Detecting patterns in the LSI term-term matrix.  In _Proceedings of the Workshop on Foundations of Data Mining and Discovery_.
  * Kontostathis and Pottenger (2006) April Kontostathis and William M. Pottenger. 2006.  A framework for understanding latent semantic indexing LSI performance.  _Information Processing & Management_, 42(1):56–73.
  * Landauer et al. (1998) Thomas K Landauer, Peter W Foltz, and Darrell Laham. 1998.  An introduction to latent semantic analysis.  _Discourse Processes_ , 25(2-3):259–284.
  * Lemaire and Denhiere (2006) Benoit Lemaire and Guy Denhiere. 2006.  Effects of high-order co-occurrences on word semantic similarity.  _Current Psychology Letters – Behaviour, Brain and Cognition_ , 18(1).
  * Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014.  Neural word embedding as implicit matrix factorization.  In _Proceedings of the 27th International Conference on Neural Information Processing Systems_ , pages 2177–2185, Montreal, Canada.
  * Levy et al. (2015) Omer Levy, Yoav Goldberg, and Ido Dagan. 2015.  Improving distributional similarity with lessons learned from word embeddings.  _Transactions of the Association for Computational Linguistics_ , 3:211–225.
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.  In _Proceedings of the International Conference on Learning Representations_.
  * Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b.  Distributed representations of words and phrases and their compositionality.  In _Proceedings of the Neural Information Processing Systems Conference_.
  * Newman-Griffis and Fosler-Lussier (2017) Denis Newman-Griffis and Eric Fosler-Lussier. 2017.  Second-order word embeddings from nearest neighbor topological features.  _arXiv preprint arXiv:1705.08488_.
  * Newo Kenmogne (2005) Regis Newo Kenmogne. 2005.  Understanding LSI via the truncated term-term matrix.  Master’s thesis, Universität des Saarlandes, Saarbrücken.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_ , pages 1532–1543, Doha, Qatar.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2227–2237, New Orleans, LA, USA.
  * Salton and McGill (1983) Gerard Salton and Michael J McGill. 1983.  _Introduction to Modern Information Retrieval_.  McGraw - Hill Book Company, New York.
  * Schulte im Walde (2010) Sabine Schulte im Walde. 2010.  Comparing computational approaches to selectional preferences: Second-order co-occurrence vs. latent semantic clusters.  In _Proceedings of the 7th International Conference on Language Resources and Evaluation_ , pages 1381–1388, Valletta, Malta.
  * Schütze (1998) Hinrich Schütze. 1998.  Automatic word sense discrimination.  _Computational Linguistics_ , 24(1):97–123.
  * Schütze and Pedersen (1993) Hinrich Schütze and Jan Pedersen. 1993.  A vector model for syntagmatic and paradigmatic relatedness.  In _Proceedings of the 9th Annual Conference of the UW Centre for the New OED and Text Research_ , pages 104–113, Oxford, UK.
  * Turney and Pantel (2010) Peter D Turney and Patrick Pantel. 2010.  From frequency to meaning: Vector space models of semantics.  _Journal of Artificial Intelligence Research_ , 37:141–188.
  * Zhuang et al. (2018) Yimeng Zhuang, Jinghui Xie, Yinhe Zheng, and Xuan Zhu. 2018.  Quantifying context overlap for training word embeddings.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 587–593.