# IMPROVED SELF-SUPERVISED MULTILINGUAL SPEECH REPRESENTATION LEARNING
COMBINED WITH AUXILIARY LANGUAGE INFORMATION

###### Abstract

Multilingual end-to-end models have shown great improvement over monolingual
systems. With the development of pre-training methods on speech, self-
supervised multilingual speech representation learning like XLSR has shown
success in improving the performance of multilingual automatic speech
recognition (ASR). However, similar to the supervised learning, multilingual
pre-training may also suffer from language interference and further affect the
application of multilingual system. In this paper, we introduce several
techniques for improving self-supervised multilingual pre-training by
leveraging auxiliary language information, including the language adversarial
training, language embedding and language adaptive training during the pre-
training stage. We conduct experiments on a multilingual ASR task consisting
of 16 languages. Our experimental results demonstrate 14.3% relative gain over
the standard XLSR model, and 19.8% relative gain over the no pre-training
multilingual model.

Index Terms—  multilingual, self-supervised, automatic speech recognition,
language interference, language information

##  1 Introduction

The performance of automatic speech recognition (ASR) has progressed
substantially in the recent past [1, 2, 3]. However, ASR has suffered from the
reliance on human annotations, which is typically impractical for most
languages around the world. For multilingual business, there is considerable
interest and importance to support multiple languages in a single ASR system.
To take advantage of the commonality among different languages, multilingual
models have been studied over the last two decades and have outperformed
monolingual models for ASR in low-resource languages [4, 5].

Despite such promising potential of multilingual models, most approaches
inevitably require abundant training data because it combines acoustic models,
language models and alignment into the same model. Therefore, there has been a
lot of interest in how to better use unlabeled speech data since the
availability of unlabeled data may be virtually unlimited [6, 7] . As a
mainstream study direction, self-supervised Learning (SSL) typically utilizes
speech pre-training to learn better representations. Based on a pre-trained
model, the recognition system can achieve comparable performance to supervised
models trained on quantities of labeled data, while only requiring fine-tuning
on a small amount of labeled data [8, 9, 10].

Self-supervised learning has been applied to multilingual modeling in
recently, namely cross-lingual speech representation learning (XLSR) [11]. And
other studies also show that multilingual pre-training outperforms monolingual
pre-training in low resource languages [12, 13]. However, according to the
studies in [12], there exists the capacity dilution and interference problem
in multilingual pre-training. Specifically, low resource languages benefit
from joint training with similar languages, while high resource languages
often suffer from the negative transfer problem, resulting in inferior
performance. Such performance degeneration becomes more significant as the
model expands to more languages [5] or more training data [14], which
obstacles the application of pre-trained models on multilingual tasks.

Motivated by that, we propose several methods to alleviate language
interference to improve the performance of multilingual pre-training by
leveraging auxiliary language information. These approaches include the
introduction of language adversarial training, language embedding and language
adaptive training during the pre-training stage, so as to balance the learning
of shared representations and language specific representations. At inference
time, we assume the language is either specified in the user’s preferences, or
determined automatically from a language identification system. We analyze the
trade-offs of different methods with regards to quality gains and number of
additional parameters, and further explore the key factors that affect the
effectiveness of each method. We conduct experiments in an ASR task consisting
of 16 languages. Our experimental results demonstrate $\sim$4% to $\sim$14%
relative gain over the standard XLSR model, and $\sim$10% to $\sim$20%
relative gain over the baseline multilingual model.

##  2 MODELING

Self-supervised approaches such as wav2vec 2.0 [8] have been extended to the
multilingual setting. The recent XLSR or XLS-R focus on the cross-lingual
setting by learning representations on unlabeled data that generalize across
languages.

XLSR builds on the pre-training approach of wav2vec 2.0 which jointly learns
contextualized speech representations as well as a discrete vocabulary of
latent speech representations. The backbone of XLSR includes a feature
encoder, a context representations encoder and a quantization module. During
pre-training, the feature encoder takes the speech frames as input and
generate higher dimensional features
$\boldsymbol{Z}=[\boldsymbol{z}_{1},\boldsymbol{z}_{2},\dots,\boldsymbol{z}_{T}]$.
Then some frames of $\boldsymbol{Z}$ are randomly selected to be masked. The
corrupted features $\boldsymbol{\tilde{Z}}$ are fed into transformer based
context network and then transformed into contextual representations
$\boldsymbol{C}=[\boldsymbol{c}_{1},\boldsymbol{c}_{2},\dots,\boldsymbol{c}_{T}]$
at the top layer [15]. The model is trained by solving a contrastive task over
masked feature encoder outputs. To provide the training targets, unmasked
feature encoder representations Z are discretized to
$[\boldsymbol{q}_{1},\boldsymbol{q}_{2},\dots,\boldsymbol{q}_{T}]$ with a
quantization module. The quantization module uses a Gumbel softmax [16] to
choose entries form the codebooks and the chosen entries are concatenated to
obtain $\boldsymbol{q}$. The goal of the contrast task is to distinguish a
true encoded sample qt among distractors $\boldsymbol{Q}_{t}$ that are sampled
uniformly from other masked steps of the same sequence:

$L=-log\frac{exp(sim(\boldsymbol{c}_{t},\boldsymbol{q}_{t})/\tau)}{\sum_{\boldsymbol{q}^{{}^{\prime}}\in\boldsymbol{Q}_{t}}exp(sim(\boldsymbol{c}_{t},\boldsymbol{q}^{{}^{\prime}})/\tau)}$
(1)

Where $sim(a,b)$ computes the cosine similarity and $\tau=0.1$ scales the
logit. The objective is augmented by a codebook diversity penalty to encourage
the model to use all codebook entries.

We train the standard XLSR as a pre-training baseline in our work.
Furthermore, we replace Transformer with Conformer [17] as the context
representations encoder, which has presented higher effectiveness in ASR task.
The major component of Conformer is a stack of conformer blocks, each of which
is a series of multi-headed self-attention, depth-wise convolution and feed-
forward layers. On the other hand, to improve computational efficiency, we use
filter-bank features as input of network instead of raw waveforms. The feature
encoder that acts as a convolutional sub-sampling block is replaced with two
2D-convolution layers, both with strides (2, 2), resulting in a 4x reduction
in the acoustic sequence length.

##  3 PRE-TRAINING WITH LANGUAGE INFORMATION

In this section, we present the different approaches to improve self-
supervised multilingual pre-training. Based on the standard XLSR model, we
introduce language information into multilingual representation learning in
different designs, to alleviate the negative impact of language interference.

###  3.1 Language adversarial pre-training

In order to guide the model to learn language-independent representations that
are more amenable to adaptation, we propose to combine language adversarial
training [18] with multilingual pre-training, called language adversarial pre-
training.

In language adversarial pre-training, a language discriminator is used to
recognize the language label using the shared features of the multilingual
pre-training model. An additional language label is given for each training
sample $\left\\{h_{i}^{m},m\right\\}$ , where $h_{i}^{m}$ denotes the output
of a hidden layer, $m\in\left\\{1,\dots,M,\right\\}$ denotes the language
label for each frame, and M is the number of languages in training sets. The
language discriminator loss function $L_{adv}$ is defined as:

$L_{adv}=-\sum_{m=1}^{M}\sum_{i=1}^{N_{m}}logP(m|h_{i}^{m};\theta^{s},\theta^{d})$
(2)

Where $N_{m}$ denotes the number of training samples of language $m$.
$\theta^{s}$ denotes the parameters of the shared layers, $\theta^{d}$ denotes
the parameters of the sub-network of the language discriminator.

Meanwhile, the gradient reversal layer (GRL) [19] is introduced to ensure the
feature distributions over all the languages are as indistinguishable as
possible for the language discriminator. Thus the shared layers can learn
language-invariant features. At the feed-forward stage, the GRL acts as an
identity transformation. During the back-propagation, however, the GRL takes
the gradient from the subsequent level and changes its sign, i.e., multiplying
by -1. The GRL has no parameters associated with it.

Thus, using the multi task learning method, the language adversarial pre-
training is to simultaneously optimize contrastive loss and language
discriminator loss:

$L=L_{c}-\lambda L_{adv}$ (3)

Where $L_{c}$ is the contrastive loss described in Eq.(1), and $\lambda$
controls the trade-off between the two loss.

###  3.2 Language embedding involved pre-training

Then we investigate the architectural extension of multilingual pre-training
by feeding a vector to represent the language [20]. It is assumed that a
universal model can use a language vector to become an “expert” on each
individual language instead of learning a representation tailored to languages
with more data. Various methods of using a language vector have been
previously described and directly compared in E2E multilingual [21] and multi-
dialect [22] models.

The language itself can be represented in a one-hot vector or an embedding
vector. In addition, given such a language vector, there are many different
position in the model where it can be inserted. In this work, language
embedding shows its effectiveness when involved in multilingual pre-training.
Moreover, our experiments have confirmed earlier observations that simply
adding it as language-specific bias terms to the low-level features is
sufficient. Thus, given a training sample with language label
$\left\\{h_{i}^{m},m\right\\}$, the multilingual pre-training representation
$h_{i}^{m}$ is enhanced by language embedding as:

$\tilde{h}_{i}^{m}=h_{i}^{m}+embedding(m)$ (4)

Where the embedding of language $m$ is trainable parameters.

Furthermore, in order to hold the learned language embedding as different as
possible and provide more effective language specific information, the
language orthogonal constraint loss is defined as:

$L_{o}=-\sum_{i=1}^{m}\sum_{j=1}^{m}||E_{i}E_{j}^{\top}-[i==j]||^{2}$ (5)

Where $E_{i}$ and $E_{j}$ are the embedding of language $i$ and $j$,
respectively. Combined with contrastive loss, the final multilingual pre-
training loss is defined as:

$L=L_{c}+\alpha L_{o}$ (6)

Where $L_{c}$ is the contrastive loss described in Eq.(1), and $\alpha$ is a
penalty coefficient of the constraint.

###  3.3 Language specific adaptive pre-training

Intuitively, multilingual model needs to learn the common representation
between different languages to improve the performance of low resource
languages from transfer learning, while retaining the language specialization
to avoid the negative impact of unrelated languages. Therefore, researchers
opt to use specific network components for each language. Adapters [23] and
adaptive weights [24] have got more attention in supervised multilingual
models due to being computationally manageable and achieve promising results.
Here we extend them to self-supervised multilingual pre-training, becoming
language specific adaptive pre-training.

Adapters are implemented by a small multi-layer perceptrons (MLP), in which
one hidden layer acts as a down-sampler for parameter efficiency. This MLP is
serialized at the end of each layer in Transformer to help the network changes
the feature distribution based on languages.

However, the language adapters added to the Transformer are essentially feed-
forward neural network layers similar to the counterpart already in the shared
Transformer body. A scenario with 20 languages consequently generates hundreds
of these layers accounting for a large amount of parameters to be optimized.
To solve this problem, adaptive weights was proposed by using a factorization
scheme that is both effective and highly scalable for multilingual system.

The main idea of adaptive weights is that each matrix multiplication
$Y=W^{\top}X$ in the multilingual model can be decomposed into a function of
shared weights $W_{S}$ and additional language dependent weights $W_{ML}$ and
$W_{BL}$:

$Y=(W_{S}\cdot W_{ML}+W_{BL})^{\top}X$ (7)

In order to encourage the model to share parameters as well as keeping the
parameters efficient, the adaptive weights are factorized by using the form of
1-rank matrices which can be compactly represented as a dot-product between
two vectors. One drawback in this method is the lacking representational power
of Rank-1 matrices. One solution is to modify the factorization into using $k$
vectors per language so that there are k independent weight factors followed
by a summation, which increases the rank of additional weight matrices:

$\bar{W}=\sum_{i}^{k}r_{i}s_{i}^{\top}$ (8)

##  4 Experiments

###  4.1 Datasets

We conduct our experiments on a inner multilingual speech recognition task,
which consist of 16 languages from different countries and regions. Train and
test set sizes vary due to availability of transcribed data and are shown in
Table 1. For all of these languages, the training data is anonymized and
transcribed by humans. Transcription of each language is cleaned and only the
scripts of the language are included. The test sets are recorded independently
of the training set and transcribed manually. Therefore, in some languages,
there may be serious mismatches between the training set and the test set.

Table 1: Number of hours in train sets and utterances in test sets. Lang. | train/h | test/utt. | Lang. | train/h | test/utt.
---|---|---|---|---|---
ml-in | 328 | 1k | ps-ar | 295 | 1k
ne-np | 397 | 1k | he-il | 477 | 1k
hy-am | 342 | 2k | af-za | 313 | 1k
ka-ge | 281 | 1k | az-az | 246 | 2k
mr-in | 269 | 1k | da-dk | 320 | 1k
te-in | 206 | 1k | fi-fi | 287 | 1k
am-et | 266 | 1k | lo-la | 434 | 1k
tg-ke | 364 | 2k | my-mm | 362 | 2k

###  4.2 Model Setups

We train a XLSR model as the pre-training baseline. The whole encoder consists
of a feature encoder and a context representation encoder. Feature encoder has
2 convolutional layers with filter size (3, 3), strides (2, 2). Two layers
have 128 and 32 channels respectively. Representation encoder consists of 16
Conformer blocks, each with encoder embedding 512, hidden dimension 2048, 8
attention heads and convolution kernel size 15. The final projection layer
dimension is 768. In total, the whole encoder including feature encoder has
about 100 million trainable parameters. We also train a multilingual ASR model
as the supervised baseline. The encoder is consistent with the pre-training
model. For the decoder, simple yet effective, we use a Transformer decoder.
Decoder consists of 6 blocks, each with decoder embedding 768, hidden
dimension 3072, 12 attention heads. The input features are 40-dimensional
filter-bank acoustic features computed every 10ms over a 25ms window.

In language adversarial pre-training, the language discriminator consists of
two layers of perceptrons, each with 512 units, and a classification layer
with 16 units. For language embedding involved pre-training, the dimension of
language embedding is consistent with its added feature dimension. During
language specific adaptive pre-training, the structure of adapter module
follows [29], and the projection dimension is set to 256. While the number of
$k$ in adaptive weights is explored in the following experiments.

###  4.3 Training Details

Pre-training. We train all the self-supervised models on 32 A100 GPUs for 400k
steps. We upsample the data from low resource languages with $\alpha$ = 0.5
and for high resource languages we use natural sampling probability with
$\alpha$ = 1.0. We use dropout 0.1 in the conformer and at the output of the
feature encoder. Layers are dropped at a rate of 0.2. We optimize with Adam
[25], warming up the learning rate for the first 8000 updates to a peak of
0.0005, and then linearly decay it. For masking, we sample p = 0.065 of all
time-steps to be starting indices and mask the subsequent 10 time-steps.

Fine-tuning. During fine-tuning, we add the same decoder as the supervised
multilingual model to the pre-training model. We also insert a linear layer
between the pre-trained model and the decoder layer as the projection block.
We fine-tune the model for 15 epochs. The learning rate warms up linearly from
0 to the peak learning rate 0.0007 for the first 8000 training steps, decays
to the 10-th epoch and then is halved for each epoch from the 11-th epoch.
During fine-tuning, the parameters of feature encoder are fixed.

###  4.4 Results and Discussion

The performance of the different pre-trained modules are fully presented in
Table 2. Standard pre-training model, XLSR, outperforms the multilingual model
without pre-training, achieving 6.5% relatively improvement. It can be seen
that with the increase of the number of languages and supervised data in the
fine-tuning stage, the general pre-training model has not been improved as
much as reported in the paper [11], even performance of some languages has
obviously deteriorated, which confirms the negative impact of language
interference in the pre-training process. Furthermore, when multilingual pre-
training is assisted by several pre-training strategies that we propose to
introduce language information, it exceeds the general pre-training model.

Among the improved multilingual pre-training, the performance of language
adversarial pre-training and language embedding involved pre-training is
similar, with 4.3% and 3.6% improvement respectively compared with the
standard pre-training model, and 10.5% and 9.7% improvement respectively
compared with the supervised multilingual model. The language specific
adaptive pre-training with adapter or adaptive weights, performs even better.
Compared with the XLSR model, they achieve 11.3% and 14.3% improvement
respectively, while compared with the supervised multilingual baseline, they
achieve 17.0% and 19.8% improvement respectively. Finally, with regards to
performance gains and number of additional parameters introduced, pre-training
with language specific adaptive weight significantly improves the multilingual
pre-training with few additional parameters.

Table 2: WER(%) on the test set of 16 languages. Models include conformer (C), standard pre-training (XLSR), language adversarial pre-training (LA), language embedding involved pre-training (LE), pre-training with language specific adapter (LSA), and pre-training with language specific adaptive weight (LSAW). Lang. | CF | XLSR | LA | LE | LSA | LSAW
---|---|---|---|---|---|---
| Params
---
Inc.(%)
- | 0 | 0.2 | 0.01 | 4.9 | 0.5
ml-in | 37.67 | 38.71 | 38.46 | 37.61 | 37.47 | 37.6
ne-np | 45.34 | 46.77 | 44.86 | 45.92 | 42.75 | 42.83
hy-am | 21.41 | 21 | 16.51 | 18.96 | 17.01 | 14.72
ka-ge | 7.94 | 7.67 | 7.48 | 7.41 | 7.40 | 7.51
mr-in | 14.31 | 15.1 | 15.05 | 14.94 | 14.80 | 13.87
te-in | 18.16 | 18.81 | 18.52 | 18.78 | 18.22 | 17.45
am-et | 20.44 | 20.24 | 20.39 | 20.06 | 20.19 | 19.81
tg-ke | 18.44 | 19.15 | 18.79 | 18.56 | 18.14 | 18.48
ps-ar | 22.73 | 23.38 | 22.83 | 22.48 | 22.61 | 20.48
he-il | 21.17 | 24.88 | 23.72 | 24.85 | 20.87 | 20.71
af-za | 24.14 | 22.74 | 22.4 | 22.69 | 22.16 | 22.42
az-az | 36.59 | 23.28 | 18.86 | 18.75 | 24.54 | 20.78
da-dk | 38.11 | 24.69 | 24.37 | 25.45 | 25.45 | 24.5
fi-fi | 56.66 | 52.07 | 47.88 | 49.92 | 27.90 | 26.71
lo-la | 20.30 | 13.34 | 12.9 | 13.87 | 12.16 | 12.33
my-mm | 37.33 | 40.54 | 41.61 | 37.79 | 34.25 | 33.22
AVG. | 27.55 | 25.77 | 24.66 | 24.88 | 22.87 | 22.09

###  4.5 Ablation Studies

Language adversarial pre-training: Intuitively, the more language-specific
information contained in the features fed into the language discriminator, the
more easily the network can learn the language-invariant features under the
effect of the gradient reversal layer (GRL). And language information is
similar to speaker information, which is a long-term stable information,
usually contained in the low layer of the network. Therefore, we explore the
impact of the position of the language discriminator inserted into the network
and the loss weight of language classification.

Table 3: Comparison of different insertion positions and loss weights in language adversarial training. Layer |  Weight |  AVG. WER
---|---|---
16 |  0.1 |  25.98
16 |  0.01 |  25.16
16 |  0.005 |  25.38
8 |  0.01 |  24.97
4 |  0.01 |  24.66
0 |  0.01 |  24.83

The results of different insertion positions and loss weights of discriminator
are summarized in Table 3, Layer 0 means the input of the context
representation encoder of the pre-training model. First of all, it can be seen
that too large or too small weight is not conducive to the language
adversarial training, and even has a negative impact on multilingual pre-
training. In addition, the lower layer insertion position works best, which is
consistent with the assumption. In this work, the best performance is achieved
by sending the output of the 4-th conformer block to the language
discriminator.

Language embedding involved pre-training:Based on the language adversarial
pre-training, we investigate the influence of the position of language
embedding inserted into network on effectiveness of pre-training. Moreover,
the effect of proposed language orthogonal constraint under different penalty
coefficients is also discussed.

As shown in Table 4, inserting language embedding into the input of
representation encoder performs best, which confirms the existing research
[21, 22] and our observation in Language adversarial pre-training.
Additionally, further improvement can be achieved by adding orthogonal
constraints, and the best penalty coefficient is 10. This shows that language
orthogonal constraints can help model learning more distinguishable language
embedding.

Table 4: Comparison of different insertion positions and orthogonal constraint in language embedding involved pre-training. Layer |  orthogonal constraint |  AVG. WER
---|---|---
0 |  0 |  25.36
8 |  0 |  25.65
16 |  0 |  25.87
0 |  1 |  25.03
0 |  10 |  24.88
0 |  50 |  25.32
Table 5: Comparison of different $k$ in language specific adaptive pre-training with adaptive weights. $k$ in adaptive weight |  AVG. WER
---|---
1 |  24.06
(1,8) |  23.47
8 |  22.09
(8,16) |  22.32
16 |  22.13

Language specific adaptive pre-training: Based on the conclusions of our
previous two investigations and the consideration of additional parameters, we
only insert adapter or adaptive weight into the input of representation
encoder in language specific adaptive pre-training. We explore the effect of
number of $k$ in adaptive weights. The results are presented in Table 5, for
the value in parentheses, the former represents the number of vectors used for
scale matrices, and the latter represents the number of vectors used for bias
matrices. It can be seen that when $k$ for scale matrices and bias matrices
both are eight, adaptive weight can achieve the best performance.

##  5 Conclusion

In this paper, we propose several methods to alleviate language interference
to improve the performance of multilingual pre-training by leveraging
auxiliary language information. These approaches include the extension of
language adversarial training, language embedding and language adaptive
training to pre-training, so as to balance the learning of shared
representations and language specific representations. We analyze the trade-
offs of different methods with regards to quality gains and number of
additional parameters introduced, and further explore the key factors that
affect the effectiveness of each method. We conduct experiments in an ASR task
consisting of 16 languages. Our experimental results demonstrate $\sim$4% to
$\sim$14% relative gain over the standard XLSR model, and $\sim$10% to
$\sim$20% relative gain over the baseline multilingual model.

## References

  * [1] Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert,  “End-to-end asr: from supervised to semi-supervised learning with modern architectures,”  arXiv preprint arXiv:1911.08460, 2019.
  * [2] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  arXiv preprint arXiv:2005.08100, 2020.
  * [3] William Chan, Daniel Park, Chris Lee, Yu Zhang, Quoc Le, and Mohammad Norouzi,  “Speechstew: Simply mix all available speech recognition data to train one large neural network,”  arXiv preprint arXiv:2104.02133, 2021.
  * [4] Anjuli Kannan, Arindrima Datta, Tara N Sainath, Eugene Weinstein, Bhuvana Ramabhadran, Yonghui Wu, Ankur Bapna, Zhifeng Chen, and Seungji Lee,  “Large-scale multilingual speech recognition with a streaming end-to-end model,”  arXiv preprint arXiv:1909.05330, 2019.
  * [5] Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Hannun, Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert,  “Massively multilingual asr: 50 languages, 1 model, 1 billion parameters,”  arXiv preprint arXiv:2007.03001, 2020.
  * [6] Alexander H Liu, Hung-yi Lee, and Lin-shan Lee,  “Adversarial training of end-to-end speech recognition using a criticizing language model,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 6176–6180.
  * [7] Wei-Ning Hsu, Ann Lee, Gabriel Synnaeve, and Awni Hannun,  “Semi-supervised speech recognition via local prior matching,”  arXiv preprint arXiv:2002.10336, 2020.
  * [8] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  Advances in Neural Information Processing Systems, vol. 33, pp. 12449–12460, 2020.
  * [9] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.,  “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”  IEEE Journal of Selected Topics in Signal Processing, 2022.
  * [10] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli,  “Data2vec: A general framework for self-supervised learning in speech, vision and language,”  arXiv preprint arXiv:2202.03555, 2022.
  * [11] Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli,  “Unsupervised cross-lingual representation learning for speech recognition,”  arXiv preprint arXiv:2006.13979, 2020.
  * [12] Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al.,  “Xls-r: Self-supervised cross-lingual speech representation learning at scale,”  arXiv preprint arXiv:2111.09296, 2021.
  * [13] Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli,  “Unsupervised cross-lingual representation learning for speech recognition,”  arXiv preprint arXiv:2006.13979, 2020.
  * [14] Bo Li, Ruoming Pang, Tara N Sainath, Anmol Gulati, Yu Zhang, James Qin, Parisa Haghani, W Ronny Huang, Min Ma, and Junwen Bai,  “Scaling end-to-end models for large-scale multilingual asr,”  in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 1011–1018.
  * [15] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [16] Eric Jang, Shixiang Gu, and Ben Poole,  “Categorical reparameterization with gumbel-softmax,”  arXiv preprint arXiv:1611.01144, 2016.
  * [17] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  arXiv preprint arXiv:2005.08100, 2020.
  * [18] Jiangyan Yi, Jianhua Tao, Zhengqi Wen, and Ye Bai,  “Language-adversarial transfer learning for low-resource speech recognition,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 3, pp. 621–630, 2018.
  * [19] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky,  “Domain-adversarial training of neural networks,”  The journal of machine learning research, vol. 17, no. 1, pp. 2096–2030, 2016.
  * [20] Shubham Toshniwal, Tara N Sainath, Ron J Weiss, Bo Li, Pedro Moreno, Eugene Weinstein, and Kanishka Rao,  “Multilingual speech recognition with a single end-to-end model,”  in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 4904–4908.
  * [21] Shubham Toshniwal, Tara N Sainath, Ron J Weiss, Bo Li, Pedro Moreno, Eugene Weinstein, and Kanishka Rao,  “Multilingual speech recognition with a single end-to-end model,”  in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 4904–4908.
  * [22] Bo Li, Tara N Sainath, Khe Chai Sim, Michiel Bacchiani, Eugene Weinstein, Patrick Nguyen, Zhifeng Chen, Yanghui Wu, and Kanishka Rao,  “Multi-dialect speech recognition with a single sequence-to-sequence model,”  in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 4749–4753.
  * [23] Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier,  “Lightweight adapter tuning for multilingual speech translation,”  arXiv preprint arXiv:2106.01463, 2021.
  * [24] Ngoc-Quan Pham, Tuan-Nam Nguyen, Sebastian Stueker, and Alexander Waibel,  “Efficient weight factorization for multilingual speech recognition,”  arXiv preprint arXiv:2105.03010, 2021.
  * [25] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.


∎

11institutetext: Wenlong Gao 22institutetext: Key Laboratory of Advanced
Process Control for Light Industry (Ministry of Education), Jiangnan
University, Wuxi 214100, China
22email<EMAIL_ADDRESS>33institutetext: Ying Chen 44institutetext:
Key Laboratory of Advanced Process Control for Light Industry (Ministry of
Education), Jiangnan University, Wuxi 214100, China
44email<EMAIL_ADDRESS>55institutetext: Yong Peng 66institutetext:
Key Laboratory of Advanced Process Control for Light Industry (Ministry of
Education), Jiangnan University, Wuxi 214100, China
66email<EMAIL_ADDRESS>
# Cascade Attentive Dropout for Weakly Supervised Object Detection

Wenlong Gao  Ying Chen  Yong Peng

(Received: date / Accepted: date)

###### Abstract

Weakly supervised object detection (WSOD) aims to classify and locate objects
with only image-level supervision. Many WSOD approaches adopt multiple
instance learning as the initial model, which is prone to converge to the most
discriminative object regions while ignoring the whole object, and therefore
reduce the model detection performance. In this paper, a novel cascade
attentive dropout strategy is proposed to alleviate the part domination
problem, together with an improved global context module. We purposely discard
attentive elements in both channel and space dimensions, and capture the
inter-pixel and inter-channel dependencies to induce the model to better
understand the global context. Extensive experiments have been conducted on
the challenging PASCAL VOC 2007 benchmarks, which achieve 49.8% mAP and 66.0%
CorLoc, outperforming state-of-the-arts.

###### Keywords:

Weakly supervised object detection Convolutional neural network Dropout

##  1 Introduction

Figure 1: Typical weakly-supervised object detection results: part-dominated
(first cow)and correct localization (second row)

Weakly Supervised Object Detection (WSOD) is attracting more and more
attention in computer vision area in recent years. The training of a
traditional object detector usually is based on a large amount of manually
labeled data, while the labeling process is time-consuming, expensive, and
inefficient. Weakly supervised object detection has come into being, which
aims to solve this problem by training a detector with only image-level
annotations instead of bounding boxes.

Most methods model WSOD as a multiple instance learning (MIL) problem, where
each image is considered as a bag and the object proposals as instances.
During the training process, the network iterates in the following two steps:
(1) training an object classifier to compute the object score of each object
proposal; (2) selecting the proposals with the highest score and their similar
proposals based on IoU. The model is prone to learn only the most
discriminative object features rather than comprehensive object features so
that it will cause part domination shown in Figure 1. For example, in the case
of a cat, the head may be more discriminative than the body in which
appearance changes dramatically due to patterns. In this case, previous
techniques can localize only the head, rather than the entire region.

Recent work have alleviated this problem by using context
informationkantorov2016contextlocnet ; wei2018ts2c , progressive
refinementtang2018pcl ; tang2017multiple ; wan2019c , and smoothed loss
functionswan2019c . For example, Contextlocnetkantorov2016contextlocnet  built
additive and contrastive guidance models to utilize their surrounding context
feature to enhance localization performance. C-MILwan2019c  divided the
instances in the bag into multiple subsets, and defines corresponding smoother
loss functions on the each subset to approximate the convex loss function.

Attention mechanism, which enables network to focus on the most informative
views, has been proven effective in many fields, such as image
classificationhu2018squeeze , image inpaintingyu2018generative , medical image
segmentationsinha2019multi , etc. However, the mechanism hurts the object
localization performance of WSOD because it focuses only on the most
discriminative features. To address this issue, we propose a cascaded
attentive dropout module (CADM) to inducing the network to learn less
discriminative features for classification but meaningful features for object
localization. Specifically, following channel attentive dropout, two
complementary attentive branch are built and randomly selected to build
spatial-attentive feature maps, where the one rewards the most discriminative
features while the other punishes them via a designed attentive dropout
strategy. A global context module (GCM), which uses sigmoid to enhance
nonlinearity and perform feature fusion through element-wise multiplication
and additions, also been introduced into the proposed network to obtain better
global context information.

In summary, the main contributions of our work can be summarized as follows:

  1. 1)

An end-to-end weakly supervised object detection network is proposed,
considering both network attention and global context information.

  2. 2)

A lightweight but effective cascade attentive dropout module is designed to
help the network learn more comprehensive features rather than only
discriminative features, which notably improve the accuracy of WSOD.

  3. 3)

An improved global context module is introduced to further boost the learned
features in a more efficient way of feature fusion, jointly optimizing the
region classification and localization.

  4. 4)

The proposed network significantly outperforms most state-of-the-art weakly
supervised object detection approaches on PASCAL VOC 2007.

##  2 Related Work

###  2.1 Weakly supervised object detection

Recent work have combined MIL and CNN to train a weakly supervised object
detector in an end-to-end way. Bilen and Vedaldibilen2016weakly  proposed a
two-stream weakly supervised deep detection network (WSDDN) to get
classification and detection scores respectively. Based on WSDDN, Tang et
al.tang2017multiple  proposed an online instance classifier refinement (OICR)
approach to refine the rough output of WSDDN, Kantorov et
al.kantorov2016contextlocnet  introduced two different kinds of context-aware
guidance to improve localization performance of the network. Tang et
al.tang2018pcl  also proposed a graph-based center cluster method to alleviate
the local optimum problem. Some work have also begun to link weak supervision
and strong supervision to comprehensively solve the WSOD problem. Zhang et
al.zhang2018w2f  designed a Weakly-supervised to fully-supervised framework
(W2F) which mines high-quality pseudo ground truth to train a fully-supervised
object detector. Wang et al.wang2018collaborative  proposed a weakly
supervised collaborative learning approach that adopts WSDDN and Faster-RCNN
as weakly and strongly supervised sub-network respectively. From the
perspective of optimization, Wang et al.wan2019c  introduce a continuation
optimization method into MIL to boost the detection performance. Arun et
al.arun2019dissimilarity  designed a novel dissimilarity coefficient based
WSOD framework which is optimized by minimizing the difference between an
annotation agnostic prediction distribution and an annotation aware
conditional distribution. Besides, some work combined the tasks of weakly
supervised object detection and segmentation into a unified end-to-end
learning architecturegao2019c ; zeng2019wsod2 .

###  2.2 Dropout

Dropout is a regularization technique originally proposed by
hinton2012improving  to alleviate the over-fitting problem of fully connected
neural networks, which randomly drops neurons with fixed probability during
network training. However, all neuron activations are retained while testing
the model, and the final output will be scaled according to the dropout
probability. In this way, the network can easily integrate several different
small models to achieve good regularization. Inspired by Dropout,
Dropconnectwan2013regularization , Monte Carlo dropoutgal2016dropout , and
many others were also introduced. This is only useful for the fully connected
layer, but useless for the convolution layer. One of the reasons may be that
the spatially adjacent pixels on the feature map are related to each other and
share contextual information. After that, DeVries et al.devries2017improved
designed Cutout to randomly drop out contiguous regions of input images,
Tompson et al.tompson2015efficient  also proposed SpatialDropout to randomly
drops partial channels of a feature map, rather than dropping the independent
random pixels. Besides, ADLchoe2019attention  was proposed to utilize the
attention mechanism to erase the maximally activated part.

Figure 2: Architecture of our proposed network. (1)Generate comprehensive
feature maps by VGG16 with CADM and GCM. (2)Generate fixed-size ROI features.
(3)Feed the proposal feature vectors to MIL and Refinement submodule to
predict categories and locations.

###  2.3 Attention mechanism

The Attention mechanism is inspired by the human vision which does not treat
all data equally but enhances or weakens them. Recent work have been proposed
to improve the localization performance of the model. For example, Hu et. al
proposed a squeeze-and-excitation network (SENet)hu2018squeeze  to model the
inter-channel dependencies, which generates a weight of $1\times 1\times C$
via a global average pooling layer and two FC layers and multiply it into the
input feature map to get an enhanced feature map. Based on SENet,
SKNetli2019selective  built multiple branches of different receptive fields
and used the information summarized by the multiple scale feature to channel-
wise guide how to allocate the representation of which kernel to focus on.
Wang et. al proposed Non-local Neural networks to fuse the global information
and bring richer semantic information to the following layers. Convolutional
Block Attention Module (CBAM)woo2018cbam  are also proposed to enhance
features in channel and spatial dimensions in a cascading manner.

##  3 The Proposed Approach

In this section, we will introduce our proposed weakly supervised object
detection architecture. As shown in Figure 2, the first stage aims to extract
enhanced feature maps $\boldsymbol{X}_{5}^{*}$ from VGG16 with a cascade
attentive dropout module (CADM) and a global context module (GCM). The
enhanced feature maps and region proposals generated by Selective
Searchuijlings2013selective  are then sent to the RoI pooling layer to produce
fixed-size RoI feature maps. At last, MIL Refinement Module utilizes proposal
feature vectors $\boldsymbol{x}$ to predict object categories and locations.
The proposed CADM, which is designed to elliminate negative effects of
discriminative features, is employed on pooling 3 feature map. Different from
ADL[21] which erased the maximally activated spatial parts, we purposely
discard attentive elements in both channel and space dimension. The remainder
of this section will discuss these components in detail.

Figure 3: Cascade attentive dropout module. This module dropouts elements in
the dimensions of channel and space respectively to induce the model to learn
more comprehensive features.

###  3.1 Cascade Attentive Dropout Module

Weakly supervised object detectors tend to learn only the most discriminative
features in imageswan2018min ; wan2019c , which will cause network
localization errors and decrease detection accuracy. In order to solve this
problem, we try to dropout elements in the dimensions of channel and space
respectively, as shown in Figure 3.

Channel-Dropout. Given a feature map
$\boldsymbol{X}_{3}\in\boldsymbol{R}^{N\times D\times H\times W}$ extracted
from CNN, the channel-dropout module takes it as input and outputs a global
information embedding via a global average pooling (GAP) layer. The embedding
can also be considered as the confidence of different channels, denoted as
$\boldsymbol{f}=(f_{1},f_{2},\cdots,f_{D})$. Note that $N$ is the mini-batch
number, $D$ is the number of channel, $W$ and $H$ are weight and height of the
feature map, respectively. After that, we refer the confidence of the channel
with the highest confidence as $f_{max}$, and set the threshold $\lambda_{1}$.
When the channel confidence $f_{i}$ is greater than $f_{max}\cdot\lambda_{1}$,
the channel $i$ is dropped; otherwise, keep the channel. Consequently, a
binary channel-dropout mask $\boldsymbol{m}_{cd}\in\boldsymbol{R}^{N\times
D\times 1\times 1}$ is generated to indicate whether each channel is dropped
or not, as shown in formula 1.

$m^{i}_{cd}=\left\\{\begin{array}[]{ll}0,&\text{ if
}f_{i}>\left(f_{\max}\cdot\lambda_{1}\right)\\\ 1&\text{ otherwise
}\end{array}\right.$ (1)

where $m^{i}_{cd}$ equal to 0 means the $i$-th channel is dropped. The binary
drop mask is then multiplied to input map $\boldsymbol{X}_{3}$ to get the
channel-dropped feature map $\boldsymbol{X}_{cd}\in\boldsymbol{R}^{N\times
D\times H\times W}$:

$\boldsymbol{X}_{cd}=\boldsymbol{X}_{3}\odot\boldsymbol{m}_{cd}$ (2)

where $\odot$ denotes broadcast element-wise multiplication.

Spatial-Dropout. A complementary symmetric structure is constructed for
spatial dropout to induce the network to learn more comprehensive features. We
first get the self-attention map $\boldsymbol{X}_{a}\in R^{N\times 1\times
H\times W}$ via a channel-wise average pooling layer. Since the activation
value of more discriminative areas in the attention map is higher, we set a
threshold $\lambda_{2}$ to erase these areas to force the network to learn
less discriminative features for classification but meaningful features for
object localization, thereby avoiding location part domination. For the self-
attention map $\boldsymbol{X}_{a}$, the maximum value of $i^{th}$ row is
recorded as $g^{i}_{max}$. When the element $g^{ij}$ in row $i$ and column $j$
of the attention map is greater than the corresponding drop threshold
$g^{i}_{max}\cdot\lambda_{2}$, the element is dropped; otherwise, the element
is retained. As a result, we can obtain a binary spatial-dropout mask
$\boldsymbol{m}_{sd}\in\boldsymbol{R}^{N\times 1\times H\times W}$:

$m^{ij}_{sd}=\left\\{\begin{array}[]{ll}0,&\text{ if
}g^{ij}>\left(g^{i}_{max}\cdot\lambda_{2}\right)\\\ 1&\text{ otherwise
}\end{array}\right.$ (3)

where $m^{ij}_{sd}$ equal to 0 means the element in row $i$ and column $j$ of
$\boldsymbol{X}_{a}$ should be discarded. When $\lambda_{2}$ decreases, more
element values will be discarded.

However, when applying spatial-dropout throughout the training peroid, the
most discriminative elements will always be ignored. As a result, the
classification performance of the network will also be significantly reduced,
which will also harm localization performance. In order to make up for the
reduction of classification ability, we set up a reward branch to further
enhance the discriminative elements. Specifically, the proposed network
activates the self-attention map $\boldsymbol{X}_{a}$ through a sigmoid
function to obtain an importance map $\boldsymbol{m}_{imp}$, where the
intensity of each pixel in the importance map is close to 1 for the most
discriminative features and close to 0 for the least discriminative features.
During the training process, the network stochastically chooses either of the
drop mask or importance map according to $drop\\_rate$, and the selected one
is merged into the input feature map $\boldsymbol{X}_{cd}$ to gain the
spatial-dropped feature $\boldsymbol{X}_{3}^{*}\in R^{N\times D\times H\times
W}$ by element-wise multiplication:

$\boldsymbol{X}_{3}^{*}=\left\\{\begin{array}[]{ll}\boldsymbol{X}_{cd}\odot\boldsymbol{m}_{sd},&\text{
if }(\alpha+drop\\_rate)\textgreater 1\\\
\boldsymbol{X}_{cd}\odot\boldsymbol{m}_{imp}&\text{ otherwise
}\end{array}\right.$ (4)

where $\alpha$ is a random number from a uniform distribution on the interval
[0, 1).

Figure 4: Global context module. (1) Generate global context features. (2)
Capture channel-wise dependencies. (3) Perform feature fusion to obtain
enhanced feature maps.

###  3.2 Global Context Module

The non-local strategywang2018non  has been proved its efficiency on modeling
long-distance dependencies of pixel pairs. NLNet wang2018non  learns a
position-independent attention map for each position, which is time-consuming.
SENethu2018squeeze  uses the global context to calibrate the weights of
different channels to adjust the channel dependence, in which the weight
calibration inhabits its use of global context. In the paper, a new global
context module like GCNetcao2019gcnet  is introduced to enhance the
understanding of the global context in a more efficient way, as shown in
Figure 4.

The module can be divided into three stages: global attention pooling,
bottleneck transform, and feature fusion. At the first stage, we obtain the
attention weights via $1\times 1$ convolutional layer $\boldsymbol{W}_{1}$ and
a softmax layer, and multiple it into the input to get the global context
features $\boldsymbol{\beta}$, which is expected to help the model better
understand the visual scene globally. After that, the model capture channel-
wise dependencies through two $1\times 1$ convolutional layers
$\boldsymbol{W}_{2},\boldsymbol{W}_{3}$. In order to reduce the difficulty of
model optimization, a layer normalization (LN) is inserted into the module
(before ReLU). Sigmoid activation is also employed to learn a non-matually-
exclusive relationship to improve original bottleneck transform of GCNet.
Finally, unlike GCNet, which uses element-wise addition to fuse features, our
enhanced features are fused into the original input by element-wise
multiplication and then addition.

Denoting $\boldsymbol{X}_{5}$ and $\boldsymbol{X}_{5}^{*}$ as the input and
output feature map of the global context module,so this module can be
formulated as :

$\boldsymbol{X}_{5}^{*ij}=\boldsymbol{X}_{5}^{ij}+\boldsymbol{X}_{5}^{ij}\cdot\operatorname{Sig}\left(\boldsymbol{W}_{3}\operatorname{ReLU}\left(\operatorname{LN}\left(\boldsymbol{W}_{2}\boldsymbol{\beta}\right)\right)\right)$
(5)

$\boldsymbol{\beta}=\sum_{i=1}^{H}\sum_{j=1}^{W}\frac{e^{\boldsymbol{W}_{1}\boldsymbol{X}_{5}^{ij}}}{\sum_{s=1}^{H}\sum_{t=1}^{W}e^{\boldsymbol{W}_{1}\boldsymbol{X}_{5}^{st}}}\cdot\boldsymbol{X}_{5}^{ij}$
(6)

where $\operatorname{Sig}$ is the nonlinear activation function Sigmoid, $i$
and $j$ is the index of query positions, and $H$ and $W$ is the height and
weight of the feature map.
$\delta(\cdot)=\operatorname{Sig}\left(\boldsymbol{W}_{3}\operatorname{ReLU}\left(\mathrm{LN}\left(\boldsymbol{W}_{2}\boldsymbol{\beta}\right)\right)\right)$
indicates the bottleneck transform to capture channel-wise dependencies.

Figure 5:  MIL and refinement module. (1) Train a basic mil classifier to
roughly classify instances. (2) Build K instance classifiers and a
distillation branch to optimize the output of the basic instance classifier.

###  3.3 MIL and Refinement Module

Following felipe2020distilling , we build three submodules to classify and
optimize instances, as shown in Figure 5. The first module trains a basic
multiple instance learning network, which branches the proposal feature
vectors into two streams to get
$\boldmath{x}^{det}=\\{\boldsymbol{x}^{det}_{1},\boldsymbol{x}^{det}_{2},\cdots,\boldsymbol{x}^{det}_{|R|}\\},\boldsymbol{x}^{cls}=\\{\boldsymbol{x}^{cls}_{1},\boldsymbol{x}^{cls}_{2},\cdots,\boldsymbol{x}^{cls}_{|R|}\\}$
by two FC layers, where $|R|$ is the number of proposals. These two matrices
are passed through a softmax layer on the classes and proposals dimensions
respectively to get the activated prediction result
$\sigma\left(\boldsymbol{x}^{det}\right),\sigma\left(\boldsymbol{x}^{cls}\right)$
. The predictions of the two branches are fused by element-wise multiplication
to obtain the category prediction score
$\boldsymbol{x}_{r}=\sigma\left(\boldsymbol{x}_{r}^{det}\right)\odot\sigma\left(\boldsymbol{x}_{r}^{cls}\right)$
of the $r^{th}$ region proposal. Finally, the prediction score
$\phi_{c}=\sum_{r=1}^{|R|}\boldsymbol{x}_{cr}$ of this image for class $c$ can
be obtained by summing up the scores over proposal dimensions. We use multi-
classes cross-entropy loss to guide the training of this submodule:

$\mathcal{L}_{cls}=-\sum_{c=1}^{C}\left\\{y_{c}\log\phi_{c}+\left(1-y_{c}\right)\log\left(1-\phi_{c}\right)\right\\}$
(7)

where $y_{c}=1$ indicates that the input image contain $c^{th}$ class object
and $y_{c}=0$ otherwise.

The second submodule builds K instance classifiers to optimize the output of
the basic instance classifier. Each classifier is implemented by a FC layer
and a softmax layer along $C+1$ categories (background is considered as
$0^{th}$class). The output of the $k^{th}$ classifier is considered as the
supervision information of the $(k+1)^{th}$ classifier. so we can train the
$k^{th}$ refined instance classifier based on the loss function
$\mathcal{L}_{ref}^{k}$ in formula 8.

$\mathcal{L}_{ref}^{k}=-\frac{1}{|R|}\sum_{r=1}^{|R|}\sum_{c=1}^{C+1}w_{r}^{k}y_{cr}^{k}\log
x_{cr}^{k}$ (8)

where $w_{r}^{k}$ is the loss weight term of $r^{th}$ region proposal to
reduce the noise of supervision and the same as tang2017multiple .
$y_{cr}^{k}$ is the pseudo grouth truth information for class $c$ from the
$(k-1)^{th}$ classifier, and $x_{cr}^{Rk}$ indicates the prediction score of
$r^{th}$ region proposal for class $c$ in the $k^{th}$ refinement branch.

Finally, we average the outputs of the K refinements agents outputs as the
supervision to guide the distillation agent, which consist of a single FC
layer and a softmax layer over class dimension. The distillation loss
$\mathcal{L}_{dis}$ is the same as the refinement loss $\mathcal{L}_{ref}$. As
a result, The final loss function of the entire network is as follows:

$\mathcal{L}=\mathcal{L}_{cls}+\mathcal{L}_{dis}+\sum^{K}_{k=1}\mathcal{L}^{k}_{ref}.$
(9)

##  4 Experiments

In this section, we will first introduce the dataset we used and the
implementation details of our proposed approach. Then we will explore the
contributions of each proposed module by the ablation experiments. Finally, we
will compare the performance of our proposed network with the-state-of-art
methods.

###  4.1 Datasets and Evaluation Metrics

we evaluate our method on the challenging PASCAL VOC2007
datasetseveringham2010pascal  which have 9963 images for 20 object classes and
are divided into three subsets: training, verification, and testing sets. The
trainval set (5011 images) is chosen to train our proposed network. As we
focus on weakly-supervised detection, only image-level labels (presence or
absence of a class in the chosen image)are utilized during training. For
testing, we evaluating our network using mean average precision
(mAP)everingham2010pascal  and correct localization (CorLoc)
metricsdeselaers2012weakly . All these metrics are based on the PASCAL
criteria of IoU > 0.5 between ground truths and predicted boxes.

###  4.2 Implementation Details

We use Selective searchuijlings2013selective  to generate original region
proposal and build our proposed network on VGG16simonyan2014very  pre-trained
on ImageNetdeng2009imagenet . We add the CADM module to the back layer of
pooling3. The last max-pooling layer is replaced by ROI pooling and the last
FC layer and softmax loss layer are replaced by the layer described in Section
3.3. We insert the global context module in front of the ROI layer.

The original input images are resized into five different scales
{480,576,688,864,1200} concerning the smallest image dimension. The resized
scale of a training image is randomly selected and the image is randomly
horizontal flipped. In this way, each image is augmented into a total of ten
images as many other WSOD methods dotang2017multiple ; tang2018pcl ;
wan2018min ; wan2019c . During network learning, we employ the SGD algorithm
with momentum 0.9, weight decay $5\times 10^{-4}$ and batch size 4. The model
iterates 50K iterations, where the learning rate is set to $5\times 10^{-4}$
for the first 20K iterations and then decreases to $5\times 10^{-5}$ in the
following 30K iterations. During testing, all ten augmented images are passed
into the network, and take the averages as the final outputs. Non-maximum
suppression is applied to all prediction with the IoU threshlod set to 0.3.
Our experiments are implemented based on PyTorch deep learning framework and a
NVIDIA GTX 1080Ti GPU.

Table 1: Ablation study : AP performance(%) of each category on the PASCAL VOC 2007 test set Methods | aero | bike | bird | boat | bottle | bus | car | cat | chair | cow
---|---|---|---|---|---|---|---|---|---|---
Baselinefelipe2020distilling  | 63.1 | 66.4 | 46.1 | 25.4 | 16.9 | 70.8 | 68.8 | 53.2 | 14.9 | 56.8
\+ CADM | 64.6 | 64.7 | 53.1 | 33.1 | 23.2 | 70.4 | 70.6 | 22.2 | 22.1 | 64.1
\+ GCM | 66.0 | 66.0 | 56.8 | 20.3 | 19.1 | 72.5 | 70.6 | 59.5 | 19.6 | 64.3
\+ Both | 66.5 | 65.6 | 56.5 | 26.8 | 19.7 | 69.9 | 69.0 | 61.3 | 21.5 | 66.9
Methods | table | dog | horse | mbike | person | plant | sheep | sofa | train | tv
Baselinefelipe2020distilling  | 41.5 | 53.7 | 42.7 | 70.0 | 2.9 | 20.6 | 42.8 | 44.8 | 50.8 | 68.3
\+ CADM | 46.2 | 27.1 | 49.3 | 70.8 | 2.6 | 19.5 | 57.1 | 55.3 | 64.6 | 69.9
\+ GCM | 42.8 | 47.8 | 42.8 | 68.5 | 2.5 | 20.0 | 47.6 | 48.7 | 62.9 | 64.2
\+ Both | 43.1 | 50.4 | 49.0 | 70.1 | 2.3 | 20.1 | 53.9 | 47.4 | 65.7 | 70.7
Table 2: Ablation study : CorLoc performance(%) of each category on the PASCAL VOC 2007 trainval set Methods | aero | bike | bird | boat | bottle | bus | car | cat | chair | cow
---|---|---|---|---|---|---|---|---|---|---
Baselinefelipe2020distilling  | 84.6 | 78.4 | 59.2 | 49.5 | 44.7 | 77.7 | 85.0 | 61.0 | 34.8 | 75.3
\+ CADM | 84.6 | 77.6 | 70.6 | 56.4 | 50.4 | 78.7 | 84.4 | 39.0 | 47.9 | 82.9
\+ GCM | 84.2 | 82.4 | 70.3 | 64.4 | 44.7 | 79.7 | 86.6 | 47.1 | 46.9 | 84.2
\+ Both | 84.2 | 75.7 | 75.1 | 50.0 | 40.8 | 77.7 | 82.3 | 68.6 | 46.3 | 82.2
Methods | table | dog | horse | mbike | person | plant | sheep | sofa | train | tv
Baselinefelipe2020distilling  | 44.1 | 70.5 | 65.0 | 88.8 | 11.4 | 57.1 | 73.2 | 51.9 | 66.2 | 82.4
\+ CADM | 52.1 | 38.4 | 70.4 | 90.4 | 10.8 | 55.7 | 81.4 | 69.4 | 78.7 | 82.1
\+ GCM | 48.7 | 40.0 | 73.5 | 88.8 | 11.0 | 55.7 | 82.5 | 66.4 | 70.3 | 82.4
\+ Both | 52.9 | 63.5 | 73.1 | 89.6 | 10.6 | 52.7 | 79.4 | 55.4 | 78.7 | 82.1
Table 3: Ablation study : average detection and localization performance(%) on PASCAL VOC 2007 Methods | mAP | CorLoc
---|---|---
Baselinefelipe2020distilling  | 46.0 | 63.0
\+ CADM | 47.5 | 65.1
\+ GCM | 48.1 | 65.5
\+ CADM + GCM | 49.8 | 66.0
Table 4: Ablation study : different way of feature fusion in GCM on PASCAL VOC 2007. $\checkmark$ indicates that the component is used. multiplication | addition | multiplication + addition | mAP
---|---|---|---
|  |  | 46
$\checkmark$ |  |  | 47.8
| $\checkmark$ |  | 47.0
|  | $\checkmark$ | 48.1
Table 5: Ablation study : different dimension of dropout in CADM on PASCAL VOC 2007. $\checkmark$ indicates that the component is used. channel-dropout | spatial-dropout | channel + spatial dropout | mAP
---|---|---|---
|  |  | 48.1
$\checkmark$ |  |  | 47.6
| $\checkmark$ |  | 48.8
|  | $\checkmark$ | 49.8

###  4.3 Ablation experiments

we conduct extensive ablation experiments on PASCAL VOC2007 to prove the
effectiveness of our proposed network and respectively validate the
contribution of each component including CADM and GCM.

####  4.3.1 Baseline

We use Boosted-OICRfelipe2020distilling  as our baseline. We re-run the
author’s code multiple times with the same configuration, but the highest mAP
we can achieve is only 46.0% due to different cuda versions and GPUs. 111More
details about the issue can refer to
https://github.com/ppengtang/pcl.pytorch/issues/9

####  4.3.2 GCM

We conduct experiments with and without GCM to illustrate the effectiveness of
GCM and denote the network with GCM as +GCM, which does not include CADM and
report the results in Table 1 , Table 2 and Table 3. The detection performance
of 13 classes and the localization performance of 11 classes have been
improved. Figure 3 shows that applying global context module to our proposed
network could improve the performance of the model by at least 2.1% mAP and
2.5% CorLoc.

We also explore the effect of different way of feature fusion on model
performance on PASCAL VOC 2007 and the results are shown in Table 4. It shows
that multiplication and then addition is more effective than simple
multiplication or addition in the fusion stage.

Figure 6: Ablation study : influence of different values of $\lambda_{1}$ (left), $\lambda_{2}$ (middle), $drop\\_rate$ (right) in CADB on model performance. Table 6: Detection performance (%) of each category on the VOC 2007 test set. Comparison to the state-of-the-arts. Methods | aero | bike | bird | boat | bottle | bus | car | cat | chair | cow
---|---|---|---|---|---|---|---|---|---|---
WSDDNbilen2016weakly  | 46.4 | 58.3 | 35.5 | 25.9 | 14.0 | 66.7 | 53.0 | 39.2 | 8.9 | 41.8
DSTLjie2017deep  | 52.2 | 47.1 | 35 | 26.7 | 15.4 | 61.3 | 66 | 54.3 | 3 | 53.6
OICRtang2017multiple  | 58.0 | 62.4 | 31.1 | 19.4 | 13.0 | 65.1 | 62.2 | 28.4 | 24.8 | 44.7
WCCNdiba2017weakly  | 49.5 | 60.6 | 38.6 | 29.2 | 16.2 | 70.8 | 56.9 | 42.5 | 10.9 | 44.1
PCLtang2018pcl  | 54.4 | 69.0 | 39.3 | 19.2 | 15.7 | 62.9 | 64.4 | 30.0 | 25.1 | 52.5
TS2Cwei2018ts2c  | 59.3 | 57.5 | 43.7 | 27.3 | 13.5 | 63.9 | 61.7 | 59.9 | 24.1 | 46.9
C-WSLgao2018c  | 62.7 | 63.7 | 40.0 | 25.5 | 17.7 | 70.1 | 68.3 | 38.9 | 25.4 | 54.5
W2Fzhang2018w2f  | 60.9 | 68.7 | 47.1 | 31.7 | 14.2 | 71.2 | 68.9 | 24.5 | 23.5 | 57.6
WeakRPNtang2018weakly  | 57.9 | 70.5 | 37.8 | 5.7 | 21.0 | 66.1 | 69.2 | 59.4 | 3.4 | 57.1
BOICRfelipe2020distilling  | 63.1 | 66.4 | 46.1 | 25.4 | 16.9 | 70.8 | 68.8 | 53.2 | 14.9 | 56.8
MELMwan2018min  | 55.6 | 66.9 | 34.2 | 29.1 | 16.4 | 68.8 | 68.1 | 43.0 | 25.0 | 65.6
ZLDNzhang2018zigzag  | 55.4 | 68.5 | 50.1 | 16.8 | 20.8 | 62.7 | 66.8 | 56.5 | 2.1 | 57.8
WSCDNwang2018collaborative  | 61.2 | 66.6 | 48.3 | 26.0 | 15.8 | 66.5 | 65.4 | 53.9 | 24.7 | 61.2
C-MILwan2019c  | 62.5 | 58.4 | 49.5 | 32.1 | 19.8 | 70.5 | 66.1 | 63.4 | 20.0 | 60.5
Ours | 66.5 | 65.6 | 56.45 | 26.8 | 19.7 | 69.9 | 69.0 | 61.3 | 21.5 | 66.9
Methods | table | dog | horse | mbike | person | plant | sheep | sofa | train | tv
WSDDNbilen2016weakly  | 26.6 | 38.6 | 44.7 | 59.0 | 10.8 | 17.3 | 40.7 | 49.6 | 56.9 | 50.8
DSTLjie2017deep  | 24.7 | 43.6 | 48.4 | 65.8 | 6.6 | 18.8 | 51.9 | 43.6 | 53.6 | 62.4
OICRtang2017multiple  | 30.6 | 25.3 | 37.8 | 65.5 | 15.7 | 24.1 | 41.7 | 46.9 | 64.3 | 62.6
WCCNdiba2017weakly  | 29.9 | 42.2 | 47.9 | 64.1 | 13.8 | 23.5 | 45.9 | 54.1 | 60.8 | 54.5
PCLtang2018pcl  | 44.4 | 19.6 | 39.3 | 67.7 | 17.8 | 22.9 | 46.6 | 57.5 | 58.6 | 63.0
TS2Cwei2018ts2c  | 36.7 | 45.6 | 39.9 | 62.6 | 10.3 | 23.6 | 41.7 | 52.4 | 58.7 | 56.6
C-WSLgao2018c  | 41.6 | 29.9 | 37.9 | 64.2 | 11.3 | 27.4 | 49.3 | 54.7 | 61.4 | 67.4
W2Fzhang2018w2f  | 43.6 | 20.9 | 47.9 | 66.0 | 11.3 | 22.3 | 56.4 | 57.7 | 61.1 | 60.1
WeakRPNtang2018weakly  | 57.3 | 35.2 | 64.2 | 68.6 | 32.8 | 28.6 | 50.8 | 49.5 | 41.1 | 30.0
BOICRfelipe2020distilling  | 41.5 | 53.7 | 42.7 | 70 | 2.9 | 20.6 | 42.8 | 44.8 | 50.8 | 68.3
MELMwan2018min  | 45.3 | 53.2 | 49.6 | 68.6 | 2.0 | 25.4 | 52.5 | 56.8 | 62.1 | 57.1
ZLDNzhang2018zigzag  | 47.5 | 40.1 | 69.7 | 68.2 | 21.6 | 27.2 | 53.4 | 56.1 | 52.5 | 58.2
WSCDNwang2018collaborative  | 46.2 | 53.5 | 48.5 | 66.1 | 12.1 | 22.0 | 49.2 | 53.2 | 66.2 | 59.4
C-MILwan2019c  | 52.9 | 53.5 | 57.4 | 68.9 | 8.4 | 24.6 | 51.8 | 58.7 | 66.7 | 63.5
Ours | 43.1 | 50.4 | 49.0 | 70.1 | 2.3 | 20.1 | 53.9 | 47.4 | 65.7 | 70.7
Table 7: Localization performance (%) of each category on the VOC 2007 trainval set. Comparison to the state-of-the-arts. Methods | aero | bike | bird | boat | bottle | bus | car | cat | chair | cow
---|---|---|---|---|---|---|---|---|---|---
WSDDNbilen2016weakly  | 65.1 | 58.8 | 58.5 | 33.1 | 39.8 | 68.3 | 60.2 | 59.6 | 34.8 | 64.5
DSTLjie2017deep  | 72.7 | 55.3 | 53.0 | 27.8 | 35.2 | 68.6 | 81.9 | 60.7 | 11.6 | 71.6
OICRtang2017multiple  | 81.7 | 80.4 | 48.7 | 49.5 | 32.8 | 81.7 | 85.4 | 40.1 | 40.6 | 79.5
WCCNdiba2017weakly  | 83.9 | 72.8 | 64.5 | 44.1 | 40.1 | 65.7 | 82.5 | 58.9 | 33.7 | 72.5
PCLtang2018pcl  | 79.6 | 85.5 | 62.2 | 47.9 | 37.0 | 83.8 | 83.4 | 43.0 | 38.3 | 80.1
TS2Cwei2018ts2c  | 84.2 | 74.1 | 61.3 | 52.1 | 32.1 | 76.7 | 82.9 | 66.6 | 42.3 | 70.6
C-WSLgao2018c  | 86.3 | 80.4 | 58.3 | 50.0 | 36.6 | 85.8 | 86.2 | 47.1 | 42.7 | 81.5
WeakRPNtang2018weakly  | 77.5 | 81.2 | 55.3 | 19.7 | 44.3 | 80.2 | 86.6 | 69.5 | 10.1 | 87.7
BOICRfelipe2020distilling  | 84.6 | 78.4 | 59.2 | 49.5 | 44.7 | 77.7 | 85.0 | 61.0 | 34.8 | 75.3
ZLDNzhang2018zigzag  | 74.0 | 77.8 | 65.2 | 37.0 | 46.7 | 75.8 | 83.7 | 58.8 | 17.5 | 73.1
WSCDNwang2018collaborative  | 85.8 | 80.4 | 73.0 | 42.6 | 36.6 | 79.7 | 82.8 | 66.0 | 34.1 | 78.1
ours | 84.2 | 75.7 | 75.1 | 50.0 | 40.8 | 77.7 | 82.3 | 68.6 | 46.3 | 82.2
Methods | table | dog | horse | mbike | person | plant | sheep | sofa | train | tv
WSDDNbilen2016weakly  | 30.5 | 43.0 | 56.8 | 82.4 | 25.5 | 41.6 | 61.5 | 55.9 | 65.9 | 63.7
DSTLjie2017deep  | 29.7 | 54.3 | 64.3 | 88.2 | 22.2 | 53.7 | 72.2 | 52.6 | 68.9 | 75.5
OICRtang2017multiple  | 35.7 | 33.7 | 60.5 | 88.8 | 21.8 | 57.9 | 76.3 | 59.9 | 75.3 | 81.4
WCCNdiba2017weakly  | 25.6 | 53.7 | 67.4 | 77.4 | 26.8 | 49.1 | 68.1 | 27.9 | 64.5 | 55.7
PCLtang2018pcl  | 50.6 | 30.9 | 57.8 | 90.8 | 27.0 | 58.2 | 75.3 | 68.5 | 75.7 | 78.9
TS2Cwei2018ts2c  | 39.5 | 57.0 | 61.2 | 88.4 | 9.3 | 54.6 | 72.2 | 60.0 | 65.0 | 70.3
C-WSLgao2018c  | 42.2 | 42.6 | 50.7 | 90.0 | 14.3 | 61.9 | 85.6 | 64.2 | 77.2 | 82.4
WeakRPNtang2018weakly  | 68.4 | 52.1 | 84.4 | 91.6 | 57.4 | 63.4 | 77.3 | 58.1 | 57.0 | 53.8
BOICRfelipe2020distilling  | 44.1 | 70.5 | 65.0 | 88.8 | 11.4 | 57.1 | 73.2 | 51.9 | 66.2 | 82.4
ZLDNzhang2018zigzag  | 49.0 | 51.3 | 76.7 | 87.4 | 30.6 | 47.8 | 75.0 | 62.5 | 64.8 | 68.8
WSCDNwang2018collaborative  | 36.9 | 68.6 | 72.4 | 91.6 | 22.2 | 51.3 | 79.4 | 63.7 | 74.5 | 74.6
Ours | 52.9 | 63.5 | 73.1 | 89.6 | 10.6 | 52.7 | 79.4 | 55.4 | 78.7 | 82.1

####  4.3.3 CADM

To verify the effectiveness of CADM, we also conduct experiments with and
without CADM. We empirically set original threshold $\lambda_{1},\lambda_{2}$
to 0.8, $drop\\_rate$ to 0.8 and denote the network with GAM as +CADM, which
does not include a global context module. From Table 1 , Table 2 and Table 3,
we can conclude that CADM does help the detector learn comprehensive features
rather than only discriminative features and brings at least 1.5% mAP and 2.1%
CorLoc improvement.

We also further investigate the effect of different
$\lambda_{1},\lambda_{2},drop\\_rate$ on detection performance, as shown in
Figure 6. When exploring the role of a certain value, the other two values are
fixed at 0.8. From the three subgraphs, we can find that different values have
a greater impact on the performance of the model, sometimes it will greatly
improve the performance of the model (49.8% vs 48.1%), but sometimes it will
hurt the performance of the model (46.8% vs 48.1%). As a result, an
appropriate threshold selection is very important.

From Table 5, we can find that using the spatial-dropout submodule can
slightly improve the performance of the model, but simply using the channel-
dropout submodule will damage the performance of the model to a certain
extent. Interestingly, when both submodules are inserted into the network in a
cascaded way, the performance of the model is greatly improved. We believe
that this is because a simple channel-dropout submodule may throw away too
many elements, and the increase in localization performance cannot make up for
the loss of classification performance, thereby compromising the final
accuracy. But when the two are combined, the more discriminative features are
moderately discarded to induce the network to learn those less discriminative
features, thereby greatly improving the performance of the model.

Table 8: Average detection and localization performance on PASCAL VOC 2007. Methods | mAP | CorLoc
---|---|---
WSDDN bilen2016weakly  | 39.2 | 53.5
DSTL jie2017deep  | 41.7 | 56.1
OICR tang2017multiple  | 42.0 | 60.6
WCCN diba2017weakly  | 42.8 | 56.7
PCL tang2018pcl  | 43.5 | 62.7
TS2C wei2018ts2c  | 44.3 | 61.0
C-WSL gao2018c  | 45.6 | 63.3
WSD+FSD1 zhang2018w2f  | 45.8 | 65.0
WeakRPN tang2018weakly  | 45.3 | 63.8
BOICR felipe2020distilling  | 46.0 | 63.0
MELM wan2018min  | 47.3 | 61.4
ZLDN zhang2018zigzag  | 47.3 | 61.2
WSCDN wang2018collaborative  | 48.3 | 64.7
C-MIL wan2019c  | 50.3 | 65.0
Ours | 49.8 | 66.0
Table 9: Average detection and localization performance on PASCAL VOC 2012. Methods | mAP | CorLoc
---|---|---
DSTL jie2017deep  | 38.3 | 58.8
OICR tang2017multiple  | 37.9 | 62.1
WCCN diba2017weakly  | 37.9 | -
PCL tang2018pcl  | 40.6 | 63.2
TS2C wei2018ts2c  | 40.0 | 64.4
C-WSL gao2018c  | 41.5 | 64.2
WSD+FSD1 zhang2018w2f  | 42.4 | 65.5
WeakRPN tang2018weakly  | 40.8 | 64.9
MELM wan2018min  | 42.4 | -
ZLDN zhang2018zigzag  | 42.9 | 61.5
WSCDN wang2018collaborative  | 43.3 | 65.2
C-MIL wan2019c  | 46.6 | 67.4
SDCN li2019weakly  | 43.5 | 67.9
BOICR felipe2020distilling  | 46.7 | 66.3
MIL-OICR+GAM+REG yang2019towards  | 46.8 | 69.5
Ours | 46.9 | 66.5

###  4.4 Comparison with state-of-the-art

Table 6 and Table 8 shows the detection performance of our proposed method and
a comparison with other state-of-the-art methods on the PASCAL VOC 2007 test
set. It can be seen that our proposed method improves the original Boosted-
OICRfelipe2020distilling  code in 3.8% mAP, and outperformed
WSDDNbilen2016weakly , OICRtang2017multiple , WSCDNwang2018collaborative ,
MELMwan2018min  and other approaches by 1.5% $\sim$ 10.6%. Meanwhile, our
method achieved the highest AP results in 6 classes(aeroplane, bird,
motorbike, cow, sheep and tv).

We also evaluated object location performance of our network and compared it
with other SOTA on the PASCAL VOC 2007 trainval set in Table 7 and Table 8.
The proposed network reached the best localization performance in 4 of the 20
classes(aeroplane, bird, chair and train). our network respectively
outperformed WSDDNbilen2016weakly , OICRtang2017multiple ,
WSCDNwang2018collaborative , MELMwan2018min , C-MILwan2019c  and other
approaches by 1.0% $\sim$ 12.5%.

We achieve a performance of 46.9% mAP and 66.5% CorLoc on Pascal VOC 2012,
which is supervior to previous work in with gain of about 0.1% $\sim$ 8.6% mAP
in Table 9.

Figure 7: Visualization of our detection results(2,4 columns) and
baselinefelipe2020distilling  detection results(1,3 columns). The bule, green
and red boxes repectively indicies the ground-truth boxes, correct
detctions(IoU > 0.5 with grouth-truth boxes ) and wrong detections. The label
in each box is the category prediction and its confidence score.

###  4.5 Visualization

The visualization of our network and baselinefelipe2020distilling  detection
results is shown in Figure 7. The first and third columns are the detection
results of felipe2020distilling , and the other two columns are our detection
results. Our proposed methods have achieved better detection results than
felipe2020distilling , and the part-dominated problem has been better
alleviated as shown in the first two rows. As can be seen from the third row
of pictures, our method can also improve the problem of too large a prediction
box.

##  5 Conclusion

In this paper, we propose a simple but effective architecture for weakly
supervised object detection. The network selectively discards the most
discriminative features in the channel and spatial dimensions based on
attention mechanism. To understand the image context information better, a
global context module is also introduced into MIL. We have conducted extensive
experiments and results show substantial and distinctive improvement of the
our proposed method.

###### Acknowledgements.

This work is supported by the National Natural Science Foundation of
China(grant no. 61573168)

## References

  * (1) Kantorov, V., Oquab, M., Cho, M., Laptev, I.: Contextlocnet: Context-aware deep network models for weakly supervised localization.  In: European Conference on Computer Vision, pp. 350–365. Springer (2016)
  * (2) Wei, Y., Shen, Z., Cheng, B., Shi, H., Xiong, J., Feng, J., Huang, T.: Ts2c: Tight box mining with surrounding segmentation context for weakly supervised object detection.  In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 434–450 (2018)
  * (3) Tang, P., Wang, X., Bai, S., Shen, W., Bai, X., Liu, W., Yuille, A.: Pcl: Proposal cluster learning for weakly supervised object detection.  IEEE transactions on pattern analysis and machine intelligence 42(1), 176–191 (2018)
  * (4) Tang, P., Wang, X., Bai, X., Liu, W.: Multiple instance detection network with online instance classifier refinement.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2843–2851 (2017)
  * (5) Wan, F., Liu, C., Ke, W., Ji, X., Jiao, J., Ye, Q.: C-mil: Continuation multiple instance learning for weakly supervised object detection.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2199–2208 (2019)
  * (6) Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141 (2018)
  * (7) Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505–5514 (2018)
  * (8) Sinha, A., Dolz, J.: Multi-scale guided attention for medical image segmentation.  arXiv preprint arXiv:1906.02849 (2019)
  * (9) Bilen, H., Vedaldi, A.: Weakly supervised deep detection networks.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2846–2854 (2016)
  * (10) Zhang, Y., Bai, Y., Ding, M., Li, Y., Ghanem, B.: W2f: A weakly-supervised to fully-supervised framework for object detection.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 928–936 (2018)
  * (11) Wang, J., Yao, J., Zhang, Y., Zhang, R.: Collaborative learning for weakly supervised object detection.  arXiv preprint arXiv:1802.03531 (2018)
  * (12) Arun, A., Jawahar, C., Kumar, M.P.: Dissimilarity coefficient based weakly supervised object detection.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9432–9441 (2019)
  * (13) Gao, Y., Liu, B., Guo, N., Ye, X., Wan, F., You, H., Fan, D.: C-midn: Coupled multiple instance detection network with segmentation guidance for weakly supervised object detection.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 9834–9843 (2019)
  * (14) Zeng, Z., Liu, B., Fu, J., Chao, H., Zhang, L.: Wsod2: Learning bottom-up and top-down objectness distillation for weakly-supervised object detection.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 8292–8300 (2019)
  * (15) Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors.  arXiv preprint arXiv:1207.0580 (2012)
  * (16) Wan, L., Zeiler, M., Zhang, S., Le Cun, Y., Fergus, R.: Regularization of neural networks using dropconnect.  In: International conference on machine learning, pp. 1058–1066 (2013)
  * (17) Gal, Y., Ghahramani, Z.: Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In: international conference on machine learning, pp. 1050–1059 (2016)
  * (18) DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552 (2017)
  * (19) Tompson, J., Goroshin, R., Jain, A., LeCun, Y., Bregler, C.: Efficient object localization using convolutional networks.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 648–656 (2015)
  * (20) Choe, J., Shim, H.: Attention-based dropout layer for weakly supervised object localization.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2219–2228 (2019)
  * (21) Li, X., Wang, W., Hu, X., Yang, J.: Selective kernel networks (2019)
  * (22) Woo, S., Park, J., Lee, J.Y., So Kweon, I.: Cbam: Convolutional block attention module.  In: Proceedings of the European conference on computer vision (ECCV), pp. 3–19 (2018)
  * (23) Uijlings, J.R., Van De Sande, K.E., Gevers, T., Smeulders, A.W.: Selective search for object recognition.  International journal of computer vision 104(2), 154–171 (2013)
  * (24) Wan, F., Wei, P., Jiao, J., Han, Z., Ye, Q.: Min-entropy latent model for weakly supervised object detection.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1297–1306 (2018)
  * (25) Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794–7803 (2018)
  * (26) Cao, Y., Xu, J., Lin, S., Wei, F., Hu, H.: Gcnet: Non-local networks meet squeeze-excitation networks and beyond.  In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0–0 (2019)
  * (27) Felipe Zeni, L., Jung, C.R.: Distilling knowledge from refinement in multiple instance detection networks.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 768–769 (2020)
  * (28) Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge.  International journal of computer vision 88(2), 303–338 (2010)
  * (29) Deselaers, T., Alexe, B., Ferrari, V.: Weakly supervised localization and learning with generic knowledge.  International journal of computer vision 100(3), 275–293 (2012)
  * (30) Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556 (2014)
  * (31) Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database.  In: 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee (2009)
  * (32) Jie, Z., Wei, Y., Jin, X., Feng, J., Liu, W.: Deep self-taught learning for weakly supervised object localization.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1377–1385 (2017)
  * (33) Diba, A., Sharma, V., Pazandeh, A., Pirsiavash, H., Van Gool, L.: Weakly supervised cascaded convolutional networks.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 914–922 (2017)
  * (34) Gao, M., Li, A., Yu, R., Morariu, V.I., Davis, L.S.: C-wsl: Count-guided weakly supervised localization.  In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 152–168 (2018)
  * (35) Tang, P., Wang, X., Wang, A., Yan, Y., Liu, W., Huang, J., Yuille, A.: Weakly supervised region proposal network and object detection.  In: Proceedings of the European conference on computer vision (ECCV), pp. 352–368 (2018)
  * (36) Zhang, X., Feng, J., Xiong, H., Tian, Q.: Zigzag learning for weakly supervised object detection.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4262–4270 (2018)
  * (37) Li, X., Kan, M., Shan, S., Chen, X.: Weakly supervised object detection with segmentation collaboration.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 9735–9744 (2019)
  * (38) Yang, K., Li, D., Dou, Y.: Towards precise end-to-end weakly supervised object detection network.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 8372–8381 (2019)