# Neural Semi-supervised Learning for Text Classification
Under Large-Scale Pretraining

Zijun Sun♣, Chun Fan♠★, Xiaofei Sun♣, Yuxian Meng♣, Fei Wu◆ and Jiwei Li◆♣
◆Department of Computer Science and Technology, Zhejiang University
♠Computer Center of Peking University, ★Peng Cheng Laboratory
♣ Shannon.AI
{zijun_sun, xiaofei_sun, yuxian_meng<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The goal of semi-supervised learning is to utilize the unlabeled, in-domain
dataset $U$ to improve models trained on the labeled dataset $D$. Under the
context of large-scale language-model (LM) pretraining, how we can make the
best use of $U$ is poorly understood: Is semi-supervised learning still
beneficial with the presence of large-scale pretraining? Should $U$ be used
for in-domain LM pretraining or pseudo-label generation? How should the
pseudo-label based semi-supervised model be actually implemented? How
different semi-supervised strategies (e.g., self-learning) affect performances
regarding $D$ of different sizes, $U$ of different sizes, etc.

In this paper, we conduct comprehensive studies on semi-supervised learning in
the task of text classification under the context of large-scale LM
pretraining. Our studies shed important lights on the behavior of semi-
supervised learning methods. We find that: (1) with the presence of in-domain
LM pretraining on $U$, open-domain LM pretraining Devlin et al. (2018) is
unnecessary, and we are able to achieve better performance with pretraining on
the in-domain dataset $U$; (2) both the in-domain pretraining strategy and the
pseudo-label based strategy introduce significant performance boosts, with the
former performing better with larger $U$, the latter performing better with
smaller $U$, and the combination leading to the largest performance gain; (3)
vanilla self-training (pretraining first on the pseudo-label dataset
$D^{\prime}$ and then fine-tuning on $D$) yields better performances when $D$
is small, while joint training on the combination of $D^{\prime}$ and $D$
yields better performances when $D$ is large.

Using semi-supervised learning strategies, we are able to achieve a
performance of around $93.8\%$ accuracy with only 50 training data points on
the IMDB dataset, and a competitive performance of 96.6$\%$ with the full IMDB
dataset. Our work marks an initial step toward understanding the behavior of
semi-supervised learning models under the context of large-scale
pretraining.111Code, models and datasets can be found at
https://github.com/ShannonAI/Neural-Semi-Supervised-Learning-for-Text-
Classification

##  1 Introduction

Because of the fact that obtaining supervised training labels is costly and
time-intensive, and that unlabeled data is relatively easy to obtain, semi-
supervised learning Chapelle et al. (2006); Zhu (2005), which utilizes in-
domain unlabeled data $U$ to improve models trained on the labeled dataset
$D$, is of growing interest. Under the context of large-scale of language
model pretraining (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019;
Lewis et al., 2019; Bao et al., 2020; Joshi et al., 2020), where a language
model is pretrained on an extremely large, open-domain dataset (denoted by
largeU, with |largeU| $\gg$ |U|), how we can make the best use of the in-
domain unlabeled dataset $U$ is poorly understood. There are basically two
ways to take advantages of the unlabeled, in-domain dataset $U$: in-domain
pretraining222To note, the pretraining on the in-domain dataset $U$ is
distinguished from the pretraining on the large-scale, open-domain dataset
largeU. The model for in-domain pretraining can be randomly initialized or
taking a pretrained model based on the open-domain dataset largeU (Gururangan
et al., 2020)., where a language model is pretrained on the in-domain dataset
$U$, and then fine-tuned on $D$; pseudo-label based approach (Lee, 2013; Reed
et al., 2015; Iscen et al., 2019; Shi et al., 2018; Arazo et al., 2020), where
unlabeled data points are assigned with labels predicted by the model trained
on $D$ (referred to as the teacher model), forming a new dataset $D^{\prime}$.
A new model (referred to as the student model) is trained for final
predictions by considering $D^{\prime}$.

Many important questions regarding the behavior of semi-supervised learning
models under the context of large-scale LM pretraining remain unanswered: Is
semi-supervised training still beneficial with the presence of large scale
pretraining on largeU? Should $U$ be used for in-domain LM pretraining or
pseudo-label generation? How should pseudo-label based semi-supervised models
be implemented? How different semi-supervised strategies (e.g., self learning)
affect performances regarding $D$ of different sizes, and $U$ of different
sizes, etc.

In this paper, we conduct comprehensive studies on the behavior of semi-
supervised learning in NLP with the presence of large-scale language model
pretraining. We use the task of text classification as an example, the method
of which can be easily adapted to different NLP tasks. Our work sheds
important lights on the behavior of semi-supervised learning models: we find
that (1) with the presence of in-domain pretraining LM on $U$, open-domain LM
pretraining Devlin et al. (2018) is unnecessary, and we are able to achieve
better performance with pretraining on the in-domain dataset $U$; (2) both the
in-domain pretraining strategy and the pseudo-label based strategy lead to
significant performance boosts, with the former performing better with larger
$U$, the latter performing better with smaller $U$, and the combination of
both performing the best; (3) for pseudo-label based strategies, self-training
(pretraining first on the pseudo-label dataset $D^{\prime}$ and then fine-
tuning on $D$) yields better performances when $D$ is small, while joint
training on the combination of $D^{\prime}$ and $D$ yields better performances
when $D$ is large.

Using semi-supervised learning models, we are able to achieve a performance of
around $93-94\%$ accuracy with only 50 training data points on the IMDB
dataset, and a competitive performance of 96.6$\%$ with the full dataset. More
importantly, our work marks an initial step toward understanding the behavior
of semi-supervised learning models in the context of large-scale pretraining.

The rest of this paper is organized as follows: related work is detailed in
Section 2. Different strategies for training semi-supervised models are shown
in Section 3. Experimental results and findings are shown in Section 4,
followed by a brief conclusion in Section 5.

##  2 Related Work

###  2.1 Semi-Supervised Learning

The goal of semi-supervised learning (Chapelle et al., 2006; Zhu, 2005) is to
use massive amount of unlabeled data to improve the models trained on labeled
data. One widely-used type of semi-supervised method is the pseudo-label based
method (Lee, 2013; Reed et al., 2015; Iscen et al., 2019; Shi et al., 2018;
Arazo et al., 2020), where unlabeled data points are assigned with labels
predicted by a trained model, forming a large pseudo labeled dataset to train
a model. Self-training (Scudder, 1965; Riloff and Wiebe, 2003) is a specific
type of pseudo-label based method that is of growing interest. Self-training
involves training two models: a “teacher” used to label unlabeled data, which
is used as an augmented labeled dataset. Then a “student” is trained on the
newly augmented dataset. This process can be iterated to further boost
performances. Self-training has been successfully applied in different fields
such as computer vision (Yalniz et al., 2019; Babakhin et al., 2019; Xie et
al., 2020; Chen et al., 2020; Zoph et al., 2020), automatic speech recognition
(ASR) Parthasarathi and Strom (2019). In Vision, Yalniz et al. (2019) adopted
the self-training paradigm in image classification, and achieves the state-of-
the-art top-1 result on ImageNet benchmark; Xie et al. (2020) proposed the
strategy of Noisy Student Training, a variant of self-training built on top of
EfficientNet (Tan and Le, 2020). In ASR, Parthasarathi and Strom (2019)
trained an ASR model on one million hours of unlabeled speech data using a
teacher-student self-training model. Park et al. (2020) proposed the concept
of normalized filtering score that filters out low-confident utterance-
transcript pairs generated by the teacher to mitigate the noise introduced by
the teacher model.

In the context of natural language processing (NLP), the concept of semi-
supervised learning has been adopted in different NLP tasks such as machine
translation (Cheng et al., 2016; Tu et al., 2016; Ramachandran et al., 2016;
Edunov et al., 2018; Clark et al., 2018), information extraction (Liao and
Veeramachaneni, 2009; Peters et al., 2017), text classification (Nigam et al.,
2006; Dai and Le, 2015; Miyato et al., 2016; Howard and Ruder, 2018;
Karamanolakis et al., 2019; Li et al., 2019a), and text generation (Zang and
Wan, 2019; Qader et al., 2019; Shang et al., 2019). Particularly, He et al.
(2019) studied the efficacy of self-training on sequence generation tasks and
found that self-training can significantly boost performances, particularly
when labeled data is scarce. Besides, they also pointed out that the core of
self-training for sequence generation tasks is the noise injected into the
neural model, which can be interpreted to smooth the latent sequence space.

Word vector models (Mikolov et al., 2013b, a; Pennington et al., 2014; Mikolov
et al., 2013a; Levy and Goldberg, 2014) and language modeling pretraining
(Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Lewis et al., 2019;
Bao et al., 2020; Joshi et al., 2020) can also be viewed as a specific type of
semi-supervised learning model, by leveraging the information of data in the
general domain. Other strategies for training semi-supervised models involve
co-training (Qiao et al., 2018; Chen et al., 2018; Han et al., 2018), low-
density separation (Grandvalet and Bengio, 2005; Dai et al., 2017).

###  2.2 Data Augmentation

Data augmentation aims at increasing the amount of training data by adding
slightly modified copies of existing data points or created new synthetic data
based on existing data points (Krizhevsky et al., 2012; Paulin et al., 2014;
Laine and Aila, 2016; Sajjadi et al., 2016; Cubuk et al., 2018; Inoue, 2018;
Cubuk et al., 2020). The concept consistency training (Rasmus et al., 2015;
Sajjadi et al., 2016; Laine and Aila, 2017; Tarvainen and Valpola, 2018;
Miyato et al., 2018; Luo et al., 2018; Athiwaratkun et al., 2019; Li et al.,
2019b; Verma et al., 2019; Liu et al., 2020) is widely used as a regulation to
force the label of modified copies to the same as the original label.

In NLP, modified copies of existing data points are generated usually by
synonym replacement and text editing (Zhang et al., 2015; Kobayashi, 2018; Wei
and Zou, 2019; Gao et al., 2019), back-translation (Sennrich et al., 2016;
Edunov et al., 2018; Xie et al., 2019), noise injection (Wang and Yang, 2015;
Xie et al., 2017, 2018), mixup (Guo et al., 2019), generation (Anaby-Tavor et
al., 2019; Wu et al., 2019; Kumar et al., 2020). Data augmentation has
introduced significant performance boost especially in low-resource scenarios
(Fadaee et al., 2017; Bergmanis et al., 2017; Şahin and Steedman, 2018; Xia et
al., 2019; Shleifer, 2019; Singh et al., 2019).

##  3 Models

###  3.1 Notations

We use the task of text classification for illustration purposes, in which the
goal is to assign a label $y$ to a given input $x$. $x$ is a sequence of
words. We have a given labeled set $D=\\{x_{i},y_{i}\\}$, $i\in[1,N_{D}]$,
where $N_{D}$ denotes the number of data points in $D$. In addition, we have
an in-domain unlabeled dataset $U$ of size $N_{U}$, where $N_{U}\gg N_{D}$.
The goal of semi-supervised learning is to explore how the unlabeled in-domain
dataset $U$ can be leveraged at the training time. At test time, inference
remains the same as the original setup.

###  3.2 In-domain LM Pretraining

A direct way to take advantage of $U$ is to pretrain a BERT (Devlin et al.,
2018), RoBERTa (Liu et al., 2019) or GPT3 (Brown et al., 2020) style language
model on $U$ by predicting a masked word given surrounding contexts or
predicting a subsequent word given proceeding contexts. Pretraining on the in-
domain data facilitates the learning of in-domain semantics and word
compositions.

For training, the LM model can be trained from scratch with random
initialization or initialized using an existing BERT or RoBERTa model trained
on an open-domain dataset, the latter of which is similar to the idea in
Gururangan et al. (2020), which continues LM pretraining in different domains
and tasks. The LM model trained on $U$ is used as initialization to be further
finetuned on $D$.

###  3.3 Pseudo-label Based Approach

Another way to take advantage of $U$ is to use the model trained on $D$ to
assign pseudo labels to data points in $U$. Specifically, the model trained on
$D$ is referred to as the teacher model. The teacher model is used to assign
pseudo labels to $U$ or a specific portion of $U$, forming the augmented
dataset $D^{\prime}$. Let $N_{D^{\prime}}$ denote the number of selected
examples in $D^{\prime}$. There are different options on how to generate
$D^{\prime}$, how $D$ and $D^{\prime}$ are combined, and how the new model can
be trained on the combination. The model trained on the combination is
referred to as the student model.

The teacher and the student can share a similar model backbone. In the text
classification task, various structures can be used as the backbone such as
LSTMs (Hochreiter and Schmidhuber, 1997; Tang et al., 2016), CNNs (Kim, 2014),
BERT Devlin et al. (2018); Chai et al. (2020), etc, with the trained model to
maximize the probability of predicting the golden label $y$ given $x$. The
student model can be randomly initialized or initialized using the teacher
model.

####  3.3.1 Different strategies to construct $D^{\prime}$

Here we discuss different strategies to generate $D^{\prime}$.

##### Naive Strategy $D^{\prime}=U$:

The teacher model is used to label all instances in $U$ to form $D^{\prime}$.
The shortcoming for this strategy is that incorrect and low-confident labels
included in $D^{\prime}$ can be severely detrimental to the student model.

##### Top-K Model Predictions:

To avoid the negative effect from the low-confident labels assigned by the
teacher, we can only pick the confident ones. The teacher model is run on each
example in $U$ to obtain the probability of all classes. Then for each class
$l$, we rank all instances in $U$ based on the corresponding probabilities.
The top-$K$ instances for each class are selected to form $D^{\prime}$.

####  3.3.2 Different Strategies for Student Training

Here we discuss how models can be trained given $D^{\prime}$ and $D$.

##### Training on $D+D^{\prime}$:

This strategy is denoted by $T(D+D^{\prime})$. The most straightforward
strategy is to train the student model on the union of $D$ and $D^{\prime}$.
The potential risk with this naive strategy is that the influence from clean
labels in $D$ can be diluted if the size of $D^{\prime}$ is large and that
incorrect labels in $D^{\prime}$ can exert negative effects on the student
model.

##### Pretraining on $D^{\prime}$ and Fine-tuning on $D$:

This strategy is denoted by $T(D^{\prime})F(D)$. To make the model more immune
to incorrectly labeled examples in $D^{\prime}$, and be able to take advantage
of the large amount of data in $D^{\prime}$ at the same time, we can first
train the student model on the newly collected dataset $D^{\prime}$ to predict
the pseudo labels, and then fine-tune the model on the original labeled
dataset $D$. This strategy ensures that the final model is fine-tuned on the
dataset with clean labels. This process actually mimics the idea of the self-
training (Devlin et al., 2018; Xie et al., 2020; Chen et al., 2020; Grill et
al., 2020) in semi-supervised learning literature.

##### Pretraining on $D+D^{\prime}$ and Fine-tuning on $D$:

This strategy is denoted by $T(D+D^{\prime})F(D)$. A minor change can be made
to the strategy described above, where the model is pretrained on the
concatenation of $D$ and $D^{\prime}$, and then fine-tuned on $D$.
Practically, we find that $T(D+D^{\prime})F(D)$ consistently outperforms
$T(D^{\prime})F(D)$, which is expected since adding examples with golden
labels for pretraining does no harm the performance. We thus only report
results for $T(D+D^{\prime})F(D)$, and omit $T(D^{\prime})F(D)$ for brevity.

##### Iterative Training

The process of training the teacher and the student can be iterated: the
student model trained in the previous iteration can be used as a new teacher
model to relabel the unlabeled dataset $U$, from which the top-$K$ examples
are regenerated to form the new $D^{\prime}$. Based on the new $D^{\prime}$, a
new student model is trained. This process is repeated until the pre-defined
value of iterations $N$ is reached.

###  3.4 Combining In-domain LM Pretraining and Pseudo-label Based Approach

The in-domain LM pretraining and the pseudo-label based strategy can be
combined: an LM model is first pretrained on the in-domain data $U$. Next, the
model is used to initialize the teacher, which will be trained on $D$. The
teacher model is then used to generate pseudo labels for $U$, which are used
to train the student. In this way, $U$ is used twice, both in LM pretraining
and pseudo-label generation for student training.

##  4 Experiments

In this section, we conduct extensive experiments to better understand the
behavior of semi-supervised learning. We give insights gathered from extensive
experimental studies and also discuss several considerations towards obtaining
a successful model. We use the following two datasets for detailed
explorations:

(1) The labeled dataset $D$ is the IMDB dataset collected by Maas et al.
(2011). This dataset contains an even number of positive and negative movie
reviews. The training and test sets respectively contain 25k and 25k examples.
The task is formalized as a binary classification task to decide the polarity
of sentiment for a review. To explore models’ behavior on training datasets of
different sizes, we use 10, 20, 50, 100, 1k, 5k and 25k for training.

For the unlabeled dataset $U$, we crawled the IMDB and collected about 3.4M
movie reviews. We release this large-scale IMDB movie review dataset to
public.

(2) The labeled dataset $D$ is the deceptive opinion spam dataset (Ott et al.,
2011; Li et al., 2014) to separate fake hotel reviews generated by Turkers and
hotel employees from genuine reviews from real customers. The task is
formulated as a three-class classification task, and $D$ contains 800/280/800
reviews, which respectively denote fake reviews from Turkers, fake reviews
from hotel employees and genuine reviews from real customers. For the
unlabeled dataset $U$, we crawled TripAdvisor, an online travel website and
collected about 1M reviews from roughly 5k hotels.

The sentiment analysis task on movie reviews is a significantly easier task
than the deceptive review detection task, since the latter requires to model
to identify subtle changes in language usage for spam generation, while the
former focuses more on identifying sentiment-indicative tokens.

##### Training Details

For in-domain LM pretraining, we use the RoBERTa structure (Liu et al., 2019)
as the backbone. We use both the small model, a 12-layer transformer with the
hidden size of each layer being 768, and a large model, a 24-layer transformer
with the hidden size of each layer being 1,024. Models are trained using using
Adam (Kingma and Ba, 2014) with $\beta=(0.9,0.98)$, $\epsilon=10^{-6}$, a
polynomial learning rate schedule, warmup for 4K steps and weight decay with
$10^{-3}$. Dropout rate is set to 0.2. The pretrained LM model on the in-
domain $U$ is then finetuned on $D$. For training the teacher and the student,
we use Adam for optimization. Learning rate, batch-size and dropout rate are
treated hyperparamters tuned on the dev set.

For reference purposes, we also implement the BiLSTM model, where the
representation at the last time step from the left-to-right direction is
concatenated with the the representation at the first time step from the
right-to-left direction, which is next fed to the softmax function for golden
label prediction. Word vectors are initialized using GloVe (Pennington et al.,
2014).

| T$(D)$ | T$(D^{\prime})$ | T$(D+D^{\prime})$ | $\text{T}(D+D^{\prime})\text{F}(D)$
---|---|---|---|---
$|D|$ $|D^{\prime}|$ | 0 | 1K | 10K | 100K | 1M | 1K | 10K | 100K | 1M | 1K | 10K | 100K | 1M
10 | 53.34 | 54.00 | 51.46 | 53.95 | 53.60 | 55.10 | 51.54 | 52.49 | 53.91 | 55.25 | 55.35 | 55.47 | 54.60
|  | (+0.66) | (-1.88) | (+0.61) | (+0.26) | (+1.76) | (-1.80) | (-0.85) | (+0.57) | (+1.91) | (+2.01) | (+2.13) | (+1.26)
20 | 54.62 | 51.73 | 53.28 | 54.16 | 55.43 | 54.35 | 54.50 | 53.92 | 54.43 | 58.15 | 58.63 | 58.77 | 55.63
|  | (-2.89) | (-1.34) | (-0.46) | (+0.81) | (-0.27) | (-0.12) | (-0.70) | (-0.19) | (+3.53) | (+4.01) | (+4.15) | (+1.01)
50 | 80.67 | 56.91 | 72.25 | 82.19 | 81.30 | 73.81 | 77.36 | 83.80 | 81.63 | 81.32 | 84.96 | 85.86 | 82.85
|  | (-23.76) | (-8.42) | (+1.52) | (+0.63) | (-6.86) | (-3.31) | (+3.13) | (+0.96) | (+0.65) | (+4.29) | (+5.19) | (+2.18)
100 | 83.12 | 65.67 | 83.49 | 85.52 | 83.95 | 80.30 | 83.50 | 84.89 | 83.60 | 81.86 | 86.25 | 87.27 | 84.40
|  | (-17.48) | (+0.37) | (+2.40) | (+0.83) | (-2.82) | (+0.38) | (+1.77) | (+0.48) | (-1.26) | (+3.13) | (+4.15) | (+1.28)
1K | 89.90 | – | 84.68 | 89.84 | 90.66 | – | 90.10 | 90.15 | 91.12 | – | 90.45 | 90.78 | 91.32
|  | – | (-5.22) | (-0.06) | (+0.76) | – | (+0.20) | (+0.25) | (+1.22) | – | (+0.55) | (+0.88) | (+1.42)
5K | 93.16 | – | 87.09 | 91.43 | 92.98 | – | 93.02 | 93.57 | 93.79 | – | 93.07 | 93.05 | 93.72
|  | – | (-6.07) | (-1.73) | (-0.18) | – | (-0.14) | (+0.41) | (+0.63) | – | (-0.09) | (-0.11) | (+0.56)
25K | 95.20 | – | – | 90.56 | 92.75 | – | – | 95.51 | 95.80 | – | – | 95.42 | 95.50
|  | – | – | (-3.24) | (-1.05) | – | – | (+0.31) | (+0.60) | – | – | (+0.22) | (+0.30)
Table 1: Results for different strategies for pseudo-label based approaches on
the IMDB dataset based on open-domain pretraining. “T” represents “Train” and
“F” represents “Fintune”. Brackets represent the used dataset(s), e.g.,
“T(D+D’)F(D)” means the model is trained on $D+D^{\prime}$ and finetuned on
$D$. The best result for each row is marked in bold.

Figure 1: Teacher perforamnces for BiLSTMs, In-domain pretraining, Open-domain
pretraining, Open-domain+In-domain pretraining on the IMDB (upper) and the
deceptive opinion spam (lower) datasets based on different sizes of the
labeled dataset. All pretraining models use RoBERTa small as the backbone.

###  4.1 Teacher Performances

We first examine teacher performances for BiLSTMs, In-domain pretraining,
Open-domain pretraining and Open-domain+In-domain pretraining on the IMDB and
the deceptive spam datasets regarding different values of $|D|$, as shown in
Figure 1. For In-domain pretraining, a randomly initialized LM model is first
pretrained on the in-domain dataset $U$, and then fine-tuned on $D$. For Open-
domain pretraining, we directly take the pretrained RoBERTa model, which is
pretrained on the open-domain dataset $largeU$. For open-domain+In-domain
pretraining, the LM model is first pretrained on $largeU$, then on the in-
domain dataset $U$, and last fine-tuned on $D$. The pseudo-label based semi-
supervised strategy is applied in none of these setups. The accuracy
progressively increases as we increase the size of the labeled dataset $D$,
which is in line with our expectation.

##### Is open-domain pretraining still necessary?

Take the IMDB dataset as an example. As can be seen from Figure 1, the
performance of In-domain pretraining (95.87 when $|D|=25k$) on the 3.4M
unlabeled reviews performs nearly the same as open-domain+In-domain
pretraining (95.82), both of which significantly outperform open-domain
pretraining (95.20). Similar phenomena are observed for the deceptive spam
dataset, where In-domain pretraining, open-domain+In-domain pretraining and
open-domain pretraining respectively obtains 82.5, 82.0 and 78.2 accuracy.
This demonstrates that with the presence of relatively large in-domain data,
the extremely time-intensive training of LM on a huge amount of open-domain
data is unnecessary.

###  4.2 In-domain Pretraining

As shown in Figures 1 and 4, the in-domain pretraining strategy has
significantly better few-shot learning abilities and requires much smaller
amount of data for training: the performance from in-domain pretraining
drastically improves as $|D|$ increases from 10 to 50, achieving a performance
of 93.8 accuracy on the IMDB dataset with only 50 training examples. This
performance is higher than BiLSTMs with 25K training examples, and similar to
the performance of vanilla RoBERTa with 5K training examples. This shows that
LM pretraining on the in-domain dataset $U$ provides the model with the
generality to rapidly figure out the necessary task-specific information for
predictions.

Comparing with existing pretraining models that have few-shot learning ability
such as GPT3, the advantages of the semi-supervised in-domain pretraining are
obvious: the model is easier to train, has significantly fewer parameters and
does not have to rely on a vast amount of training data.

####  4.2.1 Influence from the size of $|U|$

It is widely accepted that LM pretraining requires a massive amount of
training data. Results for $U$ of different sizes for LM pretraining are shown
in Figure 2. The model is randomly initialized and then trained on the in-
domain $U$ until convergence. The pretrained LM model is next fine-tuned on
$D=25k$.

As can be seen, the performance of in-domain pretraining highly relies on the
size of the in-domain dataset $U$. With smaller sizes of $U$, in-domain
pretraining underperforms open-domain pretraining, which is in line with our
expectation since small $U$ cannot provide enough evidence for learning in-
domain word semantics and compositions. Specifically, the performance of
$|U|=10k$ is only slightly better than the no-pretraining setup; the
performance for in-domain pretraining outperforms open-domain pretraining only
when the size of $U$ exceeds 1M.

Figure 2: The influence of $|U|$ for in-domain pretraining with $|D|=25k$.
Figure 3: The influence of $|U|$ for pseudo-label based approaches.  Figure 4:
The influence of $|D|$ on pseudo-label based approaches for different
$|D^{\prime}|$.  Figure 5: The influence of $|D^{\prime}|$ on pseudo-label
based approaches. $|D|$ is required to be larger than $|D|$.

###  4.3 Pseudo-label Based Approaches

Detailed results for pseudo-label based approaches based on open-domain
pretraining are shown in Table 1. The trends can be summarized as follows:

(1) Generally, both $\text{T}(D+D^{\prime})\text{F}(D)$ and
$\text{T}(D+D^{\prime})$ perform better than the vanilla setup $T(D)$.
$T(D^{\prime})$ underperforms the vanilla setup $T(D)$ when $D$ is small and
$D$ is large, but outperforms $T(D)$ when $D$ is of medium size.

(2) The performance boost introduced by pseudo-label based methods gradually
increases as the size of $D$ increases, and then shrinks. The explanation is
as follows: with a small $D$, the accuracy of the teacher model is low. Most
predicted labels on $D^{\prime}$ are thus unreliable. Therefore, the advantage
from the model trained on the noisy $D^{\prime}$ is relatively small; with a
large $D$, the teacher model is already good enough, reaching an accuracy
higher than 0.9. Though almost all predicted labels on $D^{\prime}$ are
correct, their improvement upon an already pretty good teacher model is small.
There is a sweet spot for the size of $D$, where pseudo-label based methods
introduce the largest boost: with a medium-sized $D$, where the teacher model
reaches an acceptable accuracy when the correctly labeled examples in
$D^{\prime}$ outweigh incorrectly labeled ones, the student model can take the
most advantage of $D^{\prime}$, leading to significant performance boosts of
+5$\%$. Similar trends are observed for $T(D^{\prime})$, where $T(D^{\prime})$
underperforms the vanilla setup $T(D)$ both when $D$ is small and $D$ is
large, but outperforms $T(D)$ when $D$ is of medium size.

(3) For the three pseudo-label based strategies, $T(D^{\prime})$,
$\text{T}(D+D^{\prime})\text{F}(D)$ and $\text{T}(D+D^{\prime})$,
$T(D^{\prime})$ performs the worst for all $|D|$. This is in line with our
expectation because for $T(D^{\prime})$, only $D^{\prime}$ is used for final
predictions, and thus the influence of the golden-labeled dataset $D$ is
diluted; $\text{T}(D+D^{\prime})\text{F}(D)$ works best when $|D|$ is small,
while $\text{T}(D+D^{\prime})$ works best when $|D|$ is large. Our
explanations are as follows: when $|D|$ is small, the student trained on
$D+D^{\prime}$ is inferior due to the massive amount of incorrect labels in
$D^{\prime}$. The model thus needs to be further fine-tuned on $D$, making
$\text{T}(D+D^{\prime})\text{F}(D)$ perform better than
$\text{T}(D+D^{\prime})$; When $|D|$ is large, most labels in $D^{\prime}$ are
correct and the student is more immune to the small proportion of incorrect
labels in $D^{\prime}$. Directly training on the larger $D+D^{\prime}$ dataset
provides the model with more generalization ability, while fine-tuning only on
$D$ dilutes the influence from massive amount of correct labels in
$D^{\prime}$, making the performance of $\text{T}(D+D^{\prime})\text{F}(D)$
worse than $\text{T}(D+D^{\prime})$ when $|D|$ is large.

####  4.3.1 Influence from the size of $D^{\prime}$

With a fixed in-domain dataset $U$, we select top-K examples for each label
$l$. Different values of $K$ lead to different sizes of $D^{\prime}$. There is
apparently a tradeoff between the size of $D^{\prime}$ and the confidence for
examples included in $D^{\prime}$: larger size of $D^{\prime}$ means that more
less-confident examples are selected.

Trends regarding different values of $|D^{\prime}|$ are shown in Figure 5. We
only run setups with $|D^{\prime}|$ larger than $|D|$. As can be seen, for
smaller $D$ ($|D|=50,100$), the final performance first increases as
$|D^{\prime}|$ gets larger, which means the model is taking advantage of
evidence provided by the pseudo labels. Then, the performance decreases as
$|D^{\prime}|$ continues to increase, which means the model starts suffering
from the incorrectness of the less-confident examples. For larger values of
$D$ ($|D|$ larger than 1k), performances keep increasing as $|D^{\prime}|$
gets larger. This is because the teacher model trained on $D$ is good enough
to generate confident examples.

####  4.3.2 Influence from the size of $U$

The influence of the size of $U$ is already manifested in the size of
$D^{\prime}$, since the size of $U$ should be larger than the size of
$D^{\prime}$ ($D^{\prime}$ is selected from $U$). Performances for different
sizes of $U$ are shown in Figure 3, where we randomly sample examples from the
3.4 million IMDB reviews to form $U$ of different sizes. For a given $U$, we
plot the performance achieved with the best $|D^{\prime}|$. Larger $U$ should
lead to better performances since more confident examples can be selected.

As shown in Figure 3, for $D$ of different sizes, the performance first
increases as $|U|$ grows, but then immediately plateaus. Explanations are as
follows: for larger $|D|$, the model is already good enough to provide
confidently correct labels for points in $D^{\prime}$, and the improvement
from extra confident examples is marginal.

####  4.3.3 Iterative Training

We can iterate the teacher-student pattern for pseudo-label based approaches,
where the teacher for the current iteration is initialized with the student in
the previous iteration. Results are shown in Figure 6. As can be seen,
additional performance boosts are observed for different $|D|$ as the
iterative process goes on. For larger $|D|$, the performance becomes stagnant
across the iterative process, while for smaller $|D|$, the curve keeps rising
until convergence.

Figure 6: Iterative Training for Pseudo-label Based Approaches on IMDB based
on Open-domain Pretraining.

###  4.4 Combing In-domain Pretraining and Pseudo-label Based Approaches

In-domain pretraining and pseudo-label based approaches can be combined, where
the teacher model is initialized with the pretrained model on the in-domain
dataset $U$. Results for the combined strategy are shown in Table 2. As can be
seen, both strategies lead to progressive performance boosts over Open-domain
Pretraining, with a combined boost of +1.1 for the small model, and +1.0 for
the large model.

Small |D|=25k
---
Open-domain Pretraining | 95.2
Open-domain+In-domain Pretraining | 95.8
In-domain Pretraining | 95.8
In-domain Pretraining + Pseudo-label | 96.3
Large |D|=25k
Open-domain Pretraining | 95.6
In-domain Pretraining | 96.2
In-domain Pretraining + Pseudo-label | 96.6
Table 2: Results for Combining In-domain Pretraining and Pseudo-label based
Approaches

##  5 Conclusion

In this paper, we conduct comprehensive analysis on semi-supervised learning
in NLP under the context of large-scale language model pretraining. We find
that even with the presence of large-scale LM pretraining, both the in-domain
pretraining strategy and the pseudo-label based strategy introduce additional
significant performance boost, with the former performing better with larger
$U$, the latter performing better with smaller $U$, and the combination
leading to the best performance. Using semi-supervised learning models, we are
able to achieve a performance of around $93-94\%$ accuracy with only 50
training data points on the IMDB dataset, and a competitive performance of
96.6$\%$ with the full dataset. Our work sheds light on the behavior of semi-
supervised learning models in the context of large-scale pretraining.

## References

  * Anaby-Tavor et al. (2019) Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2019.  Not enough data? deep learning to the rescue!
  * Arazo et al. (2020) Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness. 2020\.  Pseudo-labeling and confirmation bias in deep semi-supervised learning.  In _2020 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8. IEEE.
  * Athiwaratkun et al. (2019) Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, and Andrew Gordon Wilson. 2019.  There are many consistent explanations of unlabeled data: Why you should average.
  * Babakhin et al. (2019) Yauhen Babakhin, Artsiom Sanakoyeu, and Hirotoshi Kitamura. 2019.  Semi-supervised segmentation of salt bodies in seismic images using an ensemble of convolutional neural networks.  In _German Conference on Pattern Recognition_ , pages 218–231. Springer.
  * Bao et al. (2020) Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2020.  Unilmv2: Pseudo-masked language models for unified language model pre-training.
  * Bergmanis et al. (2017) Toms Bergmanis, Katharina Kann, Hinrich Schütze, and Sharon Goldwater. 2017\.  Training data augmentation for low-resource morphological inflection.  In _Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection_ , pages 31–39.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Chai et al. (2020) Duo Chai, Wei Wu, Qinghong Han, Fei Wu, and Jiwei Li. 2020.  Description based text classification with reinforcement learning.
  * Chapelle et al. (2006) Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. 2006.  _Semi-Supervised Learning_.  The MIT Press.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. 2020.  Big self-supervised models are strong semi-supervised learners.  _arXiv preprint arXiv:2006.10029_.
  * Chen et al. (2018) Yanbei Chen, Xiatian Zhu, and Shaogang Gong. 2018.  Semi-supervised deep learning with memory.  In _ECCV_ , pages 275–291.
  * Cheng et al. (2016) Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016\.  Semi-supervised learning for neural machine translation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1965–1974, Berlin, Germany. Association for Computational Linguistics.
  * Clark et al. (2018) Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc V. Le. 2018.  Semi-supervised sequence modeling with cross-view training.
  * Cubuk et al. (2018) Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. 2018\.  Autoaugment: Learning augmentation policies from data.  _arXiv preprint arXiv:1805.09501_.
  * Cubuk et al. (2020) Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. 2020.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 702–703.
  * Dai and Le (2015) Andrew M Dai and Quoc V Le. 2015.  Semi-supervised sequence learning.  In _Advances in neural information processing systems_ , pages 3079–3087.
  * Dai et al. (2017) Zihang Dai, Zhilin Yang, Fan Yang, William W. Cohen, and Ruslan Salakhutdinov. 2017\.  Good semi-supervised learning that requires a bad gan.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.  _arXiv preprint arXiv:1808.09381_.
  * Fadaee et al. (2017) Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017.  Data augmentation for low-resource neural machine translation.  _arXiv preprint arXiv:1705.00440_.
  * Gao et al. (2019) Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou, and Tie-Yan Liu. 2019.  Soft contextual data augmentation for neural machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5539–5544, Florence, Italy. Association for Computational Linguistics.
  * Grandvalet and Bengio (2005) Yves Grandvalet and Yoshua Bengio. 2005.  Semi-supervised learning by entropy minimization.  In _Advances in neural information processing systems_ , pages 529–536.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. 2020.  Bootstrap your own latent: A new approach to self-supervised learning.  _arXiv preprint arXiv:2006.07733_.
  * Guo et al. (2019) Hongyu Guo, Yongyi Mao, and Richong Zhang. 2019.  Augmenting data with mixup for sentence classification: An empirical study.  _arXiv preprint arXiv:1905.08941_.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of ACL_.
  * Han et al. (2018) Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In _Advances in neural information processing systems_ , pages 8527–8537.
  * He et al. (2019) Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ranzato. 2019.  Revisiting self-training for neural sequence generation.  _arXiv preprint arXiv:1909.13788_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  _arXiv preprint arXiv:1801.06146_.
  * Inoue (2018) Hiroshi Inoue. 2018.  Data augmentation by pairing samples for images classification.  _arXiv preprint arXiv:1801.02929_.
  * Iscen et al. (2019) Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. 2019.  Label propagation for deep semi-supervised learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5070–5079.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020.  Spanbert: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Karamanolakis et al. (2019) Giannis Karamanolakis, Daniel Hsu, and Luis Gravano. 2019.  Leveraging just a few keywords for fine-grained aspect detection through weakly supervised co-training.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1746–1751, Doha, Qatar. Association for Computational Linguistics.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kobayashi (2018) Sosuke Kobayashi. 2018.  Contextual augmentation: Data augmentation by words with paradigmatic relations.  _arXiv preprint arXiv:1805.06201_.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012.  Imagenet classification with deep convolutional neural networks.  In _Advances in neural information processing systems_ , pages 1097–1105.
  * Kumar et al. (2020) Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020.  Data augmentation using pre-trained transformer models.
  * Laine and Aila (2016) Samuli Laine and Timo Aila. 2016.  Temporal ensembling for semi-supervised learning.  _arXiv preprint arXiv:1610.02242_.
  * Laine and Aila (2017) Samuli Laine and Timo Aila. 2017.  Temporal ensembling for semi-supervised learning.
  * Lee (2013) Dong-Hyun Lee. 2013.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In _Workshop on challenges in representation learning, ICML_ , volume 3.
  * Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014.  Neural word embedding as implicit matrix factorization.  In _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_ , pages 2177–2185.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Li et al. (2014) Jiwei Li, Myle Ott, Claire Cardie, and Eduard Hovy. 2014.  Towards a general rule for identifying deceptive opinion spam.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1566–1576, Baltimore, Maryland. Association for Computational Linguistics.
  * Li et al. (2019a) Xinzhe Li, Qianru Sun, Yaoyao Liu, Shibao Zheng, Qin Zhou, Tat-Seng Chua, and Bernt Schiele. 2019a.  Learning to self-train for semi-supervised few-shot classification.
  * Li et al. (2019b) Yiting Li, Lu Liu, and Robby T Tan. 2019b.  Decoupled certainty-driven consistency loss for semi-supervised learning.  _arXiv preprint arXiv:1901.05657_.
  * Liao and Veeramachaneni (2009) Wenhui Liao and Sriharsha Veeramachaneni. 2009.  A simple semi-supervised algorithm for named entity recognition.  In _Proceedings of the NAACL HLT 2009 Workshop on Semi-Supervised Learning for Natural Language Processing_ , pages 58–65.
  * Liu et al. (2020) Lu Liu, Yiting Li, and Robby T. Tan. 2020.  Decoupled certainty-driven consistency loss for semi-supervised learning.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Luo et al. (2018) Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. 2018.  Smooth neighbors on teacher graphs for semi-supervised learning.
  * Maas et al. (2011) Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-volume 1_ , pages 142–150. Association for Computational Linguistics.
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b.  Distributed representations of words and phrases and their compositionality.  In _Advances in neural information processing systems_ , pages 3111–3119.
  * Miyato et al. (2016) Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2016.  Adversarial training methods for semi-supervised text classification.  _arXiv preprint arXiv:1605.07725_.
  * Miyato et al. (2018) Takeru Miyato, Shin ichi Maeda, Masanori Koyama, and Shin Ishii. 2018.  Virtual adversarial training: A regularization method for supervised and semi-supervised learning.
  * Nigam et al. (2006) Kamal Nigam, Andrew McCallum, and Tom M Mitchell. 2006.  Semi-supervised text classification using em.
  * Ott et al. (2011) Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T. Hancock. 2011.  Finding deceptive opinion spam by any stretch of the imagination.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 309–319, Portland, Oregon, USA. Association for Computational Linguistics.
  * Park et al. (2020) Daniel S Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V Le. 2020.  Improved noisy student training for automatic speech recognition.  _arXiv preprint arXiv:2005.09629_.
  * Parthasarathi and Strom (2019) Sree Hari Krishnan Parthasarathi and Nikko Strom. 2019.  Lessons from building acoustic models with a million hours of speech.
  * Paulin et al. (2014) Mattis Paulin, Jérôme Revaud, Zaid Harchaoui, Florent Perronnin, and Cordelia Schmid. 2014.  Transformation pursuit for image classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3646–3653.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Peters et al. (2017) Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017.  Semi-supervised sequence tagging with bidirectional language models.
  * Qader et al. (2019) Raheel Qader, François Portet, and Cyril Labbé. 2019.  Semi-supervised neural text generation by joint learning of natural language generation and natural language understanding models.
  * Qiao et al. (2018) Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan Yuille. 2018.  Deep co-training for semi-supervised image recognition.
  * Ramachandran et al. (2016) Prajit Ramachandran, Peter J Liu, and Quoc V Le. 2016.  Unsupervised pretraining for sequence to sequence learning.  _arXiv preprint arXiv:1611.02683_.
  * Rasmus et al. (2015) Antti Rasmus, Harri Valpola, Mikko Honkala, Mathias Berglund, and Tapani Raiko. 2015\.  Semi-supervised learning with ladder networks.
  * Reed et al. (2015) Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. 2015.  Training deep neural networks on noisy labels with bootstrapping.
  * Riloff and Wiebe (2003) Ellen Riloff and Janyce Wiebe. 2003.  Learning extraction patterns for subjective expressions.  In _Proceedings of the 2003 conference on Empirical methods in natural language processing_ , pages 105–112.
  * Şahin and Steedman (2018) Gözde Gül Şahin and Mark Steedman. 2018.  Data augmentation via dependency tree morphing for low-resource languages.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5004–5009, Brussels, Belgium. Association for Computational Linguistics.
  * Sajjadi et al. (2016) Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. 2016.  Regularization with stochastic transformations and perturbations for deep semi-supervised learning.  In _Advances in neural information processing systems_ , pages 1163–1171.
  * Scudder (1965) H. Scudder. 1965.  Probability of error of some adaptive pattern-recognition machines.  _IEEE Transactions on Information Theory_ , 11(3):363–371.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Improving neural machine translation models with monolingual data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96, Berlin, Germany.
  * Shang et al. (2019) Mingyue Shang, Piji Li, Zhenxin Fu, Lidong Bing, Dongyan Zhao, Shuming Shi, and Rui Yan. 2019.  Semi-supervised text style transfer: Cross projection in latent space.
  * Shi et al. (2018) Weiwei Shi, Yihong Gong, Chris Ding, Zhiheng MaXiaoyu Tao, and Nanning Zheng. 2018\.  Transductive semi-supervised deep learning using min-max features.  In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Shleifer (2019) Sam Shleifer. 2019.  Low resource text classification with ulmfit and backtranslation.
  * Singh et al. (2019) Jasdeep Singh, Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2019.  Xlda: Cross-lingual data augmentation for natural language inference and question answering.
  * Tan and Le (2020) Mingxing Tan and Quoc V. Le. 2020.  Efficientnet: Rethinking model scaling for convolutional neural networks.
  * Tang et al. (2016) Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016.  Effective LSTMs for target-dependent sentiment classification.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 3298–3307, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Tarvainen and Valpola (2018) Antti Tarvainen and Harri Valpola. 2018.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.
  * Tu et al. (2016) Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2016.  Neural machine translation with reconstruction.  _arXiv preprint arXiv:1611.01874_.
  * Verma et al. (2019) Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. 2019.  Interpolation consistency training for semi-supervised learning.
  * Wang and Yang (2015) William Yang Wang and Diyi Yang. 2015.  That’s so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using #petpeeve tweets.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2557–2563, Lisbon, Portugal. Association for Computational Linguistics.
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  Eda: Easy data augmentation techniques for boosting performance on text classification tasks.
  * Wu et al. (2019) Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019.  Conditional bert contextual augmentation.  In _International Conference on Computational Science_ , pages 84–95. Springer.
  * Xia et al. (2019) Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos, and Graham Neubig. 2019.  Generalized data augmentation for low-resource translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5786–5796, Florence, Italy. Association for Computational Linguistics.
  * Xie et al. (2019) Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019.  Unsupervised data augmentation for consistency training.  _arXiv preprint arXiv:1904.12848_.
  * Xie et al. (2020) Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. 2020.  Self-training with noisy student improves imagenet classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10687–10698.
  * Xie et al. (2018) Ziang Xie, Guillaume Genthial, Stanley Xie, Andrew Y Ng, and Dan Jurafsky. 2018\.  Noising and denoising natural language: Diverse backtranslation for grammar correction.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 619–628.
  * Xie et al. (2017) Ziang Xie, Sida I Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng. 2017.  Data noising as smoothing in neural network language models.  _arXiv preprint arXiv:1703.02573_.
  * Yalniz et al. (2019) I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. 2019.  Billion-scale semi-supervised learning for image classification.  _arXiv preprint arXiv:1905.00546_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_ , pages 5753–5763.
  * Zang and Wan (2019) Hongyu Zang and Xiaojun Wan. 2019.  A semi-supervised approach for low-resourced text generation.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in neural information processing systems_ , pages 649–657.
  * Zhu (2005) Xiaojin Jerry Zhu. 2005.  Semi-supervised learning literature survey.  Technical report, University of Wisconsin-Madison Department of Computer Sciences.
  * Zoph et al. (2020) Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le. 2020.  Rethinking pre-training and self-training.  _arXiv preprint arXiv:2006.06882_.