#  Consistency Training with Virtual Adversarial Discrete Perturbation

Jungsoo Park1 Gyuwan Kim211footnotemark: 1 Jaewoo Kang1
1Korea University 2Clova AI, NAVER Corp.
{jungsoo_park<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal
contribution. Work done during the internship at Clova AI.

###### Abstract

We propose an effective consistency training framework that enforces a
training model’s predictions given original and perturbed inputs to be similar
by adding a discrete noise that would incur the highest divergence between
predictions. This virtual adversarial discrete noise obtained by replacing a
small portion of tokens while keeping original semantics as much as possible
efficiently pushes a training model’s decision boundary. Moreover, we perform
an iterative refinement process to alleviate the degraded fluency of the
perturbed sentence due to the conditional independence assumption.
Experimental results show that our proposed method outperforms other
consistency training baselines with text editing, paraphrasing, or a
continuous noise on semi-supervised text classification tasks and a robustness
benchmark111Our code will be publicly available in
https://github.com/clovaai/vat-d.

##  1 Introduction

Building a natural language processing (NLP) system often requires an
expensive process to collect a huge amount of labeled text data. Data
augmentation (Simard et al., 1998; Vapnik, 2000) techniques are widely used in
NLP by training a model with original and synthetically generated data to
generalize well to unseen data (Zhang et al., 2015; Sennrich et al., 2016) and
especially beneficial in a low-resource scenario. To improve the effectiveness
of data augmentation, devising the augmentation strategy with a training
algorithm is a crucial research problem.

The consistency training framework (Laine and Aila, 2016; Sajjadi et al.,
2016) enforces a model to produce similar predictions of original and
perturbed inputs. This method has several advantages over other training
algorithms such as naively adding augmented samples into the training set (Wei
and Zou, 2019; Ng et al., 2020) in that it provides a richer training signal
than a one-hot label, and also applies to a semi-supervised setting given
unlabeled data.

(a) Real data distribution, which requires complex decision boundary. (b) A
simple decision boundary is drawn when samples are insufficient.

(c) Augmentations can push the decision boundary (dotted line) from the
current one (bold line). (d) Augmentations which are outside the current
decision boundary enable further pushing it.

Figure 1:  A simple illustration of the intuition behind our method visualized
in the two-dimensional space, where the augmented samples (triangle) would aid
the training given a limited number of data (circle).

For perturbing (i.e. augmenting) a text while preserving its semantic, some
approaches inject continuous noise to embedding vectors (Xie et al., 2017;
Miyato et al., 2018), and others modify text itself in discrete fashion by
edit operations Kobayashi (2018); Wei and Zou (2019) or paraphrasing with
back-translation Sennrich et al. (2015); Edunov et al. (2018). Adding
continuous noise might not be helpful to generalize to texts that are
semantically coherent but far from the original ones on the embedding space.
The augmentations with discrete noise are mostly black-box approaches based on
simple rules or fixed models without access to the training model’s internal
states, having no control to output augmentations that would aid in the
regularization of the training model. As seen in Fig. 1 (d), the augmented
samples with similar semantics but that are outside the training model’s
decision boundary (i.e. vulnerable) are the ones that would effectively
regularize the model to fit into the complex real data distribution.

To this end, we propose virtual adversarial training with discrete token
replacements (_VAT-D_). Our framework first perturbs a given input text by
replacing a small subset of tokens to maximize the divergence between the
original and the perturbed samples’ model predictions while filtering tokens
to replace for constraining the semantic similarity, and train a model to
minimize the divergence. Our method relies on the training model’s
predictions, hence applies to the data with or without label information.
VAT-D shares the advantages of virtual adversarial training (VAT) with
continuous noise (Miyato et al., 2018) in that the perturbation is model-
dependent, changing over the training time to approximate the augmented
samples that would effectively push the decision boundary. On the other hand,
_VAT-D_ differs from VAT in that the search space is discrete rather than
continuous, thus not constrained by the pre-defined norm on the embedding
space.

Simultaneously replacing multiple tokens from the sentence, the operation our
method relies on, can degrade the fluency e.g. ungrammatical, incurring the
augmented points not to lie on the data manifold. Hence, we further refine the
unnatural tokens from the perturbed sentence motivated by non-autoregressive
(NAR) generation (Gu et al., 2017). We develop a new iterative refinement
process (Lee et al., 2018, 2020) to generate more naturally perturbed
sentences while still being virtually adversarial to the training model.

Our proposed method empirically outperforms previous state-of-the-art methods
on topic classification datasets (Chang et al., 2008; Mendes et al., 2012;
Zhang et al., 2015) under various semi-supervised learning (SSL) scenarios and
ANLI robustness benchmark dataset (Nie et al., 2019) where the method improves
over the RoBERTa-Large (Liu et al., 2019) on an average of 8 points. Our
work’s main contributions can be summarized as (1) proposing a consistency
training framework with the virtual adversarial discrete word replacement for
the classification task, and (2) employing the iterative refinement method to
improve the augmented samples’ fluency.

##  2 Method

We aim to generate a perturbed sample that incurs the highest divergence of
the model’s prediction logits from the original one without significant
changes in its surface form or semantics. Our augmentation is made on-the-fly
depending on the current model to push the decision boundary during training
effectively.

In this section, we first briefly explain the concept of consistency training
and virtual adversarial training (§2.1) that our framework relies on, and then
demonstrate our proposed method, virtual adversarial token replacement (§2.2)
and iterative refinement (§2.3).

###  2.1 Background

#### Consistency Training

Consistency training (Laine and Aila, 2016; Sajjadi et al., 2016) enforces
models’ predictions to be invariant when the input is perturbed. This
regularization pushes the decision boundary to traverse a low-density region
(Verma et al., 2019). The consistency loss is formally defined as

$\displaystyle\mathcal{L}(\mathbf{x},\mathbf{x^{\prime}})=D[p(\cdot\mid\mathbf{x}),p(\cdot\mid\mathbf{x^{\prime}})]$
(1)

where $D$ is a non-negative divergence metric between two probability
distributions (e.g., KL-divergence), $x^{\prime}$ is a perturbed sample from
an input $x$ by any transformation. The loss term can be used for labeled data
in supervised learning (Jiang et al., 2019; Liu et al., 2020; Asai and
Hajishirzi, 2020) and unlabeled data in SSL (Xie et al., 2019; Miyato et al.,
2016).

#### Virtual Adversarial Training

VAT (Miyato et al., 2016, 2018) is a consistency training method, which
perturbs a given input with continuous noise to maximize the divergence from
the model’s prediction of the original input. Such virtually adversarial
examples effectively smooth the decision boundary compared to the random
perturbation (Miyato et al., 2018). The formal definition of virtual
adversarial samples is

$\displaystyle\mathbf{\hat{x}}=\underset{\mathbf{x}^{\prime}\in
Neighbor(\mathbf{x})}{\operatorname{argmax}}{\mathcal{L}(\mathbf{x},\mathbf{x^{\prime}})}.$
(2)

where the training objective is to minimize the
$\mathcal{L}(\mathbf{x},\mathbf{\hat{x}})$. Miyato et al. (2016) perturbs
input by injecting noise to the embedding space, where the constraint of the
perturbation is $\epsilon$-ball in $L^{p}$ norm centered at $x$, i.e.
$Neighbor(x)=\\{\mathbf{x^{\prime}}\mid\|\mathbf{x^{\prime}}-\mathbf{x}\|_{p}\leq\epsilon\\}$.

To obtain the virtual adversarial continuous noise
$r=(\mathbf{x^{\prime}}-\mathbf{x})$, the gradient information of the loss
function with respect to the inputs is required which signals as the direction
to maximize the loss function. Miyato et al. (2018) use the power iteration
method for second-order approximation because the first derivative with
respect to $r=0$ is zero and computing a full Hessian matrix is
computationally intractable. The original VAT paper (Miyato et al., 2018)
describes more detailed derivations.

###  2.2 Virtual Adversarial Discrete Noise

We develop the consistency training framework by perturbing inputs with
virtual adversarial discrete noise, called VAT-D. We want to perturb a given
sentence $\mathbf{x}=(x_{1},\dots,x_{M})\in V^{M}$ of sequence length $M$ into
a new sentence $\mathbf{x^{\prime}}=(x^{\prime}_{1},\dots,x^{\prime}_{M})\in
V^{M}$ of the same length, where $V$ is the word vocabulary. In contrast with
the continuous case, we constrain that $x^{\prime}$ differs from $x$ in only
small portion of positions changing their surface forms, i.e.
$Neighbor(x)=\\{\|\mathbf{x^{\prime}}-\mathbf{x}\|_{H}/M\leq\tau\\}$ where $H$
denotes hamming distance in the token-level and $\tau$ is the replacement
ratio. In this work, we only focus on the replacement for simplicity and leave
other operations, including insertion, deletion, and swap, to future work.

#### Gradient Information

The white-box approaches having an access to the training model’s internal
states, mostly rely on the gradient vectors of the loss function with respect
to the input embeddings for finding adversarial discrete noise (Ebrahimi et
al., 2017). However, for acquiring such gradient information, naively
resorting to the linear approximation like in previous works (Ebrahimi et al.,
2017; Michel et al., 2019; Cheng et al., 2019) does not apply to our problem
since the first-order term is zero when the label information is substituted
to model’s predictions (Miyato et al., 2018). Also, employing the power
iteration method like in Miyato et al. (2018) is not possible since the
perturbation is discrete, i.e. token from vocabulary.

We bypass the obstacle by simply sharpening the original examples’ predictions
to enable the linear approximation. By sharpening the predictions of the
original inputs, the first-order term does not result in zero. This is because
the modified divergence loss is not zero when $\mathbf{x^{\prime}}=\mathbf{x}$
indicating the divergence is not necessarily minimum at
$r=\mathbf{x^{\prime}}-\mathbf{x}=0$. Formally, the Eq. 1 is modified to

$\displaystyle\mathcal{\tilde{L}}(\mathbf{x},\mathbf{x^{\prime}})=D[p^{\text{{sharp}}}(\cdot\mid\mathbf{x})),p(\cdot\mid\mathbf{x}^{\prime})]$

by sharpening the prediction given an original input by the temperature $T$ as
$p^{\text{{sharp}}}(\cdot\mid\mathbf{x})=p(\cdot|\mathbf{x})^{\frac{1}{T}}/\left\|p(\cdot|~{}\mathbf{x})^{\frac{1}{T}}\right\|_{1}$.

#### Virtual Adversarial Token Replacement

Consequently, the optimization problem to find a virtual adversarial discrete
perturbation changes to

$\displaystyle\mathbf{\hat{x}}=\underset{\mathbf{x}^{\prime}\in
Neighbor(\mathbf{x})}{\operatorname{argmax}}{\mathcal{\tilde{L}}(\mathbf{x},\mathbf{x^{\prime}})}.$

Input : $\text{input sentence}~{}\mathbf{x},\text{index to
perturb}~{}\mathbf{I}$

Output : $\text{perturbed sentence}~{}\mathbf{\hat{x}}$

Function  _VA_TR( $\mathbf{x}$, $\mathbf{I}$)_:

$\mathbf{\hat{x}}\leftarrow\mathbf{x}$

for  _$m\in\mathbf{I}$_ do

$g_{x_{m}}\leftarrow\nabla_{e({x_{m})}}{\mathcal{\tilde{L}}(\mathbf{x},\mathbf{x^{\prime}})}|_{\mathbf{x}^{\prime}=\mathbf{x}}$

$\hat{x}_{m}\leftarrow\underset{x\in\text{top\\_k}(x_{m},V)}{\operatorname{argmax}}\delta(x_{m},x)^{\top}\cdot~{}g_{x_{m}}$

Replace $m$-th token of $\mathbf{\hat{x}}$ to $\hat{x}_{m}$

return _$\mathbf{\hat{x}}$_

Algorithm 1 VA_TR Module

Finally, we train the modified consistency loss function with obtained
discrete perturbation. The replacement operation of $m$-th token $x_{m}$ to
the arbitrary token $x$ can be written as $\delta(x_{m},x):=e(x)-e(x_{m})$,
where $e(\cdot)$ denotes embedding look-up since NLP models generally map a
token to its corresponding embedding in an one-to-one relationship. We induce
a virtual adversarial token which maximizes the following criteria (Ebrahimi
et al., 2017; Michel et al., 2019; Cheng et al., 2019; Wallace et al., 2019;
Park et al., 2020):

$\displaystyle\hat{x}_{m}=\underset{x\in\operatorname{top\\_k}(x_{m},V)}{\operatorname{argmax}}\delta(x_{m},x)^{\top}\cdot
g_{x_{m}}$ (3) $\displaystyle\text{where
}g_{x_{m}}=\nabla_{e({x_{m})}}{\mathcal{\tilde{L}}(\mathbf{x},\mathbf{x^{\prime}})}|_{\mathbf{x}^{\prime}=\mathbf{x}}$

$\mathbf{g}_{x_{m}}$ is the gradient vector of the sharpened consistency loss
with respect to the $m$-th token. In brief, we replace the $m$-th original
token $x_{m}$ with one of the candidates $x$ that approximately maximizes the
consistency loss. To bound the semantics similarity from the original sentence
and the perturbed one, we use a masked language model (MLM) (Devlin et al.,
2019; Liu et al., 2019) to restrict a set of possible candidates to replace
$x_{m}$. We filter top-k candidates (Cheng et al., 2019), denoted as
$\operatorname{top\\_k}(x_{m},V)$, from the vocabulary having the highest MLM
probability at position $m$ when an original sentence $x$ is given to the MLM.

Alg. 1 illustrates the procedure (VA_TR) to acquire virtual adversarial tokens
with the modified consistency loss. We randomly select token indexes to
perturb $I$, subject to the length of the sentence. Considering multiple
substitutions, an exhaustive search over all possible combinations to find the
optimal one is computationally intractable. For efficient generation during
each training step, we replace multiple tokens simultaneously instead of
greedy search or beam search, which has shown to work considerably well in
previous works (Ebrahimi et al., 2017; Cheng et al., 2019). During training,
the models are optimized with standard cross-entropy and consistency loss with
an equal weight where we utilize KL-Divergence as the divergence $D$.

In our preliminary experiment, utilizing the MLM with masking was worse than
that without masking, similar to Li et al. (2020b). We conjecture that the
loss of information when applying masking operation has evoked the perturbed
samples to significantly deviate from the original ones, resulting in a
degradation in performance. Moreover, we do not fine-tune the off-the-shelf
MLM on the training corpus but only the classification model, which is to
ensure a fair comparison with other augmentation baselines.

###  2.3 Iterative Refinement

Replacing multiple tokens simultaneously can result in a degradation in the
perturbed sentence’s fluency, such as word duplicates or words out of context,
due to the conditional independence assumption. To address the issue, we
develop an iterative refinement method adjusted to our problem setting,
motivated by the NAR literature (Lee et al., 2018; Ghazvininejad et al.,
2019). We want to generate a more fluent perturbed sentence, while still being
virtually adversarial to the model.

Input : $\text{input sentence}~{}\mathbf{x},\text{refinement steps}~{}S$

Output : $\text{perturbed sentence}~{}\mathbf{\hat{x}}$

Function  _Iterative_Refinements( $\mathbf{x}$, $S$)_:

// initial perturbation $\text{Get}~{}\mathbf{I}~{}\text{randomly with the
ratio}~{}\tau$
$\mathbf{\hat{x}}\leftarrow\textit{VA\\_TR}{}(\mathbf{\hat{x}},\mathbf{I})$

// iterative refinements

for  _$s\leftarrow 1$ to $S$_ do

$\mathbf{\hat{x}}\leftarrow\textit{VA\\_TR}{}(\mathbf{\hat{x}},\mathbf{I})$

$\text{Update}~{}\mathbf{I}~{}\text{with the least MLM scores}$

return _$\mathbf{\hat{x}}$_

Algorithm 2 Iterative_Refinements

We adopt the mask-predict (Ghazvininejad et al., 2019) algorithm with several
modifications made. The process (Iterative_Refinements) is illustrated in Alg.
2. We utilize the MLM model for the refinements, which were used to filter the
top-k candidates. We select the tokens from the perturbed sentence having the
lowest MLM scores i.e. unnatural tokens, where the number of tokens selected
per refinement step linearly decays following Ghazvininejad et al. (2019). We,
then repeat the procedure from §2.2 for only the selected indexes from above.
With updated top-k candidates from the perturbed sentence, we again pick the
token which would incur the highest divergence. We repeat this refinement
procedure for a predefined number of steps $S$. Note that we only refine the
tokens among the initially perturbed ones.

The virtual adversarial scores from Eq. 3 are pre-computed before the
refinements, which does not change throughout the iterations since we are
searching for the adversarial sample from the original one. Thus, no further
backward operations are required. However, since we do not want the perturbed
sentence to deviate from the original one, we summed the MLM scores of the
perturbed ones with that of the original ones during refinements for acquiring
top-k possible candidates.

##  3 Experimental Setup

###  3.1 Dataset

We experiment on four datasets in text classification including three topic
classification datasets and ANLI (Nie et al., 2019) with additional natural
language inference (NLI) training datasets. The former evaluate our method’s
effectiveness in SSL following previous works (Xie et al., 2019; Chen et al.,
2020a). The latter is for evaluating the robustness of the models under the
standard supervised training framework.

The three topic classification benchmarks consist of AG News (Zhang et al.,
2015), DBpedia (Mendes et al., 2012), and YAHOO! Answers (Chang et al., 2008).
We follow the experimental setting from Chen et al. (2020a), where we train
with a limited number of labeled data in diverse settings, namely, 10, 200,
2500 per class. We randomly sample the labeled, unlabeled, and development set
and report the performance on the official test set. Since the sampling
procedure can incur high variance in the final performance, we report the
average of five different seeds’ distinct runs for the robust evaluation. The
dataset statistics and split information is presented in Table 1.

Dataset | Genre | Class | Unlabel | Dev | Test
---|---|---|---|---|---
AG_NEWS | News | 4 | 20k | 20k | 19k
YAHOO! | QA | 10 | 50k | 20k | 60k
DBPedia | Wikipedia | 14 | 70k | 20k | 50k

Table 1: Data statistics for the topic classification datasets following the
experimental setting from Chen et al. (2020a).

ANLI (Nie et al., 2019) is an NLI testbed recently introduced for evaluating
the robustness of the models in natural language understanding. The dataset
consists of three rounds (A1-A3), each consisting of a train-dev-test set with
increasing difficulty, where the data is generated by human-and-model-in-the-
loop fashion to fool the strong pre-trained models (Devlin et al., 2019; Yang
et al., 2019; Liu et al., 2019). Following the experimental settings from Nie
et al. (2019), we train the model with two different settings, training with
only the ANLI dataset or additionally training with other NLI datasets,
including SNLI (Bowman et al., 2015), MNLI (Williams et al., 2017), and FEVER
(Thorne et al., 2018). The dataset statistics regarding ANLI is in Table 2.

###  3.2 Baseline

We compare our method with various baselines for the topic classification
datasets. For the ANLI dataset, we compare with the baselines (Devlin et al.,
2019; Yang et al., 2019; Liu et al., 2019; Jiang et al., 2019) that have
reported numbers on the official validation and test set. For the SSL setup,
we use the following baselines:

#### BERT (Devlin et al., 2019)

We use the pre-trained BERT-base-uncased model and finetune it for the
classification dataset using only standard cross-entropy loss.

#### EDA (Wei and Zou, 2019)

EDA is a simple data augmentation strategy based on word unit operations such
as synonym replacement or deletion. We perturb the unlabeled samples using
EDA222https://github.com/jasonwei20/eda_nlp and exploit them for consistency
training.

#### UDA (Xie et al., 2019)

UDA paraphrases the sentence using the back-translation. We employ the WMT-19
DE$\leftrightarrow$EN model from fairseq333https://github.com/pytorch/fairseq
(Ott et al., 2019) to do the back-translation on unlabeled samples, and
exploit them for consistency training.

#### VAT (Miyato et al., 2016, 2018)

We re-implement VAT with or without entropy minimization loss (Grandvalet and
Bengio, 2005) which is to ensure a fair comparison with other methods where
the predictions are sharpened. We apply consistency loss to the unlabeled
samples.

Dataset | Genre | Train | Dev | Test
---|---|---|---|---
A1 | Wikipedia | 17k | 1k | 1k
A2 | Wikipedia | 45k | 1k | 1k
A3 | Various | 100k | 1.2k | 1.2k
ANLI | Various | 162k | 3.2k | 3.2k
MNLI | Various | 392k | - | -
Fever | Wikipedia | 208k | - | -
SNLI | Image Captions | 549k | - | -

Table 2: Data statistics for the ANLI with three rounds (A1-A3) and concerning
NLI datasets for the training.

Method | AG_NEWS | YAHOO! | DBpedia
---|---|---|---
10 | 200 | 2500 | 10 | 200 | 2500 | 10 | 200 | 2500
BERT (Devlin et al., 2019) | 79.4 | 88.6 | 91.6 | 58.2 | 70.1 | 73.9 | 97.8 | 98.8 | 99.1
EDA (Wei and Zou, 2019) | 83.8 | 88.9 | 91.8 | 62.0 | 70.6 | 73.8 | 98.4 | 98.8 | 99.1
UDA (Xie et al., 2019) | 83.8 | 88.5 | 91.6 | 62.0 | 70.4 | 73.7 | 98.2 | 98.8 | 99.1
VAT (Miyato et al., 2016) | 82.3 | 88.9 | 91.8 | 62.4 | 70.7 | 74.1 | 98.4 | 98.8 | 99.1
VAT + Ent-Min (Miyato et al., 2018) | 82.0 | 89.1 | 92.0 | 64.3 | 70.8 | 74.2 | 98.5 | 98.9 | 99.2
VAT-D | 86.2 | 89.8 | 92.3 | 65.3 | 71.7 | 74.1 | 98.4 | 99.0 | 99.2
VAT-D + Iterative Refinement | 86.9 | 90.0 | 92.3 | 66.0 | 71.7 | 74.1 | 98.4 | 99.1 | 99.2

Table 3: Accuracy on topic classification datasets under the various SSL
settings. 10, 200, 2500 denote the number of labeled samples per class used
during training. We average five different runs with a differently indexed
dataset. The numbers in the bold denote the best score.

###  3.3 Training Details

We exploit the unlabeled data from the topic classification datasets and the
labeled data from the ANLI for consistency loss. Throughout the experiments,
we tune the learning rate in between {2e-5, 3e-5}. For the evaluation of topic
classification datasets, we use BERT-base-uncased (Devlin et al., 2019) as a
backbone model and its wordpiece (Schuster and Nakajima, 2012) vocabulary as
$V$. As for the ANLI, we use RoBERTa-Large (Liu et al., 2019) as a backbone
model and its vocabulary as $V$.

During SSL experiments, we set the batch size to 8 for the labeled and 24 for
the unlabeled data with the maximum sequence length of 256. Also, we adopt
some training tactics from UDA (Xie et al., 2019) during SSL to our method and
the baselines, namely training signal annealing and sharpening, which
moderately improves the performance. The former is not to train overconfident
samples, which is applied to the labeled data, and the latter is for
sharpening the original samples’ predictions for the unlabeled data. However,
we did not apply sharpening to the VAT baseline, but instead the entropy-
minimization following the original work (Miyato et al., 2018). We utilized a
single 24G GPU for the training. For the ANLI, we follow the hyper-parameter
suggestion from the official
repository.444https://github.com/facebookresearch/anli, where we train with
the batch size of 128 and the maximum sequence length of 156. We experimented
with eight 32G GPUs for the training.

Throughout the experiments, we set the replacement ratio $\tau$ as 0.25 and
top-k as 10. We sharpen the predictions with $T$ as 0.5 for topic
classification datasets (including baselines) and 0.75 for the ANLI. We
exclude the original token from the sentence not to be included in top-k
candidates of itself. For the refinements, we additionally conducted three
steps from the perturbed sequence, i.e. $S=3$. We verified not much variation
in adjusting the hyperparameters of our methods, such as top-k and replacement
ratio. Our implementation is based on
huggingface555https://github.com/huggingface/transformers (Wolf et al., 2020).
In total, our method requires a single backward operation and a forward
operation for generating the virtual adversarial samples, where each
refinement step would additionally incur a single forward step. Our method
takes approximately 2.5 times the standard training without refinements and
3.5 times with them, where the time complexity is similar or less than the
recently introduced relevant methods (Jiang et al., 2019; Ng et al., 2020).

##  4 Experimental Results

###  4.1 Semi Supervised Text Classification

Table 3 shows the experimental results on topic classification datasets under
SSL setup. Our method outperforms the baselines by up to 7.5 points from the
BERT model finetuned with standard cross-entropy loss and 3.1 points from
other methods utilizing the consistency regularization loss. The accuracy
gained from the proposed method with or without iterative refinement stands
out from the baselines, especially when the number of labeled samples is
limited. The refinements further bring a moderate gain in the performance.
However, since all the methods have already achieved high accuracy in the
DBpedia, the difference among methods is not significant.

Among the baselines, VAT (Miyato et al., 2016, 2018) with entropy minimization
(Grandvalet and Bengio, 2005) performs reasonably well. The finding supports
the claim that a transformation during consistency training should be done
with regard to the training model (Miyato et al., 2018). We further conduct an
ablation study in §5.1 for the verification. However, as the number of labeled
samples is limited, the local perturbation shows its limitation in strongly
regularizing the model compared to our method. For the baseline VAT (Miyato et
al., 2018), we can inspect the improvements when applying the entropy
minimization (Grandvalet and Bengio, 2005) loss, which justifies utilizing the
sharpening operation for SSL since both operations are equivalent.

###  4.2 Adversarial Natural Language Inference

Method | Dev | Test
---|---|---
A1 | A2 | A3 | ALL | A1 | A2 | A3 | ALL
MNLI + SNLI + ANLI + FEVER
BERT(Nie et al., 2019) | 57.4 | 48.3 | 43.5 | 49.3 | - | - | - | 44.2
XLnet (Nie et al., 2019) | 67.6 | 50.7 | 48.3 | 55.1 | - | - | - | 52.0
RoBERTa (Nie et al., 2019) | 73.8 | 48.9 | 44.4 | 53.7 | - | - | - | 49.7
SMART (Jiang et al., 2019) | 74.5 | 50.9 | 47.6 | 57.1 | 72.4 | 49.8 | 50.3 | 57.1
VAT-D | 74.5 | 54.2 | 50.8 | 59.2 | 72.4 | 51.8 | 49.5 | 57.4
VAT-D + Iterative Refinement | 74.4 | 54.3 | 50.8 | 59.2 | 71.9 | 53.0 | 49.8 | 57.7
ANLI
RoBERTa (Nie et al., 2019) | 71.3 | 43.3 | 43.0 | 51.9 | - | - | - | -
SMART (Jiang et al., 2019) | 74.2 | 49.5 | 49.2 | 57.1 | 72.4 | 50.3 | 49.5 | 56.9
VAT-D | 74.8 | 52.1 | 51.1 | 58.8 | 72.1 | 51.4 | 51.7 | 57.9
VAT-D + Iterative Refinement | 75.3 | 52.0 | 50.8 | 58.8 | 71.4 | 51.9 | 50.3 | 57.4

Table 4: Accuracy on the ANLI benchmark. The numbers of the baselines are from
the original papers (Nie et al., 2019; Jiang et al., 2019). The upper section
is for training with all the NLI datasets, and the bottom is for training with
only the ANLI.

Table 4 shows the experimental results on the ANLI dataset with different
training settings: training with all the NLI datasets, or training with only
the ANLI dataset. Our method improves over baselines, including RoBERTa-Large
(Liu et al., 2019) and SMART (Jiang et al., 2019) in both settings. By using
RoBERTa-large (Liu et al., 2019) and our training framework, we further
improve on an average of 8.0 points in the test set from training with cross-
entropy loss only. SMART is the method for preventing aggressive finetuning by
inducing smoothness regularization, i.e. a variation of VAT, and Bregman
proximal point optimization. Our method outperforms the SMART on an average of
1 point from the test set.

Our method’s accuracy improvement over baselines in the experiment implies
that the methodology can be applied to the standard training framework where
the unlabeled samples are not available. The result also implies that the
method can be exploited for the two-sequence classification tasks without
specific modifications.

##  5 Ablations

###  5.1 Effectiveness of the White-box Search

Our central intuition behind the proposed method is to generate the augmented
samples concerning the model, i.e. vulnerable to the model. This section
further conducts an ablation study for whether such virtual adversarial search
is crucial in discrete space. We select the token among the top-K candidates
that would incur the highest divergence from the model’s prediction. We
compare with other sampling strategies among the top-K candidates, namely,
uniform sampling (Uniform), selecting the token with maximum MLM probability
(Argmax), and sampling from MLM probabilities (Sampling). We match other
training details except for the sampling strategy for a fair comparison.

Table 5 illustrates the result of comparisons. The virtual adversarial search
among candidates outperforms other search strategies in discrete space,
especially when the number of labeled samples is limited. The result
demonstrates that the virtual adversarial search is indeed the crucial
component during perturbation. Furthermore, Fig. 2 shows the indexes selected
from top-k distribution (Sampling and VAT-D, YAHOO! dataset). The distribution
of indexes selected from our method resembles the uniform distribution;
however, as in the loss plot from Fig. 2, our method searches for the diverse
yet adversarial candidates to the model, i.e. incurring high divergence.

###  5.2 Enhancing the Fluency by Refinements

Method | AG_NEWS | YAHOO!
---|---|---
10 | 200 | 2500 | 10 | 200 | 2500
VAT-D | 86.2 | 89.8 | 92.3 | 65.3 | 71.7 | 74.1
Uniform | 83.8 | 89.3 | 91.8 | 63.2 | 70.8 | 73.8
Argmax | 83.2 | 89.0 | 91.9 | 63.7 | 70.9 | 73.7
Sampling | 84.8 | 89.3 | 91.8 | 63.5 | 70.9 | 73.8

Table 5:  Accuracy according to different sampling strategies from top-K
candidates.

(a) Top-k Distribution (b) Consistency Loss

Figure 2: Index distribution from the top-k candidates sorted by MLM scores
(a) and consistency loss of different sampling strategies during training (b).

Type | Original | Perturbed
---|---|---
Refinement | N/A | 0 | 1 | 2 | 3
Perplexity | 1.42 | 1.87 | 1.78 | 1.68 | 1.67

Table 6: Average perplexity of the original sentences and perturbed sentences
during iterative refinement.

We adopt and modify the iterative refinement-based method from NAR, i.e. mask-
predict algorithm (Ghazvininejad et al., 2019), for improving the fluency of
the perturbed sentences. We analyze the perturbed sentences’ perplexity
according to refinement steps acquired by MLM (unlabeled samples from
AG_NEWS). Table 6 shows that (1) perturbed sentences overall have higher
perplexity than the original, but (2) refinements always lower the perplexity,
leading to more natural and fluent sentences. Note that this does not
necessarily imply further refining will enhance the training task’s
performance for the possibility of deviating the semantic from the original
sentence.

##  6 Related Works

#### Consistency Regularization

Consistency regularization (Laine and Aila, 2016; Sajjadi et al., 2016) has
been mainly explored in the context of SSL (Chapelle et al., 2009; Oliver et
al., 2018). A line of research in text-domain (Miyato et al., 2016; Clark et
al., 2018; Xie et al., 2019) explored the idea. However, the framework is also
applicable when only the labeled samples are available (Miyato et al., 2018;
Jiang et al., 2019; Asai and Hajishirzi, 2020). Our method is applicable and
efficient in both settings. Existing studies explored varying perturbation
methods. Injecting norm-constrained continuous noise to the embedding space
(Miyato et al., 2016; Jiang et al., 2019; Liu et al., 2020; Chen et al.,
2020b; Sano et al., 2019) and directly perturbing the text (Clark et al.,
2018; Minervini and Riedel, 2018; Li et al., 2019; Xie et al., 2019; Asai and
Hajishirzi, 2020) via discrete noise are the primary approaches for the
perturbation. Our method perturbs the sentence by the discrete noise, yet the
noise is generated concerning the training mode.

#### Adversarial Training

Our method extends the white-box-based adversarial training framework
(Goodfellow et al., 2014), which has recently begun to be widely explored in
the NLP domain (Miyato et al., 2016; Ebrahimi et al., 2017; Michel et al.,
2019; Wang et al., 2019; Zhu et al., 2019; Jiang et al., 2019; Cheng et al.,
2019; Liu et al., 2020). Miyato et al. (2016) first adopted the adversarial
training framework on text classification by perturbing embedding space with
continuous adversarial noise. Cheng et al. (2019) introduced an adversarial
training framework on machine translation by discrete word replacements.
However, the method relies on the label information, thus can not be applied
to SSL, whereas ours can be. On the other hand, there are black-box approaches
for generating the adversarial attacks or test sets (Jia and Liang, 2017;
Alzantot et al., 2018; Ribeiro et al., 2018, 2019; Li et al., 2020b, a;
Gardner et al., 2020) to evaluate the vulnerability of the NLP models. Li et
al. (2020b, a) replace tokens to find the sentence that maximizes the
divergence via multiple iterations, unlike our method, which utilizes gradient
information during training.

#### Augmentation

Synthetically generated training examples are utilized to augment an existing
dataset. Existing word-level augmentation methods (Zhang et al., 2015; Xie et
al., 2017; Wei and Zou, 2019) are based on heuristics. Back-translation from
or through another language has been applied in the context of machine
translation (Sennrich et al., 2015) and consistency training (Xie et al.,
2019). Other methods include utilizing contextual language models (Kobayashi,
2018; Wu et al., 2019) or generative models (Anaby-Tavor et al., 2020; Yang et
al., 2020). Unlike previous works, our method is subject to the training
model, thus approximating the augmented points, efficiently filling in gaps
from the training data.

##  7 Conclusion

We present VAT-D, a consistency training framework with virtual adversarial
discrete perturbation. In contrast to other methods, we utilize the gradients
information to approximate the samples that would incur the highest divergence
from the models’ predictions for efficiently pushing the decision boundary.
Our method applies to SSL since we do not require label information and is
beneficial for the standard training framework. We further employ the idea of
iterative refinement from NAR literature to our problem setting for enhancing
the fluency of the perturbed samples. Our method does not require domain-
specific knowledge or fine-tuning except for the training model. We
demonstrate our method’s effectiveness in topic classification datasets and
ANLI under various settings.

## References

  * Alzantot et al. (2018) Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018.  Generating natural language adversarial examples.  _arXiv preprint arXiv:1804.07998_.
  * Anaby-Tavor et al. (2020) Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020.  Do not have enough data? deep learning to the rescue!  In _AAAI_ , pages 7383–7390.
  * Asai and Hajishirzi (2020) Akari Asai and Hannaneh Hajishirzi. 2020.  Logic-guided data augmentation and regularization for consistent question answering.  _arXiv preprint arXiv:2004.10157_.
  * Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015\.  A large annotated corpus for learning natural language inference.  _arXiv preprint arXiv:1508.05326_.
  * Chang et al. (2008) Ming-Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008.  Importance of semantic representation: Dataless classification.  In _Aaai_ , volume 2, pages 830–835.
  * Chapelle et al. (2009) Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. 2009.  Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews].  _IEEE Transactions on Neural Networks_ , 20(3):542–542.
  * Chen et al. (2020a) Jiaao Chen, Zichao Yang, and Diyi Yang. 2020a.  Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification.  _arXiv preprint arXiv:2004.12239_.
  * Chen et al. (2020b) Luoxin Chen, Weitong Ruan, Xinyue Liu, and Jianhua Lu. 2020b.  Seqvat: Virtual adversarial training for semi-supervised sequence labeling.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8801–8811.
  * Cheng et al. (2019) Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019.  Robust neural machine translation with doubly adversarial inputs.  In _ACL_.
  * Clark et al. (2018) Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc V Le. 2018.  Semi-supervised sequence modeling with cross-view training.  _arXiv preprint arXiv:1809.08370_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_.
  * Ebrahimi et al. (2017) Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2017.  Hotflip: White-box adversarial examples for text classification.  In _ACL_.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.  _arXiv preprint arXiv:1808.09381_.
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. 2020\.  Evaluating models’ local decision boundaries via contrast sets.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 1307–1323.
  * Ghazvininejad et al. (2019) Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019.  Mask-predict: Parallel decoding of conditional masked language models.  _arXiv preprint arXiv:1904.09324_.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014.  Explaining and harnessing adversarial examples.  In _ICLR_.
  * Grandvalet and Bengio (2005) Yves Grandvalet and Yoshua Bengio. 2005.  Semi-supervised learning by entropy minimization.  In _Advances in neural information processing systems_ , pages 529–536.
  * Gu et al. (2017) Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2017\.  Non-autoregressive neural machine translation.  _arXiv preprint arXiv:1711.02281_.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial examples for evaluating reading comprehension systems.  _arXiv preprint arXiv:1707.07328_.
  * Jiang et al. (2019) Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2019.  Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.  _arXiv preprint arXiv:1911.03437_.
  * Kobayashi (2018) Sosuke Kobayashi. 2018.  Contextual augmentation: Data augmentation by words with paradigmatic relations.  _arXiv preprint arXiv:1805.06201_.
  * Laine and Aila (2016) Samuli Laine and Timo Aila. 2016.  Temporal ensembling for semi-supervised learning.  _arXiv preprint arXiv:1610.02242_.
  * Lee et al. (2018) Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018.  Deterministic non-autoregressive neural sequence modeling by iterative refinement.  _arXiv preprint arXiv:1802.06901_.
  * Lee et al. (2020) Jason Lee, Raphael Shu, and Kyunghyun Cho. 2020.  Iterative refinement in the continuous space for non-autoregressive neural machine translation.  _arXiv preprint arXiv:2009.07177_.
  * Li et al. (2020a) Dianqi Li, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. 2020a.  Contextualized perturbation for textual adversarial attack.  _arXiv preprint arXiv:2009.07502_.
  * Li et al. (2020b) Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020b.  Bert-attack: Adversarial attack against bert using bert.  _arXiv preprint arXiv:2004.09984_.
  * Li et al. (2019) Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019.  A logic-driven framework for consistency of neural models.  _arXiv preprint arXiv:1909.00126_.
  * Liu et al. (2020) Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. 2020.  Adversarial training for large neural language models.  _arXiv preprint arXiv:2004.08994_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Mendes et al. (2012) Pablo N Mendes, Max Jakob, and Christian Bizer. 2012.  Dbpedia: A multilingual cross-domain knowledge base.  In _LREC_ , pages 1813–1817. Citeseer.
  * Michel et al. (2019) Paul Michel, Xian Li, Graham Neubig, and Juan Miguel Pino. 2019.  On evaluation of adversarial perturbations for sequence-to-sequence models.  _arXiv preprint arXiv:1903.06620_.
  * Minervini and Riedel (2018) Pasquale Minervini and Sebastian Riedel. 2018.  Adversarially regularising neural nli models to integrate logical background knowledge.  _arXiv preprint arXiv:1808.08609_.
  * Miyato et al. (2016) Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2016.  Adversarial training methods for semi-supervised text classification.  _arXiv preprint arXiv:1605.07725_.
  * Miyato et al. (2018) Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018.  Virtual adversarial training: a regularization method for supervised and semi-supervised learning.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(8):1979–1993.
  * Ng et al. (2020) Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi. 2020.  Ssmba: Self-supervised manifold based data augmentation for improving out-of-domain robustness.  _arXiv preprint arXiv:2009.10195_.
  * Nie et al. (2019) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019.  Adversarial nli: A new benchmark for natural language understanding.  _arXiv preprint arXiv:1910.14599_.
  * Oliver et al. (2018) Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. 2018.  Realistic evaluation of deep semi-supervised learning algorithms.  In _Advances in neural information processing systems_ , pages 3235–3246.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _NAACL-HLT 2019: Demonstrations_.
  * Park et al. (2020) Jungsoo Park, Mujeen Sung, Jinhyuk Lee, and Jaewoo Kang. 2020.  Adversarial subword regularization for robust neural machine translation.  _arXiv preprint arXiv:2004.14109_.
  * Ribeiro et al. (2019) Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019.  Are red roses red? evaluating consistency of question-answering models.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6174–6184.
  * Ribeiro et al. (2018) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018.  Semantically equivalent adversarial rules for debugging nlp models.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 856–865.
  * Sajjadi et al. (2016) Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. 2016.  Regularization with stochastic transformations and perturbations for deep semi-supervised learning.  In _Advances in neural information processing systems_ , pages 1163–1171.
  * Sano et al. (2019) Motoki Sano, Jun Suzuki, and Shun Kiyono. 2019.  Effective adversarial regularization for neural machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 204–210.
  * Schuster and Nakajima (2012) Mike Schuster and Kaisuke Nakajima. 2012.  Japanese and korean voice search.  In _ICASSP_.
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015.  Improving neural machine translation models with monolingual data.  _arXiv preprint arXiv:1511.06709_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Edinburgh neural machine translation systems for wmt 16.  _arXiv preprint arXiv:1606.02891_.
  * Simard et al. (1998) Patrice Y Simard, Yann A LeCun, John S Denker, and Bernard Victorri. 1998.  Transformation invariance in pattern recognition—tangent distance and tangent propagation.  In _Neural networks: tricks of the trade_ , pages 239–274. Springer.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  Fever: a large-scale dataset for fact extraction and verification.  _arXiv preprint arXiv:1803.05355_.
  * Vapnik (2000) Vladimir Vapnik. 2000.  Vicinal risk minimization.  In _NIPS_.
  * Verma et al. (2019) Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. 2019.  Interpolation consistency training for semi-supervised learning.  _arXiv preprint arXiv:1903.03825_.
  * Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019.  Universal adversarial triggers for attacking and analyzing nlp.  In _EMNLP_.
  * Wang et al. (2019) Dilin Wang, Chengyue Gong, and Qiang Liu. 2019.  Improving neural language modeling via adversarial training.  In _ICLR_.
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  Eda: Easy data augmentation techniques for boosting performance on text classification tasks.  _arXiv preprint arXiv:1901.11196_.
  * Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017.  A broad-coverage challenge corpus for sentence understanding through inference.  _arXiv preprint arXiv:1704.05426_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2019) Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019.  Conditional bert contextual augmentation.  In _International Conference on Computational Science_ , pages 84–95. Springer.
  * Xie et al. (2019) Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019.  Unsupervised data augmentation for consistency training.  _arXiv preprint arXiv:1904.12848_.
  * Xie et al. (2017) Ziang Xie, Sida I Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng. 2017.  Data noising as smoothing in neural network language models.  _arXiv preprint arXiv:1703.02573_.
  * Yang et al. (2020) Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020.  G-daug: Generative data augmentation for commonsense reasoning.  _arXiv preprint arXiv:2004.11546_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _NeurIPS_.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in neural information processing systems_ , pages 649–657.
  * Zhu et al. (2019) Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Thomas Goldstein, and Jingjing Liu. 2019\.  Freelb: Enhanced adversarial training for language understanding.  _arXiv preprint arXiv:1909.11764_.

##  Appendix A Validation Scores on the Topic Classification Datasets

Method | AG_NEWS | YAHOO! | DBpedia
---|---|---|---
10 | 200 | 2500 | 10 | 200 | 2500 | 10 | 200 | 2500
BERT (Devlin et al., 2019) | 79.7 | 89.0 | 92.0 | 58.3 | 70.0 | 73.8 | 97.9 | 98.9 | 99.1
EDA (Wei and Zou, 2019) | 84.4 | 89.3 | 92.2 | 61.9 | 70.4 | 73.8 | 98.6 | 98.8 | 99.2
VAT (Miyato et al., 2016) | 82.3 | 89.4 | 92.3 | 62.2 | 70.5 | 74.2 | 98.6 | 98.9 | 99.2
VAT + Ent-Min (Miyato et al., 2018) | 82.2 | 89.9 | 92.4 | 64.2 | 70.8 | 74.3 | 98.7 | 99.0 | 99.2
UDA (Xie et al., 2019) | 84.3 | 88.7 | 92.0 | 61.6 | 70.2 | 73.7 | 98.2 | 98.9 | 99.2
VAT-D | 86.6 | 90.3 | 92.5 | 65.2 | 71.5 | 74.2 | 98.5 | 99.0 | 99.2
VAT-D + Iterative Refinements | 87.4 | 90.5 | 92.5 | 65.7 | 71.5 | 74.2 | 98.4 | 99.1 | 99.2

Table A.1: Accuracy on the validation data set from the§4.1. We average five
different runs with a differently indexed dataset.