# Low-Rank+Sparse Tensor Compression for Neural Networks

Cole Hawkins
University of California, Santa Barbara Haichuan Yang, Meng Li, Liangzhen Lai,
Vikas Chandra
Facebook Inc Work completed during an internship at Facebook

###### Abstract

Low-rank tensor compression has been proposed as a promising approach to
reduce the memory and compute requirements of neural networks for their
deployment on edge devices. Tensor compression reduces the number of
parameters required to represent a neural network weight by assuming network
weights possess a coarse higher-order structure. This coarse structure
assumption has been applied to compress large neural networks such as VGG and
ResNet. However modern state-of-the-art neural networks for computer vision
tasks (i.e. MobileNet, EfficientNet) already assume a coarse factorized
structure through depthwise separable convolutions, making pure tensor
decomposition a less attractive approach. We propose to combine low-rank
tensor decomposition with sparse pruning in order to take advantage of both
coarse and fine structure for compression. We compress weights in SOTA
architectures (MobileNetv3, EfficientNet, Vision Transformer) and compare this
approach to sparse pruning and tensor decomposition alone.

##  1 Introduction

Using machine learning to enable multiple applications on edge devices
requires multiple task-specific persistent models (Yang et al., 2020). These
models are used for tasks ranging from computer vision (Howard et al., 2019)
to automatic speech recognition (Wang et al., 2021). The trend towards
multiple applications and multiple models is constrained by the fact that off-
chip memory reads incur high latency and power costs (Sze et al., 2017).
Therefore, in this paper we target memory cost reduction. In this area low-
rank tensor compression is a popular approach (Garipov et al., 2016; Novikov
et al., 2015; Lebedev et al., 2014) that can achieve orders of magnitude
compression, but can lead to significant accuracy loss.

Low-rank tensor compression has achieved impressive headline compression
numbers (Garipov et al., 2016) and is suitable for on-device acceleration
(Zhang et al., May 2021; Deng et al., 2019) due to its reliance on standard
dense linear algebra operations. However it is often applied to massively
overparameterized architectures such as VGG or ResNet (Novikov et al., 2015;
Hawkins & Zhang, 2019). Recent work benchmarking sparse pruning (Blalock et
al., 2020) raises the issue that compression techniques applied to
overparameterized architectures may not reach the compression/accuracy Pareto
frontier of SOTA compressed networks such as EfficientNet. Further, tensor
compression enforces low-rank factorized structure on neural network weights.
This factorized structure is already exploited by SOTA computer vision
backbones via depth-wise separable convolutions. This motivates us to consider
the following question: how can low-rank tensor compression contribute to SOTA
computer vision architectures? In this paper we investigate whether low-rank
tensor compression can be combined with sparse pruning to capture
complementary coarse and fine structures respectively and outperform either
sparse pruning or low-rank tensor factorization alone.

Specifically, we investigate two forms of low-rank plus sparse decomposition
of neural network weights. First we consider additive structure in which the
neural network weights can be decomposed as the sum of a low-rank component
and a sparse component. Second we consider a low-rank or sparse structure in
which the entries of neural network weights are taken from either a sparse
pruned weight or a low-rank weight.

###  1.1 Related Work

Existing work has studied the combination of low-rank matrices and sparse
representations to compress VGG-style models (Yu et al., 2017) and ResNets
(Gui et al., 2019). These methods apply additive low-rank plus sparse
compression to the weights of neural networks and can achieve compression
results that outperform sparse compression or low-rank compression alone. More
recent work suggests that low-rank structure is a viable compression method
for the EfficientNet model class (Gu et al., 2021) on CIFAR-10/100 computer
vision tasks. An open question is whether low-rank matrix or tensor structure
remains for efficient modern networks trained on ImageNet.

###  1.2 Contributions

Our main contribution is our study of low-rank structure remaining in recent
SOTA efficient neural networks for computer vision tasks. We observe that
trends in architecture design (Howard et al., 2017; Sandler et al., 2018;
Howard et al., 2019) have removed low-rank structure from such networks by
building in factorized weights. Therefore the direct application of low-rank
matrix and tensor methods is challenging. To the best of our knowledge, our
work is the first to consider the combination of low-rank tensor compression
with sparse pruning. Further, our work is the first to study low-rank+sparse
compression of weights for SOTA architectures that rely on efficient depthwise
separable convolutions.

Figure 1: High-level proposed algorithm flow.

##  2 Background

###  2.1 Tensor Formats

We follow the notation and exposition of (Hawkins et al., 2020) to describe
the low-rank tensor formats that we employ. This paper uses lower-case letters
(e.g., $a$) to denote scalars, bold lowercase letters (e.g., $\mathbf{a}$) to
represent vectors, bold uppercase letters (e.g., $\mathbf{A}$) to represent
matrices, and bold calligraphic letters (e.g., $\mathbfcal{A}$) to denote
tensors. A tensor is a generalization of a matrix, or a multi-way data array.
An order-$d$ tensor is a $d$-way data array
$\mathbfcal{A}\in\mathbb{R}^{I_{1}\times I_{2}\times\dots\times I_{d}}$, where
$I_{n}$ is the size of mode $n$. The $(i_{1},i_{2},\cdots,i_{d})$-th element
of $\mathbfcal{A}$ is denoted as $a_{i_{1}i_{2}\cdots i_{d}}$. An order-$3$
tensor is shown in Fig. 2 (a).

The number of entries in a tensor with mode size $n$ and $d$ modes is $n^{d}$.
This exponential dependence on the tensor order is often referred to as the
“curse of dimensionality" and can lead to high costs in both computing and
storage. Fortunately, many practical tensors have a low-rank structure, and
this property can be exploited to reduce such costs. We will use four tensor
decomposition formats (CP, Tucker, and TT/TTM) to reduce the parameters of
neural networks.

First, we introduce a necessary decomposition to describe the CP format.

###### Definition 1

A $d$-way tensor $\mathbfcal{A}\in\mathbb{R}^{I_{1}\times\cdots\times I_{d}}$
is rank-1 if it can be written as a single outer product of $d$ vectors

$\mathbfcal{A}=\mathbf{u}^{(1)}\circ\dots\circ\mathbf{u}^{(d)},\;{\text{with}}\;\mathbf{u}^{(n)}\in\mathbb{R}^{I_{n}}\;\text{for}\;n=1,\cdots,d.$

Each element of $\mathbfcal{A}$ is $a_{i_{1}i_{2}\cdots
i_{d}}=\prod\limits_{n=1}^{d}u_{i_{n}}^{(n)}$, where $u_{i_{n}}^{(n)}$ is the
$i_{n}$-th element of the vector $\mathbf{u}^{(n)}$.

###### Definition 2

The CP factorization (Carroll & Chang, 1970; Harshman et al., 1994) expresses
tensor $\mathbfcal{A}$ as the sum of multiple rank-1 tensors:

$\mathbfcal{A}=\sum_{j=1}^{R}\mathbf{u}_{j}^{(1)}\circ\mathbf{u}_{j}^{(2)}\dots\circ\mathbf{u}_{j}^{(d)}.$
(1)

Here $\circ$ denotes an outer product operator. The minimal integer $R$ that
ensures the equality is called the CP rank of $\mathbfcal{A}$. To simplify
notation we collect the rank-1 terms of the $n$-th mode into a factor matrix
$\mathbf{U}^{(n)}\in\mathbb{R}^{I_{n}\times R}$ with
$\mathbf{U}^{(n)}(:,j)=\mathbf{u}_{j}^{(n)}$. A rank-$R$ CP factorization can
be described with $d$ factor matrices $\\{\mathbf{U}^{(n)}\\}_{n=1}^{d}$ using
$R\sum_{n}I_{n}$ parameters.

(a)

(b)

(c)

(d)

Figure 2: Taken from Hawkins et al. (2020). (a): An order-3 tensor, (b) and
(c): representations in CP and Tucker formats respectively, where low-rank
factors are color-coded to indicate the corresponding modes. (d): TT
representation of an order-$d$ tensor, where the purple lines and squares
indicate $\mathbfcal{G}^{(n)}(:,i_{n},:)$, which is the $i_{n}$-th slice of
the TT core $\mathbfcal{G}^{(n)}$ obtained by fixing its second index.

Next we introduce a prerequisite definition for the low-rank Tucker format.

###### Definition 3

The mode-$n$ product of a tensor
$\mathbfcal{A}\in\mathbb{R}^{I_{1}\times\dots\times I_{n}\times\dots\times
I_{d}}$ with a matrix $\mathbf{U}\in\mathbb{R}^{J\times I_{n}}$ is

$\begin{split}\mathbfcal{B}&=\mathbfcal{A}\times_{n}\mathbf{U}\Longleftrightarrow
b_{i_{1}\dots i_{n-1}ji_{n+1}\dots i_{d}}=\sum_{i_{n}=1}^{I_{n}}a_{i_{1}\dots
i_{d}}u_{ji_{n}}.\end{split}$ (2)

The result is still a $d$-dimensional tensor $\mathbfcal{B}$, but the mode-$n$
size becomes $J$. In the special case $J=1$, the $n$-th mode diminishes and
$\mathbfcal{B}$ becomes an order-$d-1$ tensor. The Tucker format is defined by
a series of mode$-n$ products between a small core tensor and several factor
matrices.

###### Definition 4

The Tucker factorization (Tucker, 1966) expresses a $d$-way tensor
$\mathbfcal{A}$ as a series of mode-$n$ products:

$\mathbfcal{A}=\mathbfcal{G}\times_{1}\mathbf{U}^{(1)}\times_{2}\dots\times_{d}\mathbf{U}^{(d)}.$
(3)

Here $\mathbfcal{G}\in\mathbb{R}^{R_{1}\times\dots\times R_{d}}$ is a small
core tensor, and $\mathbf{U}^{(n)}\in\mathbb{R}^{I_{n}\times{R}_{n}}$ is a
factor matrix for the $n$-th mode. The Tucker rank is the tuple
$(R_{1},\dots,R_{d})$. A Tucker factorization with ranks $R_{n}=R$ requires
$R^{d}+R\sum_{n}I_{n}$ parameters.

The product structure permits more expressive cross-dimension interactions
than the CP model but may not result in large parameter reductions due to the
exponential dependence on the tensor order $d$.

Finally we introduce two formats, Tensor-Train and Tensor-Train Matrix. Like
the Tucker format, these formats permit more expressive interactions across
dimensions. Unlike the Tucker format these formats remove the exponential
dependence on the tensor order $d$.

###### Definition 5

The tensor-train (TT) factorization (Oseledets, 2011) expresses a $d$-way
tensor $\mathbfcal{A}$ as a collection of matrix products:

$a_{i_{1}i_{2}\dots
i_{d}}=\mathbfcal{G}^{(1)}(:,i_{1},:)\mathbfcal{G}^{(2)}(:,i_{2},:)\dots\mathbfcal{G}^{(d)}(:,i_{d},:).$
(4)

Each TT-core $\mathbfcal{G}^{(n)}\in\mathbb{R}^{R_{n-1}\times I_{n}\times
R_{n}}$ is an order-$3$ tensor. The tuple $(R_{0},R_{1},\dots,R_{d})$ is the
TT-rank and $R_{0}=R_{d}=1$.

The TT format uses $\sum_{n}R_{n-1}I_{n}R_{n}$ parameters in total and leads
to more expressive interactions than the CP format.

The TT-format is well-suited to tensors, but is equivalent to low-rank matrix
format when the order is $d=2$. We can extend the TT-format to matrices by
reshaping along each mode. Let $\mathbf{A}\in\mathbb{R}^{I\times J}$ be a
matrix. We assume that $I$ and $J$ can be factored as follows:

$I=\prod_{n=1}^{d}I_{n},J=\prod_{n=1}^{d}J_{n}.$ (5)

We can reshape $\mathbf{A}$ into a tensor $\mathbfcal{A}$ with dimensions
$I_{1}\times\dots\times I_{d}\times J_{1}\times\dots\times J_{d}$, such that
the $(i,j)$-th element of $\mathbf{A}$ uniquely corresonds to the
$(i_{1},i_{2},\cdots,i_{d},j_{1},j_{2},\cdots,j_{d})$-th element of
$\mathbfcal{A}$. The TT decomposition can extended to compress the resulting
order-$2d$ tensor as follows.

###### Definition 6

The tensor-train matrix (TTM) factorization expresses an order-$2d$ tensor
$\mathbfcal{A}$ as $d$ matrix products:

$a_{i_{1}\dots i_{d}j_{1}\dots
j_{d}}=\mathbfcal{G}^{(1)}(:,i_{1},j_{1},:)\mathbfcal{G}^{(2)}(:,i_{2},j_{2},:)\dots\mathbfcal{G}^{(d)}(:,i_{d},j_{d},:).$
(6)

Each TT-core $\mathbfcal{G}^{(n)}\in\mathbb{R}^{R_{n-1}\times I_{n}\times
J_{n}\times R_{n}}$ is an order $4$ tensor. The tuple
$(R_{0},R_{1},R_{2},\dots,R_{d})$ is the TT-rank and as before
$R_{0}=R_{d}=1$. This TTM factorization requires
$\sum_{n}R_{n-1}I_{n}J_{n}R_{n}$ parameters to represent $\mathbfcal{A}$.

We provide a visual representation of the CP, Tucker, and TT formats in Fig. 2
(b) – (d).

###  2.2 Folding to high-order tensors.

In the previous section we described how to adapt the TT format to the matrix
case by factorizing the dimensions of a matrix and viewing the matrix as a
tensor. Our goal is to apply tensor decompositions to the weights of many
architectures that include $1\times 1$ convolutions (MobileNet, EfficientNet)
or fully-connected layers (Vision Transformer). These matrix-format weights
are well-suited to the TTM decomposition, but not the CP and Tucker formats
which collapse to low-rank matrix format. To apply the CP and Tucker
decompositions to these matrix weights we can perform a reshape operation. A
weight matrix $\mathbf{W}\in\mathbb{R}^{I\times J}$ can be folded into an
order-$d$ tensor $\mathbfcal{A}\in\mathbb{R}^{I_{1}\times\dots\times I_{d}}$
where $IJ=\prod_{n}I_{n}$. We can also fold $\mathbf{W}$ to an order-$2d$
tensor $\mathbfcal{A}\in\mathbb{R}^{I_{1}\times\cdots\times I_{d}\times
J_{1}\times\cdots\times J_{d}}$ such that $w_{ij}=a_{i_{1}\cdots
i_{d}j_{1}\cdots j_{d}}$. While a convolution filter is already a tensor, we
can reshape it to a higher-order tensor with reduced mode sizes. Reshaping
imposes additional higher-order and hierarchical tensor structure.

###  2.3 Sparse Pruning

We prune a tensor weight $\mathbfcal{S}$ by applying a binary tensor mask
$\mathbfcal{M}$ to $\mathbfcal{S}$ using the entry-wise product

$(\mathbfcal{M}\odot\mathbfcal{S})_{i_{1}\dots i_{d}}=m_{i_{1}\dots
i_{d}}\times s_{i_{1}\dots i_{d}}.$

We focus on global magnitude pruning LeCun et al. (1990) throughout this work
and use a cubic pruning schedule taken from Zhu & Gupta (2017). At each step
we mask the bottom $100\times(s_{t}-s_{t-1})$ percent of the non-masked
weights in $\mathbfcal{S}$ by setting their corresponding entries in
$\mathbfcal{M}$ to zero where

$s_{t}=s_{f}\left(\frac{t}{n}\right)^{3}$ (7)

and $s_{f}$ is the desired final sparsity ratio, $t$ is the current step, and
$n$ is the total number of steps. This process gradually increases the
sparsity in $\mathbfcal{M}$ until the target sparsity $s_{f}$ is reached.

###  2.4 Distillation

The goal of knowledge distillation is to transfer the knowledge of an
expensive teacher model to a cheaper student model (Hinton et al., 2015). In
this setting “expensive" refers to the memory cost of storing the neural
network weights. The expensive teacher model $f_{t}$ is a neural network
pretrained on ImageNet. The student model $f_{s}$ is the compressed teacher
model obtained through sparse pruning and/or tensor compression. Given a
training example with input $\mathbf{x}$ and label $\mathbf{y}$ the knowledge
distillation loss is

$\mathcal{L}\left(\mathbf{x},\mathbf{y}\right)=\alpha
l(\mathbf{x},\mathbf{y})+(1-\alpha)T^{2}KL\left(f_{s}(\mathbf{x})/T||f_{t}(\mathbf{x})/T\right)$
(8)

where $l$ is any standard loss function and $KL$ is the KL-divergence. In our
setting $l$ is the cross-entropy loss. The KL term in the loss function
penalizes the student output distribution when it differs from the teacher
output distribution. Both outputs are temperature scaled by $T$. We set
$\alpha=0.9$ and $T=3$ which is a standard choice (Hinton et al., 2015; Cho &
Hariharan, 2019). We note that another reasonable loss function is fine tuning
on the training data only (set $\alpha=1$) which we do not explore in this
work.

##  3 Method: Low-Rank + Sparse Tensor Decomposition

At a high level, our algorithm takes a trained neural network $f_{t}$ as input
and then outputs a low-rank plus sparse tensor-compressed neural network
$f_{s}$ by Phase 1: low-rank plus sparse tensor decomposition and Phase 2:
iterative pruning and fine-tuning. This is outlined in Figure 1. The technical
description is given in Algorithm 1.

0: Trained NN $f_{t}$, tensor ranks $\\{\mathbf{r}_{i}\\}_{i=1}^{L}$, sparsity
target $s_{f}$, number of epochs $e$

for $i$ in $[1,2,3,\dots]$ do

Compress layer $i$ weight $\mathbfcal{A}_{i}$ with target rank
$\mathbf{r}_{i}$ using Equation 12 or Equation 14.

end for

for $t$ in $[1,2,3,\dots,e]$ do

Global magnitude pruning on sparse residuals
$\\{\mathbfcal{S}_{l}\\}_{i=1}^{L}$ with target sparsity $s_{t}$ from Equation
7.

Train for one epoch using loss function $\mathcal{L}$ from Equation 8.

end for

Algorithm 1 Low-Rank + Sparse Compression and Fine-Tuning

###  3.1 Reconstruction

We consider two approaches to represent the tensor $\mathbfcal{A}$. We assume
that the reconstruction $\mathbfcal{A}=h(\mathbfcal{L},\mathbfcal{S})$
combines the low-rank and sparse components and is performed before any
forward propagation. Depending on the specific form of $h$, which will be
specified below, direct contraction of the layer activation with
$\mathbfcal{S}$ and $\mathbfcal{L}$ respectively may be more efficient.

#### Additive:

Given weight tensor $\mathbfcal{A}$ we reconstruct

$\mathbfcal{A}=\mathbfcal{L}+\mathbfcal{M}\odot\mathbfcal{S}$ (9)

where $\mathbfcal{L}$ is a low-rank tensor whose weights are maintained in
low-rank format and $\mathbfcal{S}$ is a sparse residual whose weights will be
pruned. The binary mask $\mathbfcal{M}$ controls the sparsity of
$\mathbfcal{S}$. After decomposition, each forward pass of the neural network
will reconstruct the weight
$\mathbfcal{A}\leftarrow\mathbfcal{L}+\mathbfcal{M}\odot\mathbfcal{S}$,
permitting backpropagation directly to the low-rank tensor factors comprising
$\mathbfcal{L}$ and the sparse unpruned entries of $\mathbfcal{S}$. For
example, when the low-rank tensor $\mathbfcal{L}$ is in CP format we store the
factor matrices $\\{\mathbf{U}^{(i)}\\}$ and not the full tensor
$\mathbfcal{L}$.

#### Masking:

We identify a drawback of the reconstruction specified in Equation 9. First,
representational capacity is wasted when the low-rank tensor $\mathbfcal{L}$
and the sparse tensor $\mathbfcal{S}$ both contribute to an entry in
$\mathbfcal{A}$. Each nonzero entry in $\mathbfcal{S}$ has no constraint, and
so the given entry could be represented by $\mathbfcal{S}$ alone. This would
permit the low-rank factors of $\mathbfcal{L}$ to fit other entries in
$\mathbfcal{A}$ where the sparse pruned $\mathbfcal{S}$ is zero and cannot
contribute. Therefore we propose an alternate “masking" reconstruction method
in which we reconstruct $\mathbfcal{A}$ as

$\mathbfcal{A}=(1-\mathbfcal{M})\odot\mathbfcal{L}+\mathbfcal{M}\odot\mathbfcal{S}.$
(10)

Under this reconstruction each entry of $\mathbfcal{A}$ is represented by
either $\mathbfcal{L}$ or $\mathbfcal{S}$, but not both.

###  3.2 Initialization

The sparse weight $\mathbfcal{S}$ will be iteratively pruned. Therefore our
goal is to capture any possible coarse-grained structure in $\mathbfcal{A}$
using $\mathbfcal{L}$, and retain fine structure in $\mathbfcal{S}$ that will
be well-suited to sparse pruning. We describe two methods of decomposition.
The tensor $\mathbfcal{A}$ may be reshaped before performing factorization.

#### Residual:

A simple approach is to obtain $\mathbfcal{L}$ by minimizing the residual to
$\mathbfcal{A}$. We focus on the Tensor-Train format for notational
convenience and use

$\mathbfcal{L}=\llbracket\mathbfcal{G}^{(1)},\dots,\mathbfcal{G}^{(d)}\rrbracket$
(11)

to represent the reconstruction of the low-rank TT-format tensor
$\mathbfcal{L}$ from its factors $\\{\mathbfcal{G}^{(i)}\\}_{i=1}^{d}$.

$\begin{split}\mathbfcal{G}^{(1)},\dots,\mathbfcal{G}^{(d)}&=\operatorname*{arg\,min}_{\mathbfcal{G}^{(1)},\dots,\mathbfcal{G}^{(d)}}\left\|\llbracket\mathbfcal{G}^{(1)},\dots,\mathbfcal{G}^{(d)}\rrbracket-\mathbfcal{A}\right\|_{2}\\\
\mathbfcal{L}&=\llbracket\mathbfcal{G}^{(1)},\dots,\mathbfcal{G}^{(d)}\rrbracket\\\
\mathbfcal{S}&=\mathbfcal{A}-\mathbfcal{L}.\end{split}$ (12)

We obtain an approximate solution to the minimization problem through standard
decomposition methods implemented in Tensorly (Kossaifi et al., 2016). For the
TT and Tucker formats the decomposition to obtain the factors of
$\mathbfcal{L}$ requires $d-1$ tensor unfoldings and singular value
decompositions. The CP format uses alternating least squares (ALS) iterations.
In general, the decomposition time for the networks we consider requires less
than five minutes and is inconsequential compared to the resources used during
the fine-tuning phase which takes approximately 12hrs.

The decomposition given in Equation 12 provides an exact reconstruction of the
the original tensor $\mathbfcal{A}$ before the sparse component is pruned at
the first iteration. We note that this naive decomposition has one clear
weakness: it does not take into account the fact that many entries of
$\mathbfcal{S}$ will be pruned. A natural approach would be to consider robust
PCA or other decomposition methods that explicitly target a low-rank plus
sparse decomposition. We attempted this approach but did not observe any clear
gains. We use the residual initialization approach with the additive approach.

#### Masking:

In order to directly encourage the low-rank factors to capture only the low-
magnitude entries (which will be pruned) we consider an alternate
decomposition. We mask the top-$K\%$ of entries using the operator
$P_{\Omega}$:

$P_{\Omega}(\mathbfcal{A})_{i_{1}\dots i_{d}}=\begin{cases}0\hskip
21.68121pt\text{if }\hskip 7.22743pta_{ijk}\in TopK(\mathbfcal{A})\\\
a_{ijk}\hskip 7.22743pt\text{ otherwise}\end{cases}$ (13)

Then we obtain our low-rank decomposition
$\left(\mathbfcal{S},\mathbfcal{L}\right)$ by minimizing the reconstruction
loss and considering only the bottom $K\%$ of entries in $\mathbfcal{A}$.

$\begin{split}\mathbfcal{G}^{(1)},\dots,\mathbfcal{G}^{(d)}&=\operatorname*{arg\,min}_{\mathbfcal{G}^{(1)},\dots,\mathbfcal{G}^{(d)}}\left\|P_{\Omega}\left(\llbracket\mathbfcal{G}^{(1)},\dots,\mathbfcal{G}^{(d)}\rrbracket-\mathbfcal{A}\right)\right\|_{2}\\\
\mathbfcal{L}&=\llbracket\mathbfcal{G}^{(1)},\dots,\mathbfcal{G}^{(d)}\rrbracket\\\
\mathbfcal{S}&=\mathbfcal{A}-\mathbfcal{L}.\end{split}$ (14)

This decomposition differs from the one obtained in Equation 12 in that the
tensor decomposition does not fit any of the high-magnitude entries which will
remain unpruned. We use this initialization approach with the “masking"
reconstruction so that the initialization ignores high-magnitude entries that
will remain un-pruned.

##  4 Experiments

In this section we report results obtained on ImageNet-1K. All results are
obtained by training on a machine with 32 CPU cores, 8 Tesla V100 GPUs, and
128GB memory. We train each method for 50 epochs using a batch size of 256 and
distillation settings $\alpha=0.9,T=3$. Each run takes approximately 12hrs.
All reported ImageNet accuracy metrics are validation accuracy. In all
experiments we set the tensor rank so that the low-rank factors required to
represent $\mathbfcal{L}$ use $10\%$ as many elements as are in
$\mathbfcal{A}$.

###  4.1 Reconstruction and Initialization

First we compare two variants of our method: additive reconstruction with
residual initialization and masking reconstruction with masking
initialization. For this experiment, and all other MobileNet experiments, we
use SGD with momentum $0.9$ base learning rate $0.1$ and a decay of $0.7$
applied every $5$ epochs. The decay rate was selected using a small grid
search over $\\{0.5,0.7,0.9\\}$ and selecting the best ImageNet-1K validation
accuracy for the sparse pruned MobileNet-v3-Large baseline. In Figure 3 we
compare the results of our residual initialization strategy with our masking
initialization strategy. We observe no clear gains from either strategy. Based
on these results we use the simpler residual initialization strategy with
additive reconstruction for the remainder of our experiments.

(a)

(b)

Figure 3: Comparison of initialization strategies for low-rank+sparse tensor
decomposition for the (a) MobileNet-v3-Small and (b) MobileNet-v3-Large
architectures. There are no major distinctions between initialization
strategies.

###  4.2 MobileNetv3 Results

(a)

(b)

(c)

(d)

Figure 4: ImageNet top-5 accuracy comparison between our LR+S method with
CP/TT/Tucker tensor format and the sparse pruning baseline. Results for (a)
Mobilenet-v3-Small (b) MobileNet-v3-Large (c) EfficientNet-B0 (d) DEIT-Small.

In Figures 4(a) and 4(b) we plot a comparison between our method with varying
tensor formats (CP,Tucker,TT) and the sparse pruning baseline on
MobileNet-v3-Small and MobileNet-v3-Large respectively. We observe that the
combination of low-rank and sparse representations is competitive with the
sparse pruning baseline. However we do not observe clear performance
advantages.

###  4.3 Other Models: EfficientNet and DEIT

To test if our conclusions hold across architectural families we test on
EfficientNet-B0 (Tan & Le, 2019). We use the same optimizer settings as the
MobileNet experiments, and report our results in Figure 4(c). We also test our
results on DEIT (Touvron et al., 2021) which is a highly accurate Vision
Transformer (Dosovitskiy et al., 2020). Specifically, we the the DEIT-Small
model with resolution 224 and patch size 16. We train with the Adam optimizer
with learning rate $1e-4$. In Figures 4(c) and 4(d) we observe similar results
as those from the MobileNet family of models. Low-rank plus sparse
decomposition is competitive with sparse pruning across models, but does not
significantly outperform sparse pruning.

###  4.4 Effect on Model Weights

In this section we investigate the effects of our compression methods on the
model weights. We highlight a key challenge: many pre-trained models do not
possess weights that are close to low-rank. In particular, models with a large
number of FC layers or 1x1 convolutions are not “approximately low-rank" even
after reshaping from matrix to tensor format. In Figure 5 we plot the layer-
wise relative errors of the low-rank decomposition, computed as

$Err=\frac{\|\mathbfcal{A}-\mathbfcal{L}\|_{2}}{\|\mathbfcal{A}\|_{2}}$ (15)

We plot the result of CP-format decomposition applied to the MobilNetv3-small
and EfficientNet-B0 models. We observe that many layers have a high relative
error, but also that several layers stand out in that the low-rank tensor
component fits the original tensor $\mathbfcal{A}$ almost exactly.

(a)

(b)

Figure 5: Layer-wise relative error of the low-rank tensor-train factorization
for each layer of (a) Mobilenet-v3-Small and (b) EfficientNet-B0. Other tensor
types are similar. Layers with high relative error are the pointwise
convolutions.

(a) MobileNetv3-Small spatial convolution before and after removing low-rank
tensor factors.

(b) Effect of tensor factorization on MobileNetv3-Small spatial convolution
weights.

(c) MobileNetv3-Small pointwise convolution before and after removing low-rank
tensor factors.

(d) Effect of tensor factorization on MobileNetv3-Small pointwise convolution
weights.

Figure 6: (a) Spatial convolutions possess regular structure that can be
captured by low-rank tensor factorization. (b) Tensor factorization on spatial
convolutions introduces overhead but leads to a weight distribution (in
orange) that is easier to prune. (c) Point-wise convolutions posses less
regular structure, and tensor factorization induces fewer 0 entries. (d)
Tensor factorization on pointwise convolutions introduces overhead and leads
to a weight distribution (in orange) that is not easier to prune.

We found that many, but not all, spatial convolutions were fit well by tensor
decomposition and have low relative error. We provide an example in Figure 6
which gives representative examples of pointwise and spatial convolutions in
MobileNetv3-Small. We detail the effect of our proposed compression method on
the model weights. Our motivation is to “preprocess" the weights by removing
low-rank structure so that the remaining weight values are better-suited to
sparse pruning. Therefore we investigate the effects of our decomposition
approach on the weight distributions. We show how tensor compression can
remove repeated coarse structure present in spatial convolutions, resulting in
a sharply peaked weight distribution compared to the weights of the pretrained
model. In contrast tensor decomposition captures less structure in the
pointwise convolution and does not induce a weight distribution that is
better-suited to sparse pruning.

###  4.5 Low-Rank Only

We also considered compressing models using only post-training tensor
decomposition. We take a model and perform tensor decomposition on all
convolutional kernels and FC layers except the last layer. This idea was first
studied in Lebedev et al. (2014) on larger, more compressible architectures.
We report our results in Figure 7. We followed the same experimental setup as
before, and observed that it was difficult to achieve similar accuracy with
low-rank decomposition alone. We hypothesize that this is the natural result
of the observation from Figure 5. Since the model weights are not
“aproximately low-rank" it is much more challenging to decompose a model and
then fine-tune. It is possible that a longer training schedule (>>50 epochs),
or training from scratch in low-rank tensor format, would resolve this issue.

(a)

(b)

Figure 7: ImageNet top-5 accuracy comparison between sparse pruning and low-
rank tensor decomposition on (a) MobileNet-v3-Small (b) MobileNet-v3-Large

## References

  * Blalock et al. (2020) Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag.  What is the state of neural network pruning?  _arXiv preprint arXiv:2003.03033_ , 2020.
  * Carroll & Chang (1970) J Douglas Carroll and Jih-Jie Chang.  Analysis of individual differences in multidimensional scaling via an N-way generalization of “eckart-young” decomposition.  _Psychometrika_ , 35(3):283–319, 1970.
  * Cho & Hariharan (2019) Jang Hyun Cho and Bharath Hariharan.  On the efficacy of knowledge distillation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 4794–4802, 2019.
  * Deng et al. (2019) Chunhua Deng, Fangxuan Sun, Xuehai Qian, Jun Lin, Zhongfeng Wang, and Bo Yuan.  TIE: energy-efficient tensor train-based inference engine for deep neural network.  In _ISCA_ , pp. 264–278, 2019.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Garipov et al. (2016) Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov.  Ultimate tensorization: compressing convolutional and FC layers alike.  _arXiv:1611.03214_ , 2016.
  * Gu et al. (2021) Jiaqi Gu, Hanqing Zhu, Chenghao Feng, Mingjie Liu, Zixuan Jiang, Ray T Chen, and David Z Pan.  Towards memory-efficient neural networks via multi-level in situ generation.  _arXiv preprint arXiv:2108.11430_ , 2021.
  * Gui et al. (2019) Shupeng Gui, Haotao N Wang, Haichuan Yang, Chen Yu, Zhangyang Wang, and Ji Liu.  Model compression with adversarial robustness: A unified optimization framework.  _Advances in Neural Information Processing Systems_ , 32:1285–1296, 2019.
  * Harshman et al. (1994) Richard A Harshman, Margaret E Lundy, et al.  PARAFAC: Parallel factor analysis.  _Computational Statistics and Data Analysis_ , 18(1):39–72, 1994.
  * Hawkins & Zhang (2019) Cole Hawkins and Zheng Zhang.  Bayesian tensorized neural networks with automatic rank selection.  _arXiv:1905.10478_ , 2019.
  * Hawkins et al. (2020) Cole Hawkins, Xing Liu, and Zheng Zhang.  Towards compact neural networks via end-to-end training: A bayesian tensor approach with automatic rank determination.  _arXiv preprint arXiv:2010.08689_ , 2020.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv:1503.02531_ , 2015.
  * Howard et al. (2019) Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 1314–1324, 2019.
  * Howard et al. (2017) Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ , 2017.
  * Kossaifi et al. (2016) Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic.  Tensorly: Tensor learning in python.  _arXiv preprint arXiv:1610.09555_ , 2016.
  * Lebedev et al. (2014) Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky.  Speeding-up convolutional neural networks using fine-tuned cp-decomposition.  _arXiv:1412.6553_ , 2014.
  * LeCun et al. (1990) Yann LeCun, John S Denker, and Sara A Solla.  Optimal brain damage.  In _NIPS_ , pp. 598–605, 1990.
  * Novikov et al. (2015) Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov.  Tensorizing neural networks.  In _NIPS_ , pp. 442–450, 2015.
  * Oseledets (2011) Ivan V Oseledets.  Tensor-train decomposition.  _SIAM J. Sci. Computing_ , 33(5):2295–2317, 2011\.
  * Sandler et al. (2018) Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4510–4520, 2018.
  * Sze et al. (2017) Vivienne Sze, Yu-Hsin Chen, Joel Emer, Amr Suleiman, and Zhengdong Zhang.  Hardware for machine learning: Challenges and opportunities.  In _IEEE Custom Integrated Circuits Conference_ , pp. 1–8, 2017\.
  * Tan & Le (2019) Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International Conference on Machine Learning_ , pp. 6105–6114. PMLR, 2019.
  * Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International Conference on Machine Learning_ , pp. 10347–10357. PMLR, 2021.
  * Tucker (1966) Ledyard R Tucker.  Some mathematical notes on three-mode factor analysis.  _Psychometrika_ , 31(3):279–311, 1966.
  * Wang et al. (2021) Dilin Wang, Yuan Shangguan, Haichuan Yang, Pierce Chuang, Jiatong Zhou, Meng Li, Ganesh Venkatesh, Ozlem Kalinli, and Vikas Chandra.  Noisy training improves e2e asr for the edge.  _arXiv preprint arXiv:2107.04677_ , 2021.
  * Yang et al. (2020) Lei Yang, Zheyu Yan, Meng Li, Hyoukjun Kwon, Liangzhen Lai, Tushar Krishna, Vikas Chandra, Weiwen Jiang, and Yiyu Shi.  Co-exploration of neural architectures and heterogeneous asic accelerator designs targeting multiple tasks.  In _2020 57th ACM/IEEE Design Automation Conference (DAC)_ , pp. 1–6. IEEE, 2020.
  * Yu et al. (2017) Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao.  On compressing deep models by low rank and sparse decomposition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 7370–7379, 2017.
  * Zhang et al. (May 2021) Kaiqi Zhang, Cole Hawkins, Xiyuan Zhang, Cong Hao, and Zheng Zhang.  On-fpga training with ultra memory reduction: A low-precision tensor method.  _ICLR Workshop of Hardware Aware Efficient Training_ , May 2021.
  * Zhu & Gupta (2017) Michael Zhu and Suyog Gupta.  To prune, or not to prune: exploring the efficacy of pruning for model compression.  _arXiv preprint arXiv:1710.01878_ , 2017.


# Data Augmentation for Intent Classification with Off-the-shelf Large
Language Models

Gaurav Sahu
University of Waterloo
<EMAIL_ADDRESS>
&Pau Rodriguez
ServiceNow Research
&Issam H. Laradji
ServiceNow Research
Parmida Atighehchian
ServiceNow Research
&David Vazquez
ServiceNow Research
&Dzmitry Bahdanau
ServiceNow Research
Work done during an internship at ServiceNow Research

###### Abstract

Data augmentation is a widely employed technique to alleviate the problem of
data scarcity. In this work, we propose a prompting-based approach to generate
labelled training data for intent classification with off-the-shelf language
models (LMs) such as GPT-3. An advantage of this method is that no task-
specific LM-fine-tuning for data generation is required; hence the method
requires no hyper-parameter tuning and is applicable even when the available
training data is very scarce. We evaluate the proposed method in a few-shot
setting on four diverse intent classification tasks. We find that GPT-
generated data significantly boosts the performance of intent classifiers when
intents in consideration are sufficiently distinct from each other. In tasks
with semantically close intents, we observe that the generated data is less
helpful. Our analysis shows that this is because GPT often generates
utterances that belong to a closely-related intent instead of the desired one.
We present preliminary evidence that a prompting-based GPT classifier could be
helpful in filtering the generated data to enhance its quality.111Our code is
available at: https://github.com/ElementAI/data-augmentation-with-llms

##  1 Introduction

A key challenge in creating task-oriented conversational agents is gathering
and labelling training data. Standard data gathering options include manual
authoring and crowd-sourcing. Unfortunately, both of these options are tedious
and expensive. Data augmentation is a widely used strategy to alleviate this
problem of data acquisition.

Figure 1: Generation Process. Given a seed intent (here, music_likeness) and
$K$(=10) available examples for that intent, we construct a prompt following
the shown template. Note that the last line of the prompt is incomplete (there
is no new line character.) We then feed this prompt to a GPT-3 engine, which
generates some completions of the prompt. In this example,  red text denotes
unfaithful examples and  blue text denotes faithful examples. Note: For
brevity, we only show ten generated sentences.

There are two particularly promising paradigms for data augmentation in
natural language processing that use pretrained language models (LMs) (Peters
et al., 2018; Devlin et al., 2018). The first family of methods fine-tunes an
LM on task-specific data and generates new examples using the fine-tuned LM
(Wu et al., 2018; Kumar et al., 2019, 2021; Anaby-Tavor et al., 2020; Lee et
al., 2021). A limitation of these methods is that, in a real-world scenario,
task-specific data is scarce and fine-tuning an LM can quickly become the
bottleneck. The second family of methods sidesteps this bottleneck by
employing off-the-shelf pretrained LMs such as GPT-3 (Brown et al., 2020) to
directly generate text without any task-specific fine-tuning. In particular,
data generation by the GPT3Mix approach by Yoo et al. (2021) boosts
performance on multiple classification tasks; however, they only consider
tasks with few (up to 6) classes and easy-to-grasp class boundaries (e.g.,
positive and negative).

This work studies the applicability of massive off-the-shelf LMs, such as
GPT-3 and GPT-J Wang and Komatsuzaki (2021) to perform effective data
augmentation for intent classification (IC) tasks. In IC, the end goal is to
predict a user’s intent given an utterance, i.e., what the user of a task-
oriented chatbot wants to accomplish. Data augmentation for IC is particularly
challenging because the generative model must distinguish between a large
number (in practice up to several hundreds) of fine-grained intents that can
be semantically very close to each other. Prior methods such as GPT3Mix prompt
the model with the names of all classes as well as a few examples from
randomly chosen classes. We test GPT3Mix for one and observe that such
approaches are poorly suitable for intent classification tasks with tens or
hundreds of possible intents. Instead, in this study, we use a simple prompt
structure that focuses on a single seed intent (see Figure 1) as it combines
the intent’s name and all available examples.

Our experiments primarily focus on few-shot IC on four prominent datasets:
CLINC150 (Larson et al., 2019), HWU64 (Xingkun Liu and Rieser, 2019),
Banking77 (Casanueva et al., 2020), and SNIPS (Coucke et al., 2018). We also
consider a partial few-shot setup to compare to the Example Extrapolation
(Ex2) approach by Lee et al. (2021) who use a similar prompt but fine-tune the
LM instead of using it as is. The main findings of our experiments are as
follows: (1) GPT-generated samples boost classification accuracy when the
considered intents are well-distinguished from each other (like in CLINC150,
SNIPS). (2) On more granular datasets (namely HWU64 and Banking77), we find
that GPT struggles in distinguishing between different confounding intents.
(3) A small-scale study to further understand this behaviour suggests that GPT
could be used as a classifier to filter out unfaithful examples and enhance
the quality of the generated training set. Additionally, we investigate how
valuable the generated data could be if relabelled by a human. Using an oracle
model, we show that (4) the human labelling of GPT-generated examples can
further improve the performance of intent classifiers, and that (5) LM-
generated data has a higher relabelling potential compared to edit-based
augmentation techniques, such as Easy Data Augmentation (EDA) (Wei and Zou,
2019).

##  2 Method

We consider training an intent classifier, where an intent is a type of
request that the conversational agent supports; e.g. the user may want to
change the language of the conversation, play a song, transfer money between
accounts, etc. However, collecting many example utterances that express the
same intent is difficult and expensive. Therefore, this paper experiments with
a straightforward method to augment the training data available for an intent:
creating prompts from the available examples and feeding them to a large
language model such as GPT-3 (Brown et al., 2020). Figure 1 illustrates the
process of data generation for an intent with $K$ available examples.

##  3 Experimental Setup

###  3.1 Datasets

We use four intent classification datasets in our experiments with varying
levels of granularity among intents. CLINC150 (Larson et al., 2019), HWU64
Xingkun Liu and Rieser (2019) are multi-domain datasets, each covering a wide
range of typical task-oriented chatbot domains, such as playing music and
setting up alarms. Importantly, the CLINC150 task also contains examples of
out-of-scope (OOS) utterances that do not correspond to any of CLINC’s 150
intents. Banking77 Casanueva et al. (2020) is a single domain dataset with
very fine-grained banking-related intents. Finally, the SNIPS (Coucke et al.,
2018) dataset contains 7 intents typical for the smart speaker usecase. We
refer the reader to Table 1 for exact statistics of all used datasets.

| CLINC150 | SNIPS | HWU64 | Banking77
---|---|---|---|---
domains | 10 | 1 | 18 | 1
intents | 150 | 7 | 64 | 77
train examples | 15000 | 13084 | 8954* | 9002*
(100) |  |  |
val. examples | 3000 | 700 | 1076* | 1001*
(100) |  |  |
test examples | 4500 | 700 | 1076 | 3080
(1000) |  |  |
Table 1: Statistics of the intent classification datasets that we use in our
experiments. * indicates that we split the original data into training and
validation instead of using a split provided by the dataset authors. For
CLINC150, the number of out-of-scope examples in different data partitions is
given in parenthesis.

###  3.2 Setup

The main data-scarce setup that we consider in this work is the few-shot
setup, where only $K=10$ training examples are available for every intent of
interest. Additionally, to compare to example extrapolation with fine-tuned
language models as proposed by Lee et al. (2021), we consider a partial few-
shot setup. In the latter setting, we limit the amount of training data only
for a handful of few-shot intents222We use the truncation heuristic provided
by Lee et al. (2021):
https://github.com/google/example_extrapolation/blob/master/preprocess_clinc150.py
and use the full training data for others. When data augmentation is
performed, we augment the few-shot intents to have $N$ examples, where $N$ is
the median number of examples per intent of the original data.

To precisely describe the training and test data in all settings, we will use
$D_{part}$ to refer to dataset parts, i.e. train, validation, and test. In
addition, we use $D_{F}$ and $D_{M}$ to refer to data-scarce and data-rich
intents (the latter only occur in the partial few-shot setting). This notation
is defined for all parts, therefore, $D_{part}=D_{\\{F,part\\}}\cup
D_{\\{M,part\\}}$, $\forall\,part\in\\{train,val,test\\}$. When GPT-3 or a
baseline method is used to augment the training data we generate $N-K$
examples per intent and refer to the resulting data as $\tilde{D}_{F,train}$.
We experiment with four different-sized GPT-3
models333https://beta.openai.com/docs/engines by OpenAI and GPT-J by
EleutherAI444https://github.com/kingoflolz/mesh-transformer-jax/ to obtain
$\tilde{D}$. The four GPT-3 models are: Ada, Babbage, Curie, and Davinci. In
order, Ada is the smallest model and Davinci is the largest. Model sizes of
GPT-3 engines are not known precisely but are estimated by Eleuther AI to be
between 300M and 175B parameters555https://blog.eleuther.ai/gpt3-model-sizes/.

###  3.3 Training and Evaluation

We fine-tune BERT-large (Devlin et al., 2018) on the task of intent
classification by adding a linear layer on top of the [CLS] token (Wolf et
al., 2019). In all setups we use the original validation set for tuning the
classifier’s training hyperparameters. We chose to use the full validation set
as opposed to a few-shot one to avoid issues with unstable hyperparameter
tuning and focus on assessing the quality of the generated data.

#### Full few-shot.

In this setup, we treat all the intents as few-shot and evaluate our method on
the following three scenarios: (i) Baseline: all the intents are truncated to
$K=10$ samples per intent, (ii) Augmented: $\tilde{D}_{\\{F,train\\}}$ is
generated using GPT and models are trained on
$D_{\\{F,train\\}}\cup\tilde{D}_{\\{F,train\\}}$ and (iii) EDA-baseline: same
as above, but $\tilde{D}_{\\{F,train\\}}$ is generated using Easy Data
Augmentation (EDA) by Wei and Zou (2019). For each scenario, we report the 1)
overall in-scope accuracy on the complete test set $D_{test}$, i.e. intent
classification accuracy excluding OOS samples in the test set, and 2) few-shot
classification accuracy of the models on $D_{\\{F,test\\}}$. For CLINC150, we
also report out-of-scope recall (OOS recall) on $D_{test}$ that we compute as
the percentage of OOS examples that the model correctly labelled as such.

The purpose of this setting is to estimate what further gains can be achieved
if the data generated by GPT were labelled by a human. We train an oracle
$\mathcal{O}$ on the full training data $D_{train}$. We also use $\mathcal{O}$
to assess the quality of the generated data. Namely, we compute fidelity of
the generated data as the ratio of generated utterances that the oracle labels
as indeed belonging to the intended seed intent. A higher fidelity value means
that the generated samples are more faithful to original data distribution.

#### Partial few-shot.

In this setup, we train $\mathcal{S}$ intent classifiers, choosing different
few-shot intents every time to obtain $D_{F}$. We then average the metrics
across these $\mathcal{S}$ runs. For CLINC150, $\mathcal{S}=10$ corresponding
to the 10 different domains, whereas for SNIPS, $\mathcal{S}=7$ corresponding
to the 7 different intents. We evaluate our method on the following three
scenarios introduced by Lee et al. (2021): (i) Baseline: models are trained
without data augmentation on $D_{\\{F,train\\}}\cup D_{\\{M,train\\}}$. (ii)
Upsampled: $D_{\\{F,train\\}}$ is upsampled to have $N$ examples per intent.
Then models are trained on upsampled $D_{\\{F,train\\}}\cup
D_{\\{M,train\\}}$. (iii) Augmented: models are trained on
$D_{\\{F,train\\}}\cup\tilde{D}_{\\{F,train\\}}\cup D_{\\{M,train\\}}$. For
each scenario in this setup, we report the overall in-scope classification
accuracy (and OOS Recall for CLINC150).

For both partial few-shot and full few-shot settings, we report means and
standard deviations over 10 repetitions of each experiment.

##  4 Experimental Results

| CLINC150 | HWU64 | Banking77 | SNIPS
---|---|---|---|---
Model | IA (96.93) | OR (42.9) | IA (92.75) | IA (93.73) | IA (98.57)
EDA | 92.66 (0.40) | 43.81 (2.03) | 83.67 (0.48) | 83.96 (0.66) | 92.50 (1.61)
Baseline (Ours) | 90.28(0.49) | 50.18(1.14) | 81.43 (0.57) | 83.35 (0.59) | 89.69 (1.63)
Augmented
Ada (Ours) | 91.31 (0.34) | 21.69 (1.57) | 79.68 (0.83) | 79.30 (0.42) | 94.27 (0.52)
Babbage (Ours) | 92.72 (0.33) | 22.99 (2.39) | 81.86 (0.78) | 80.31 (0.41) | 94.74 (0.67)
Curie (Ours) | 93.37 (0.21) | 25.85 (1.49) | 82.85 (0.70) | 83.50 (0.44) | 94.73 (0.62)
GPT-J (Ours) | 93.25 (0.19) | 24.02 (1.45) | 81.78 (0.56) | 82.32 (0.90) | 95.19 (0.61)
Davinci (Ours) | 94.07 (0.18) | 27.36 (1.08) | 82.79 (0.93) | 83.60 (0.45) | 95.77 (0.86)
Augmented + Relabelled
EDA | 93.43 (0.22) | 48.56 (1.84) | 85.58 (0.73) | 84.82 (0.57) | 94.91 (0.66)
Ada (Ours) | 95.09 (0.16) | 41.38 (1.77) | 88.53 (0.61) | 88.45 (0.19) | 97.03 (0.18)
Babbage (Ours) | 95.39 (0.17) | 40.58 (1.63) | 89.49 (0.32) | 88.86 (0.26) | 96.89 (0.49)
Curie (Ours) | 95.08 (0.19) | 40.09 (2.38) | 89.78 (0.47) | 88.30 (4.64) | 96.86 (0.31)
GPT-J (Ours) | 95.11 (0.13) | 43.94 (1.76) | 89.52 (0.54) | 88.94 (0.40) | 97.33 (0.38)
Davinci (Ours) | 95.08 (0.13) | 40.76 (1.37) | 89.53 (0.45) | 88.89 (0.31) | 97.03 (0.38)
Table 2: Full few-shot results on CLINC150, HWU64, Banking77, and SNIPS
datasets. IA: Inscope Accuracy (mean (std)). OR: OOS-Recall (mean (std)).
Towards the top of the table, we also report the test set performance
(enclosed in parentheses) when all examples are used for fine-tuning (without
any augmentation.)

|  | CLINC150 | SNIPS
---|---|---|---
|  | Overall | Few-shot | Overall | Few-shot
| Classifier | IA | OR | A | IA | A
Baseline♠ | T5 | 97.4 | - | 93.7 | 95.2 | 74.0
Upsampled♠ | T5 | 97.4 | - | 94.4 | 95.9 | 80.0
Augmented (Ex2)♠ | T5 | 97.4 | - | 95.6 | 97.8 | 94.0
Baseline (ours) | BERT | 96.28 (0.06) | 39.14 (0.82) | 91.36 (0.47) | 95.47 (0.45) | 78.38 (3.34)
Upsample (ours) | BERT | 96.20 (0.05) | 40.21 (0.59) | 90.93 (0.19) | 95.29 (0.37) | 79.28 (2.05)
Augmented (Ada) | BERT | 96.16 (0.05) | 34.37 (0.27) | 92.60 (0.15) | 97.30 (0.24) | 94.41 (0.72)
Augmented (Babbage) | BERT | 96.39 (0.06) | 35.71 (0.46) | 93.66 (0.21) | 97.46 (0.25) | 95.31 (0.74)
Augmented (Curie) | BERT | 96.41 (0.06) | 36.77 (0.93) | 93.90 (0.21) | 97.37 (0.19) | 94.79 (0.64)
Augmented (GPT-J) | BERT | 96.38 (0.05) | 35.91 (0.94) | 93.85 (0.25) | 97.59 (0.21) | 96.08 (0.39)
Augmented (Davinci) | BERT | 96.45 (0.03) | 37.52 (0.54) | 94.28 (0.24) | 97.66 (0.21) | 96.52 (0.35)

Table 3: Partial few-shot results on CLINC150 and SNIPS datasets. Refer to
Section 3.3 for more details. IA: Inscope accuracy (mean (std)). OR: OOS
Recall (mean (std)). A: Accuracy (mean (std)). ♠ Lee et al. (2021).

(a) Temperature v/s Few-shot accuracy

(b) Temperature v/s OOS recall

(c) Temperature v/s Fidelity

Figure 2: Partial few-shot validation performance for different GPT-3 models
and temperatures. (a) few-shot accuracy, (b) OOS recall of intent classifiers
trained on augmented sets, and (c) fidelity measured as the accuracy of the
oracle on the augmented sets.

(a) Temperature v/s Few-shot inscope accuracy

(b) Temperature v/s Fidelity

Figure 3: Full few-shot validation performance for different GPT-J
temperatures on different datasets. (a) few-shot inscope accuracy of intent
classifiers trained on augmented sets, and (b) fidelity (oracle accuracy) of
augmented sets generated by GPT-J with different temperatures.

#### Full few-shot.

Table 2 shows the results of our few-shot experiments. For CLINC150 and SNIPS,
data augmentation with GPT-3 is very effective as it leads to respective
accuracy improvements of up to approximately 3.7% and 6% on these tasks. These
improvements are larger than what the baseline EDA method brings, namely 2.4%
and 2.9% for CLINC150 and SNIPS. Importantly, using larger GPT models for data
augmentation brings significantly bigger gains. Data augmentation results on
Banking77 and HWU64 are, however, much worse, with no or little improvement
upon the plain few-shot baseline. We present a thorough investigation of this
behaviour in Section 4.1. One can also see that data augmentation with GPT
models lowers the OOS recall.

Next, we observe that relabelling EDA and GPT-generated sentences by the
oracle gives a significant boost to accuracies across the board, confirming
our hypothesis that human inspection of generated data could be fruitful.
Importantly, we note that the magnitude of improvement for EDA is less than
for GPT models. This suggests that GPT models generate more diverse data that
can eventually be more useful after careful human inspection. Lastly,
relabelling also improves OOS recall on CLINC150, which is due to the fact
that much of the generated data was labelled as OOS by the oracle.

#### Partial few-shot.

Table 3 shows the results of our partial few-shot experiments on CLINC150 and
SNIPS. By augmenting the dataset with GPT-generated samples, the few-shot
accuracy improves by up to 2.92% on CLINC150 and 18.14% on SNIPS compared to
the baseline setting. Our method achieves competitive results compared to Ex2
(Lee et al., 2021), both in terms of absolute accuracies and the relative
gains brought by data augmentation. Note that Ex2 uses T5-XL Roberts et al.
(2020) with nearly 3 billion parameters as its base intent classifier, while
our method uses BERT-large with only 340 million parameters.

Davinci generated sentences | Seed Intent | Oracle Prediction
---|---|---
HWU64
play a song with the word honey | music_likeness | play_music
you are playing music | music_likeness | play_music
‘let me hear some of that jazz!’ | music_likeness | play_music
i really like myspace music | play_music | music_likeness
i love the start lucky country music | play_music | music_likeness
thank you for the music | play_music | music_likeness
please play the next song | music_settings | play_music
play background music | music_settings | play_music
play the hour long loop of rock song | music_settings | play_music
need you to play that song one more time | play_music | music_settings
skip that song, its turkish | play_music | music_settings
pickup the beat or a temp track or audio plugin | play_music | music_settings
Banking77
My last attempt to top up didn’t seem to work, any success? | topping_up_by_card | top_up_failed
I tried to top off my wallet using my card but it says “top up failed". | topping_up_by_card | top_up_failed
I cannot top-up by my cellular phone number? How do I do that? | topping_up_by_card | top_up_failed
Can you transfer money to my Ola prepaid option? Or help me top up my card to money. They never accept my card so I always have to suffer | top_up_failed | topping_up_by_card
Hi my app is activated on activate.co.in, but unable to top up my phone. I tried credit card, debit card and Paytm but fails | top_up_failed | topping_up_by_card
I try to top up my card but it’s not going through. It’s still on pending status. Do I need to wait or did I do something wrong | top_up_failed | pending_top_up
I tried top-up with my card but notification shows that ‘Pending’. This has been happening since last night. Can you tell me what’s going on | top_up_failed | pending_top_up
Top up didn’t go through. | pending_top_up | top_up_failed
Did my master card top-up fail? | pending_top_up | top_up_failed

Table 4: Davinci-generated sentences for closely-related intents in HWU64 and Banking77 datasets. Highlighted sub-strings indicate a difference with respect to the seed intent. Fidelity (3 intents) | HWU64 | Banking77
---|---|---
w/o filtering (468) | 60.26 | 57.31
w/ filtering (371) | 72.51 | 65.54
3-way accuracy |  |
Davinci | 86.36 | 78.75
10-shot BERT-large | 82.95 | 65.54
Full data BERT-large | 94.32 | 95.00
Table 5: The impact and the accuracy of using GPT-3 as a 3-way classifier on
close intent triplets from HWU64 and Banking77 datasets. For fidelity,
generated examples are rejected if the GPT-3 classifier labels them as not
belonging to the seed intent. Classification accuracies are reported on the
reduced validation+test sets where we only consider examples from the three
confounding intents. Figure 4: Distribution of labels as predicted by the
oracle for lowest-fidelity intents in Banking77 and HWU64 datasets
(“topping_up_by_card" and “music_likeness," respectively). Green areas denote
the portion of generated sentences deemed fit by the oracle for the lowest-
fidelity intents in the two datasets. Red and Blue areas respectively
correspond to the most common and the second most common alternative intent
predicted by the oracle.

###  4.1 Analysis

Effect of GPT sampling temperature. We investigate the impact of generation
temperature on the quality and fidelity of generated data. We perform this
investigation on the CLINC150 dataset using the partial few-shot setup.
Results in Figure 2 show that, for all engines, the generated data leads to
the highest classification accuracy when the generation temperature is around
1.0, although lower temperatures result in higher OOS recall. We also observe
that the fidelity of the generated samples decreases as we increase the
temperature (i.e. higher diversity, see Figure 2(c)). This suggests that
higher fidelity does not always imply better quality samples as the language
model may simply copy or produce less diverse utterances at lower
temperatures. In Appendix A, we perform a human evaluation, reaching similar
conclusions as when using an oracle to approximate fidelity.

Fidelity on different datasets. Our results in Section 4 show that data
augmentation gains are much higher on CLINC150 and SNIPS than on HWU64 and
Banking77. To contextualize these results, we report the fidelity of GPT-J-
generated data for all these tasks in Figure 3(b). Across all generation
temperatures, the fidelity of the generated data is higher for CLINC150 and
SNIPS than for HWU64 and Banking77. For all datasets, the fidelity is higher
when the generation temperature is lower; however, Figure 3(a) shows that low-
temperature data also does improve the model’s performance.

#### Data generation for close intents.

To better understand the lower fidelity and accuracy on HWU64 and Banking77
datasets, we focus on intents with the lowest fidelities. Here, by intent
fidelity, we mean the percentage of the intent’s generated data that the
oracle classified as indeed belonging to the seed intent. In the Banking77
dataset, the lowest-fidelity intent is “topping_up_by_card.” For this intent,
only 33% of the Davinci-generated sentences were labelled as
“topping_up_by_card” by the oracle, implying that two-thirds of the sentences
did not fit that intent, “top_up_failed" and “top_up_card_charge" being the
two most common alternatives chosen by the oracle. Similarly, only 50% of the
Davinci-generated sentences abide by the lowest-fidelity “music_likeness"
intent in the HWU64 dataset, “play_music” and “general_quirky" being the most
common intents among the “unfaithful" sentences. Figure 4 visualizes this high
percentage of unfaithful generated sentences. It also shows the proportion of
the two most common alternatives that the oracle preferred over the seed
intent. Table 4 presents generated sentences for confounding intents in the
HWU64 and Banking77 datasets. There are clear indications of mix-up of
intents, e.g., Davinci generates, “play a song with the word honey," which
should belong to “play_music" rather than “music_likeness." There are also
instances where the LM mixes two intents; for instance, Davinci generates “Hi
my app is activated on activate.co.in, but unable to top up my phone. I tried
credit card, debit card and Paytm but fails," which could belong to either
“topping_up_by_card" intent (as it mentions about using credit card in the
context of a top up) or “top_up_failed" (as the top up ultimately fails).

###  4.2 Can GPT Models Understand Close Intents?

We perform extra investigations to better understand what limits GPT-3’s
ability to generate data accurately. We hypothesize that one limiting factor
can be GPT-3’s inability to understand fine-grained differences in the
meanings of utterances. To verify this hypothesis, we evaluate how accurate
GPT-3 is at classifying given utterances as opposed to generating new ones.
Due to the limited prompt size of 2048 tokens, we can not prompt GPT-3 to
predict all the intents in the considered datasets. We thus focus on the close
intent triplets from HWU64 and Banking77 datasets that we use in Table 4. We
compare the 3-way accuracy of a prompted GPT-3 classifier to the similarly-
measured 3-way performance of conventional BERT-large classifiers. We prompt
GPT-3 with 10 examples per intent (see Figure 5). For comparison, we train
BERT-large classifiers on either the same 10 examples or the full training
set. Table 5 shows that the Davinci version of GPT-3 performs in between the
10-shot and the full-data conventional classifiers. This suggests that while
GPT-3’s understanding of nuanced intent differences is imperfect, it could
still be sufficient to improve the performance of the downstream few-shot
model. Inspired by this finding, we experiment with using GPT-3’s
classification abilities to improve the quality of generated data. Namely, we
reject the generated utterances that GPT-3 classifies as not belonging to the
seed intent. For both HWU64 and Banking77, this filtering method significantly
improves the fidelity of the generated data for the chosen close intent
triplets.

###  4.3 Comparison with GPT3Mix

To test our initial hypothesis that prior methods such as GPT3Mix are not
suitable for intent classification, we experiment with the said method on the
CLINC150 dataset using Curie. Specifically, we include an enumeration of the
150 intent names in the prompt and randomly select one example for $K$
intents. We observe a poor in-scope accuracy of 86.33% in the Augmented
scenario666Average of 10 runs with a standard deviation of 1.17. Furthermore,
the generated samples have low fidelity (27.96%). We also test a mixture of
GPT3Mix prompt and our prompt where we include all the $K$ examples for the
seed intent instead of 1 example per $K$ randomly sampled intents. This mixed
variant also performs poorly on CLINC150 and only achieves an in-scope
accuracy of 86.05%777Average of 10 runs with a standard deviation of 0.59 and
a fidelity of 33.56%. Our interpretation of this result is that GPT cannot
handle the long list of 150 intent names in the prompt.

Figure 5: Using GPT-3 as a classifier. Given a triplet of close intents, we
mix and shuffle the multiple seed examples available for each of them. Then,
we append an incomplete line to the prompt with just the generated sentence
and feed it to GPT-3 multiple times. Among the responses, we choose the most
generated in-triplet intent as the predicted intent (“music_likeness" in the
above example). Note: For brevity, we don’t show all the seed examples and
predictions.

##  5 Related Work

The natural language processing literature features diverse data augmentation
methods. Edit-based methods such as Easy Data Augmentation apply rule-based
changes to the original utterances to produce new ones (Wei and Zou, 2019). In
back-translation methods (Sennrich et al., 2016) available examples are
translated to another language and back. Recently, data augmentation with
fine-tuned LMs has become the dominant paradigm (Wu et al., 2018; Kumar et
al., 2019, 2021; Anaby-Tavor et al., 2020; Lee et al., 2021). Our simpler
method sidesteps LM-fine-tuning and directly uses off-she-shelf LMs as is.

The data augmentation approach that is closest to the one we use here is
GPT3Mix by Yoo et al. (2021). A key part of the GPT3Mix prompt is a list of
names of all possible classes (e.g. “The sentiment is one of ‘positive’ or
‘negative”’). The LM is then expected to pick a random class from the list and
generate a new example as well as the corresponding label. However, this
approach does not scale to intent classification setups, which often feature
hundreds of intents (see Section 4.3). Therefore, we choose a different prompt
that encourages the model to extrapolate between examples of a seed intent
similarly to (Lee et al., 2021).

Other work on few-shot intent classification explores fine-tuning dialogue-
specific LMs as classifiers as well as using similarity-based classifiers
instead of MLP-based ones on top of BERT (Vulić et al., 2021). We believe that
improvements brought by data augmentation would be complementary to the gains
brought by these methods.

Lastly, our method to filter out unfaithful GPT generations is related to the
recent work by Wang et al. (2021) that proposes using GPT3 for data labelling.
A crucial difference with respect to our work, however, is that we use GPT-3
for rejecting mislabelled samples rather than proposing labels for unlabelled
samples.

##  6 Conclusion

We propose a prompt-based method to generate intent classification data with
large pretrained language models. Our experiments show that generated data can
be helpful as additional labelled data for some tasks, whereas, for other
tasks, the generated data needs to be either relabelled or filtered to be
helpful. We show that a filtering method that recasts the same GPT model as a
classifier can be effective. Our filtering method, however, requires knowing
the other intents that the generated data is likely to belong to instead of
the seed intent. Future work can experiment with heuristics for approximately
identifying the most likely actual intents for the generated utterances. This
would complete a data generation and filtering pipeline that, according to our
preliminary results in Section 4.2 here, could be effective. Other filtering
methods could also be applied, such as looking at the likelihood of the
generated utterances as explored in a concurrent work by Meng et al. (2022).
Lastly, an interesting future work direction is identifying which generated
utterances most likely need a human inspection.

##  7 Ethical Considerations

As discussed for the GPT3Mix method in Yoo et al. (2021), using large language
models for data augmentation presents several challenges: they exhibit social
biases and are prone to generating toxic content. Therefore, samples generated
using our prompting-based approach need to be considered carefully.

To address such ethical concerns, human inspection would be the most reliable
way to identify and filter out problematic generations. The practitoners who
apply our method may also consider debiasing the language model before using
it for generation (Schick and Schütze, 2021).

## References

  * Anaby-Tavor et al. (2020) Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020.  Do Not Have Enough Data? Deep Learning to the Rescue! _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):7383–7390.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  _arXiv:2005.14165 [cs]_.  ArXiv: 2005.14165.
  * Casanueva et al. (2020) Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020.  Efficient intent detection with dual sentence encoders.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 38–45, Online. Association for Computational Linguistics.
  * Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018.  Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces.  _arXiv:1805.10190 [cs]_.  ArXiv: 1805.10190.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2019_.
  * Kumar et al. (2021) Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2021.  Data Augmentation using Pre-trained Transformer Models.  _arXiv:2003.02245 [cs]_.  ArXiv: 2003.02245.
  * Kumar et al. (2019) Varun Kumar, Hadrien Glaude, Cyprien de Lichy, and Wlliam Campbell. 2019.  A Closer Look At Feature Space Data Augmentation For Few-Shot Intent Classification.  In _Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)_ , pages 1–10, Hong Kong, China. Association for Computational Linguistics.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019.  An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1311–1316, Hong Kong, China. Association for Computational Linguistics.
  * Lee et al. (2021) Kenton Lee, Kelvin Guu, Luheng He, Tim Dozat, and Hyung Won Chung. 2021.  Neural Data Augmentation via Example Extrapolation.  _arXiv:2102.01335 [cs]_.  ArXiv: 2102.01335.
  * Meng et al. (2022) Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022.  Generating training data with language models: Towards zero-shot language understanding.  _arXiv preprint arXiv:2202.04538_.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep Contextualized Word Representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT 2018_.  ArXiv: 1802.05365.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference.  _arXiv:2001.07676 [cs]_.  ArXiv: 2001.07676.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Improving Neural Machine Translation Models with Monolingual Data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96, Berlin, Germany. Association for Computational Linguistics.
  * Vulić et al. (2021) Ivan Vulić, Pei-Hao Su, Samuel Coope, Daniela Gerz, Paweł Budzianowski, Iñigo Casanueva, Nikola Mrkšić, and Tsung-Hsien Wen. 2021.  ConvFiT: Conversational fine-tuning of pretrained language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1151–1168, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wang et al. (2021) Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021.  Want To Reduce Labeling Cost? GPT-3 Can Help.  _arXiv:2108.13487 [cs]_.  ArXiv: 2108.13487.
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6382–6388, Hong Kong, China. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Wu et al. (2018) Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2018.  Conditional BERT Contextual Augmentation.  _arXiv:1812.06705 [cs]_.  ArXiv: 1812.06705.
  * Xingkun Liu and Rieser (2019) Pawel Swietojanski Xingkun Liu, Arash Eshghi and Verena Rieser. 2019.  Benchmarking natural language understanding services for building conversational agents.  In _Proceedings of the Tenth International Workshop on Spoken Dialogue Systems Technology (IWSDS)_ , pages xxx–xxx, Ortigia, Siracusa (SR), Italy. Springer.
  * Yoo et al. (2021) Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyoung Park. 2021\.  GPT3Mix: Leveraging large-scale language models for text augmentation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2225–2239, Punta Cana, Dominican Republic. Association for Computational Linguistics.

## Appendix

##  Appendix A Human Evaluation

In Figure 2 we evaluate the fidelity of the samples generated by GPT-3 with
respect to the original set of sentences used to prompt it. Fidelity is
approximated by the classification performance of an "oracle" intent
classifier trained on the whole dataset ($D_{train}\cup D_{test}$) and
evaluated over the generated samples. In order assess whether the oracle
predictions are comparable to those of a human, we perform a human evaluation
study.

Figure 6: Human evaluation. Error rate of human evaluators at the task of
finding whether any sentence in a group of 5 was generated by GPT-3 or not.
Each color represents a different GPT-3 engine. Higher error rate indicates
that humans could not correctly identify generated samples and thus it also
indicates higher fidelity. The standard error is displayed as a vertical line
on top of each bar. Figure 7: Human evaluation tool. Example of a question for
the human evaluators. Human evaluators are asked to flag which example is
GPT-3 generated if any among the 5 presented ones.

We consider that a model produces sentences with high fidelity if a human is
unable to distinguish them from a set of human-generated sentences belonging
to the same intent. Therefore, for each intent in the CLINC150 dataset, we
sample five random examples and we randomly choose whether to replace one of
them by a GPT-3 generated sentence from the same intent. We generate sentences
with each of the four GPT-3 models considered in the main text with two
different temperatures (0.8 and 1.0). The sentence to replace is randomly
selected. Finally, the five sentences are displayed to a human who has to
choose which of the sentences is generated by GPT-3, if any.

The task is presented to human evaluators in the form of a web application
(see Figure 7). We placed a button next to each sentence in order to force
human evaluators to individually consider each of the examples. Once
annotated, the evaluator can either submit, discard, or leave the task to
label later. We used a set of 15 voluntary evaluators from multiple
backgrounds, nationalities, and genders. Each evaluator annotated an average
of 35 examples, reaching a total of 500 evaluated tasks.

For each model and temperature, we report the error rate of humans evaluating
whether a task contains a GPT-generated sample. We consider that evaluators
succeeds at a given task when they correctly find the sentence that was
generated by GPT or when they identify that none of them was generated. Thus,
the error rate for a given model and temperature is calculated as #failed /
total_evaluated.

Results are displayed in Figure 6. We find that human evaluators tend to make
more mistakes when the temperature used to sample sentences from GPT-3 is
smaller. This result is expected since lowering the temperature results in
sentences closer to those prompted to GPT-3, which are human-made. We also
observe that models with higher capacity such as Davinci tend to generate more
indistinguishable sentences than lower-capacity models such as Ada, even for
higher temperatures. These results are in agreement with the "oracle" fidelity
results introduced in Figure 2.