#  ConveRT: Efficient and Accurate
Conversational Representations from Transformers
github.com/PolyAI-LDN/polyai-models

Matthew Henderson, Iñigo Casanueva, Nikola Mrkšić,
Pei-Hao Su, Tsung-Hsien Wen and Ivan Vulić
<EMAIL_ADDRESS>
PolyAI Limited, London, UK

###### Abstract

General-purpose pretrained sentence encoders such as BERT are not ideal for
real-world conversational AI applications; they are computationally heavy,
slow, and expensive to train. We propose ConveRT (Conversational
Representations from Transformers), a pretraining framework for conversational
tasks satisfying all the following requirements: it is effective, affordable,
and quick to train. We pretrain using a retrieval-based response selection
task, effectively leveraging quantization and subword-level parameterization
in the dual encoder to build a lightweight memory- and energy-efficient model.
We show that ConveRT achieves state-of-the-art performance across widely
established response selection tasks. We also demonstrate that the use of
extended dialog history as context yields further performance gains. Finally,
we show that pretrained representations from the proposed encoder can be
transferred to the intent classification task, yielding strong results across
three diverse data sets. ConveRT trains substantially faster than standard
sentence encoders or previous state-of-the-art dual encoders. With its reduced
size and superior performance, we believe this model promises wider
portability and scalability for Conversational AI applications.

##  1 Introduction

Dialog systems, also referred to as conversational systems or conversational
agents, have found use in a wide range of applications. They assist users in
accomplishing well-defined tasks such as finding and booking restaurants,
hotels, and flights Hemphill et al. (1990); Williams (2012); El Asri et al.
(2017), with further use in tourist information Budzianowski et al. (2018),
language learning Raux et al. (2003); Chen et al. (2017), entertainment Fraser
et al. (2018), and healthcare Laranjo et al. (2018); Fadhil and Schiavo
(2019). They are also key components of intelligent virtual assistants such as
Siri, Alexa, Cortana, and Google Assistant.

Data-driven task-oriented dialog systems require domain-specific labelled
data: annotations for intents, explicit dialog states, and mentioned entities
Williams (2014); Wen et al. (2017b, a); Ramadan et al. (2018); Liu et al.
(2018); Zhao et al. (2019b). This makes the scaling and maintenance of such
systems very challenging. Transfer learning on top of pretrained models
(Devlin et al., 2019; Liu et al., 2019, inter alia) provides one avenue for
reducing the amount of annotated data required to train models capable of
generalization.

Pretrained models making use of language-model (LM) based learning objectives
have become prevalent across the NLP research community. When it comes to
dialog systems, _response selection_ provides a more suitable pretraining task
for learning representations that can encapsulate conversational cues. Such
models can be pretrained using large corpora of natural unlabelled
_conversational_ data Henderson et al. (2019b); Mehri et al. (2019). Response
selection is also directly applicable to retrieval-based dialog systems, a
popular and elegant approach to framing dialog Wu et al. (2017); Weston et al.
(2018); Mazaré et al. (2018); Gunasekara et al. (2019); Henderson et al.
(2019b).111Retrieval-based dialog is popular because posing dialog as response
selection Gunasekara et al. (2019) simplifies system design Boussaha et al.
(2019). Unlike modular or end-to-end task-oriented systems, retrieval-based
ones do not rely on dedicated modules for language understanding, dialog
management, and generation. They mitigate the requirements for explicit task-
specific semantics hand-crafted by domain experts Henderson et al. (2014);
Mrkšić et al. (2015, 2017).

Response Selection is a task of selecting the most appropriate response given
the dialog history Wang et al. (2013); Al-Rfou et al. (2016); Yang et al.
(2018); Du and Black (2018); Chaudhuri et al. (2018). This task is central to
retrieval-based dialog systems, which typically encode the context and a large
collection of responses in a joint semantic space, and then retrieve the most
relevant response by matching the query representation against the encodings
of each candidate response. The key idea is to: 1) make use of large
unlabelled conversational datasets (such as Reddit conversational threads) to
pretrain a neural model on the general-purpose response selection task; and
then 2) fine-tune this model, potentially with additional network layers,
using much smaller amounts of task-specific data.

Dual-encoder architectures pretrained on response selection have become
increasingly popular in the dialog community Cer et al. (2018); Humeau et al.
(2020); Henderson et al. (2019b). In recent work, Henderson et al. (2019a)
show that standard pretraining LM-based architectures cannot match the
performance of dual encoders when applied to dialog tasks such as response
retrieval.

Scalability and Portability. A fundamental problem with pretrained models is
their large number of parameters (see Table 2 later): they are typically
highly computationally expensive to both train and run Liu et al. (2019). Such
high memory footprints and computational requirements hinder quick deployment
as well as their wide portability, scalability, and research-oriented
exploration. The need to make pretrained models more compact has been
recognized recently, with a line of work focused on building more efficient
pretraining and fine-tuning protocols Tang et al. (2019); Sanh et al. (2019).
The desired reductions have been achieved through techniques such as
distillation Sanh et al. (2019), quantization-aware training Zafrir et al.
(2019), weight pruning Michel et al. (2019) or weight tying Lan et al. (2019).
However, the primary focus so far has been on optimizing the LM-based models,
such as BERT.

ConveRT. This work introduces a more compact pretrained response selection
model for dialog. ConveRT is only 59MB in size, making it significantly
smaller than the previous state-of-the-art dual encoder (444MB). It is also
more compact than other popular sentence encoders, as illustrated in Table 2.
This notable reduction in size and training acceleration are achieved through
combining 8-bit embedding quantization and quantization-aware training,
subword-level parameterization, and pruned self-attention. Furthermore, the
lightweight design allows us to reserve additional parameters to improve the
expressiveness of the dual-encoder architecture; this leads to improved
learning of conversational representations that can be transferred to other
dialog tasks Casanueva et al. (2020); Bunk et al. (2020).

Multi-Context Modeling. ConveRT moves beyond the limiting single-context
assumption made by Henderson et al. (2019b), where only the immediate
preceding context was used to look for a relevant response. We propose a
multi-context dual-encoder model which combines the immediate context with
previous dialog history in the response selection task. The multi-context
ConveRT variant remains compact (73MB in total), while offering improved
performance on a range of established response selection tasks. We report
significant gains over the previous state-of-the-art on benchmarks such as
Ubuntu DSTC7 Gunasekara et al. (2019), AmazonQA Wan and McAuley (2016) and
Reddit response selection Henderson et al. (2019a), both in single-context and
multi-context scenarios. Moreover, we show that sentence encodings learned by
the model can be transferred to other dialog tasks, reaching strong intent
classification performance over three evaluation sets. Pretrained dual-encoder
models, both single-context and multi-context ones, are shared as TensorFlow
Hub modules at github.com/PolyAI-LDN/polyai-models.222Finally, our more
compact neural response selection architecture is well aligned with the recent
socially-aware initiatives on reducing costs and improving fairness and
inclusion in NLP research and practice Strubell et al. (2019); Mirzadeh et al.
(2019); Schwartz et al. (2019). Cheaper training (pretraining the proposed
dual-encoder model on the entire Reddit costs only 85 USD) and quicker
development cycles offer new opportunities for more researchers and
practitioners to tap into the construction of neural task-based dialog
systems.

##  2 Methodology

Pretraining on Reddit Data. We assume working with English throughout the
paper. Simplifying the conversational learning task to response selection, we
can relate target dialog tasks to general-domain conversational data such as
Reddit Al-Rfou et al. (2016). This allows us to fine-tune the parameters of
the task-specific response selection model, starting from the general-domain
response selection model pretrained on Reddit. Similar to Henderson et al.
(2019b), we choose Reddit for pretraining due to: 1) its organic
conversational structure; and 2) its unmatched size, as the public repository
of Reddit data comprises 727M (input, response) pairs.333github.com/PolyAI-
LDN/conversational-datasets

###  2.1 More Compact Response Selection Model

We propose ConveRT \-- Conversational Representations from Transformers -- a
compact dual-encoder pretraining architecture, leveraging subword
representations, transformer-style blocks, and quantization, as illustrated in
Figure 1. ConveRT satisfies all the following requirements: it is effective,
affordable, and quick to train.

Figure 1: Single-context ConveRT dual-encoder model architecture. Its multi-
context extension is illustrated in Figure 2. It is possible to transfer
learned encodings at different network layers (e.g., $r_{x}$ or the final
$h_{x}$) to other tasks such as intent detection or value extraction (see §4).
Note that the model uses two different feed-forward network (FFN) layers: 1)
feed-forward 1 is the standard FFN layer also used by Vaswani et al. (2017),
and 2) feed-forward 2 contains 3 fully-connected non-linear feed-forward
layers followed by a linear layer which maps to the final encodings $h_{x}$
and $h_{y}$ (note that the two feed-forward 2 networks do not share
parameters, while the feed-forward 1 parameters are shared).

Input and Response Representation. Prior to training, we obtain a vocabulary
of subwords $V$ shared by the input side and the response side: we randomly
sample and lowercase 10M sentences from Reddit, and then iteratively run any
subword tokenization algorithm.444In the actual implementation, we use the
same subword tokenization as Vaswani et al. (2018). We run it for 4 iterations
and retain only subwords occurring at least 250 times, containing no more than
20 UTF8 characters, also disallowing more than 4 consecutive digits. The final
vocabulary $V$ contains 31,476 subword tokens. During training and inference,
if we encounter an OOV character it is treated as a subword token, where its
ID is computed using a hash function, and it gets assigned to one of 1,000
additional ‘‘buckets’’ reserved for the OOVs. We therefore reserve parameters
(i.e., embeddings) for the 31,476 subwords from $V$ and for the additional
1,000 OOV-related buckets. At training and inference, after the initial word-
level tokenization on UTF8 punctuation and word boundaries, input text $x$ is
split into subwords following a simple left-to-right greedy prefix matching
Vaswani et al. (2018). We tokenize all responses $y$ during training in
exactly the same manner.

Input and Response Encoder Networks. The subword embeddings then go through a
series of transformations on both the input and the response side. The
transformations are based on the standard Transformer architecture Vaswani et
al. (2017). Before going through the self-attention blocks, we add positional
encodings to the subword embedding inputs. Previous work (e.g., BERT and
related models) (Devlin et al., 2019; Lan et al., 2019, inter alia) learns a
fixed number of positional encodings, one for each position in the sequence,
allowing the model to represent a fixed number of positions. Instead, we learn
two positional encoding matrices of different sizes- $M^{1}$ of dimensionality
[47, 512] and $M^{2}$ of dimensionality [11, 512]. An embedding at position
$i$ is added to: $M^{1}_{i\mod 47}+M^{2}_{i\mod 11}$.555Note that since 47 and
11 are coprime, this gives $47\cdot 11=517$ different possible positional
encodings. Similar to the original (non-learned) positional encodings from
Vaswani et al. (2017), the rationale behind this choice of positional encoding
is to allow the model to generalize to unseen sequence lengths.

The next layers closely follow the original Transformer architecture with some
notable differences. First, we set maximum relative attention Shaw et al.
(2018) in the six layers to the following respective values: [3, 5, 48, 48,
48, 48].666We zero out in training and inference the attention scores for
pairs of words if they are further apart than the set maximum relative
attention values. This also helps the architecture to generalize to long
sequences and distant dependencies: earlier layers are forced to group
together meanings at the phrase level before later layers model larger
patterns. We use single-headed attention throughout the network.777Multi-
headed attention requires running computations on 4-tensors: [batch, time,
head, embedding], while for single-headed attention, this reduces to
3-tensors, and effectively speeds up training without hurting performance.

Before going into a softmax, we add a bias to the attention scores that
depends only on the relative positions:
$\alpha_{ij}\rightarrow\alpha_{ij}+B_{n-i+j}$ where $B$ is a learned bias
vector. This helps the model understand relative positions, but is much more
computationally efficient than computing full relative positional encodings
Shaw et al. (2018). Again, it also helps the model generalize to longer
sequences.

Six Transformer blocks use a 64-dim projection for computing attention
weights, a 2,048-dim kernel (feed-forward 1 in Figure 1), and 512-dim
embeddings. Note that all Transformer layers use parameters that are fully
shared between the input side and the response side. As in the Universal
Sentence Encoder (use) Cer et al. (2018), we use square-root-of-N reduction to
convert the embedding sequences to fixed-dimensional vectors. Two self-
attention heads each compute weights for a weighted sum, which is scaled by
the square root of the sequence length; the length is computed as the number
of constituent subwords.888In fact, rather than computing the self-attended
sequence, then reducing it, we reduce the attention weights accordingly, and
then directly apply them via matrix multiplication to the input sequence to
get the final reduced representation, that is, we _fuse_ these two operations.
This is more computationally efficient, avoiding another 3-tensor
multiplication. The outputs of the reduction layer, labelled $r_{x}$ and
$r_{y}$ in Figure 1, are 1,024-dimensional vectors that are fed to the two
‘‘side-specific’’ (i.e., they do not share parameters) feed-forward networks.

In other words, the vectors $r_{x}$ and $r_{y}$ go through a series of $N_{f}$
$l$-dim feed-forward hidden layers ($N_{f}=3$; $l=1,024$) with skip
connections, layer normalization, and orthogonal initialization. The
activation function used in these networks and throughout the architecture is
the fast GeLU approximation Hendrycks and Gimpel (2016):
$GeLU(x)=x\sigma(1.702x)$. The final layer is linear and maps the text into
the final L2-normalized 512-dim representation: $h_{x}$ for the input text,
and $h_{y}$ for the corresponding response text (Figure 1).

Input-Response Interaction. The relevance of each response to the given input
is then quantified by the score $S(x,y)$, computed as cosine similarity with
annealing between the encodings $h_{x}$ and $h_{y}$. It starts at 1 and ends
at $\sqrt{d}$, linearly increasing over the first 10K training batches.
Training proceeds in batches of $K$ (input, response) pairs
$(x_{1},y_{1}),\ldots,(x_{K},y_{K})$. The aim of the objective is to
distinguish between the true relevant response ($y_{i}$) and irrelevant
responses (i.e., negative samples) $y_{j},j\neq i$ for each input sentence
$x_{i}$. The training objective for a single batch of $K$ pairs is as follows:
$J=\sum_{i=1}^{K}S(x_{i},y_{i})-\sum_{i=1}^{K}\log\sum_{j=1}^{K}e^{S(x_{i},y_{j})}$.
The goal is to maximize the score of positive training pairs $(x_{i},y_{i})$
and minimize the score of pairing each input $x_{i}$ with $K^{\prime}$
negative examples, which are responses that are not associated with the input
$x_{i}$: for simplicity, all other $K-1$ from the current batch are used as
negative examples.

Quantization. Very recent work has shown that large models of language can be
made more compact by applying quantization techniques Han et al. (2016): e.g.,
quantized versions of Transformer-based machine translation systems Bhandare
et al. (2019) and BERT Shen et al. (2019); Zhao et al. (2019a); Zafrir et al.
(2019) are now available. In this work, we focus on enabling quantization-
aware conversational pretraining on the response selection task. We show that
the dual-encoder ConveRT model from Figure 1 can be also be trained in a
quantization-aware manner. Rather than the standard 32-bits per parameter, all
embedding parameters are represented using only 8 bits, and other network
parameters with just 16 bits; they are trained in a quantization-aware manner
by adapting the mixed precision training scheme from Micikevicius et al.
(2018). It keeps shadow copies of each variable with 32bit Floating Point
(FP32) precision, but uses FP16-cast versions in the computations and
inference models. Some operations in the graph, however, require FP32
precision to be numerically stable: layer normalization, L2-normalization, and
softmax in attention layers.

Again, following Micikevicius et al. (2018), the final loss is scaled by 128,
and the updates to the shadow FP32 variables are scaled back by 1/128: this
allows the gradient computations to stay well represented by FP16 (e.g., they
will not get rounded to zero). The subword embeddings are stored using 8-bits
per parameter, and the quantization range is adjusted dynamically through
training. It is updated periodically to contain all of the embedding values
that have so-far been learned, with room for growth above and below - 10% of
the range, or 0.01 - whichever is larger. Finally, quantization also allows
doubling the batch size, which also has a favourable effect of increasing the
number of negative examples in training.

Figure 2: Multi-context ConveRT. It models 1) the interaction between the
immediate context and its accompanying response, 2) the interaction of the
response with up to 10 earlier contexts from the conversation history, as well
as 3) the interaction of the full context with the response. Transformer
layers refer to the standard Transformer architecture also used in the single-
context encoder model in Figure 1; the feed-forward 2 blocks are the same as
with the single-context encoder architecture, see Figure 1. The block mean
refers to simple averaging of two context encodings $h_{x}$ and $h_{z}$.

Multi-Context ConveRT. Figure 1 depicts a single-context dual encoder
architecture. Intuitively, the single-context assumption is limiting for
modeling multi-turn conversations, where strong conversational cues can be
found in earlier dialog history, and there has been a body of work on
leveraging richer dialog history for response selection Chaudhuri et al.
(2018); Zhou et al. (2018); Humeau et al. (2020). Taking a simple illustrative
example:

Student: I’m very interested in representation learning.
Teacher: Do you have any experience in PyTorch?
Student: Not really.
Teacher: And what about TensorFlow?

Selecting the last Teacher’s response would be very difficult given only the
immediate preceding context. However, the task becomes easier when taking into
account the entire context of the conversation. We thus construct a multi-
context dual-encoder model by using up to 10 more previous messages in a
Reddit thread. The extra 10 contexts are concatenated from most recent to
oldest, and treated as an extra feature in the network, as shown in Figure 2.
Note that all context representations are still independent from the
representation of a candidate response, so we can still do efficient response
retrieval and training. The full training objective is a linear combination of
three sub-objectives: 1) ranking responses given the immediate context (i.e.,
this is equal to the single-context model from §2.1), 2) ranking responses
given only the extra (non-immediate) contexts, and 3) ranking responses given
the averaged representation of the immediate context and additional
contexts.999Combining multiple objectives in a dual-encoder framework has also
been done by Al-Rfou et al. (2016) and Henderson et al. (2017). Note that more
sophisticated solutions to fusing dialog history are possible such as using
attention over older contexts as done by Vlasov et al. (2019) on the much
smaller MultiWOZ 2.1 dataset Eric et al. (2019), but we have opted for simple
concatenation as an efficient solution for training on the large Reddit data.
The multiple objectives result in quicker learning, and also give useful
diagnostic probes into the performance of each feature throughout training.

##  3 Experimental Setup

Training Data and Setup. We base all our (pre)training on the large Reddit
conversational corpus Henderson et al. (2019a) derived from 3.7B Reddit
comments: it comprises 727M (input, response) pairs for single-context
modeling -- 654M pairs are reserved for training, the rest is used for
testing. We truncate sequences to 60 subwords, embedding size is set to 512
for all subword embeddings and bucket embeddings, and the final encodings
$h_{x}$, $h_{y}$, $h_{z}$, and $h_{x,z}$ are all 512-dimensional. The hidden
layer size of feed forward 2 networks is set to 1,024 (with $N_{f}=3$ hidden
layers used).

We train using ADADELTA with $\rho=0.9$ Zeiler (2012), batch size of 512, and
a learning rate of 1.0 annealed to 0.001 with cosine decay over training.
L2-regularization of $10^{-5}$ is used, subword embedding gradients are
clipped to 1.0, and label smoothing of 0.2 is applied.101010The label
smoothing technique Szegedy et al. (2016) reduces overfitting by preventing a
network to assign full probability to the correct training example Pereyra et
al. (2017). It means that each positive example in each batch is assigned the
probability of 0.8, while the remaining probability mass is evenly
redistributed across in-batch negative examples.

We pretrain the model on Reddit on 12 GPU nodes with one Tesla K80 each for 18
hours; this is typically sufficient to reach convergence. The total
pretraining cost is roughly $85 on Google Cloud Platform. This pretraining
regime is orders of magnitude cheaper and more efficient than the prevalent
pretrained NLP models such as BERT, GPT-2, XLNet, and RoBERTa Strubell et al.
(2019).

Baselines. We report results on the response selection tasks and compare
against the standard set of baselines Henderson et al. (2019a). First, we
compare to a simple keyword matching baseline based on tf-idf query-response
scoring Manning et al. (2008), and then with a representative sample of
publicly available neural encoders that embed inputs and responses into a
vector space relying on various pretraining objectives: (1) The larger variant
of Universal Sentence Encoder Cer et al. (2018) (use-large); (2) The large
variant of BERT Devlin et al. (2019) (bert-large). We also compare to two
recent dual-encoder architectures: (3) use-qa is a dual question-answer
encoder version of the USE (large) model Chidambaram et al. (2019).111111Note
that use-qa encodes inputs/contexts and responses using separate sub-networks,
while ConveRT (Figure 1) relies on full parameter sharing in the Transformer
layers. (4) polyai-dual is the best-performing dual-encoder model from
Henderson et al. (2019b) pretrained on Reddit response selection. For baseline
models 1-3, we report the results with the map response selection variant
Henderson et al. (2019a): it showed much stronger performance than a simpler
similarity-based variant which directly ranks responses according to their
cosine similarity with the context vector. map learns to (linearly) map the
response vectors to the input vector space.

Response Selection: Evaluation Tasks. We report response selection performance
on Reddit test set Henderson et al. (2019a) with both single-context and
multi-context ConveRT variants. For multi-context ConveRT, the averaged
representation of (immediate and previous) context is used in evaluation. The
models are applied directly on the Reddit test data without any further fine-
tuning. We also evaluate on two other well-known response selection problems
in different domains. (1) amazonQA Wan and McAuley (2016) is an e-commerce
data set which contains information about Amazon products in the form of
question-answer pairs:out of 3.6M (single-context) QA pairs, 300K pairs are
reserved for testing. (2) dstc7-ubuntu is based on the Ubuntu v2 corpus Lowe
et al. (2017): it contains 1M+ conversations in a highly technical domain
(i.e., Ubuntu technical support). dstc7-ubuntu uses 100K conversations for
training, 10K for validation, and 5K conversations are used for testing
Gunasekara et al. (2019).

For dstc7-ubuntu we fine-tune for 60K training steps: it takes around 2h on 12
GPU workers. The learning rate starts at 0.1, and is annealed to 0.0001 using
cosine decay over training. We use a batch size of 256, and dropout of 0.2
after the embedding and self-attention layers. We use the same fine-tuning
regime for amazonQA. For dstc7-ubuntu, extra contexts are prepended with
numerical strings 0--9 to help the model identify their position. We also
release the fine-tuned models.

We evaluate with a standard IR-inspired evaluation measure: Recall@k, used in
prior work on retrieval-based dialog Chaudhuri et al. (2018); Henderson et al.
(2019b); Gunasekara et al. (2019). Given a set of $N$ responses to the given
input, where only one response is relevant, it indicates whether the relevant
response occurs in the top $k$ ranked candidates. We denote this measure as
$\mathbf{R}_{N}@k$, and set $N=100;k=1$: $\mathbf{R}_{100}@1$.

Intent Classification: Task, Data, Setup. Pretrained sentence encoders have
become particularly popular due to the success of training models for
downstream tasks on top of their learned representations, greatly improving
the results compared to training from scratch, especially in low-data regimes
(see Table 1). Therefore, we also probe the usefulness of ConveRT encodings
for transfer learning in the intent classification task: the model must
classify the user’s utterance into one of several predefined classes, that is,
intents (e.g., within e-banking intents can be card lost or replace card). We
use three internal intent classification datasets from three diverse domains,
see Table 1, divided into train, dev and test sets using a 80/10/10 split.

We use the pretrained ConveRT encodings $r_{x}$ on the input side (see Figure
1) as input to an intent classification model. We also experimented with later
$h_{x}$ encodings on the input side, but stronger results were observed with
$r_{x}$. We train a 2-layer feed-forward net with dropout on top of $r_{x}$.
SGD with a batch size of 32 is used, with early stopping after 5 epochs
without improvement on the validation set. Layer sizes, dropout rate and
learning rate are selected through grid search. We compare against two other
standard sentence encoders again: use-large and bert-large. For ConveRT and
use-large we keep the encoders fixed and train the classifier layers on top of
the sentence encodings. For bert-large, we train on top of the CLS token and
we fine-tune all its parameters.

|  # intents |  # examples
---|---|---
Banking (customer service) |  77 |  14.6K
Shopping (online shopping) |  10 |  13.8K
Company FAQ |  110 |  3.3K
Table 1: Intent classification data sets.

##  4 Results and Discussion

|  Embedding |  Network |  Total | Size after
---|---|---|---|---
|  parameters |  parameters |  size | quantization
USE Cer et al. (2018) |  256 M |  2 M |  1033 MB | 261 MB *
BERT-BASE Devlin et al. (2019) |  23 M |  86 M |  438 MB | 196 MB */ 110 MB **
BERT-LARGE Devlin et al. (2019) |  31 M |  304 M |  1341 MB | 639 MB */ 336 MB **
GPT-2 Radford et al. (2019) |  80 M |  1462 M |  6168 MB | 3004 MB *
POLYAI-DUAL Henderson et al. (2019b) |  104 M |  7 M |  444 MB | 118 MB
ConveRT (this work) |  16 M |  13 M |  116 MB | 59 MB
Table 2: Comparison of the proposed compact dual-encoder architecture for
response selection to existing public standard sentence embedding models. (*)
The size after quantization assumes embeddings can be quantized to 8 bits and
network parameters to 16 bits, which has not been verified for the public
models. (**) Best-case model size estimates of the BERT model after full 8-bit
quantization based on the work of Zafrir et al. (2019).

Model Size, Training Time, Cost. Table 2 lists encoders from prior work along
with their model size, and estimated model size after quantization. The
reported numbers indicate the gains achieved through subword-level
parameterization and quantization of ConveRT. Besides reduced training costs,
ConveRT offers a reduced memory footprint and quicker training. We pretrain
all our models for 18 hours only (on 12 16GB T4 GPUs), while a model
compression technique DistilBERT Sanh et al. (2019) (i.e., it reports $\approx
40\%$ relative reduction of the original BERT) trains on 8 16GB V100 GPUs for
90 hours, and larger models like RoBERTa require 1 full day of training on
1024 32GB V100 GPUs. The achieved size reduction and quick training also allow
for quicker development and insightful ablation studies (see later in Table
4), and using quantization also improves training efficiency in terms of
examples per second.

|  Reddit |  AmazonQA
---|---|---
tf-idf |  26.4 |  51.8
use-large-map |  47.7 |  61.9
bert-large-map |  24.0 |  44.1
use-qa-map |  46.6 |  70.7
polyai-dual |  61.3 |  71.3
ConveRT (single-context) |  68.2 |  84.3
ConveRT (multi-context) |  71.8 |  \--
Table 3: $\mathbf{R}_{100}@1\times 100\%$ scores on Reddit test set and amazonQA. polyai-dual and ConveRT networks are fine-tuned on the training portion of amazonQA. Note that amazonQA by design supports only single-context response selection. Model Configuration |
---|---
ConveRT |  68.2
A: Multi-headed attention (8 64-dim heads) |  68.5
B: No relative position bias |  67.8
C: Without gradually increasing max attention span |  67.7
D: Only 1 OOV bucket |  68.0
E: 1-headed (instead of 2-headed) reduction |  67.7
F: No skip connections in feed forward 2 |  67.8
D + E + F |  66.7
B + C + D + E + F |  66.6
Table 4: An ablation study illustrating the importance of different components in ConveRT: single-context response selection on Reddit ($\mathbf{R}_{100}@1$). Each experiment has been run for 966K steps (batch size 512). |  $\mathbf{R}_{100}@1$ |  MRR
---|---|---
Best DSTC7 System |  64.5 |  73.5
GPT* |  48.9 |  59.5
BERT* |  53.0 |  63.2
Bi-encoder Humeau et al. (2020) |  70.9 |  78.1
ConveRT (single-context) |  38.2 |  49.2
ConveRT (multi-context) |  71.2 |  78.8
Table 5: Results on dstc7-ubuntu. (*) Scores for GPT and BERT taken from Vig
and Ramea (2019).

Response Selection on Reddit. The results are summarized in Table 3. Even
single-context ConveRT achieves peak performance in the task, with substantial
gains over the previous best reported score of Henderson et al. (2019b). It
also substantially outperforms all the other models which were not pretrained
directly on the response selection task, but on a standard LM task instead.
The strongest baselines, however, are two dual-encoder architectures (i.e.,
use-large, use-qa and polyai-dual); this illustrates the importance of
explicitly distinguishing between inputs/contexts and responses when modeling
response selection.

Table 3 also shows the importance of leveraging additional contexts (see
Figure 2). Multi-context ConveRT achieves a state-of-the-art Reddit response
selection score of 71.8% We observe similar benefits in other reported
response selection tasks. We also note the results of 1) using only the sub-
network that models the interaction between the immediate context and the
response (i.e., the $h_{x}^{T}h_{y}$ interaction), and 2) artificially
replacing the concatenated extra contexts $z$ with an empty string. The
respective scores are 65.7% and 65.6%. This suggests that multi-context
ConveRT is also applicable to single-context scenarios when no extra contexts
are provided for the target task.

Ablation Study. The efficient training regime also allows us to perform a
variety of diagnostic experiments and ablations. We report results with
variants of single-context ConveRT in Table 4. They indicate that replacing
single-headed with multi-headed attention leads to slight improvements, but
this comes at a cost of slower (and consequently - more expensive) training.
Using 1 instead of 1,000 OOV buckets leads only to a modest decrease in
performance. Most importantly, the ablation study indicates that the final
performance actually comes from the synergistic effect of applying a variety
of components and technical design choices such as skip connections, 2-headed
reductions, relative position biases, etc. While removing only one component
at a time yields only modest performance losses, the results show that the
loss adds up as we remove more components, and different components indeed
contribute to the final score.121212Furthermore, quick development and short
training times also allow us to treat some of the component choices as hyper-
parameter choices. It effectively means that such configuration choices can
also be fine-tuned similar to any other hyper-parameter to optimize the final
retrieval performance.

Other Response Selection Tasks. The results on the amazonQA task are provided
in Table 3. We see similar trends as with Reddit evaluation. Fine-tuned
ConveRT reaches a new state-of-the-art score, and the strongest baselines are
again dual-encoder networks. Fine-tuned polyai-dual, which was pretrained on
exactly the same data, cannot match ConveRT’s performance.131313Interestingly,
directly applying ConveRT to amazonQA without any fine-tuning also yields a
reasonably high score of 67.0%. Moreover, learning the mapping function
between inputs and responses (again without any fine-tuning) for ConveRT the
same way as is done for use-qa-map results in the score of 71.6%, which
outperforms use-qa-map (70.7%). The gap to the fine-tuned model’s performance,
however, indicates the importance of in-domain fine-tuning.

The results on dstc7-ubuntu are summarized in Table 5 First, they suggest very
competitive performance of multi-context ConveRT model: it outperforms the
best-scoring system from the official DSTC7 challenge Gunasekara et al.
(2019). It is an encouraging finding, given that multi-context ConveRT relies
on simple context concatenation without any additional attention mechanisms.
We leave the investigation of such more sophisticated models to integrate
additional contexts for future work. Multi-context ConveRT can also match or
even surpass the performance of another dual-encoder architecture from Humeau
et al. (2020). Their dual encoder (i.e., bi-encoder) is based on the BERT-base
architecture Humeau et al. (2020): it relies on 12 Transformer blocks, 12
attention heads, and a hidden size dimensionality of 768 (while we use 512).
Training with that model is roughly 5$\times$ slower, and the pretraining
objective is more complex: they use the standard BERT pretraining objective
plus next utterance classification. Moreover, their model is trained on 32
v100 GPUs for 14 days, which makes it roughly 50$\times$ more expensive than
ConveRT.

|  Banking |  Shopping | Company FAQ
---|---|---|---
use-large |  92.2 |  94.0 | 62.4
bert-large |  93.2 |  94.3 | 61.2
ConveRT |  92.7 |  94.5 | 64.3
Table 6: Intent classification results.

Intent Classification. The results are summarized in Table 6: we report the
results of two strongest baselines for brevity. The scores show very
competitive performance of ConveRT encodings $r_{x}$ transferred to another
dialog task. They outperform use-large in all three tasks and bert-large in
2/3 tasks. Note that, besides quicker pretraining, intent classifiers based on
ConveRT encodings train 40 times faster than bert-large-based ones, as only
the classification layers are trained for ConveRT. In sum, these preliminary
results suggest that ConveRT as a sentence encoder can be useful beyond the
core response selection task. The usefulness of ConveRT-based sentence
representations have been recently confirmed on other intent classification
datasets Casanueva et al. (2020), with different intent classifiers Bunk et
al. (2020), and in another dialog task: turn-based value extraction Coope et
al. (2020); Bunk et al. (2020). In future work, we plan to investigate other
possible applications of transfer, especially for low-data setups.

##  5 Conclusion

We have introduced ConveRT, a new light-weight model of neural response
selection for dialog, based on Transformer-backed dual-encoder networks, and
have demonstrated its state-of-the-art performance on an array of response
selection tasks and in transfer learning for intent classification tasks. In
addition to offering more accurate conversational pretraining models this work
has also resulted in more compact conversational pretraining. The quantized
versions of ConveRT and multi-context ConveRT take up only 59 MB and 73 MB,
respectively, and train for 18 hours with a training cost estimate of only 85
USD. In the hope that this work will motivate and guide further developments
in the area of retrieval-based task-oriented dialog, we publicly release
pretrained ConveRT models.

## References

  * Al-Rfou et al. (2016) Rami Al-Rfou, Marc Pickett, Javier Snaider, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2016.  Conversational contextual cues: The case of personalization and history for response ranking.  _CoRR_ , abs/1606.00372.
  * Bhandare et al. (2019) Aishwarya Bhandare, Vamsi Sripathi, Deepthi Karkada, Vivek Menon, Sun Choi, Kushal Datta, and Vikram Saletore. 2019.  Efficient 8-bit quantization of transformer neural machine language translation model.  _CoRR_ , abs/1906.00532.
  * Boussaha et al. (2019) Basma El Amel Boussaha, Nicolas Hernandez, Christine Jacquin, and Emmanuel Morin. 2019.  Deep retrieval-based dialogue systems: A short review.  _CoRR_ , abs/1907.12878.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  MultiWOZ - A large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  In _Proceedings of EMNLP_ , pages 5016--5026.
  * Bunk et al. (2020) Tanja Bunk, Daksh Varshneya, Vladimir Vlasov, and Alan Nichol. 2020.  DIET: Lightweight language understanding for dialogue systems.  _CoRR_ , abs/2004.09936.
  * Casanueva et al. (2020) Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020.  Efficient intent detection with dual sentence encoders.  _CoRR_ , abs/2003.04807.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Universal sentence encoder for English.  In _Proceedings of EMNLP_ , pages 169--174.
  * Chaudhuri et al. (2018) Debanjan Chaudhuri, Agustinus Kristiadi, Jens Lehmann, and Asja Fischer. 2018.  Improving response selection in multi-turn dialogue systems by incorporating domain knowledge.  In _Proceedings of CoNLL_ , pages 497--507.
  * Chen et al. (2017) Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017.  A survey on dialogue systems: Recent advances and new frontiers.  _CoRR_ , abs/1711.01731.
  * Chidambaram et al. (2019) Muthuraman Chidambaram, Yinfei Yang, Daniel Cer, Steve Yuan, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019.  Learning cross-lingual sentence representations via a multi-task dual-encoder model.  In _Proceedings of the 4th Workshop on Representation Learning for NLP_ , pages 250--259.
  * Coope et al. (2020) Sam Coope, Tyler Farghly, Daniela Gerz, Ivan Vulić, and Matthew Henderson. 2020\.  Span-ConveRT: Few-shot span extraction for dialog with pretrained conversational representations.  In _Proceedings of ACL_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171--4186.
  * Du and Black (2018) Wenchao Du and Alan Black. 2018.  Data augmentation for neural online chats response selection.  In _Proceedings of the 2nd International Workshop on Search-Oriented Conversational AI_ , pages 52--58.
  * El Asri et al. (2017) Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017.  Frames: A corpus for adding memory to goal-oriented dialogue systems.  In _Proceedings of SIGDIAL_ , pages 207--219.
  * Eric et al. (2019) Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, and Dilek Hakkani-Tür. 2019.  Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines.  _CoRR_ , abs/1907.01669.
  * Fadhil and Schiavo (2019) Ahmed Fadhil and Gianluca Schiavo. 2019.  Designing for health chatbots.  _CoRR_ , abs/1902.09022.
  * Fraser et al. (2018) Jamie Fraser, Ioannis Papaioannou, and Oliver Lemon. 2018.  Spoken conversational AI in video games: Emotional dialogue management increases user engagement.  In _Proceedings of IVA_.
  * Gunasekara et al. (2019) Chulaka Gunasekara, Jonathan K. Kummerfeld, Lazaros Polymenakos, and Walter Lasecki. 2019.  DSTC7 task 1: Noetic end-to-end response selection.  In _Proceedings of the 1st Workshop on NLP for Conversational AI_ , pages 60--67.
  * Han et al. (2016) Song Han, Huizi Mao, and William J. Dally. 2016.  Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding.  In _Proceedings of ICLR_.
  * Hemphill et al. (1990) Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990.  The ATIS Spoken Language Systems Pilot Corpus.  In _Proceedings of the Workshop on Speech and Natural Language_ , HLT ’90, pages 96--101.
  * Henderson et al. (2017) Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017.  Efficient natural language response suggestion for smart reply.  _CoRR_ , abs/1705.00652.
  * Henderson et al. (2019a) Matthew Henderson, Pawel Budzianowski, Iñigo Casanueva, Sam Coope, Daniela Gerz, Girish Kumar, Nikola Mrkšić, Georgios Spithourakis, Pei-Hao Su, Ivan Vulić, and Tsung-Hsien Wen. 2019a.  A repository of conversational datasets.  In _Proceedings of the 1st Workshop on Natural Language Processing for Conversational AI_.
  * Henderson et al. (2014) Matthew Henderson, Blaise Thomson, and Jason D. Wiliams. 2014.  The Second Dialog State Tracking Challenge.  In _Proceedings of SIGDIAL_ , pages 263--272.
  * Henderson et al. (2019b) Matthew Henderson, Ivan Vulić, Daniela Gerz, Iñigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019b.  Training neural response selection for task-oriented dialogue systems.  In _Proceedings of ACL_ , pages 5392--5404.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  Gaussian error linear units (GELUs).  _arXiv preprint arXiv:1606.08415_.
  * Humeau et al. (2020) Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020.  Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring.  In _Proceedings of ICLR_.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A Lite BERT for self-supervised learning of language representations.  In _Proceedings of ICLR_.
  * Laranjo et al. (2018) Liliana Laranjo, Adam G. Dunn, Huong Ly Tong, Ahmet Baki Kocaballi, Jessica Chen, Rabia Bashir, Didi Surian, Blanca Gallego, Farah Magrabi, Annie Y.S. Lau, and Enrico Coiera. 2018.  Conversational agents in healthcare: A systematic review.  _Journal of the American Medical Informatics Association_ , 25(9):1248--1258.
  * Liu et al. (2018) Bing Liu, Gökhan Tür, Dilek Hakkani-Tür, Pararth Shah, and Larry P. Heck. 2018.  Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems.  In _Proceedings of NAACL-HLT_ , pages 2060--2069.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Lowe et al. (2017) Ryan Thomas Lowe, Nissan Pow, Iulian Vlad Serban, Laurent Charlin, Chia-Wei Liu, and Joelle Pineau. 2017.  Training end-to-end dialogue systems with the ubuntu dialogue corpus.  _Dialogue & Discourse_, 8(1):31--65.
  * Manning et al. (2008) Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008.  _Introduction to Information Retrieval_.  Cambridge University Press.
  * Mazaré et al. (2018) Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018\.  Training millions of personalized dialogue agents.  In _Proceedings of EMNLP_ , pages 2775--2779.
  * Mehri et al. (2019) Shikib Mehri, Evgeniia Razumovskaia, Tiancheng Zhao, and Maxine Eskenazi. 2019.  Pretraining methods for dialog context representation learning.  In _Proceedings of ACL_ , pages 3836--3845.
  * Michel et al. (2019) Paul Michel, Omer Levy, and Graham Neubig. 2019.  Are sixteen heads really better than one? In _Proceedings of NeurIPS_.
  * Micikevicius et al. (2018) Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018.  Mixed precision training.  In _Proceedings of ICLR_.
  * Mirzadeh et al. (2019) Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh. 2019\.  Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher.  _CoRR_ , abs/1902.03393.
  * Mrkšić et al. (2015) Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2015\.  Multi-domain dialog state tracking using recurrent neural networks.  In _Proceedings of ACL_ , pages 794--799.
  * Mrkšić et al. (2017) Nikola Mrkšić, Ivan Vulić, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. 2017.  Semantic specialisation of distributional word vector spaces using monolingual and cross-lingual constraints.  _Transactions of the ACL_ , pages 314--325.
  * Pereyra et al. (2017) Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017.  Regularizing neural networks by penalizing confident output distributions.  _CoRR_ , abs/1701.06548.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8).
  * Ramadan et al. (2018) Osman Ramadan, Paweł Budzianowski, and Milica Gašić. 2018.  Large-scale multi-domain belief tracking with knowledge sharing.  In _Proceedings of ACL_ , pages 432--437.
  * Raux et al. (2003) Antoine Raux, Brian Langner, Alan W. Black, and Maxine Eskénazi. 2003.  LET’s GO: Improving spoken dialog systems for the elderly and non-natives.  In _Proceedings of EUROSPEECH_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _CoRR_ , abs/1910.01108.
  * Schwartz et al. (2019) Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2019.  Green AI.  _CoRR_ , abs/1907.10597.
  * Shaw et al. (2018) Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018.  Self-attention with relative position representations.  In _Proceedings of NAACL-HLT_ , pages 464--468.
  * Shen et al. (2019) Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2019.  Q-BERT: Hessian based ultra low precision quantization of BERT.  _CoRR_ , abs/1909.05840.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of ACL_ , pages 3645--3650.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016.  Rethinking the inception architecture for computer vision.  In _Proceedings of CVPR_ , pages 2818--2826.
  * Tang et al. (2019) Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019\.  Distilling task-specific knowledge from BERT into simple neural networks.  _CoRR_ , abs/1903.12136.
  * Vaswani et al. (2018) Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018.  Tensor2Tensor for neural machine translation.  In _Proceedings of the 13th Conference of the Association for Machine Translation in the Americas_ , pages 193--199. Association for Machine Translation in the Americas.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of NeurIPS_ , pages 6000--6010.
  * Vig and Ramea (2019) Jesse Vig and Kalai Ramea. 2019.  Comparison of transfer-learning approaches for response selection in multi-turn conversations.  In _Proceedings of DSTC-7_.
  * Vlasov et al. (2019) Vladimir Vlasov, Johannes E. M. Mosig, and Alan Nichol. 2019.  Dialogue transformers.  _CoRR_ , abs/1910.00486.
  * Wan and McAuley (2016) Mengting Wan and Julian McAuley. 2016.  Modeling ambiguity, subjectivity, and diverging viewpoints in opinion question answering systems.  In _Proceedings of ICDM_ , pages 489--498.
  * Wang et al. (2013) Hao Wang, Zhengdong Lu, Hang Li, and Enhong Chen. 2013.  A dataset for research on short-text conversations.  In _Proceedings of EMNLP_ , pages 935--945.
  * Wen et al. (2017a) Tsung-Hsien Wen, Yishu Miao, Phil Blunsom, and Steve J. Young. 2017a.  Latent intention dialogue models.  In _Proceedings of ICML_ , pages 3732--3741.
  * Wen et al. (2017b) Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017b.  A network-based end-to-end trainable task-oriented dialogue system.  In _Proceedings of EACL_ , pages 438--449.
  * Weston et al. (2018) Jason Weston, Emily Dinan, and Alexander Miller. 2018.  Retrieve and refine: Improved sequence generation models for dialogue.  In _Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI_ , pages 87--92.
  * Williams (2012) Jason Williams. 2012.  A critical analysis of two statistical spoken dialog systems in public use.  In _Proceedings of SLT_.
  * Williams (2014) Jason D. Williams. 2014.  Web-style ranking and SLU combination for dialog state tracking.  In _Proceedings of SIGDIAL_ , pages 282--291.
  * Wu et al. (2017) Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017.  Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots.  In _Proceedings of ACL_ , pages 496--505.
  * Yang et al. (2018) Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-Yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Learning semantic textual similarity from conversations.  In _Proceedings of the 3rd Workshop on Representation Learning for NLP_ , pages 164--174.
  * Zafrir et al. (2019) Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019.  Q8BERT: Quantized 8bit BERT.  _CoRR_ , abs/1910.06188.
  * Zeiler (2012) Matthew D. Zeiler. 2012.  ADADELTA: an adaptive learning rate method.  _CoRR_ , abs/1212.5701.
  * Zhao et al. (2019a) Sanqiang Zhao, Raghav Gupta, Yang Song, and Denny Zhou. 2019a.  Extreme language model compression with optimal subwords and shared projections.  _CoRR_ , abs/1909.11687.
  * Zhao et al. (2019b) Tiancheng Zhao, Kaige Xie, and Maxine Eskénazi. 2019b.  Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models.  In _Proceedings of NAACL-HLT_ , pages 1208--1218.
  * Zhou et al. (2018) Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu. 2018.  Multi-turn response selection for chatbots with deep attention matching network.  In _Proceedings of ACL_ , pages 1118--1127.


# Robustmix: Improving Robustness by Regularizing the Frequency Bias of Deep
Nets

Jonas Ngnawé
Université Laval and Mila-Quebec AI Institute
<EMAIL_ADDRESS>
&Marianne N. Abemgnigni11footnotemark: 1
University of Göttingen
<EMAIL_ADDRESS>
Jonathan Heek
Google AI
<EMAIL_ADDRESS>
&Yann Dauphin
Google AI
<EMAIL_ADDRESS>
These authors contributed equally, work done during AI residency at Google.

###### Abstract

Deep networks have achieved impressive results on a range of well-curated
benchmark datasets. Surprisingly, their performance remains sensitive to
perturbations that have little effect on human performance. In this work, we
propose a novel extension of Mixup called Robustmix that regularizes networks
to classify based on lower-frequency spatial features. We show that this type
of regularization improves robustness on a range of benchmarks such as
ImageNet-C and Stylized ImageNet. It adds little computational overhead and
furthermore does not require a priori knowledge of a large set of image
transformations. We find that this approach further complements recent
advances in model architecture and data augmentation attaining a state-of-the-
art mean corruption error (mCE) of 44.8 with an EfficientNet-B8 model and
RandAugment, which is a reduction of 16 mCE compared to the baseline.

##  1 Introduction

Deep neural networks have achieved state-of-the-art accuracy across a range of
benchmark tasks such as image segmentation (Ren et al., 2015) and speech
recognition (Hannun et al., 2014). These successes have led to the widespread
adoption of neural networks in many real-life applications. However, while
these networks perform well on curated benchmark datasets, their performance
can suffer greatly in the presence of small data corruptions (Szegedy et al.,
2014; Goodfellow et al., 2014; Moosavi-Dezfooli et al., 2017; Athalye et al.,
2018; Hendrycks & Dietterich, 2018). This poses significant challenges to the
application of deep networks.

Hendrycks & Dietterich (2018) show that the accuracy of a standard model on
ImageNet can drop from 76% to 20% when evaluated on images corrupted with
small visual transformations. This shows modern networks are not robust to
certain small shifts in the data distribution. That is a concern because such
shifts are common in many real-life applications. Secondly, Szegedy et al.
(2014) show the existence of _adversarial_ perturbations which are
imperceptible to humans but have a disproportionate effect on the predictions
of a network. This raises significant concerns about the safety of using deep
networks in critical applications such as self driving cars (Sitawarin et al.,
2018).

These problems have led to numerous proposals to improve the robustness of
deep networks. Some of these methods such as those proposed by Hendrycks et
al. (2019) require a priori knowledge of the visual transformations in the
test domain. Others, such as Geirhos et al. (2018) use a deep network to
generate transformations which comes with significant computation cost.

This paper proposes a new data augmentation technique to improve the
robustness of deep networks by regularizing frequency bias. This new
regularization technique is based on Mixup and has many advantages compared to
related robustness regularizers: (1) it does not require knowledge of a large
set of priori transformations, (2) it is inexpensive, and (3) it doesn’t have
many hyper-parameters. The key idea is to bias the network to rely more on
lower spatial frequencies to make predictions.

We demonstrate on ImageNet-C that this method works well with recent advances
and reaches a state-of-the-art mCE of 44.8, 85.0 clean accuracy with
EfficientNet-B8 and RandAugment(Cubuk et al., 2019). This is an improvement of
16 mCE compared to the baseline EfficientNet-B8 and matches ViT-L/16
(Dosovitskiy et al., 2020), which is trained on $300\times$ more data. We find
that our implementation of the method with DCT transform adds negligible
overhead in our experiments. We find that Robustmiximproves accuracy on
Stylized-ImageNet by up to 15 points and we show that it can increase
adversarial robustness.

##  2 Related Work

The proposed approach can be seen as a generalization of Mixup (Zhang et al.,
2018), a data augmentation method that regularizes models by training them on
linear interpolations of two input examples and their respective labels. These
new examples are generated as follows

$\displaystyle\tilde{x}$
$\displaystyle=\texttt{mix}(x_{1},x_{2},\lambda),\qquad\text{where~{}}x_{1},x_{2}\text{~{}are~{}input
images}$ $\displaystyle\tilde{y}$
$\displaystyle=\texttt{mix}(y_{1},y_{2},\lambda),\qquad\text{where~{}}y_{1},y_{2}\text{~{}are~{}labels}$

with mix being the linear interpolation function

$\displaystyle\texttt{mix}(x_{1},x_{2},\lambda)=\lambda
x_{1}+(1-\lambda)x_{2}$ (1)

where $\lambda\sim\mathrm{Beta}(\alpha,\alpha)$, $\alpha$ is the Mixup
coefficient hyper-parameter.

Zhang et al. (2018) show that Mixup improves the accuracy of networks and can
also improve the robustness of the network. In the past years, several
versions of Mixup were proposed with application in Fairness (Chuang & Mroueh,
2021), 3D reconstruction (Cheng et al., 2022), semi-supervised learning
(Beckham et al., 2019), as well as robustness ((Mai et al., 2021; Yun et al.,
2019; Faramarzi et al., 2020; Kim et al., 2020; Verma et al., 2019)). The
novel version we propose here is frequency-based and does not include
additional learnable parameters.

Augmix (Hendrycks et al., 2019) is a data augmentation technique to improve
robustness by training on a mix of known image transformations. It adds little
computational overhead but requires knowledge of a diverse set of domain-
specific transformations. Hendrycks et al. (2019) mixes a set of 9 different
augmentations to reach $68.4$ mCE on ImageNet. In contrast, the proposed
method does not rely on specific image augmentations but on the more general
principle that natural images are a kind of signal where most of the energy is
concentrated in the lower frequencies.

The idea of frequency filtering is popular in Deep learning frameworks and has
numerous applications including unsupervised domain adaptation (Yang & Soatto
(2020)) and adversarial perturbation attacks (Guo et al. (2018); Li et al.
(2021)). Unlike the latter papers which focus on measuring the accuracy of a
model after an adversarial attack, we focus on common (noise) corruptions by
measuring mCE as a robustness assessment.

Zhang (2019) uses low pass filters directly inside the model to improve the
frequency response of the network.Wang et al. (2019) uses a differentiable
neural network to extract textual information from images without modeling the
lower-frequency. Our method also makes use of low-pass filtering but does not
completely remove high-frequency features. Additionally, we only use frequency
filtering during training, and therefore no computational overhead is incurred
during evaluation.

##  3 Method

Figure 1: Illustration of the method. In order to better illustrate the
method, we display the Fourier spectrum of the images next to them. We can see
that even though 90% of the higher frequencies belong to the image of a dog,
Robustmixassigns more weight to the boathouse label because it assigns more
weight to the lower frequencies.

In this section, we introduce a novel extension of Mixup called _Robustmix_
that increases robustness by regularizing the network to focus more on the
low-frequency features in the signal.

Motivation Wang et al. (2020) suggest that convolutional networks trade
robustness for accuracy in their use of high-frequency image features. Such
features can be perturbed in ways that change the prediction of the model,
even though humans cannot perceive the change. This can lead models to make
puzzling mistakes, such as with adversarial examples. Our aim is to increase
robustness while retaining accuracy by regularizing how high-frequency
information is used by the model.

Robustmix We propose to regularize the sensitivity of the model to each
frequency band by extending Mixup’s linear interpolations with a new type of
band interpolation. The key insight is that we can condition the sensitivity
to each band using images that mix the frequency bands of two different
images. Suppose that we mix the lower-frequency band of an image of a
boathouse with the high-frequency band of an image of a dog. We can encourage
sensitivity to the lower band by training the model to predict "dog" for this
mixed image. However, this approach is too simplistic because it completely
disregards the impact of the image in the high band. Indeed, the ablation
study in section 4.4 shows that it is insufficient.

Figure 2: Plot of the cumulative energy in ImageNet images as a function of
the frequency cutoff.

Instead, we interpolate the label of such mixed images according to an
estimate of the importance of each frequency band. We propose to use the
relative amount of energy in each band as an estimate of the importance. Thus
the sensitivity of the model to high-frequency features will be proportional
to their energy contribution in natural images. And as we can see in Figure 2,
most of the spectral energy in natural images is concentrated in the lower end
of the spectrum. This should limit the ability of high-frequency perturbations
to change the prediction unilaterally.

Furthermore, we use linear interpolations of images like in mixup within each
band instead of raw images. This closely reflects the more common case where
the features in the bands are merely corrupted instead of entirely swapped. It
also has the benefit of encouraging linearity inside the same frequency band.

Specifically, the mixing formula for Robustmix is given by

$\displaystyle\tilde{x}$
$\displaystyle=\texttt{Low}(\texttt{mix}(x_{1},x_{2},\lambda_{L}),c)+\texttt{High}(\texttt{mix}(x_{1},x_{2},\lambda_{H}),c)$
(2) $\displaystyle\tilde{y}$
$\displaystyle=\lambda_{c}\texttt{mix}(y_{1},y_{2},\lambda_{L})+(1-\lambda_{c})\texttt{mix}(y_{1},y_{2},\lambda_{H})$
(3)

where $\lambda_{L},\lambda_{H}\sim\mathrm{Beta}(\alpha,\alpha)$, $\alpha$ is
the Mixup coefficient hyper-parameter, and
$\texttt{Low}(\cdot,c),\texttt{High}(\cdot,c)$ are a low pass and high pass
filter respectively with a uniformly sampled cutoff frequency $c\in[0,1]$. And
$\lambda_{c}$ is the coefficient that determines how much weight is given to
the lower frequency band. It is given by the relative amount of energy in the
lower frequency band for natural images

$\displaystyle\lambda_{c}=\frac{E[\|\texttt{Low}(x_{i},c)\|^{2}]}{E[\|x_{i}\|^{2}]}.$
(4)

This coefficient can be efficiently computed on a mini-batch of examples.

Implementation Computational overhead is an important consideration for data
augmentation techniques since training deep networks is computationally
intensive and practitioners have limited computational budgets. We note that
many popular techniques such as Mixup (Zhang et al., 2018) add little
overhead.

The frequency separation is implemented using a Discrete Cosine Transform
(DCT) to avoid the complex multiplication required by a Discrete Fourier
Transform. We multiply the images with the 224x224 DCT matrix directly because
the spatial dimensions are relatively small and (non-complex) matrix
multiplication is well-optimized on modern accelerators. A batch of images is
transformed into frequency space and the low and high-pass filtered images
must be transformed back to image space. Additionally, we must apply the DCT
transform over the x and y dimension separately. Thus, 6 DCT matrix
multiplications are required which results in $0.2$ GFLOPs per image. In
contrast, just the forward pass of ResNet50 requires $3.87$ GFLOPs (Hasanpour
et al., 2016).

In our implementation of Robustmix, we reorder commutative operations (low
pass and mixing) in order to compute the DCT only a single time per minibatch.
The pseudocode is provided in Algorithm 1, where $\mathrm{reverse}$ is a
function that reverses the rows of its input matrix.

Algorithm 1 Robustmix

Input: Minibatch of inputs $X\in\mathbb{R}^{N\times H\times W\times D}$ and
labels $Y\in\mathbb{R}^{N\times C}$, $\alpha\in\mathbb{R}$

Output: Augmented minibatch of inputs $\tilde{X}\in\mathbb{R}^{N\times W\times
H\times D}$ and labels $\tilde{Y}\in\mathbb{R}^{N\times C}$

$\lambda_{L},\lambda_{H}\sim\mathrm{Beta}(\alpha,\alpha)$ and $c\sim U(0,1)$

$L\leftarrow\texttt{Low}(X,c)$

$H\leftarrow 1-L$

$\lambda_{c}\leftarrow\frac{\|L\|^{2}}{\|X\|^{2}}$

$\tilde{X}\leftarrow\texttt{mix}(L,\mathrm{reverse}(L),\lambda_{L})+\texttt{mix}(H,\mathrm{reverse}(H),\lambda_{H})$

$\tilde{Y}\leftarrow\texttt{mix}(Y,\mathrm{reverse}(Y),\lambda_{c}*\lambda_{L}+(1-\lambda_{c})*\lambda_{H})$

##  4 Results

###  4.1 Datasets and Metrics

The results presented in this paper rely on the mCE measurement on ImageNet-C,
the clean accuracies on ImageNet and Stylized-ImageNet (SIN) as well as the
shape bias on SIN. These measurements are found in a range of paper studying
robustness (Hendrycks & Dietterich, 2018; Hendrycks et al., 2019; Geirhos et
al., 2018; Laugros et al., 2020). The Stylized-ImageNet benchmark is meant to
distinguish between a bias towards shape or texture. We believe our results on
Stylized-ImageNet complement the standard robustness results because it shows
that the inductive bias is more human-like, in the sense that it is more
sensitive to shape than texture.

ImageNet. ImageNet (Deng et al., 2009) is a classification dataset that
contains 1.28 million training images and 50000 validation images with 1000
classes. We evaluate the common classification accuracy which will be referred
to as clean accuracy. We use the standard Resnet preprocessing resulting in
images of size 224x224 (He et al., 2015). The standard models, without any
additional data augmentation process, will be qualified as the baseline.

ImageNet-C. This dataset is made of 15 types of corruption drawn from four
main categories: noise, blur, weather, and digital (Hendrycks & Dietterich,
2018). These corruptions are applied to the validation images of ImageNet at 5
different intensities or levels of severity. Following (Hendrycks &
Dietterich, 2018), we evaluate the robustness of our method by reporting its
mean corruption error (mCE) normalized with respect to AlexNet errors:

$\text{mCE}=\frac{\sum\limits_{\text{corruption
}c}{\text{CE}}_{c}}{\text{Total Number of Corruptions}},\;$ $\text{with
}{\text{CE}}_{c}=\frac{\sum\limits_{\text{severity
}s}E_{c,s}}{\sum_{s}E_{c,s}^{\mathrm{AlexNet}}}$

Stylized-ImageNet. Stylized-ImageNet (SIN) is constructed from ImageNet by
replacing the texture in the original image using style transfer, such that
the texture gives a misleading cue about the image label (Geirhos et al.,
2018). The 1000 classes from ImageNet are reduced to 16 shape categories, for
instance, all labels for dog species are grouped under one "dog" label, same
for "chair", "car", etc. There are 1280 generated cue conflict images (80 per
category). With SIN, we evaluate the classification accuracy (SIN accuracy)
and measure the model’s shape bias. Following Geirhos et al. (2018), the
model’s bias towards shape versus texture is measured as

$\text{shape bias}=\frac{\text{correct shapes}}{\text{correct shapes + correct
textures}}.$

###  4.2 Experimental Setup

We chose to do evaluations on residual nets (ResNet-50 and ResNet-152) and
EfficientNets (EfficientNet-B0, EfficientNet-B1, EfficientNet-B5, and
EfficientNet-B8). Experiments were run on 8x8 TPUv3 instances for the bigger
EfficientNets (EfficientNet-B5 and EfficientNet-B8), and the other experiments
were run on 4x4 TPUv3 slices. For the Resnet models, we use the same standard
training setup outlined in Goyal et al. (2017). However, we use cosine
learning rate Loshchilov & Hutter (2016) with a single cycle for Resnets
trained for 600 epochs.

###  4.3 Robustness Results

ImageNet-C First, we evaluate the effectiveness of the proposed method in
improving the robustness to visual corruptions considered in ImageNet-C. In
Table 1, we can see that Robustmix consistently improves robustness to the
considered transformations, with a 15-point decrease in mCE over the baseline
for ResNet-50. Robustmix with ResNet-50 achieves 61.2 mCE without degrading
accuracy on the clean dataset compared to the baseline. In fact, we find a
small improvement over the baseline of 0.8% on the clean error. While Mixup
yields a larger gain of 1.9% on the clean accuracy, we find that Robustmix
improves mCE by up to 6 points more than Mixup. These results also compare
favorably to Augmix, which needs to be combined with training on Stylized
ImageNet (SIN) to reduce the mCE by 12 points. And this improvement comes at a
significant cost to the accuracy due to the use of the Stylized ImageNet
dataset. We also observe a similar trade-off between accuracy and robustness
as we can observe in Figure 3. We observe that Mixup consistently produces
lower clean error for smaller models, but the accuracy gap with Robustmix
disappears as the model gets bigger.

While it is not directly comparable to ViT-L/16 due to its use of $300\times$
more data, we see that EfficientNet-B8 with Robustmixand RandAugment has
better robustness at $44.8$ mCE. It is also competitive with DeepAugment
(Hendrycks et al., 2020) which requires training additional specialized image-
to-image models on tasks such as super-resolution to produce augmented images.
By comparison, our approach does not rely on extra data or extra-trained
models.

Method |  | Clean
---
Accuracy
mCE | Size |  | Extra
---
Data
ResNet-50 Baseline (200 epochs) | 76.3 | 76.9 | 26M | 0
ResNet-50 Baseline (600 epochs) | 76.3 | 78.1 | 26M | 0
ResNet-50 BlurPool (Zhang, 2019) | 77.0 | 73.4 | 26M | 0
ResNet-50 Mixup (200 epochs) | 77.5 | 68.1 | 26M | 0
ResNet-50 Mixup (600 epochs) | 78.2 | 67.5 | 26M | 0
ResNet-50 Augmix | 77.6 | 68.4 | 26M | 0
ResNet-50 Augmix + SIN | 74.8 | 64.9 | 26M | 0
ResNet-50 Robustmix (600 epochs) | 77.1 | 61.2 | 26M | 0
EfficientNet-B0 Baseline | 76.8 | 72.4 | 5.3M | 0
EfficientNet-B0 Mixup ($\alpha=0.2$) | 77.1 | 68.3 | 5.3M | 0
EfficientNet-B0 Robustmix ($\alpha=0.2$) | 76.8 | 61.9 | 5.3M | 0
EfficientNet-B1 Baseline | 78.1 | 69.4 | 7.8M | 0
EfficientNet-B1 Mixup ($\alpha=0.2$) | 78.9 | 64.7 | 7.8M | 0
EfficientNet-B1 Robustmix ($\alpha=0.2$) | 78.7 | 57.8 | 7.8M | 0
EfficientNet-B5 Baseline | 82.7 | 65.6 | 30M | 0
EfficientNet-B5 Mixup ($\alpha=0.2$) | 83.3 | 58.9 | 30M | 0
EfficientNet-B5 Robustmix ($\alpha=0.2$) | 83.3 | 51.7 | 30M | 0
EfficientNet-B5 RandAug+Robustmix ($\alpha=0.2$) | 83.8 | 48.7 | 30M | 0
BiT m-r101x3 (Kolesnikov et al., 2020) | 84.7 | 58.27 | 387.9M | 12.7M
| ResNeXt-101 $32\times 8d$+DeepAugment+AugMix
---
(Hendrycks et al., 2020)
79.9 | 44.5 | 88.8M |  | Extra
---
models
ViT-L/16 (Dosovitskiy et al., 2020) | 85.2 | 45.5 | 304.7M | 300M
RVT-$B^{*}$ (Mao et al., 2022) | 82.7 | 46.8 | 91.8M | PAAS+Patch-wise111PAAS: Position-Aware Attention Scaling + a Simple and general patch-wise augmentation method for patch sequences.
EfficientNet-B8 Baseline | 83.4 | 60.8 | 87.4M | 0
EfficientNet-B8 Robustmix ($\alpha=0.4$) | 84.4 | 49.8 | 87.4M | 0
EfficientNet-B8 RandAug+Robustmix ($\alpha=0.4$) | 85.0 | 44.8 | 87.4M | 0
Table 1: Comparison of various models based on ImageNet accuracy and
ImageNet-C robustness (mCE). The robustness results for BiT and ViT are as
reported by Paul & Chen (2021)(Table 3).

Our experiments also show that Doublemix combines well with RandAugment (RA),
improving further both accuracy and mCE. We removed augmentations from RA that
overlap with corruptions in ImageNet-C (contrast, color, brightness,
sharpness, and Cut-out) (Hendrycks et al., 2019).

Figure 3: Highlighting the tradeoff between mCE and Clean Error for various
models.

In our cross-validation of $\alpha$, we found small values less than $0.2$
perform poorly both on accuracy and mCE. Values of $\alpha$ such that
$0.2\leq\alpha\leq 0.5$ not only give the best accuracies and mCEs but also
the best trade-off of mCE versus accuracy as bigger values of $\alpha$ have
shown giving good values for accuracy but do not do as well on mCE. In our
experiments, we found that we typically achieve good results with a frequency
cutoff $c$ sampled between $[0,1]$ as described in Algorithm 1. However, for
ResNet-50 trained with a training budget that is too limited (200 instead of
600 epochs) and its smaller versions (ResNet-18 and ResNet-34), it can be
beneficial to fix a minimum $c\geq\tau$ for the cutoff by sampling in the
interval $[\tau,1]$. The minimum cutoff determines the range at which band
mixing will occur. We can remove band interpolation entirely and recover
standard Mixup by setting $\tau=1$. For Resnet-50 with too few training
epochs, we found that a good value for the minimum is $0.1$, but we found much
better results can be achieved with 600 epochs without any modifications to
Algorithm 1.

Stylized-ImageNet. We confirm that our method indeed increases both accuracy
on Stylized ImageNet and the shape bias as shown in table 3. For ResNet-50,
Robustmix almost doubles the shape bias from baseline (from 19 to 37) and
improves it by 63% over Mixup; while relative improvements on SIN accuracy are
of 72% and 33% respectively over baseline and Mixup. The same observation is
for EfficientNet-B5 which improves shape bias by nearly 50% and SIN accuracy
by nearly 60 % over the baseline.

Method | Mixed Image | Label |  | Test
---
Accuracy
mCE
| Robustmix- Full
---
(inband mixups and energy weighting)
Equation 2 | Equation 3 | 77.1 | 61.2
Robustmix without energy weighting | Equation 2 |  | Equation 3
---
with $\lambda_{c}$
replaced by $c$
77.6 | 67.7
| Robustmix without inband mixups
---
and with energy weighting
($\lambda_{L}=1,\lambda_{H}=0$)
$\texttt{Low}(x_{1},c)+\texttt{High}(x_{2},c)$ | $\lambda_{c}y_{1}+(1-\lambda_{c})y_{2}$ | 68.6 | 75.3
| Robustmix without inband mixups
---
and without energy weighing
($\lambda_{L}=1,\lambda_{H}=0$
and cutoff c as label coefficient)
$\texttt{Low}(x_{1},c)+\texttt{High}(x_{2},c)$ | $cy_{1}+(1-c)y_{2}$ | 74.8 | 77.5

Table 2: Comparison of Robustmix with simplified cases. The results are reported on ResNet50. Method/Parameters |  | SIN
---
Accuracy
| Shape
---
Bias
ResNet-50 Baseline | 15.6 | 19.25
ResNet-50 Mixup | 20.1 | 22.7
ResNet-50 Robustmix | 26.8 | 37.0
EfficientNet-B5 Baseline | 25.3 | 44.4
EfficientNet-B5 Mixup | 28.75 | 48.3
EfficientNet-B5 Robustmix | 40.3 | 66.1
Table 3: Accuracy and shape bias computed on Stylized ImageNet.

###  4.4 Ablation Study

In order to measure the effect of Robustmix we apply some simplifications both
with respect to the image mixing as well as the labeling. The results are
compiled in table 2. It can be noticed from the first two lines that ablating
the energy weighting results in a significant drop in mCE even though there is
a slight accuracy improvement. However, keeping the energy weighting but not
applying the inband mixups is largely detrimental both to accuracy and
robustness. These results show that Robustmixachieves a better combination of
mCE and accuracy than these ablations.

###  4.5 Analysis and Discussion

Low-frequency bias In order to quantify the degree to which models rely on
lower frequencies, we measure how much accuracy drops as we remove higher-
frequency information with a low-pass filter. Figure 4 shows that Robustmixis
comparatively more robust to the removal of high frequencies. This indicates
that models trained with Robustmix rely significantly less on these high-
frequency features to make accurate predictions.

Figure 4: Test accuracy on ImageNet samples passed through a low-pass filter
with increasing cut-off. As expected, we observe that Robustmixis more robust
to the removal of high frequencies than Mixup. The comparison is done here on
ResNet-50 models.

##  5 Conclusion

In this paper, we have introduced a new method to improve robustness called
Robustmix which regularizes models to focus more on lower spatial frequencies
to make predictions. We have shown that this method yields improved robustness
on a range of benchmarks including ImageNet-C and Stylized ImageNet. In
particular, this approach attains an mCE of 44.8 on ImageNet-C with
EfficientNet-B8, which is competitive with models trained on $300\times$ more
data.

Our method offers a promising new research direction for robustness with a
number of open challenges. We have used a standard DCT-based low-pass filter
on images and L2 energy metric to determine the contribution of each label.
This leaves many alternatives to be explored, such as different data
modalities like audio; more advanced frequency separation techniques like
Wavelets; and alternative contribution metrics for mixing labels.

## References

## References

  * Athalye et al. (2018) Athalye, A., Engstrom, L., Ilyas, A., and Kwok, K.  Synthesizing robust adversarial examples.  In _International conference on machine learning_ , pp. 284–293. PMLR, 2018.
  * Beckham et al. (2019) Beckham, C., Honari, S., Verma, V., Lamb, A. M., Ghadiri, F., Hjelm, R. D., Bengio, Y., and Pal, C.  On adversarial mixup resynthesis.  In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/f708f064faaf32a43e4d3c784e6af9ea-Paper.pdf.
  * Cheng et al. (2022) Cheng, T.-Y., Yang, H.-R., Trigoni, N., Chen, H.-T., and Liu, T.-L.  Pose adaptive dual mixup for few-shot single-view 3d reconstruction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pp. 427–435, 2022.
  * Chuang & Mroueh (2021) Chuang, C.-Y. and Mroueh, Y.  Fair mixup: Fairness via interpolation.  _arXiv preprint arXiv:2103.06503_ , 2021.
  * Cubuk et al. (2019) Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V.  Randaugment: Practical automated data augmentation with a reduced search space. arxiv e-prints, page.  _arXiv preprint arXiv:1909.13719_ , 2019.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L., Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 248–255, 2009.  doi: 10.1109/CVPR.2009.5206848.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2020.
  * Faramarzi et al. (2020) Faramarzi, M., Amini, M., Badrinaaraayanan, A., Verma, V., and Chandar, S.  Patchup: A regularization technique for convolutional neural networks.  _arXiv preprint arXiv:2006.07794_ , 2020.
  * Geirhos et al. (2018) Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  In _International Conference on Learning Representations_ , 2018.
  * Goodfellow et al. (2014) Goodfellow, I. J., Shlens, J., and Szegedy, C.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ , 2014.
  * Goyal et al. (2017) Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  _arXiv preprint arXiv:1706.02677_ , 2017.
  * Guo et al. (2018) Guo, C., Frank, J. S., and Weinberger, K. Q.  Low frequency adversarial perturbation, 2018.  URL https://arxiv.org/abs/1809.08758.
  * Hannun et al. (2014) Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., Prenger, R., Satheesh, S., Sengupta, S., Coates, A., et al.  Deep speech: Scaling up end-to-end speech recognition.  _arXiv preprint arXiv:1412.5567_ , 2014.
  * Hasanpour et al. (2016) Hasanpour, S. H., Rouhani, M., Fayyaz, M., and Sabokrou, M.  Lets keep it simple, using simple architectures to outperform deeper and more complex architectures.  _arXiv preprint arXiv:1608.06037_ , 2016.
  * He et al. (2015) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition, 2015.
  * Hendrycks & Dietterich (2018) Hendrycks, D. and Dietterich, T.  Benchmarking neural network robustness to common corruptions and perturbations.  In _International Conference on Learning Representations_ , 2018.
  * Hendrycks et al. (2019) Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B.  Augmix: A simple data processing method to improve robustness and uncertainty.  In _International Conference on Learning Representations_ , 2019.
  * Hendrycks et al. (2020) Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  _arXiv preprint arXiv:2006.16241_ , 2020.
  * Kim et al. (2020) Kim, J.-H., Choo, W., and Song, H. O.  Puzzle mix: Exploiting saliency and local statistics for optimal mixup.  In _International Conference on Machine Learning_ , pp. 5275–5285. PMLR, 2020.
  * Kolesnikov et al. (2020) Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N.  Big transfer (bit): General visual representation learning.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_ , pp. 491–507. Springer, 2020.
  * Laugros et al. (2020) Laugros, A., Caplier, A., and Ospici, M.  Addressing neural network robustness with mixup and targeted labeling adversarial training, 2020.
  * Li et al. (2021) Li, X.-C., Zhang, X.-Y., Yin, F., and Liu, C.-L.  F-mixup: Attack cnns from fourier perspective.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , pp. 541–548. IEEE, 2021.
  * Loshchilov & Hutter (2016) Loshchilov, I. and Hutter, F.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ , 2016.
  * Mai et al. (2021) Mai, Z., Hu, G., Chen, D., Shen, F., and Shen, H. T.  Metamixup: Learning adaptive interpolation policy of mixup with metalearning.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2021\.
  * Mao et al. (2022) Mao, X., Qi, G., Chen, Y., Li, X., Duan, R., Ye, S., He, Y., and Xue, H.  Towards robust vision transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12042–12051, 2022.
  * Moosavi-Dezfooli et al. (2017) Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., and Frossard, P.  Universal adversarial perturbations.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1765–1773, 2017.
  * Paul & Chen (2021) Paul, S. and Chen, P.-Y.  Vision transformers are robust learners.  _arXiv preprint arXiv:2105.07581_ , 2021.
  * Ren et al. (2015) Ren, S., He, K., Girshick, R., and Sun, J.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28:91–99, 2015.
  * Sitawarin et al. (2018) Sitawarin, C., Bhagoji, A. N., Mosenia, A., Chiang, M., and Mittal, P.  Darts: Deceiving autonomous cars with toxic signs.  _arXiv preprint arXiv:1802.06430_ , 2018.
  * Szegedy et al. (2014) Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R.  Intriguing properties of neural networks.  In _2nd International Conference on Learning Representations, ICLR 2014_ , 2014.
  * Verma et al. (2019) Verma, V., Lamb, A., Beckham, C., Najafi, A., Mitliagkas, I., Lopez-Paz, D., and Bengio, Y.  Manifold mixup: Better representations by interpolating hidden states.  In Chaudhuri, K. and Salakhutdinov, R. (eds.), _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 6438–6447. PMLR, 09–15 Jun 2019.  URL https://proceedings.mlr.press/v97/verma19a.html.
  * Wang et al. (2019) Wang, H., He, Z., Lipton, Z. C., and Xing, E. P.  Learning robust representations by projecting superficial statistics out, 2019.  URL https://arxiv.org/abs/1903.06256.
  * Wang et al. (2020) Wang, H., Wu, X., Huang, Z., and Xing, E. P.  High-frequency component helps explain the generalization of convolutional neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8684–8694, 2020.
  * Yang & Soatto (2020) Yang, Y. and Soatto, S.  Fda: Fourier domain adaptation for semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4085–4095, 2020.
  * Yun et al. (2019) Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 6023–6032, 2019.
  * Zhang et al. (2018) Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D.  mixup: Beyond empirical risk minimization.  In _International Conference on Learning Representations_ , 2018.
  * Zhang (2019) Zhang, R.  Making convolutional networks shift-invariant again.  In _International conference on machine learning_ , pp. 7324–7334. PMLR, 2019.