# D3GU: Multi-target Active Domain Adaptation
via Enhancing Domain Alignment

Lin Zhang† Linghan Xu† Saman Motamed† Shayok Chakraborty‡ Fernando De la
Torre† †Robotics Institute, Carnegie Mellon University  ‡Department of
Computer Science, Florida State University

###### Abstract

Unsupervised domain adaptation (UDA) for image classification has made
remarkable progress in transferring classification knowledge from a labeled
source domain to an unlabeled target domain, thanks to effective domain
alignment techniques. Recently, in order to further improve performance on a
target domain, many Single-Target Active Domain Adaptation (ST-ADA) methods
have been proposed to identify and annotate the salient and exemplar target
samples. However, it requires one model to be trained and deployed for each
target domain and the domain label associated with each test sample. This
largely restricts its application in the ubiquitous scenarios with multiple
target domains. Therefore, we propose a Multi-Target Active Domain Adaptation
(MT-ADA) framework for image classification, named D3GU, to simultaneously
align different domains and actively select samples from them for annotation.
This is the first research effort in this field to our best knowledge. D3GU
applies Decomposed Domain Discrimination (D3) during training to achieve both
source-target and target-target domain alignments. Then during active
sampling, a Gradient Utility (GU) score is designed to weight every unlabeled
target image by its contribution towards classification and domain alignment
tasks, and is further combined with KMeans clustering to form GU-KMeans for
diverse image sampling. Extensive experiments on three benchmark datasets,
Office31, OfficeHome, and DomainNet, have been conducted to validate
consistently superior performance of D3GU for MT-ADA 111Code is available at
https://github.com/lzhangbj/D3GU.

Figure 1:  Illustration of MT-ADA task on OfficeHome with source domain (art),
three target domains (clipart, product, real), and five classification classes
(alarm_clock, backpack, batteries, bed, bike). The pretraining stage trains a
model $\theta_{0}$ through unsupervised domain adaptation on labeled source
images $L_{\mathcal{S}}$ and unlabeled target images
$\bigcup\\{U_{\mathcal{T}_{i}}\\}_{i=1}^{3}$. Multiple active learning stages
then follow to iterate between active sampling and domain adapted training. At
the $j$-th active learning stage, the active sampling function selects and
annotates some unlabeled target images to expand the labeled target set
$\bigcup\\{L_{\mathcal{T}_{i}}\\}_{i=1}^{3}$ using the model $\theta_{j-1}$
trained at the previous stage. The domain adapted training step then finetunes
$\theta_{j-1}$ to obtain $\theta_{j}$ using all images and annotations. Model
performance is evaluated on target domains at test time. Section 3 describes
more details.

##  1 Introduction

Deep image classification models trained in one domain (source) often fail to
generalize well to other domains (targets) due to ubiquitous domain shifts.
Domain shifts can occur due to a variety of reasons, such as changing
conditions (e.g., background, location, pose changes) and different image
formats (e.g., photos, NIR images, paintings, sketches) [9]. To address this
problem, the challenging Unsupervised domain adaptation (UDA) paradigm has
been extensively studied in recent years. Without any annotated data in the
target domain $\mathcal{T}$, UDA aims to train a robust classification model
for the target by transferring relevant knowledge from the labeled source
domain $\mathcal{S}$. Various methods have emerged to approach the problem by
aligning source and target feature distributions (Domain Alignment) [27, 48,
5, 43, 25, 10, 1, 8, 49, 44, 16, 23, 19]. The most popular line of research
designs adversarial domain discriminators in parallel with the classification
head [48, 20, 27]. Through adversarial training of the domain discriminator,
the feature encoder learns a domain-invariant feature space with informative
supervision from the source, while addressing the domain shifts between the
source and the target. The success of domain adaptation thus heavily relies on
effective alignment between domains.

Along with UDA, it has also been empirically established that a small number
of labeled data in the target domain can largely improve the performance of
domain adaptation [22]. Active Learning, the process of selecting labeled
data, has then been identified as the critical factor for classification
performance. This has motivated researchers to apply it to domain adaptation
and formulate the problem of Single-Target Active Domain Adaptation (ST-ADA).
ST-ADA aims to identify the most informative unlabeled target samples for
manual annotation and efficiently mitigate the lack of classification
knowledge in the target domain [20, 30, 34, 6, 7, 46, 14]. However, ST-ADA is
not an ideal solution for many real-world scenarios where multiple target
domains are available. Specifically, applying ST-ADA on multiple target
domains requires independently training one model on each target domain. At
test time, the domain label of a test sample should also be known to select
its target model for inference. The whole system is therefore restricted by
huge disk storage linearly increasing with number of target domains involved
and its usage scenarios, necessitating a framework to be designed for Multi-
Target Active Domain Adaptation (MT-ADA).

Therefore, we propose the first learning framework for MT-ADA image
classification. As depicted in Figure 1, our framework consists of an initial
pretraining stage, followed by multiple active learning stages. The
pretraining stage conducts unsupervised domain adaptation with all labeled
source images and unlabeled target images. Each active learning stage consists
of two steps: (i) an active sampling step automatically selecting unlabeled
target images from the union target pool for manual annotation within a preset
budget. (ii) a domain adapted training step utilizing all images and available
annotations to obtain an improved classification model. The iteration between
these two steps gradually expands the labeled target set and improves
classification performance on target domains. Unlike the aforementioned ST-ADA
solution, our framework only maintains one classification model during both
training and inference, and is domain-label-free at test time.

However, since every domain has its domain-specific classification
information, domain alignment in MT-ADA becomes much more challenging. To
address this problem, we focus on improving domain alignment efficacy for MT-
ADA during both training and active sampling. To this end, we first propose
decomposed domain discrimination (D3) for training, which is deployed in
pretraining stage and every domain adapted training step. It decomposes multi-
domain alignment into source-target and target-target domain alignment and
balances them appropriately. Consequently, the well-aligned feature space can
facilitate the encoding of domain-invariant features. Then during active
sampling, we conceptualize domain adaptation as a multi-task learning process
with two objectives: classification and domain alignment. We weight target
images with their contributions to both tasks, which are computed as each
target image’s feature gradients from classification and domain alignment
losses, supervised by corresponding pseudo-labels. The computed weights, named
Gradient Utility (GU), are further combined with KMeans clustering (GU-KMeans)
to select diverse target samples for annotation. GU-KMeans thus is able to
automatically distribute the preset budget among target domains and identify
images that contribute maximally to domain adaptation. Our unified framework,
named D3GU, effectively boosts MT-ADA performance via enhancing domain
alignments during both training and active sampling.

In summary, our contribution is three-fold:

  * •

To address the domain shifts during training in the multi-target setting, we
introduce decomposed domain discrimination (D3) to appropriately achieve
source-target and target-target domain alignments.

  * •

We propose GU-KMeans, an active sampling strategy utilizing gradients from
classification and domain alignment losses to select informative target
images.

  * •

To the best of our knowledge, this is the first work addressing the MT-ADA
problem for image classification. Extensive experiments on Office31 [32],
OfficeHome [41], and DomainNet [29] have demonstrated its state-of-the-art
performance.

##  2 Related work

###  2.1 Unsupervised domain adaptation

To mitigate the domain shifts between source and target domains, modern
unsupervised domain adaptation methods rely on various domain alignment
techniques to align the distributions of source and target features. One line
of research is to design various metrics to decrease the distance between
source and target distributions [26, 25, 5, 43]. It is however difficult to
comprehensively compute distribution disparity with handcrafted metrics. A
more prominent line of work resorted to adversarial domain discriminators to
automatically align domains via min-max optimization [48, 20, 27]. The pioneer
work, DANN [48], included a binary domain discriminator in the network
architecture and reversed its gradient flowing to the feature encoder during
back-propagation. Recent UDA works were mostly built on the domain
discriminator by proposing some improvement in architectures [1, 44, 45],
training pipelines [16], and loss functions [8, 49, 23, 18, 40, 19]. Due to
DANN’s popularity and the convention to use it on ADA [7, 20, 34, 30], our
work is also built upon it to enhance domain alignment.

###  2.2 Multi-target domain adaptation

Real-world applications often require adaptation to multiple target domains.
Research works for domain adaptation on multiple target domains can be
classified into blended-target adaptation and multi-target adaptation,
determined by the availability of target domain labels. When target domain
labels are missing, blended-target domain adaptation methods [15, 50, 47]
merge all targets into one and focus on source-target alignment. In this
paper, we instead study multi-target domain adaptation (MTDA) [13, 28, 39,
31], where target domain labels are provided for training data. Previous MTDA
methods proposed sophisticated training pipelines and architectures, including
decomposed mapping [13], knowledge distillation [28, 39], and curriculum
learning [31]. On the contrary, we propose a simple but effective adversarial
domain discrimination technique to enhance multi-domain alignment, with
minimal modifications to the training pipeline and architecture.

###  2.3 Active domain adaptation

Traditional active learning methods aim to selectively annotate images by
measuring their importance as data uncertainty [33, 11, 37, 36] and diversity
[35, 38, 12]. Recently, active domain adaptation (ADA), which instead actively
annotates a few images from the target domain to address domain shifts, has
received increasing research attention. AADA [20] weighted entropy score with
target probability at the equilibrium state of adversarial training. TQS [7]
jointly considered committee uncertainty, and domainness for active learning.
CLUE [30] integrated entropy score into diversity sampling via KMeans
clustering. SDM [46] boosted active domain adaptation accuracy with the help
of a margin loss. LAMDA [34] selected domain-representative candidates.
However, these methods treated classification and domain alignment as two
independent tasks and failed to explicitly consider their correlations. In
this paper, we leverage the domain discriminator to propose an active sampling
criterion called gradient utility, which explicitly measures each target
image’s contributions to classification and domain alignment.

##  3 Preliminary

###  3.1 Multi-target active domain adaptation

MT-ADA aims at training a single classification model for $K$ classes on $N$
target domains $\\{\mathcal{T}_{i}\\}_{i=1}^{N}$ by transferring
classification knowledge from one labeled source domain $\mathcal{S}$. It
utilizes labeled source images $L_{\mathcal{S}}$ and unlabeled target images
$\\{U_{\mathcal{T}_{i}}\\}_{i=1}^{N}$, then selects and annotates images from
$\\{U_{\mathcal{T}_{i}}\\}_{i=1}^{N}$ at multiple stages to build up labeled
target set $\\{L_{\mathcal{T}_{i}}\\}_{i=1}^{N}$ to join training. As
visualized in Figure 1, the learning process is composed of an initial
unsupervised domain adapted pretraining stage (stage-$0$) followed by $s$
stages of active learning. The same domain adaptation method is used in all
stages for training. The pretraining stage trains a model $\theta_{0}$ with
only $L_{\mathcal{S}}$ and $\\{U_{\mathcal{T}_{i}}\\}_{i=1}^{N}$ in the UDA
manner, since $\\{L_{\mathcal{T}_{i}}\\}_{i=1}^{N}=\emptyset$ initially. At
the $j$-th active learning stage with annotation budget $b_{j}$, an active
sampling algorithm first selects $b_{j}$ images from unlabeled target union
$\\{U_{\mathcal{T}_{i}}\\}_{i=1}^{N}$ for annotation by utilizing the previous
stage’s trained model $\theta_{j-1}$, and then moves the selected images from
unlabeled target set to the labeled target set. Since the selection pool is
the target union, numbers of images selected from each target domain, which
have a sum of $b_{j}$, are automatically determined by the active sampling
algorithm. Through domain adapted training, an improved model $\theta_{j}$ is
thus obtained by finetuning $\theta_{j-1}$ on labeled source set
$L_{\mathcal{S}}$, unlabeled target set $\\{U_{\mathcal{T}_{i}}\\}_{i=1}^{N}$,
and labeled target set $\\{L_{\mathcal{T}_{i}}\\}_{i=1}^{N}$. The active
learning stages repeat until the annotation budget runs out. At test time, we
measure model’s average classification accuracy on all target domains.

###  3.2 Adversarial domain adaptation

Following previous convention [7, 20, 34, 30], we choose DANN [48] for domain
adaptation due to its high efficiency. DANN is the exemplar of adversarial
domain adaptation [48, 10, 27]. Initially designed for single-target domain
adaptation, DANN has been adapted to multi-target domain adaptation by merging
all target domains into a union domain
$\tilde{\mathcal{T}}=\bigcup\\{\mathcal{T}_{i}\\}_{i=1}^{N}$ [39, 28, 42].
Specifically, as Figure 1 shows, the model is composed of three parts: a
backbone feature encoder $F$, a classification head $C$, and a domain
discrimination head $D$. During training, an input image
$\boldsymbol{x}\in\mathcal{R}^{3\times H\times W}$ with height $H$ and width
$W$ from either source domain $\mathcal{S}$ or target domain
$\mathcal{\tilde{T}}$ is first encoded into a feature vector, i.e.,
$\boldsymbol{z}=F(\boldsymbol{x})$, which is then passed to $C$ and $D$
simultaneously. The classification head and a softmax layer predict
$\boldsymbol{z}$ into class probabilities
$\boldsymbol{p}=\text{Softmax}(C(\boldsymbol{z}))\in\mathcal{R}^{K}$. When
$\boldsymbol{x}$ is labeled with class $y$, the classification loss is
computed as:

$\displaystyle\mathcal{L}_{cls}(\boldsymbol{x},y)$
$\displaystyle=-\sum\nolimits_{k=1}^{K}\mathbbm{1}_{[k=y]}\,\text{log}\,\boldsymbol{p}_{k}$
(1)

Parallelly, the domain discrimination branch predicts $\boldsymbol{z}$ into
domain logits $\boldsymbol{\phi}^{d}=D(\boldsymbol{z})\in\mathcal{R}^{2}$,
which is then activated into domain probabilities
$\boldsymbol{q}\in\mathcal{R}^{2}$ by softmax layer. The two channels of
$\boldsymbol{\phi}^{d}$ and $\boldsymbol{q}$ represent source and target
domains, respectively. Suppose the domain label of $\boldsymbol{x}$ is $m$, a
binary domain classification task is formulated with loss:

$\displaystyle\mathcal{L}_{dom}^{bin}(\boldsymbol{x},m)$
$\displaystyle=-\sum\nolimits_{t\in\\{\mathcal{S},\mathcal{T}\\}}\mathbbm{1}_{[t=m]}\,\text{log}\,\boldsymbol{q}_{t}$
(2)

We denote it as binary domain discrimination. To achieve domain alignment, a
domain-invariant feature encoder is needed to confuse the domain
discriminator. Therefore, we adversarially train $D$ to minimize
$\mathcal{L}_{dom}^{bin}$ and $F$ to maximize $\mathcal{L}_{dom}^{bin}$, which
is implemented by reversing the gradients flowing from $D$ to
$\boldsymbol{z}$. In practice, during all training steps,
$\mathcal{L}_{dom}^{bin}$ is computed on all images because domain labels are
always provided, while $\mathcal{L}_{cls}$ is only computed on labeled images.
At test time, only the backbone $F$ and classification head $C$ are kept for
inference.

##  4 Proposed method

###  4.1 Decomposed domain discrimination

Although binary domain discrimination already demonstrates appealing
performance, the missing target domain labels make it focus on source-target
alignment but ignore target-target alignment. As shown in Figure 2, this
results in under-alignment between target domains, the distributions of which
can be quite distinct and cause drastic category mismatch [50]. To ensure
better alignment between multiple domains, all-way domain discrimination [31]
is used to align every domain with all others. By predicting a logit for every
domain, it estimates the probabilities of the input belonging to every domain
out of all $(1+N)$ domains, i.e., $\boldsymbol{q}\in\mathcal{R}^{(1+N)}$. The
corresponding loss function is then changed from Equation 2 to:

$\displaystyle\mathcal{L}_{dom}^{aw}(\boldsymbol{x},m)$
$\displaystyle=-\sum\nolimits_{t\in\\{\mathcal{S},\mathcal{T}_{1},...,\mathcal{T}_{N}\\}}\mathbbm{1}_{[t=m]}\,\text{log}\,\boldsymbol{q}_{t}$
(3)

Similarly, we apply minmax optimization on $\mathcal{L}_{dom}^{aw}$ for
adversarial training. In this way, every domain is aligned with the others,
resulting in a more compact feature space.

However, while domain adaptation assumes most of the classification knowledge
is shared and transferable between domains (“+”), every domain also has its
own domain-specific classification knowledge (“-”), the alignment of which
adversely affects the classification in the shared feature space. This is
known as negative transfer [24, 51]. All-way domain discrimination applies the
same degree of alignment between all domains, thus emphasizing alignment
between unlabeled target domains where classification supervision is missing.
Consequently, it can easily lead to negative transfer and over-alignment, as
in Figure 2.

To mitigate this dilemma, we hereby propose Decomposed Domain Discrimination
(D3) to decompose domain alignment into source-target alignment and target-
target alignment. D3 uses a domain discriminator $D$ that predicts
$(N+2)$-dimensional domain logits
$\boldsymbol{\phi}^{d}\in\mathcal{R}^{(2+N)}$, corresponding to the source
domain $\mathcal{S}$, union target domain $\tilde{\mathcal{T}}$, and each
individual target domains $\mathcal{T}_{1},...,\mathcal{T}_{N}$, sequentially.
It then simultaneously computes $\mathcal{L}_{dom}^{bin}$ on $\mathcal{S}$ and
$\tilde{\mathcal{T}}$ for source-target alignment, and
$\mathcal{L}_{dom}^{aw}$ on $\mathcal{S}$ and
$\\{\mathcal{T}_{i}\\}_{i=1}^{N}$ for target-target alignment. Decomposed
domain discrimination is thus formulated as a weighted sum of the two losses:

$\displaystyle\mathcal{L}_{dom}^{dec}=\mathcal{L}_{dom}^{bin}+\alpha\mathcal{L}_{dom}^{aw}$
(4)

where loss weight for $\mathcal{L}_{dom}^{bin}$ is always $1.0$ to necessitate
and prioritize source-target alignment since annotations mostly come from the
source domain. $\alpha$ is the hyperparameter that controls the balance
between the source-target alignment and target-target alignment to avoid
either under-alignment or over-alignment. We show in experiments that, despite
its simplicity, decomposed domain discrimination effectively facilitates
encoding domain-invariant features and improves domain adaptation performance.

Figure 2: Each domain has its transferable positive knowledge (“+”), and
negative knowledge (“-”) that is domain-specific and cannot be shared. Binary
domain discrimination can lead to under-alignment between positives in the
target domains while all-way domain discrimination leads to over-alignment of
the negatives. Decomposed domain discrimination instead balances the two to
enhance domain alignment.

###  4.2 Gradient utility for sample selection

Traditional active sampling functions consider unlabeled samples’ uncertainty
as the major criterion for selection, i.e., samples closer to the decision
boundaries are preferred. However, domain adaptation is a two-task learning
process, where the classification task aims at learning classification
knowledge from source domain and domain alignment task is responsible for
knowledge transferring to the target domains. Merely applying the uncertainty-
based methods to active domain adaptation, without addressing domain mismatch,
can result in querying a sub-optimal set of samples. Existing active domain
adaptation research therefore considers domain gap [20] and domain
representatives [34]. However, they treat classification and domain alignment
as two independent tasks and fail to consider their correlation.

Figure 3: Gradient utility $\varphi$ measures each target sample’s
contributions towards classification ($\varphi_{cls}$) and domain alignment
($\varphi_{da}$). Target samples with the highest $\varphi$, rather than the
most uncertain ones, are upweighted the most.

We propose a novel technique to address this challenge by leveraging the
domain discriminator. As mentioned in Section 3.2, the encoded feature
$\boldsymbol{z}$ acts like a bridge between two task-specific branches $C$ and
$D$. Our method thus quantifies the contributions of each sample in the
encoded feature space, resulting from the active annotation, by measuring its
gradient utilities towards these two branches.

Assuming the ideal situation where the classification label $y$ of a sample
$\boldsymbol{x}$ is known, we firstly compute the negative gradient of the
classification loss $\mathcal{L}_{cls}$ (supervised by the label $y$) w.r.t.
its encoded feature $\boldsymbol{z}$:

$\displaystyle\tilde{\boldsymbol{g}}_{cls}(\boldsymbol{x},y)$
$\displaystyle=-\frac{\partial\mathcal{L}_{cls}(\boldsymbol{x},y)}{\partial\boldsymbol{z}}$
(5)

which represents the first-order displacement of $\boldsymbol{z}$ in the
feature space with both direction and magnitude, resulting from annotating it.
Larger displacement induces larger changes to the feature space, leading to
more contributions to the classification task. We can quantify such
contribution with gradient utility towards classification, which is the
magnitude of the displacement:

$\displaystyle\varphi_{cls}(\boldsymbol{x},y)$
$\displaystyle=\|\tilde{\boldsymbol{g}}_{cls}(\boldsymbol{x},y)\|_{2}$ (6)

However, $\boldsymbol{x}$’s contribution to domain alignment task cannot be
computed similarly, since the domain label $m$ is already known and active
annotation does not augment any extra supervision. We therefore need another
method to quantify the contribution resulting from annotating it with label
$y$. To this end, we exploit the correlation between the classification and
domain alignment tasks. Any attempt to optimize the performance of one task is
likely to alter the encoded feature representations $\boldsymbol{z}$, which
will affect the performance of the other task. Such task correlation can be
computed by the cosine similarity between the two task gradients w.r.t. the
encoded feature $\boldsymbol{z}$. Formally,

$\displaystyle\tilde{\boldsymbol{g}}_{da}(\boldsymbol{x},\mathcal{S})$
$\displaystyle=-\frac{\partial\mathcal{L}_{dom}(\boldsymbol{x},\mathcal{S})}{\partial\boldsymbol{z}}$
(7) $\displaystyle\text{corr}_{cls-da}(\boldsymbol{x},y,\mathcal{S})$
$\displaystyle=\frac{\tilde{\boldsymbol{g}}_{cls}(\boldsymbol{x},y)\boldsymbol{\cdot}\tilde{\boldsymbol{g}}_{da}(\boldsymbol{x},\mathcal{S})}{\|\tilde{\boldsymbol{g}}_{cls}(\boldsymbol{x},y)\|_{2}\cdot\|\tilde{\boldsymbol{g}}_{da}(\boldsymbol{x},\mathcal{S})\|_{2}}$
(8)

where
$\mathcal{L}_{dom}\in\\{\mathcal{L}_{dom}^{bin},\mathcal{L}_{dom}^{aw},\mathcal{L}_{dom}^{dec}\\}$
depending on which domain discrimination method is used, and
$\tilde{\boldsymbol{g}}_{da}(\boldsymbol{x},\mathcal{S})$ is the negative
gradient of the domain discrimination loss $\mathcal{L}_{dom}$ to align
$\boldsymbol{x}$ with the source domain. Now we can derive target
$\boldsymbol{x}$’s gradient utility towards domain alignment, resulting from
annotating $x$ with label $y$:

$\displaystyle\varphi_{da}(\boldsymbol{x},y)$
$\displaystyle=\varphi_{cls}(\boldsymbol{x},y)\cdot\text{corr}_{cls-
da}(\boldsymbol{x},y,\mathcal{S})$ (9)

which measures how $\boldsymbol{x}$’s displacement in the feature space
induced by classification affects domain alignment. We sum the gradient
utilities towards both classification and domain alignment to form the final
Gradient Utility (GU):

$\displaystyle\varphi(\boldsymbol{x},y)$
$\displaystyle=\varphi_{cls}(\boldsymbol{x},y)+\varphi_{da}(\boldsymbol{x},y)$
(10)

which measures $\boldsymbol{x}$’s contribution to both tasks. We weight
samples using their GUs during active sampling, as illustrated in Figure 3. In
practice, the assumption on the availability of the ground truth label $y$
does not hold when sampling. We therefore use the predicted pseudo-label
instead, which also shows strong empirical performance.

###  4.3 GU-KMeans for diverse sample selection

To further augment the selected target subset with diversity, at the $j$-th
active sampling step, we cluster the unlabeled target images into $b_{j}$
clusters, where $b_{j}$ is the annotation budget mentioned in Section 3.1. The
image closest to the centroid of each cluster is selected for annotation.
KMeans is employed for clustering in our experiments following [30, 3]. The
proposed gradient utilities act as sample weights when computing the
centroids:

$\displaystyle w(\boldsymbol{x})$
$\displaystyle=\text{norm}(\varphi(\boldsymbol{x}))^{\beta}$ (11)

where $w$ is the sample weight and the “norm” operator divides all scores by
the maximum to normalize them into the range $[0,1]$ for computational
stability. A parameter $\beta$ is used as the exponent on the score to
tradeoff gradient utility and diversity in active sampling. The larger $\beta$
is, the more importance we put on gradient utility than diversity. The final
active sampling method is named as GU-KMeans.

##  5 Experiments and results

In the absence of benchmarks for MT-ADA for image classification, we develop
our own codebase on three conventional domain adaptation datasets. We present
experimental setup in Section 5.1, introduce baselines in Section 5.2, analyze
the performance of our framework in Section 5.3, and provide more analysis in
Section 5.4. Additional details and results are in the supplementary material.

###  5.1 Experimental setup

Datasets: We conduct multi-target active domain adaptation experiments on 3
benchmark datasets: Office31 [32], OfficeHome [41], and DomainNet [29].
Office31 consists of $4,652$ images classified into $31$ classes from $3$
domains {amazon, dslr, webcam}. OfficeHome consists of $15,500$ images
spanning $65$ classes and $4$ domains {art, clipart, product, real}. DomainNet
is a large-scale dataset with $0.6$ million images spanning $345$ classes and
$6$ domains {clipart, infograph, painting, quickdrarw, real, sketch}. Since
Office31 and OfficeHome do not provide train and test splits, we follow the
conventional protocol to use all images for both training and testing [46, 30,
34, 7]. We use the official train/test splits in DomainNet for experiments.

Evaluation metric: For each dataset, when taking one domain as the source, we
use all the remaining domains as targets, and report the averaged
classification accuracy over all target domains. For example, on OfficeHome,
reported accuracy corresponding to “art” refers to the average classification
accuracy on clipart, product, and real, when taking art as the source. All
reported results are the average of three random trials, unless otherwise
specified.

Implementation: We use a ResNet-50 [21] pretrained on ImageNet [17, 2] as the
backbone. Encoded feature dimension is $256$ on Office31/OfficeHome and $512$
on DomainNet. The classification head is a single layer neural network and the
domain discriminator is a three-layer neural network with dropout value of
$0.5$. $\alpha$ in Equation 4 is set to $1.0$ on Office31/OfficeHome and
$0.05$ on DomainNet. $\beta$ in Equation 11 is set to $4.0$. We use SGD
optimizer in unsupervised pretraining with learning rate $0.001$ and domain
adapted training with learning rate $0.0003$, both of which are inversely
decayed during training. We conduct $4$ active learning stages with equal
annotation budget at each stage.

Method | Office31 | OfficeHome
---|---|---
budget=30/stage-1 | budget=120/stage-4 | budget=100/stage-1 | budget=400/stage-4
amzn | dslr | web | AVG | amzn | dslr | web | AVG | art | clip | prod | real | AVG | art | clip | prod | real | AVG
random(b) | 85.38 | 81.14 | 81.44 | 82.65 | 92.64 | 85.03 | 85.75 | 87.80 | 63.37 | 62.43 | 56.78 | 63.64 | 61.55 | 68.20 | 68.47 | 61.99 | 68.15 | 66.70
entropy(b) | 89.62 | 81.11 | 81.53 | 84.08 | 98.16 | 86.44 | 87.20 | 90.60 | 63.27 | 63.80 | 57.17 | 63.44 | 61.92 | 70.05 | 71.35 | 65.25 | 70.19 | 69.21
margin(b) | 89.80 | 82.24 | 83.46 | 85.16 | 98.92 | 87.89 | 87.75 | 91.52 | 63.75 | 64.98 | 57.90 | 64.31 | 62.73 | 71.12 | 71.86 | 65.79 | 70.64 | 69.85
coreset(b) | 85.52 | 80.50 | 81.08 | 82.36 | 90.51 | 83.22 | 83.41 | 85.71 | 62.00 | 62.75 | 55.23 | 62.94 | 60.73 | 66.48 | 67.22 | 60.38 | 66.47 | 65.14
BADGE(b) | 89.16 | 81.61 | 82.86 | 84.54 | 98.54 | 86.53 | 87.82 | 90.97 | 64.25 | 64.65 | 57.38 | 64.57 | 62.71 | 70.70 | 72.14 | 64.57 | 70.28 | 69.42
AADA(a) | 89.32 | 80.70 | 82.76 | 84.26 | 98.34 | 86.01 | 87.10 | 90.48 | 63.72 | 64.56 | 57.85 | 64.89 | 62.76 | 70.28 | 71.65 | 64.87 | 70.32 | 69.28
SDM(a) | 89.85 | 81.94 | 83.55 | 85.11 | 99.06 | 87.62 | 88.32 | 91.67 | 64.23 | 64.85 | 58.09 | 65.65 | 63.21 | 71.03 | 72.72 | 65.88 | 70.92 | 70.13
LAMDA(a) | 90.74 | 82.34 | 83.49 | 85.52 | 98.57 | 87.60 | 88.30 | 91.49 | 65.44 | 64.37 | 58.55 | 65.74 | 63.52 | 70.87 | 71.61 | 65.46 | 70.58 | 69.63
CLUE(a) | 90.74 | 84.21 | 84.12 | 86.36 | 97.93 | 87.74 | 88.86 | 91.51 | 66.02 | 65.36 | 59.40 | 65.79 | 64.14 | 71.75 | 71.33 | 64.89 | 70.56 | 69.63
D3GU | 91.14 | 84.23 | 85.16 | 86.84 | 99.16 | 88.55 | 89.29 | 92.33 | 65.96 | 66.53 | 59.29 | 66.30 | 64.52 | 72.36 | 72.65 | 65.75 | 71.43 | 70.55
Table 1: MT-ADA accuracies with total budget 120 and 400 on Office31 and
OfficeHome, respectively. We conducted 4 active learning stages with equal
budgets. Pretraining stage and active learning training stages apply the same
domain discrimination, which is indicated by “(a)”(all-way) and “(b)”(binary)
postfixes in “Method” column. We mark the best results in bold and underline
the second-best ones. The best average results across all the source domains
on each dataset are marked in red. Same annotations are used for Table 2 and
3.

###  5.2 Active sampling baselines

For replication purposes and fair comparison, we re-implemented the following
active sampling baselines on our codebase, including both classical active
sampling methods denoted as AL, and state-of-the-art active domain adaptation
sampling methods denoted as ADA.

Entropy(_AL_) selects samples with the largest entropy scores, i.e.,
$\mathcal{H}(\boldsymbol{x})=-\sum_{k=1}^{C}\boldsymbol{p}_{k}(\boldsymbol{x})\
\text{log}\ \boldsymbol{p}_{k}(\boldsymbol{x})$, where $C$ and
$\boldsymbol{p}$ refer to the number of classes and predicted class
probabilities, respectively.

Margin(_AL_) selects samples with the smallest prediction margins
$\mathcal{M}(\boldsymbol{x})=\boldsymbol{p}_{c_{1}}(\boldsymbol{x})-\boldsymbol{p}_{c_{2}}(\boldsymbol{x})$,
where $c_{1}/c_{2}$ stands for the most/second-most confident predicted
classes.

Coreset [35](_AL_) selects representative target subset with greedy KCenter
clustering.

BADGE [4](_AL_) uses KMeans++ clustering on the classification gradient
embedding space to select the most uncertain and diverse subset.

AADA [20](_ADA_) selects images with the largest augmented entropy scores,
i.e.,
$G(\boldsymbol{x})=\frac{1-\boldsymbol{q}_{\mathcal{S}}(\boldsymbol{x})}{\boldsymbol{q}_{\mathcal{S}}(\boldsymbol{x})}\mathcal{H}(\boldsymbol{x})$,
where $\boldsymbol{q}$/$\mathcal{S}$ refers to predicted domain
probabilities/source domain.

CLUE [30](_ADA_) performs KMeans clustering on all target images with entropy
scores as clustering weights. It however ignores domain shifts.

LAMDA [34](_ADA_) selects a subset of target images whose maximal mean
discrepancy with the unlabeled target set is minimized. Selected images are
representatives of the target domain but do not optimize classification
accuracy.

SDM [46](_ADA_) improves margin sampling by further selecting images with
strong correlations between the loss and the margin sampling function.

###  5.3 Main results

Method | C | I | P | Q | R | S | AVG
---|---|---|---|---|---|---|---
SDM(b) | 43.06 | 43.26 | 43.82 | 40.42 | 42.97 | 46.41 | 43.32
CLUE(b) | 43.79 | 46.48 | 44.41 | 40.03 | 43.01 | 46.71 | 44.07
LAMDA(b) | 44.08 | 46.34 | 44.75 | 40.73 | 43.25 | 46.94 | 44.35
D3GU | 44.40 | 47.23 | 44.82 | 40.90 | 43.23 | 47.13 | 44.62
Table 2: MT-ADA accuracies on DomainNet with total budget ${=}10,000$ in $4$
active learning stages. Capital letters are short for source domains. Refer to
supplementary material for more results.

We compare our unified framework D3GU with baselines on MT-ADA task on
Office31/OfficeHome in Table 1 and DomainNet in Table 2. In Table 3, we
compare different domain adaptation methods at the pretraining stage
(stage-$0$). We mainly compare with ADA baselines since they generally perform
better than AL methods on ADA tasks. Results of ADA baselines are trained with
domain adaptation methods whose empirical performances are better, i.e., all-
way domain discrimination on Office31 and OfficeHome, and binary domain
discrimination on DomainNet, as indicated in Table 3. We include more results
in the supplementary material, including comparison between GU-KMeans and
other sampling algorithms under the same domain adaptations on both MT-ADA and
ST-ADA tasks.

DA | Office31 | OfficeHome | DomainNet
---|---|---|---
amzn | dslr | web | art | clip | prod | real | C | I | P | Q | R | S
binary | 79.83 | 79.83 | 80.37 | 60.65 | 58.33 | 52.92 | 61.70 | 31.28 | 28.28 | 30.62 | 11.07 | 30.99 | 35.75
all-way | 81.12 | 80.43 | 80.62 | 61.50 | 58.80 | 54.57 | 62.70 | 30.28 | 27.90 | 30.59 | 10.59 | 30.25 | 34.11
D3 | 81.13 | 80.16 | 80.66 | 61.79 | 59.26 | 54.77 | 63.07 | 31.43 | 28.46 | 30.69 | 11.43 | 30.75 | 35.59
Table 3: Unsupervised domain adaptation results of $3$ domain adaptations (DA)
on Office31, OfficeHome, and DomainNet.

Performance of D3GU: As shown in Tables 1 and 2, D3GU consistently outperforms
all baselines on almost every MT-ADA setting on three benchmark datasets.
Under the settings where it does not achieve the best results, its accuracies
still stay very close to the best ones. In Table 1, while SDM demonstrates
competitive performance at stage-$4$ (AVG=$91.67/70.13$ on
Office31/OfficeHome), it performs relatively poor at stage-$1$
(AVG=$85.11/63.21$ on Office31/OfficeHome). In comparison, D3GU demonstrates
consistent state-of-the-art performance at both stages. This unanimously
corroborates the promise of D3GU for real-world multi-target active domain
adaptation applications.

Performance of decomposed domain discrimination: As shown in Table 3, neither
binary domain discrimination nor all-way discrimination shows consistent
advantages on all three datasets. On DomainNet, where the $6$ domains and
large number of images make domain shift a much more challenging problem to
solve, all-way domain discrimination loses its advantage and falls behind
binary discrimination by a large margin. This further suggests the existence
of negative transfer. On the contrary, D3 always demonstrates the best or
second-best accuracies, highlighting its superior and robust performance.

Performance of GU-KMeans: As shown in Figure 4, with the same domain
discrimination methods, GU-KMeans still outperforms the other ADA baselines on
all three datasets by a large margin. We also provide more experiment results
in the supplementary material, which further shows GU-KMeans’s state-of-the-
art performance as an active sampling method on Office31, OfficeHome, and
DomainNet.

Figure 4: Plot of MT-ADA accuracies at $4$ active learning stages. We plot
classification accuracies averaged over all source domains on each dataset
(AVG in Tables 1, 2).

###  5.4  Further analysis

Analysis of each component of GU-KMeans: In Table 4, we validate the
effectiveness of each component of GU-KMeans with _real_ as source on
OfficeHome. We first establish two baselines that select target images whose
gradient utility scores ($\varphi_{cls}$ and $\varphi_{cls}+\varphi_{da}$) are
highest, similar to entropy sampling. While entropy score sampling improves
random sampling by a large margin with budget = $400$, gradient utility
towards classification $\varphi_{cls}$ proves to be a more effective way to
measure samples’ contribution to classification (Cls), further improving
entropy sampling significantly in both budget settings. By further considering
samples’ contribution to domain alignment (Da) by introducing $\varphi_{da}$,
performance keeps increasing to $70.84$ with budget = $400$, already
outperforming the second best accuracy $70.64$ of _margin_ sampling as
indicated in Table 1. GU-KMeans further boosts the performance by introducing
KMeans for diversity (Div). These results show the usefulness of each
component of GU-KMeans as an active sampling method.

Ablation on decomposed loss weight parameter: We plot the unsupervised domain
adaptation pretraining results with various $\alpha$ values (Equation 4) in
the top subfigure of Figure 5, with source domain quickdraw on DomainNet.
quickdraw is the most challenging source domain in DomainNet, where the
objects are only outlined by several black lines. With binary discrimination
as the baseline, $\alpha$ around $0.2$ gives the best tradeoff between source-
target alignment and target-target alignment. The larger value of $\alpha$
leads to over-alignment and makes performance worse, with the worst
performance achieved with only all-way discrimination. We report distances
between domains of different $\alpha$ values to quantify domain alignment in
the supplementary material.

Ablation on utility-diversity tradeoff parameter: As shown in the bottom
subfigure of Figure 5, we ablate the utility-diversity tradeoff parameter
$\beta$ (Equation 11), taking art as source domain on OfficeHome. By
introducing vanilla gradient utility as weight, performance of KMeans sampling
is boosted by a large margin ($\beta=1.0$). While the best accuracy is
obtained when $\beta\approx 4.0$, any value of $\beta$ in the range
$[2.0,6.0]$ gives competitive performance.

Analysis of the number of samples selected from each target domain: In Figure
6, taking art as source domain on OfficeHome, we plot the accuracies and the
numbers of selected samples in each target domain at every active learning
stage, with annotation budget being $100$ at each stage. Note that active
sampling always relies on the model trained at previous stage. As shown in the
figure, D3GU tends to select more samples from the domain with worse
performance. We also include selected samples’ distribution visualizations in
the supplementary material.

Method | Cls | Da | Div | b=100 | b=400
---|---|---|---|---|---
random | ✗ | ✗ | ✗ | 63.64 | 68.15
entropy | ✓ | ✗ | ✗ |  63.44-0.20 |  70.19+2.04
$\varphi_{cls}$ | ✓ | ✗ | ✗ |  64.48+1.04 |  70.61+0.42
$\varphi_{cls}+\varphi_{da}$ | ✓ | ✓ | ✗ |  64.90+0.42 |  70.84+0.23
GU-KMeans | ✓ | ✓ | ✓ |  65.00+0.10 |  71.11+0.27
Table 4: Analysis of GU-KMeans with real as source on OfficeHome.“b” is short
for budget. Figure 5: Top: UDA pretrain performances of different $\alpha$
values in Equation 4 with quickdraw as source domain on DomainNet. Bottom:
Effect of $\beta$ values in Equation 11 on GU-KMeans $+$ binary domain
discrimination, with art as source domain on OfficeHome. Figure 6: Accuracy
and selected image counts for each target domain at every active learning
stage using D3GU. Experiments conducted with art as source on OfficeHome.

##  6 Conclusion

In this paper, we propose D3GU, the first framework of its kind to address the
challenging but unexplored task of multi-target active domain adaptation (MT-
ADA) for image classification. We introduced decomposed domain discrimination
to properly align source-target and target-target domains. We proposed an
active sampling strategy that computes the gradient utilities of each target
sample towards classification and domain alignment tasks, and combine it with
KMeans for diversity. Extensive analysis on three datasets corroborates the
promise of D3GU for MT-ADA. We hope our work will motivate the development of
other MT-ADA algorithms for image recognition. When images and domains scale
up, training may require extensive computing resources, impacting the
environment negatively. We leave it as a future work to design more efficient
algorithms.

##  7 Acknowledgement

This research was supported in part by the National Science Foundation under
Grant Number: IIS-2143424 (NSF CAREER Award).

## References

  * [1] Kumar Abhishek, Sattigeri Prasanna, Wadhawan Kahini, Karlinsky Leonid, Feris Rogerio, T. Freeman William, and Wornell Gregory.  Co-regularized alignment for unsupervised domain adaptation.  In NeurIPS, 2018.
  * [2] Paszke Adam, Gross Sam, Massa Francisco, Lerer Adam, Bradbury James, Chanan Gregory, Killeen Trevor, Lin Zeming, Gimelshein Natalia, Antiga Luca, Desmaison Alban, Köpf Andreas, Yang Edward, DeVito Zach, Raison Martin, Tejani Alykhan, Chilamkurthy Sasank, Steiner Benoit, Fang Lu, Bai Junjie, and Chintala Soumith.  Pytorch: An imperative style, high-performance deep learning library.  In NeuraIPS, 2019.
  * [3] Parvaneh Amin, Abbasnejad Ehsan, Teney Damien, Haffari Reza, van den Hengel Anton, and Shi Javen Qinfeng.  Active learning by feature mixing.  In CVPR, 2022.
  * [4] Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal.  Deep batch active learning by diverse, uncertain gradient lower bounds.  2020\.
  * [5] Sun Baochen, Feng Jiashi, and Saenko Kate.  Return of frustratingly easy domain adaptation.  In AAAI, 2016.
  * [6] Xie Binhui, Yuan Longhui, Li Shuang, Liu Chi Harold, Cheng Xinjing, and Wang Guoren.  Active learning for domain adaptation: An energy-based approach.  In AAAI, 2022.
  * [7] Fu Bo, Cao Zhangjie, Wang Jianmin, and Long Mingsheng.  Transferable query selection for active domain adaptation.  In CVPR, 2021.
  * [8] Chen Chao, Chen Zhihong, Jiang Boyuan, and Jin Xinyu.  Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation.  In AAAI, 2019.
  * [9] Gabriela Csurka.  Domain adaptation for visual applications: A comprehensive survey.  In Domain Adaptation in Computer Vision Applications, 2017.
  * [10] Tzeng Eric, Hoffman Judy, Saenko Kate, and Darrell Trevor.  Adversarial discriminative domain adaptation.  In CVPR, 2017.
  * [11] Yarin Gal, Riashat Islam, and Zoubin Ghahramani.  Deep bayesian active learning with image data.  In International Conference on Machine Learning, pages 1183–1192. PMLR, 2017.
  * [12] Yonatan Geifman and Ran El-Yaniv.  Deep active learning over the long tail.  2017\.
  * [13] Behnam Gholami, Pritish Sahu, Ognjen Rudovic, Konstantinos Bousmalis, and Vladimir Pavlovic.  Unsupervised multi-target domain adaptation: An information theoretic approach.  2020\.
  * [14] Rangwani Harsh, Jain Arihant, K Aithal Sumukh, and Babu R. Venkatesh.  S3vaada: Submodular subset selection for virtual adversarial active domain adaptation.  In ICCV, 2021.
  * [15] Yu Huanhuan, Hu Menglei, and Chen Songcan.  Multi-target unsupervised domain adaptation without exactly shared categories.  In arXiv, 2018.
  * [16] Na Jaemin, Jung Heechul, Jin Chang Hyung, and Hwang Wonjun.  Fixbi: Bridging domain spaces for unsupervised domain adaptation.  In CVPR, 2021.
  * [17] Deng Jia, Dong Wei, Socher Richard, Li Li-Jia, Li Kai, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [18] Hu Jian, Zhong Haowen, Yang Fei, Gong Shaogang, Wu Guile, and Yan Junchi.  Learning unbiased transferability for domain adaptation by uncertainty modeling.  In ECCV, 2022.
  * [19] Huang Jiaxing, Guan Dayan, Lu Aoran, Xiao aand Shijian, and Shao Ling.  Category contrast for unsupervised domain adaptation in visual tasks.  In CVPR, 2022.
  * [20] Su Jong-Chyi, Tsai Yi-Hsuan, Sohn Kihyuk, and Buyu Liu.  Active adversarial domain adaptation.  In CVPRW, 2019.
  * [21] He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Sun Jian.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [22] Saito Kuniaki, Kim Donghyun, Sclaroff Stan, Darrell Trevor, and Saenko Kate.  Semi-supervised domain adaptation via minimax entropy.  In ICCV, 2019.
  * [23] Hu Lanqing, Kan Meina, Shan Shiguang, and Chen Xilin.  Unsupervised domain adaptation with hierarchical gradient synchronization.  In CVPR, 2020.
  * [24] Rosenstein Michael T., Marx Zvika, Pack Kaelbling Leslie, and G. Dietterich Thomas.  To transfer or not to transfer.  In NeurIPS workshop on transfer learning, 2005.
  * [25] Long Mingsheng, Zhu Han, Wang Jianmin, and I. Jordan Michael.  Deep transfer learning with joint adaptation network.  In ICML, 2017.
  * [26] Long Mingsheng, Cao Yue, Wang Jianmin, and I. Jordan Michael.  Learning transferable features with deep adaptation networks.  In ICML, 2015.
  * [27] Long Mingsheng, Cao Zhangjie, Wang Jianmin, and I. Jordan Michael.  Conditional adversarial domain adaptation.  In NeurIPS, 2018.
  * [28] Le Thanh Nguyen-Meidine, Atif Belal, Madhu Kiran, Jose Dolz, Louis-Antoine Blais-Morin, and Eric Granger.  Unsupervised multi-target domain adaptation through knowledge distillation.  In WACV, 2021.
  * [29] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In ICCV, 2019.
  * [30] Viraj Prabhu, Arjun Chandrasekaran, Kate Saenko, and Judy Hoffman.  Active domain adaptation via clustering uncertainty-weighted embeddings.  In ICCV, 2021.
  * [31] Subhankar Roy, Evgeny Krivosheev, Zhun Zhong, Nicu Sebe, and Elisa Ricci.  Curriculum graph co-teaching for multi-target domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5351–5360, 2021.
  * [32] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.  Adapting visual category models to new domains.  In ECCV, 2010.
  * [33] Greg Schohn and David Cohn.  Less is more: Active learning with support vector machines.  In ICML, volume 2, page 6. Citeseer, 2000.
  * [34] Hwang Sehyun, Lee Sohyun, Kim Sungyeon, Ok Jungseul, and Kwak Suha.  Combating label distribution shift for active domain adaptation.  In ECCV, 2022.
  * [35] Ozan Sener and Silvio Savarese.  Active learning for convolutional neural networks: A core-set approach.  In ICLR, 2018.
  * [36] Inkyu Shin, Dong-Jin Kim, Jae Won Cho, Sanghyun Woo, Kwanyong Park, and In So Kweon.  Labor: Labeling only if required for domain adaptive semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8588–8598, 2021.
  * [37] Yawar Siddiqui, Julien Valentin, and Matthias Nießner.  Viewal: Active learning with viewpoint entropy for semantic segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9433–9443, 2020.
  * [38] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell.  Variational adversarial active learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5972–5981, 2019.
  * [39] Isobe Takashi, Jia Xu, Chen Shuaijun, He Jianzhong, Shi Yongjie, Liu Jianzhuang, Lu Huchuan, and Wang Shengjin.  Multi-target domain adaptation with collaborative consistency learning.  In CVPR, 2021.
  * [40] Kalluri Tarun, Sharma Astuti, and Chandraker Manmohan.  Memsac: Memory augmented sample consistency for large scale domain adaptation.  In ECCV, 2022.
  * [41] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In CVPR, 2017.
  * [42] Kumar Vikash, Lal Rohit, Patil Himanshu, and Chakraborty Anirban.  Conmix for source-free single and multi-target domain adaptation.  In WACV, 2023.
  * [43] Zellinger Werner, Lughofer Edwin, Saminger-Platz Susanne, Grubinger Thomas, and Natschlager Thomas.  Central moment discrepancy (cmd) for domain-invariant representation learning.  In ICLR, 2017.
  * [44] Chang Woong-Gi, You Tackgeun, Seo Seonguk, Kwak Suha, and Han Bohyung.  Domain-specific batch normalization for unsupervised domain adaptation.  In CVPR, 2019.
  * [45] Gu Xiang, Sun Jian, and Xu Zongben.  Spherical space domain adaptation with robust pseudo-label loss.  In CVPR, 2020.
  * [46] Ming Xie, Yuxi Li, Yabiao Wang, Zekun Luo, Zhenye Gan, Zhongyi Sun, Mingmin Chi, Chengjie Wang, and Pei Wang.  Learning distinctive margin toward active domain adaptation.  In CVPR, 2022.
  * [47] Peng Xingchao, Huang Zijun, Sun Ximeng, and Saenko Kate.  Domain agnostic learning with disentangled representations.  In ICML, 2019.
  * [48] Ganin Yaroslav and Lempitsky Victor.  Unsupervised domain adaptation by backpropagation.  In ICML, 2015.
  * [49] Gao Zhiqiang, Zhang Shufei, Huang Kaizhu, Wang Qiufeng, and Zhong Chaoliang.  Gradient distribution alignment certificates better adversarial domain adaptation.  In ICCV, 2021.
  * [50] Chen Ziliang, Zhuang Jingyu, Liang Xiaodan, and Lin Liang.  Blending-target domain adaptation by adversarial meta-adaptation networks.  In CVPR, 2019.
  * [51] Wang Zirui, Dai Zihang, Poczos Barnabas, and Carbonell Jaime.  Characterizing and avoiding negative transfer.  In CVPR, 2019.


# Stylistic Retrieval-based Dialogue System with Unparallel Training Data

Hao Fu,1 Yan Wang,2 Ruihua Song,3 Tianran Hu,4 Jian-Yun Nie5
1Meituan Group
2Tencent AI Lab
3Gaoling School of Artificial Intelliegent, Renmin University of China
4Computer Science, College of William & Mary
5Department of Computer Science and Operations Research, Universite De
Montreal
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

> The ability of a dialog system to express consistent language style during
> conversations has a direct, positive impact on its usability and on user
> satisfaction. Although previous studies have demonstrated that style
> transfer is feasible with a large amount of parallel data, it is often
> impossible to collect such data for different styles. In this paper, instead
> of manually constructing conversation data with a certain style, we propose
> a flexible framework that adapts a generic retrieval-based dialogue system
> to mimic the language style of a specified persona without any parallel
> data. Our approach is based on automatic generation of stylized data by
> learning the usage of jargon, and then rewriting the generic conversations
> to a stylized one by incorporating the jargon. In experiments we implemented
> dialogue systems with five distinct language styles, and the result shows
> our framework significantly outperforms baselines in terms of the average
> score of responses’ relevance and style degree, and content diversity. A/B
> testing on a commercial chatbot shows that users are more satisfied with our
> system. This study demonstrates the feasibility of building stylistic
> dialogue systems by simple data augmentation.

## Introduction

Many researches show that incorporating role-specific language style makes a
dialogue system more human-like and attractive (?). As shown in Figure 1, in
stylistic dialogue generation, the responses are expected to be not only
relevant to the context, but also consistent with the desired style. Recently,
on the basis of parallel data of an utterance and its stylized version, some
studies show the effect of stylistic dialogue generation to account for
emotions (?), response attitudes (?), personas (?), and gender identification.
(?; ?).

Figure 1: Responses of our chitchat dialogue systems with different character
settings.

Despite their success, previous studies are generally conducted in a fully
supervised setting requiring stylistic dialogue corpus. However, it is
expensive and not always feasible to create such a corpus, due to the lack of
parallel data which consists of many triplets in {style, context, response}
format. For a target style, previous work either looks for specific data
sources such as movie transcripts (?), or recruits human crowd-workers to
create corpora (?). On the other hand, although most state-of-the-art business
chatbots are built upon information retrieval technology, there isn’t any
practical way to train a stylistic retrieval-based dialogue agent. The only
way to prevent the dialogue agent from contradicting its desired style is to
carefully annotate all the corpus and deliberately filter out those
contradicted pairs, which is expensive and inefficient.

In this paper, we present a light-weight approach that enables a generic
retrieval-based dialogue system (which does not embed any persona) to mimic
the language style of a specific role. Our framework only requires minimum
human inputs (e.g., 500 keywords) for each style, and does not need any
paralleled data. Instead of building an entirely new dialogue system for each
style from scratch, our framework functions as a “patch” to stylize a generic
retrieval-based dialogue system. The main idea is to rewrite a generic
dialogue repository into a stylistic one. Specifically, we consider that a
target style is represented by a set of jargon phrases (words that are
frequently used by a particular group), and propose a jargon-based rewriting
schema to tint the responses in dialogue repository with the desired style.

One problem in this rewriting schema is that the jargon in the rewritten
stylistic responses may have never occurred in the original repository, and
the re-ranking model trained on a generic repository may not understand them.
To alleviate this problem, we further propose an alignment process to align
each stylized response to an internal response that substitute the jargon
phrases with synonyms in plain language. In the re-ranking process the model
computes matching degrees based on these internal responses and then the
aligned stylized responses are proposed as the final output.

To evaluate our framework, we implement five dialogue agents with distinct
personas and styles – trendy teenagers who prefer informal expression and
slang, computer junkies who adopt many technical terms in their language,
physics fans who frequently use physics jargon, Pokémon fans who like using
game jargon, and martial arts fans who frequently mention tricks or characters
in martial arts novels. The stylistic dialogues are evaluated by both human
annotators and automatic metrics. Extensive experiments show that our system
properly balances the trade-off between relevance and style intensity, and
yields responses with better comprehensive performance. Online A/B test
indicates that our implemented chatbot with the trendy teenager character
setting improves users engagement.

This paper has three main contributions:

  1. 1.

We endow the capability of responding in specific language style to retrieval-
based dialogue systems. More importantly, this method can be used to patch any
generic dialogue systems for various personas, without any extra fine-tuning
or re-training of dialogue models.

  2. 2.

Our method gets rid of the dependency of paralleled data. Instead we only need
a small set of jargon phrases and synonyms to rewrite the dialogue repository.

  3. 3.

Offline experimental results and online A/B test show the superiority of our
approach, as well as the effect of stylistic response on user engagement.

In the remaining part, we review related work. Then we describe our approach
in details. Experimental results of both offline evaluation and online A/B
test are presented finally.

Figure 2: Framework of a generic dialogue system and our stylized dialogue
system.

## Related Work

### Retrieval-based Dialogue Systems

Recently there has been growing interest in building intelligent chatbots to
complete tasks or chitchat. This paper focuses on retrieval-based dialogue
system that first recall several response candidates from dialogue corpus via
lexical-similarity between context, and then learns a semantic matching
function between the conversation context and response candidates for response
selection (?; ?; ?; ?; ?) Such systems generally produce more fluent and
diverse responses and are widely used in commercial chatbots (?).

### Persona-based Dialogue System

? introduce persona-based dialogue models which capture speaker’s role and
represent it as distributed embeddings. Approaches have been proposed to learn
a persona model from user profiles (?; ?) and personalized dialogue (?; ?; ?;
?; ?). However, most persona-based models focus on the consistency of dialogue
with the persona, but do not address the language style problem, which is an
important aspect of persona-based dialogue..

Some recent studies focus on specific language styles. ? introduce a Star Trek
chatbot that imitates the speaking style of characters in the show. The
chatbot is composed of two models that handle Star Trek style input and
everyday conversation separately. ? propose a set of models for generating
polite dialogues. The models are guided by a politeness classifier in
generating responses. ? first extract a non-stylistic prototype from a generic
dialogue system and then generate stylistic response via GPT2. ? and ? attempt
to using monolingual stylistic data to increase the style intensity of
dialogue response. We notice that all these studies focus on a single style,
and are difficult to extend to other styles. Compared to these methods, our
framework is retrieval-based, and can be used cover for a broader range of
personas due to its light-weight nature.

### Text Style Transfer

Style transfer of text is a related task of stylizing dialogue systems. The
task is transferring a sentence to a specified language style, while keeping
the content of the sentence unchanged. A representative approach is learning
disentangled latent representations of content and style. ? propose a cross-
aligned auto-encoder with shared content and separated style representations.
? adversarially combine variational auto-encoder and style classifier. Several
studies leverage adversarial networks with different discriminators and
training objectives (?; ?; ?; ?; ?).

Another approach separates content and style via explicit sequence editing. ?
show that style transfer can be largely achieved by lexical substitution. ?
propose a “delete, retrieve, and generate” framework that replaces style-
related words and re-generate the sentence for better fluency. Similar
approaches based on pre-trained language model or reinforcement learning are
also explored (?; ?; ?; ?; ?).

Our approach to stylize dialogue is consistent with the above studies (e.g.
with the observation of ? on lexical substitution). However, we do not employ
a heavy mechanism to implement the transfer but exploit a small set of style-
related terms to generate stylized sentences. We compare with (?) and (?) in
experiments.

Style |  Jargon phrase |  Synonym
---|---|---
Trendy Teenager |  极好的 (snatched) |  really good
智商税 (stupid tax) |  pay for ignorance
Computer |  access token |  key
transfer learning |  learn by analogy
Physics |  polarization |  single direction
sonic boom |  very fast
Pokémon |  fire blast |  burn
Squirtle |  turtle
Martial Arts |  裘千尺 11footnotemark: 1 |  an antagonist
(Qiu Chianchi) |
Table 1: Samples of annotated parallel phrases.

## Our Proposed Framework

### Main Idea

Our proposed framework is shown in Figure 2. To construct a stylized
retrieval-based dialogue system, our main idea is to rewrite the source corpus
$\mathcal{X}$ to a stylized corpus $X^{Y}$. The rewriting is based on a jargon
- synonym set that contains about 500 jargon phrases and their synonyms in
plain language (annotated by human). The first step is Response Rewriting that
transforms every response $r$ in $\mathcal{X}$ to a stylized response $r^{Y}$
with jargon phrases via a style transfer model. After response rewriting, two
main challenges still remain: 1) The ranking model trained on a generic
repository may not understand the stylized responses $r^{Y}$, and 2) the
stylized responses $r^{Y}$ may be irrelevant to the original context $c$. To
alleviate these problems, we further propose two modules, Response Alignment
module and Context Rewriting module, respectively. The former attempts to
align $r^{Y}$ to a synonymous response $r^{\prime}$ in plain language through
the jargon - synonym set, so that the reranking are based on $r^{\prime}$
instead of the $r^{Y}$. For the latter, we rewrite the context of $c$ to
$c^{\prime}$ by adding some context words that are related to the jargon so
that $c^{\prime}$ is more relevant to the stylized response. For example, for
the jargon “Markov chain”, we rewrite its origin context “Isn’t life hard for
everyone?” to “Isn’t life hard for everyone? predict; history; trend;
uncertain; short-term”.

After Rewriting and Alignment, we can “patch” a generic retrieval-based
dialogue system to adapt new language styles. As shown in Figure 2, after
replacing the all context-response pairs $(c,r)$ in a generic conversation
repository $\mathcal{X}$ to stylized pairs $(c^{\prime},r^{\prime})$, the
dialogue system searches for stylized contexts $c^{\prime}$ and selects a
synonymous response $r_{i}^{{}^{\prime}}$ as an intermediate output. Then, the
corresponded stylized response $r_{i}^{Y}$ is used as the response of input
conversation.

### Jargon phrases & synonyms

We collect a set of jargon phrases from topic-specific websites and ask human
experts to annotate them with synonyms in plain language. For instance, for
the trendy teenager persona, we collected a list of Internet slang words from
Fanjian222http://www.fanjian.net/, a website maintaining a large database of
Chinese Internet slang. We took a slang word as a jargon phrase $s^{Y}$. For
each slang word, three human annotators were asked to assign a synonymous
phrase $s^{X}$ in plain language. The annotators reviewed and discussed each
other’s annotation until they reached an agreement. Insulting slang words were
removed. Samples of the phrases are presented in Table 1. The synonym phrases
mostly explain certain aspects of the jargon phrases in plain language. We
collected on average 523 parallel jargon phrase - synonym pairs per style.

## Approach

In this section, we introduce detailed procedures of Response Rewriting,
Response Alignment, and Context Rewriting, given a set of jargon phrases and
synonyms. The specified language style is implemented in the first step. The
second step guarantees the synonymity of $r^{\prime}$and $r^{Y}$. The third
one fixes the relevance between $c^{\prime}$ and $r^{\prime}$. We further
describe rewriting response and rewriting context in details.

### Response Rewriting

In this section, we introduce detailed procedures of rewriting responses by
jargon phrases. Given an original response $r$, the goal of rewriting is to
obtain a fluent and stylized response $r^{Y}$. A trivial way to achieve the
goal is letting $r^{Y}$ be a common sentence like “I am multi-threaded” for
any $r$. To avoid this, we require that $r^{Y}$ has as many overlapping words
with $r$ as possible so that it is more specific. We generate a number of
candidates for $r^{Y}$, and then filter out ungrammatical ones by a fine-tuned
BERT model.

#### Generating Candidate Response

Given a response $r=\\{w_{1},w_{2},...,w_{n}\\}$, which is a sequence of
words, we select a sub-sequence $s_{i,j}=\\{w_{i},w_{i+1},...,w_{j}\\}$ and
substitute it with a jargon phrase $s^{Y}$. The resulting response
$r^{Y}=\\{w_{1},...,w_{i-1},s^{Y},w_{j+1},...,w_{n}\\}$ is proposed as a
candidate for further filtering.

This process is similar to that used in existing style transfer models (?; ?;
?). These models substitute style-related words and keep others, so that the
content is unchanged. To increase the coverage, we allow arbitrary words to be
substituted. This relaxation yields $\mathcal{O}(n^{2})$ possible
substitutions, so we adopt the following constraints for pruning:

  * •

##### Word constraint:

We only consider words and numbers for substitution. Punctuation marks are not
substituted, because they play an important role in grammar. This constraint
avoids generating grammatical errors by the change of punctuation.

  * •

##### Length constraint:

We try to keep $s_{i,j}$ short, so that most words of $r$ are preserved. This
choice is made in order to avoid altering too much the initial sentence.
Specifically, we limit the length of $s_{i,j}$ to be less than 5 words.

  * •

##### Context constraint:

We also require the sub-sequence $s_{i,j}$ to share the same preceding or
succeeding words with a jargon phrase. For example, suppose we find sentences
like “… is fully multi-threaded …” in the stylized corpus $\mathcal{T}$. We
may consider applying the substitution only if $s_{i,j}$ succeeds the words
“is fully”. This constraint helps filter out most of obviously ungrammatical
candidates.

#### Response Filtering

We further filter stylized response candidates that are not fluent via a
language model. Since a stylized response candidate is not an entirely new
sentence, we only need to estimate the fluency of the substituted part.

We start with the pre-trained BERT model (?), and fine-tune it with a non-
conversational stylized corpus $\mathcal{T}$. We mask the jargon phrases in
$\mathcal{T}$ and train the model to recover them. Then we use this model to
predict the probability of $s^{Y}$ in a given $r^{Y}$: $p(s^{Y}|r^{Y})$. The
candidate responses with low $p(s^{Y}|r^{Y})$ will be filtered.

### Aligning Responses

After the Response Rewriting process, it is possible that the generated
stylized response $r^{Y}$ is not synonymous to the original response $r$.
Moreover, the re-ranking model in the generic dialogue system may not
understand $r^{Y}$ as well. To alleviate these problems, we substitute
stylized phrases in $r^{Y}$ with their synonyms in plain language. That is,
for each $s^{Y}$ in $r^{Y}$, we substitute $s^{Y}$ with the synonymous
$s^{X}$. The resulting synonymous response $r^{\prime}$ is referred to as the
modified response. Such direct substitution ensures the synonymity of
$r^{\prime}$ and $r^{Y}$. Note that $r^{\prime}$ could be ungrammatical. This
is acceptable because $r^{\prime}$ is only used internally by the dialogue
system to match with user input and is not shown to users.

### Context Rewriting

Now we have pairs of synonymous response $r^{\prime}$ and stylized response
$r^{Y}$, and a remaining problem is that the $r^{\prime}$ and $r^{Y}$ may be
totally irrelevant to the original context $c$ after style transfer. So we
attempt to modify $c$ to a more relevant one, $c^{\prime}$, regarded as the
stylized context. To obtain $c^{\prime}$, we select keywords similar to the
stylized phrase $s^{Y}$ to append to $c$. For example, given $s^{y}$=“being
multi-threaded”, the stylized context $c^{\prime}$ could be [$c$ \+
“simultaneously” + “moreover”]. We considered several choices of the keywords,
including frequently co-occurred words, nearest words in word embedding space,
and simply $s^{X}$ itself. We found fastText embedding (?) yields more
reasonable keywords. We choose the top 5 words with the highest cosine
similarity to the stylized phrase $s^{Y}$.

## Evaluation

We conducted both offline experiment and online A/B test to evaluate the
performance of our framework. Each system is given an utterance in the testing
set as input. The output response is evaluated in terms of its quality.

### Dataset333All datasets will be released.

To train a stylized dialogue system, our framework does not need paralleled
data. Instead, we need a generic dialogue corpus $\mathcal{X}$, a stylized
non-conversational corpus $\mathcal{T}$, and stylized jargon phrase -
synonymous pairs.

##### Source dialogue corpus:

We use the dialogue corpus collected by (?) from Sina Weibo, a popular
twitter-like service in China. The contexts are extracted from posts. Comments
on the posts are taken as responses. The corpus covers various topics from
different users, so we consider it to be in a generic language style. We
normalized the tokens and removed non-content words like “retweeted from”.
Contexts and responses were tokenized by
Jieba444https://github.com/fxsjy/jieba toolkit. We sampled 100 posts for
testing, and took the remaining 4.4 million context-response pairs as the
source dialogue corpus.

##### Stylized non-conversational corpus:

Stylized non-conversational corpus are used for Response Filtering only. For
each stylized phrase, we searched Sina Weibo and retrieved the first 500
matched posts. Comments were not retrieved. The posts are pre-processed
similarly to the source dialogue corpus. We collected five corpora for the
Internet slang style and the computer style.

##### Jargon phrase - synonymous pairs:

As mentioned before, We collected on average 523 parallel jargon phrase -
synonym pairs per style.

| Trendy Teenager | Computer | Physics | Pokémon | Martial Arts | Average
---|---|---|---|---|---|---
|  R |  S |  D1 |  D2 |  R |  S |  D1 |  D2 |  R |  S |  D1 |  D2 |  R |  S |  D1 |  D2 |  R |  S |  D1 |  D2 |  R |  S |  D1 |  D2 |  RSA
Generic |  1.3 |  0.7 |  .51 |  .84 |  1.6 |  0.1 |  .51 |  .84 |  1.6 |  0.0 |  .51 |  .84 |  1.4 |  0.0 |  .51 |  .84 |  1.3 |  0.1 |  .51 |  .84 |  1.45 |  0.17 |  .51 |  .84 |  0.81
MTask+MMI |  1.1 |  0.7 |  .19 |  .40 |  1.0 |  0.1 |  .19 |  .38 |  1.1 |  0.0 |  .21 |  .40 |  1.0 |  0.0 |  .19 |  .39 |  1.1 |  0.0 |  .19 |  .37 |  1.07 |  0.16 |  .19 |  .39 |  0.62
CAAE-direct |  0.6 |  1.6 |  .28 |  .48 |  0.1 |  1.7 |  .19 |  .41 |  0.2 |  1.0 |  .25 |  .53 |  0.4 |  1.5 |  .27 |  .51 |  0.3 |  1.1 |  .25 |  .48 |  0.32 |  1.36 |  .25 |  .48 |  0.84
CAAE-patch |  0.8 |  1.0 |  .32 |  .51 |  0.2 |  0.5 |  .15 |  .30 |  0.3 |  0.1 |  .21 |  .43 |  0.4 |  0.3 |  .27 |  .50 |  0.4 |  0.3 |  .21 |  .50 |  0.42 |  0.45 |  .23 |  .45 |  0.44
DelRetGen-direct |  0.9 |  1.7 |  .41 |  .77 |  0.3 |  1.9 |  .38 |  .74 |  0.5 |  1.2 |  .47 |  .80 |  0.7 |  1.6 |  .45 |  .77 |  0.7 |  1.3 |  .43 |  .76 |  0.62 |  1.53 |  .43 |  .77 |  1.08
Ours-direct |  1.1 |  1.5 |  .55 |  .84 |  1.1 |  0.9 |  .55 |  .84 |  1.0 |  0.8 |  .56 |  .84 |  1.0 |  1.2 |  .57 |  .83 |  0.9 |  1.0 |  .56 |  .84 |  1.00 |  1.10 |  .56 |  .84 |  1.05
Ours-patch |  1.1 |  1.7 |  .50 |  .81 |  0.7 |  1.8 |  .47 |  .81 |  0.9 |  1.3 |  .49 |  .83 |  0.9 |  1.7 |  .50 |  .83 |  0.8 |  1.2 |  .50 |  .84 |  0.86 |  1.53 |  .49 |  .82 |  1.20
Table 2: Evaluation results of stylized dialogue. R: relevance, S: style, D1:
distinct-1, D2: distinct-2, *-m: simplified variant without modifying response
or context

### Experiment Settings

##### Generic dialogue system:

We built a retrieval-based dialogue system as the start point for mimicking
different styles. We use Elasticsearch (?) to build a conversation repository
containing the source dialogue corpus $\mathcal{X}$. Given an input utterance,
we retrieve the top 100 matched contexts and their paired responses. A pre-
trained dialogue model selects a proper response as output. The dialogue model
was trained with a different corpus other than the corpus $\mathcal{X}$ used
in this paper.

##### Our approach:

We “patch” the generic dialogue system with the proposed method. For each
$(c,r)$ pair in $\mathcal{X}$, we try to rewrite it to obtain stylized
responses. If the rewriting fails, we simply copy the pair, i.e., let
$c^{\prime}=c$ and $r^{Y}=r^{\prime}=r$.

For ablation study, we consider a simplified variant, where the generic
dialogue system is not patched and its output response $r$ is directly
rewritten as $r^{Y}$. We denote our approach and the simplified variant as
Ours-patch and Ours-direct.

##### Baselines:

We compare with a generation-based approach by ?. It uses multi-task learning
to personalize a dialogue model. We train the model with our dialogue corpus
$\mathcal{X}$ and non-conversational corpus $\mathcal{T}$. We improve its
diversity by interpolating the MMI model by ?. We set its encoder and decoder
to be 2-layer LSTMs with 500 hidden units. Its vocabulary (60k) covers words
in $\mathcal{X}$ and $\mathcal{T}$. We refer to this approach as MTask+MMI.

We also consider leveraging text style transfer models. Given an output
response $r$ of the generic dialogue system, we use a style transfer model to
transform $r$ to a stylized response. Two representative models are used, the
cross-aligned auto-encoder by ? and the “delete, retrieve, and generate” model
by ?. We denote the two approaches as CAAE-direct and DelRetGen-direct. In
addition, we consider integrating the style transfer models into our
framework. We replace the procedures of Response Rewriting with one of the
style transfer models. We keep the remaining procedures and “patch” the
generic dialogue system accordingly. This approach is denoted as CAAE-
patch555We tried to optimize the original implementation of DelRetGen, but
still can not rewrite the corpus in a reasonable time..

### Metrics

We consider three criteria for response’s quality: relevance, style degree,
and content diversity.

##### Relevance:

Relevance refers to the extent of a response being proper for an input
utterance. We follow the same guideline as ? to rate Relevance on a scale of 0
to 2.

##### Style degree:

style degree refers to how likely the specified language style can be
identified. We rate style degree on a scale of 0 to 2.

  * •

Strong style (2): Obvious usage of words in the specified language style can
be found.

  * •

Weak style (1): Two cases are included: (a) Stylized words are used, but it is
still similar to everyday conversation, e.g., “What’s the name of this
software?” (b) Common words are used, but they have special meaning for the
target persona. For example, “I have an abstract girlfriend”, where “abstract”
has a specific meaning in programming.

  * •

No style (0): The response is not fluent or does dot show any identifiable
style.

##### Diversity:

Diversity of response is recognized as an important metric of dialogue systems
(?). Dialogue systems with good diversity produce more interesting responses
on various topics. We measure the diversity of responses by the number of
distinct unigrams and bigrams divided by total number of words in the
responses, i.e., Distinct-1 (D1) and Distinct-2 (D2).

For each style, we obtained 700 responses from the 100 testing input
utterances by the evaluated systems. An input utterance and a shuffled list of
responses were presented to 3 annotators each time. The annotators were asked
to rate each response according to the above guideline.

To measure agreement, we use a generalized form of Scott’s pi coefficient with
linear weighting (?). The coefficients indicate fair agreement among ratings
(0.37 and 0.44 for relevance; 0.56 and 0.79 for style degree).

User Query |  好久没吃了，好想吃哇
---|---
Haven’t eat it for a long time and really want to eat it.
Trendy Teenagers |  好久没吃了啊，减肥不存在的，能不诱惑我吗？
Haven’t eat it for a lone time. Losing weight is non-existent. Could you not
tempt me?
Computer Junkies |  想吃，贪婪算法告诉我怎么吃吗？
Want to eat. Would greedy algorithms tell me how to eat?
Physics Fans |  好饿啊。看到这汉堡就控制运动速度。
So hungry. I have to control the movement velocity when I see this hamburger.
Pokémon Fans |  哈哈，我爱舔舌头吃！
Haha, I love eating by lick.
Martial Arts Fans |  不是好久没吃过，苏荃从来都没吃过…
Not for a long time, Su Quan666A character in the novel series The Legend of
Lu Xiaofeng.has never eaten it.
Figure 3: Sample responses by our approach.

### Evaluation Result

Table 2 shows evaluation results of stylized dialogue. The average of
relevance and style degree ratings (denoted as RSA) is included for overall
performance. The multi-task approach MTask+MMI does not produce stylized
response as expected. We attribute it to the low overlap between the dialogue
corpus and the target corpus. On average, only 81.4% words of the target
corpus (excluding stop words) appear in the vocabulary of dialogue corpus. The
decoder fails to correlate non-overlapping words with the utterance’s
encoding, even after we extended its vocabulary.

For CAAE-patch, its relevance increases after modifying context and response,
but its style degree decreases dramatically. The average lengths of response
are respectively 7.4 and 5.7 tokens for CAAE-direct and CAAE-patch. The root
cause for this behavior is the low content diversity. The model tends to
produce short and common responses. The dialogue model has to choose a “safe”
but more relevant one, when no other relevant and informative response is
available.

Our simplified approach (Ours-direct) achieves a higher relevance and a lower
style degree than our complete approach (ours-patch). Among the responses
produced by the base system, the simplified approach fails to transfer any
response for 25% of queries, whereas ours-patch succeed to return at least one
stylized responses for all queries. If we count the queries on which both
approaches succeed, the relevance is 0.78 and 0.86, and the style degree is
1.43 and 1.52, for the simplified and the complete approaches. The diversity
remains similar. This strongly supports our idea of distinguishing
transferring style from keeping meaning unchanged does work well. In general,
our complete approach (ours-patch) achieves the best overall performance in
terms of RSA (t-tests with $p<10^{-4})$. Figure 3 illustrates our approach’s
responses.

Figure 4: Relevance and style degree change when we trigger different ratios
of stylized responses

### Changes with Trigger Rate

There is a trade-off between style degree and relevance. When we allow more
stylized responses to return during conversations, a chatbot has more chance
to exhibit as its style. However, there is a risk of lowering the relevance.
Thus we use trigger rate as our variant and draw curves to show how relevance
and style degree change at the same time.

Based on our human labeled data, we simulate the results when different
trigger rates are applied. For example, given a style, e.g., trendy teenagers,
we set a threshold of the ensemble regression model output to filter out 90%
stylized responses. Thus we get 10% trigger rate. Then we average the
relevance score and the style degree as the corresponding evaluation results,
e.g., 1.3 and 1.8 for the trendy teenager style. Similarly, we can filter out
different ratio of stylized responses and get relevance and style degree
scores at different trigger rate. Figure 4 shows us the curves for five
styles.

Take the trendy teenager style as an example. We observe that relevance
remains high until about 15% stylized responses are triggered. Then relevance
goes down gradually. Even if all stylized responses are triggered, the
relevance score is around 1.1, which means above fair. The style curve has a
similar trend and the highest style degree is achieved when 20% stylized
responses are triggered. At that position, the relevance score is close to the
maximum. Therefore, if our goal is to add some sense of style while keeping
high relevance, we can choose the trigger rate of 20%. It means that in almost
one fifth turns users can get the stylized responses. That is a proper ratio
as even a real trendy teenager does not show her/his style at all utterances.

As Figure 4 shows, other styles have similar trends. We can find a proper
trade-off between relevance and style degree in the curves. Our approach
provides the flexibility to adjust a dialogue system according to the
requirements of relevance and style degree. Some styles may be more difficult
than some other styles to transfer. For example, the computer style has lowest
relevance. Although the style degree is as high as 1.8, the relevance score is
only 0.7 when the trigger rate is 10%. This indicate that there is a big gap
between the dialogue corpus and the stylized non-conversational corpus for the
computer style. We guess that Weibo users like publishing content about their
life, more than about techniques.

| F | S | D1 | D2 | FSA
---|---|---|---|---|---
Source | 1.86 | 0.18 | .50 | .84 | 1.02
CAAE | 1.09 | 1.30 | .14 | .32 | 1.20
DelRetGen | 1.01 | 1.41 | .41 | .77 | 1.21
Ours | 1.43 | 1.19 | .53 | .84 | 1.31
Table 3: Evaluation results of transferred responses by single models. LM:
Language Model

### Analysis of Style Transfer

In this study, we propose a first study to transfer a generic corpus to a
stylized one. To evaluate how effective our transfer approach is, we further
compare our style transfer model with two baseline methods, CAAE and
DelRetGen. The evaluation metrics of style transfer are similar to that of
stylized dialogue, and we only change the relevance metric to fluency metric.
The fluency metric is rated on a scale of 0 to 2 that represent completely
non-sense (0), understandable with minor errors (1), and fluent without error
(2), respectively.

Table 3 shows the evaluation results. As expected, source responses are rated
with perfect fluency and low style degree. They slightly exhibit Internet
slang’s style, because the dataset is collected from social media. On average,
our approach achieves much better fluency and diversity but lower style degree
compared to the two baselines. CAAE tends to generate “safe” sentences which
make use of only a few target style phrases. To see this, we counted distinct
target style phrases in transferred responses. The averaged counts are
respectively 81, 172, and 181, for CAAE, DelRetGen, and our approach.
DelRetGen improves diversity by involving more target style phrases than CAAE.
However, its generation step tends to generate frequent words, limiting its
overall diversity. In general, our approach outperforms the baselines on
overall performance (FSA, D1, and D2; t-tests with $p<10^{-4}$).

### Online Evaluation

Compared to the generic dialogue system, our framework (as well as other
stylized systems) produces more stylized but less relevant responses. We
wonder if it has a positive or negative impact on user satisfaction. We
conducted A/B test on a commercial social chatbot during one month. We select
the trendy teenager style as our testbed.

We randomly selected a fraction of users as the treatment group. They were
served with the patched system. In case where the patched system fails to find
a relevant response, we fallback to the generic system. Users in the control
group were always served with the generic system. After the A/B test, we
collected 12K conversations for the treatment group. A conversation is a
sequence $\\{(u_{1},r_{1}),(u_{2},r_{2})\\}$, where $u$ is a user utterance
and $r$ is a response. We let $r_{1}$ be the patched system’s response. The
same number of conversations were sampled from the control group.

We consider two automatic metrics to measure user satisfaction. The first
metric is the sentiment of user’s follow-up utterance. Positive sentiment
reflects better user satisfaction. We use a sentiment classifier (?) to
classify the follow-up utterance $u_{2}$. The classifier outputs labels of
$\\{-1,0,1\\}$, for negative, neutral, and positive sentiment. We take the
average label for each group as Sentiment in Table 4. The second metric is the
fraction of overlapping words between $r_{1}$ and $u_{2}$ (See Automatic
follow-up in Table 4). It indicates whether the user is following $r_{1}$ or
starting a new topic.

We also use human annotation for evaluation. We sampled 250 conversations from
each group, and asked 5 human annotators to rate user’s satisfaction on a
scale of -2 to +2:

  * •

Satisfied (+2) The user is satisfied with the conversation and shows positive
emotion.

  * •

Follow (+1) The user understands the response $r_{1}$ and follows the topic.

  * •

Neutral (0) The user keeps chatting but to ignore $r_{1}$.

  * •

Not-follow (-1) The user is confused about $r_{1}$ or simply exits the
chatbot.

  * •

Unsatisfied (-2) The user is upset about the conversation and shows negative
emotion.

Table 4 shows the comparison in the above metrics. The patched system performs
consistently better than the baseline system across all metrics.

| Treatment | Control
---|---|---
Sentiment | 0.0250 | 0.0217
Automatic follow-up | 0.1120 | 0.10445
Human rating | 0.488 | 0.436
Table 4: Evaluation result of online A/B test.

## Conclusion and Future Work

We propose a light-weight “patching” framework that enables a retrieval-based
dialogue system to mimic the language style of a specified character setting.
Our framework is flexible and less expensive to apply. Our approach is based
on transforming a generic dialogue corpus to a stylized corpus. We design a
series of strategies to implement the transformation. Five character settings
are implemented. The experiments show that, with several hundreds of manually
annotated jargon phrases, our approach is able to produce fluent, relevant,
stylized, and diverse responses. A/B test indicates that users are more
satisfied about the stylized system.

As future work, we plan to improve relevance further and experiment with
combining two or more styles in one chatbot. Theoretically our proposed
framework is additive, because it provides a way to put different but non-
contradictory personalities together. For example, a chatbot could be a
computer junkie and a martial arts fan at the same time. When both character
settings have response candidates, we can choose one of them by pre-defined
priors. It would be interesting to implement such combination and test its
effectiveness.

## References

  * [1]
  * [Bojanowski, Grave, Joulin, and Mikolov Bojanowski et al.2017] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ 5 (2017).
  * [Devlin, Chang, Lee, and Toutanova Devlin et al.2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * [Fu, Zhou, Chen, and Li Fu et al.2019] Yao Fu, Hao Zhou, Jiaze Chen, and Lei Li. 2019\.  Rethinking Text Attribute Transfer: A Lexical Analysis.  (2019).
  * [Fu, Tan, Peng, Zhao, and Yan Fu et al.2018] Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018.  Style transfer in text: Exploration and evaluation. In _The 32nd AAAI Conference on Artificial Intelligence_.
  * [Gormley and Tong Gormley and Tong2015] Clinton Gormley and Zachary Tong. 2015.  _Elasticsearch: the definitive guide: a distributed real-time search and analytics engine_.  ” O’Reilly Media, Inc.”.
  * [Goyal, Srinivasan, Anandhavelu, and Sancheti Goyal et al.2021] Navita Goyal, Balaji Vasan Srinivasan, N Anandhavelu, and Abhilasha Sancheti. 2021.  Multi-Style Transfer with Discriminative Feedback on Disjoint Corpus. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 3500–3510.
  * [Gwet Gwet2014] Kilem L Gwet. 2014\.  _Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters_.  Advanced Analytics, LLC.
  * [Herzig, Shmueli-Scheuer, Sandbank, and Konopnicki Herzig et al.2017] Jonathan Herzig, Michal Shmueli-Scheuer, Tommy Sandbank, and David Konopnicki. 2017.  Neural response generation for customer service based on personality traits. In _Proceedings of the 10th International Conference on Natural Language Generation_.
  * [Hu, Yang, Liang, Salakhutdinov, and Xing Hu et al.2017] Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017.  Toward controlled generation of text. In _Proceedings of the 34th International Conference on Machine Learning_.
  * [Jena, Vashisht, Basu, Ungar, and Sedoc Jena et al.2017] Grishma Jena, Mansi Vashisht, Abheek Basu, Lyle Ungar, and Joao Sedoc. 2017.  Enterprise to computer: Star trek chatbot.  _arXiv preprint arXiv:1708.00818_ (2017).
  * [John, Mou, Bahuleyan, and Vechtomova John et al.2019] Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019.  Disentangled Representation Learning for Non-Parallel Text Style Transfer. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * [Krishna, Wieting, and Iyyer Krishna et al.2020] Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020\.  Reformulating Unsupervised Style Transfer as Paraphrase Generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
  * [Li, Galley, Brockett, Gao, and Dolan Li et al.2016a] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a.  A Diversity-Promoting Objective Function for Neural Conversation Models. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * [Li, Galley, Brockett, Spithourakis, Gao, and Dolan Li et al.2016b] Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016b.  A Persona-Based Neural Conversation Model. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_.
  * [Li, Jia, He, and Liang Li et al.2018] Juncen Li, Robin Jia, He He, and Percy Liang. 2018\.  Delete, Retrieve, Generate: a Simple Approach to Sentiment and Style Transfer. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * [Li, Li, Mou, Jiang, Lyu, and King Li et al.2020] Jingjing Li, Zichao Li, Lili Mou, Xin Jiang, Michael R Lyu, and Irwin King. 2020\.  Unsupervised Text Generation by Learning from Search. In _Advances in neural information processing systems_.
  * [Liu, Neubig, and Wieting Liu et al.2021] Yixin Liu, Graham Neubig, and John Wieting. 2021.  On Learning Text Style Transfer with Direct Rewards. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 4262–4273.
  * [Luan, Brockett, Dolan, Gao, and Galley Luan et al.2017] Yi Luan, Chris Brockett, Bill Dolan, Jianfeng Gao, and Michel Galley. 2017.  Multi-Task Learning for Speaker-Role Adaptation in Neural Conversation Models. In _Proceedings of the 8th International Joint Conference on Natural Language Processing_.
  * [Luo, Huang, Zeng, Nie, and Sun Luo et al.2019] Liangchen Luo, Wenhao Huang, Qi Zeng, Zaiqing Nie, and Xu Sun. 2019.  Learning personalized end-to-end goal-oriented dialog. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * [Madotto, Lin, Wu, and Fung Madotto et al.2019] Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, and Pascale Fung. 2019.  Personalizing dialogue agents via meta-learning. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * [Niu and Bansal Niu and Bansal2018] Tong Niu and Mohit Bansal. 2018.  Polite Dialogue Generation Without Parallel Data.  _Transactions of the Association for Computational Linguistics_ 6 (2018).
  * [Shang, Lu, and Li Shang et al.2015] Lifeng Shang, Zhengdong Lu, and Hang Li. 2015.  Neural Responding Machine for Short-Text Conversation. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_.
  * [Shen, Lei, Barzilay, and Jaakkola Shen et al.2017] Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017\.  Style transfer from non-parallel text by cross-alignment. In _Advances in neural information processing systems_.
  * [Shum, He, and Li Shum et al.2018] Heung-Yeung Shum, Xiao-dong He, and Di Li. 2018\.  From Eliza to XiaoIce: challenges and opportunities with social chatbots.  _Frontiers of Information Technology & Electronic Engineering_ 19, 1 (2018), 10–26.
  * [Song, Wang, Zhang, Zhang, and Liu Song et al.2021] Haoyu Song, Yan Wang, Kaiyan Zhang, Wei-Nan Zhang, and Ting Liu. 2021.  BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data.  _arXiv preprint arXiv:2106.06169_ (2021).
  * [Song, Wang, Zhang, Liu, and Liu Song et al.2020] Haoyu Song, Yan Wang, Wei-Nan Zhang, Xiaojiang Liu, and Ting Liu. 2020.  Generate, delete and rewrite: A three-stage framework for improving persona consistency of dialogue generation.  _arXiv preprint arXiv:2004.07672_ (2020).
  * [Su, Cai, Wang, Baker, Korhonen, Collier, and Liu Su et al.2020] Yixuan Su, Deng Cai, Yan Wang, Simon Baker, Anna Korhonen, Nigel Collier, and Xiaojiang Liu. 2020.  Stylistic Dialogue Generation via Information-Guided Reinforcement Learning Strategy.  _CoRR_ abs/2004.02202 (2020).  arXiv:2004.02202
  * [Su, Yan, Cai, Baker, Korhonen, and Collier Su et al.2021] Yixuan Su, Wang Yan, Deng Cai, Simon Baker, Anna Korhonen, and Nigel Collier. 2021\.  Prototype-to-style: Dialogue generation with style-aware editing on retrieval memory.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ (2021).
  * [Sudhakar, Upadhyay, and Maheswaran Sudhakar et al.2019] Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. 2019\.  Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_.
  * [Tao, Wu, Xu, Hu, Zhao, and Yan Tao et al.2019] Chongyang Tao, Wei Wu, Can Xu, Wenpeng Hu, Dongyan Zhao, and Rui Yan. 2019\.  Multi-representation fusion network for multi-turn response selection in retrieval-based chatbots. In _WSDM_. 267–275.
  * [Wu, Ren, Luo, and Sun Wu et al.2019a] Chen Wu, Xuancheng Ren, Fuli Luo, and Xu Sun. 2019a.  A Hierarchical Reinforced Sequence Operation Method for Unsupervised Text Style Transfer. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * [Wu, Zhang, Zang, Han, and Hu Wu et al.2019b] Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019b.  Mask and infill: applying masked language model to sentiment transfer. In _Proceedings of the 28th International Joint Conference on Artificial Intelligence_.
  * [Wu, Wu, Xing, Zhou, and Li Wu et al.2017] Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017.  Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-Based Chatbots. In _ACL_. 496–505.
  * [Xu, Xu, Zeng, Zhang, Ren, Wang, and Li Xu et al.2018] Jingjing Xu, SUN Xu, Qi Zeng, Xiaodong Zhang, Xuancheng Ren, Houfeng Wang, and Wenjie Li. 2018.  Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * [Yan, Song, and Wu Yan et al.2016] Rui Yan, Yiping Song, and Hua Wu. 2016.  Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In _ACM SIGIR_. 55–64.
  * [Yang, Hu, Dyer, Xing, and Berg-Kirkpatrick Yang et al.2018] Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. 2018.  Unsupervised text style transfer using language models as discriminators. In _Advances in Neural Information Processing Systems_.
  * [Zeng, Song, Lin, and Sakai Zeng et al.2019] Zhaohao Zeng, Ruihua Song, Pingping Lin, and Tetsuya Sakai. 2019.  Attitude Detection for One-Round Conversation: Jointly Extracting Target-Polarity Pairs. In _Proceedings of the 12th ACM International Conference on Web Search and Data Mining_.
  * [Zhang, Dinan, Urbanek, Szlam, Kiela, and Weston Zhang et al.2018a] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018a.  Personalizing Dialogue Agents: I have a dog, do you have pets too?. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_.
  * [Zhang, Zhu, Wang, Zhao, and Liu Zhang et al.2019] Wei-Nan Zhang, Qingfu Zhu, Yifa Wang, Yanyan Zhao, and Ting Liu. 2019.  Neural personalized response generation as domain adaptation.  _World Wide Web_ 22, 4 (2019).
  * [Zhang, Li, Zhu, Zhao, and Liu Zhang et al.2018b] Zhuosheng Zhang, Jiangtong Li, Pengfei Zhu, Hai Zhao, and Gongshen Liu. 2018b.  Modeling Multi-turn Conversation with Deep Utterance Aggregation. In _Proceedings of the 27th International Conference on Computational Linguistics_. 3740–3752.
  * [Zheng, Chen, Zhang, Huang, Mao, and Huang Zheng et al.2020] Yinhe Zheng, Zikai Chen, Rongsheng Zhang, Shilei Huang, Xiaoxi Mao, and Minlie Huang. 2020\.  Stylized dialogue response generation using stylized unpaired texts.  _arXiv preprint arXiv:2009.12719_ (2020).
  * [Zhou, Huang, Zhang, Zhu, and Liu Zhou et al.2018a] Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018a.  Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory. In _AAAI-18, New Orleans, Louisiana, USA, February 2-7, 2018_. 730–739.
  * [Zhou, Li, Dong, Liu, Chen, Zhao, Yu, and Wu Zhou et al.2018b] Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu. 2018b.  Multi-turn response selection for chatbots with deep attention matching network. In _ACL_. 1118–1127.
  * [Zhu, Zhang, Liu, and Wang Zhu et al.2021] Qingfu Zhu, Weinan Zhang, Ting Liu, and William Yang Wang. 2021\.  Neural Stylistic Response Generation with Disentangled Latent Variables.  (2021).