# Mortality Prediction Models with Clinical Notes Using Sparse Attention at
the Word and Sentence Levels ††thanks: Citation: Technical Reports at the
Department of Medical Informatics, Amsterdam UMC, 2021.
https://kik.amc.nl/KIK/reports/TR2021-01.pdf

Miguel Rios
Centre for Translation Studies
University of Vienna
<EMAIL_ADDRESS>
&Ameen Abu-Hanna
Department of Medical Informatics, Amsterdam UMC
University of Amsterdam
<EMAIL_ADDRESS>

###### Abstract

Intensive Care in-hospital mortality prediction has various clinical
applications. Neural prediction models, especially when capitalising on
clinical notes, have been put forward as improvement on currently existing
models. However, to be acceptable these models should be performant and
transparent. This work studies different attention mechanisms for clinical
neural prediction models in terms of their discrimination and calibration.
Specifically, we investigate sparse attention as an alternative to dense
attention weights in the task of in-hospital mortality prediction from
clinical notes. We evaluate the attention mechanisms based on: i) local self-
attention over words in a sentence, and ii) global self-attention with a
transformer architecture across sentences. We demonstrate that the sparse
mechanism approach outperforms the dense one for the local self-attention in
terms of predictive performance with a publicly available dataset, and puts
higher attention to prespecified relevant directive words. The performance at
the sentence level, however, deteriorates as sentences including the
influential directive words tend to be dropped all together.

_K_ eywords sparse attention $\cdot$ clinical notes $\cdot$ mortality
prediction

##  1 Introduction

Deep learning has become a promising approach for clinical prediction models
(Rajkomar et al., 2018; Shickel et al., 2019). Moreover, natural language
processing (NLP) applications based on neural network (NN) models have shown
the potential to benefit the task of mortality prediction (Kemp et al., 2019).
However, NN models based on clinical notes are difficult to interpret, due to
their black-box nature. Decision makers need transparency in the way words in
an input clinical note contribute to the overall prediction. One approach to
gain transparency is the attention mechanism, which have shown improvements
across NLP applications (Vaswani et al., 2017; Devlin et al., 2019) for text
representation. The attention mechanism assigns weights to an input vector
representation of a layer in a NN (Bahdanau et al., 2015), and it has been
used to identify the relative importance of a word (Clark et al., 2019; Huang
et al., 2019; Vashishth et al., 2019).

Caicedo-Torres and Gutiérrez (2020) propose a mortality prediction model with
clinical notes based on a convolutional architecture, where the max pooling
layer is used to inform word importance. Lovelace et al. (2019) incorporate an
attention mechanism on top of the convolutional representation model to
highlight words from the input clinical notes.

Current attention mechanisms produce a dense weight distribution over the
input vector representation. The dense attention will assign a weight to all
words in the context even if they are irrelevant for the current prediction.
Moreover, attention weights may not be interpretable given that their
alterations do not lead to a change in predictions (Serrano and Smith, 2019;
Jain and Wallace, 2019). In contrast, sparse attention can lead to more
accurate models and more transparent presentation by assigning zero weights to
some words (Martins and Astudillo, 2016; Niculae and Blondel, 2017; Correia et
al., 2019).

In this paper, we investigate the predictive performance of mortality
prediction models, in terms of discrimination and calibration, using clinical
notes. In particular, we study the effect of sparse versus dense attention
mechanisms within a hierarchical architecture: self-attention is applied at
the local level to weigh words in a sentence, and at the global level to weigh
sentences. We show preliminary results for the task of in-hospital mortality
prediction using a publicly available dataset. First, we compare the
predictive performance between a model based on local self-attention over
words using the standard dense attention, and two sparse attention mechanisms
$\operatorname{sparsemax}$ and $\operatorname{entmax}$. The sparse attention
mechanisms show better performance compared to dense attention. A preliminary
evaluation showed that $\operatorname{sparsemax}$ consistently identified
prespecified directive words known to be associated with mortality. Finally,
we use the different attention mechanisms on a global self-attention model
across sentences on clinical notes. In contrast to the previous experiment,
the global attention model with the dense mechanism outperforms the sparse
ones. The latter seems to throw sentences out with the directive words.

##  2 Sparse Attention

The Self-attention or scaled dot-product attention is a main component in
current state-of-the-art NLP models (Vaswani et al., 2017). Self-attention
computes the representation of a sequence by processing each of its positions
and associating the position with other positions for indications of
significance. The self-attention layer is defined as follows:

$\displaystyle\operatorname{Att}(\boldsymbol{Q},\boldsymbol{K},\boldsymbol{V})=\boldsymbol{\pi}\left(\frac{\boldsymbol{Q}\boldsymbol{K}^{\top}}{\sqrt{d}}\right)\boldsymbol{V}$
(1a)

where $\boldsymbol{Q}$ indicates queries, $\boldsymbol{K}$ keys,
$\boldsymbol{V}$ values, and $d$ vector dimensionality. The queries are a
linear transformation of the input vector $x$ with a parameter matrix $W_{q}$
defined as $\boldsymbol{Q}=W_{q}x$, and similarly for keys
$\boldsymbol{K}=W_{k}x$, and values $\boldsymbol{V}=W_{v}x$. The standard
$\boldsymbol{\pi}$ mapping is based on $\operatorname{softmax}$, and it is
used to normalise the attention weights. Moreover, self-attention can be
extended to multiple heads that potentially allows the model to focus on
different positions (Vaswani et al., 2017).

Martins and Astudillo (2016) propose $\operatorname{sparsemax}$ as a
differentiable mapping that provides a sparse alternative to
$\operatorname{softmax}$. Sparsemax projects the input vector into the
probability simplex, where it is likely to be in the boundary of the simplex
and then becoming sparse. Moreover, the $\operatorname{entmax}$ mapping
produces sparse distributions by defining interpolations between
$\operatorname{softmax}$ and $\operatorname{sparsemax}$ (Peters et al., 2019;
Correia et al., 2019). The $\operatorname{entmax}$ becomes an intermediate
mapping between $\operatorname{sparsemax}$ and $\operatorname{softmax}$.
Correia et al. (2019) propose a sparse attention mechanism for transformer
architectures by replacing the $\pi$ mapping by $\operatorname{sparsemax}$,
and $\operatorname{entmax}$.

Most mortality prediction models based on free text represent multiple notes
of an ICU stay as a sequence of sentences. The model hierarchically composes
word representations (i.e. word encoder) into a sentence, and sentences (i.e.
sentence encoder) into a patient representation for predicting mortality
(Grnarova et al., 2016; Si and Roberts, 2019). For example, Grnarova et al.
(2016) use a hierarchy of convolutional neural networks (CNN) on the word and
sentence encoders for mortality prediction given complete ICU stays.

We use a word encoder with one layer self-attention and one head as a baseline
to represent local dependencies over words in a sentence. The clinical notes
of a patient for an ICU stay $x_{i,t}$ consist of $i$ words and $t$ sentences.
The input $x_{i,t}$ is first represented with pre-trained word embeddings, and
then projected into a linear layer followed by self-attention (Eq. 1) to
encode words of each sentence. Finally, the features of the word encoder are
summarised by averaging to produce a patient representation for a linear
prediction layer. For the local self-attention, we define the following
models: Att-softmax as baseline with $\operatorname{softmax}$ mapping for
dense attention weights, Att-sparsemax with the attention weights based on the
$\operatorname{sparsemax}$ mapping, and Att-entmax with the
$\operatorname{entmax}$ mapping.

A limitation of a local architecture with self-attention is the lack of long
distance dependencies across sentence representations. We use a hierarchical
architecture for modelling long distance relations within an ICU stay. We
replace the CNN with a transformer layer (Vaswani et al., 2017), which is
based on self-attention and positional embeddings, on both levels of the
hierarchy (Pappagari et al., 2019; Zhang et al., 2019). For the global
attention, we define the following hierarchical transformers: Tr-softmax, Tr-
entmax, and Tr-sparsemax. We describe the self-attention and hierarchical
transformer architectures in more detail in Appendix A.

Model | AUC-ROC$\uparrow$ | AUC-PR$\uparrow$ | Brier$\downarrow$
---|---|---|---
Att-softmax | 0.824 $\pm$ 0.003 | 0.435 $\pm$ 0.008 | 0.085 $\pm$ 0.001
Att-entmax | 0.827 $\pm$ 0.004 | 0.445 $\pm$ 0.010 | 0.084 $\pm$ 0.002
Att-sparsemax | 0.834 $\pm$ 0.004 | 0.467 $\pm$ 0.011 | 0.087 $\pm$ 0.001
Tr-softmax | 0.839 $\pm$ 0.005 | 0.462 $\pm$ 0.005 | 0.093 $\pm$ 0.016
Tr-entmax | 0.828 $\pm$ 0.007 | 0.433 $\pm$ 0.014 | 0.114 $\pm$ 0.004
Tr-sparsemax | 0.801 $\pm$ 0.009 | 0.373 $\pm$ 0.025 | 0.196 $\pm$ 0.076
Table 1: In-hospital mortality results on the test dataset with 5 runs $\pm$
standard deviation on each model reporting AUC-ROC, AUC-PR, and Brier score.

(a)

(b)

Figure 1: Receiver operating characteristic curve (a) and calibration curve
(b) for in-hospital mortality.

##  3 Experiments

The Medical Information Mart for Intensive Care (MIMIC-III) database includes
ICU information such as demographics, vital measurements, laboratory test
results, procedures, medications, caregiver notes, imaging reports, and
mortality for critical care patients (Johnson et al., 2016). Harutyunyan et
al. (2019) propose a public benchmark from MIMIC-III for modelling mortality,
length of stay, physiologic decline, and phenotype classification. We use the
in-hospital mortality benchmark cohort for extracting clinical notes in
English based on the first 48 hours of an ICU stay to develop a prediction
model. The cohort excludes ICU stays with missing length-of-stay, patients
under 18, multiple ICU stays, stays under 48 hours, and without observations
on the first 48 hours. The mortality class is defined by comparing the date of
death across hospital admissions and discharge times with a resulting
mortality rate of 13.2%. To preprocess the extracted notes, we tokenize with
NLTK111NLTK tokenizer for English: https://www.nltk.org/index.html, lowercase,
and exclude duplicate and discharge notes. The training/validation/test
datasets consist of data from $14,681$, $3,222$ and $3,236$ patients,
respectively. We report the mean and standard deviation with 5 random runs for
the area under the receiver operator characteristic curve (AUC-ROC), area
under the precision-recall curve (AUC-PR), and the Brier score.

We pretrain GloVe (Pennington et al., 2014) word embeddings on the benchmark
training split with: 100D, minimum frequency of 5 words, window of 15 words,
20 epochs, and further fine-tune them on the downstream task. For the
prediction model, we use the following hyperparameters: Adam optimiser (Kingma
and Ba, 2014), learning rate $1\mathrm{e}{-4}$, epochs 30, hidden size 128,
batch size 16, and dropout 0.2. For each training instance the max number of
words $i$ in a sentence is 50 and the max number of sentences $t$ is 1000.
Finally, we perform hyper-parameter and model selection tuning with the
validation dataset based on the AUC-ROC.

(a)

(b)

Figure 2: Attention heatmaps corresponding positive class instance. Att-
softmax, Att-entmax and Att-sparsemax respectively in (a). Tr-softmax, Tr-
entmax and Tr-sparsemax in (b).

###  3.1 In-hospital Mortality Results

We report the performance on the task of in-hospital mortality prediction
using sparse attention, and compare it to dense attention. Table 1 shows the
AUC-ROC, AUC-PR, and Brier score results of the validation and test datasets.
The Att-sparsemax shows competitive results compared to entmax and softmax on
both metrics AUC-ROC and AUC-PR. The Brier score increases for the sparse
attention models, where the models with sparse attention assign higher output
probabilities to both classes. Figure 1 shows the ROC and calibration curves.
The sparse attention models show competitive performance compared to the
baseline on ROC. However, we observe from the calibration curve that the
sparse attention underestimates predictions for the local attention models.

Moreover, the addition of sparse attention on the second level of the
hierarchy for the transformer models, hurts the performance measures and
calibration. The sparse transformer models show a marked miscalibration in
Figure 1 where it underpredicts the true probabilities.

###  3.2 Qualitative Analysis

We show example attention heatmaps from the local self-attention model for
sentences that include any of the following directives: do not resuscitate
(dnr), do not intubate (dni), and comfort measures only (cmo). The directive
words are known to be strongly correlated with mortality. For example, if the
family of a patient has decided for comfort measures only then this refers to
palliative treatment of a patient thought to be dying.

Jain and Wallace (2019) define explainable attention when the inputs with the
highest attention weights are relevant for a given prediction. Figure 2 shows
attention heatmaps for two sentences that contain any of the directive words.
The heatmaps denote the relative importance for each word embedding (row) with
respect to its context (column), the darker the cell, the higher the weight.
One can observe that the sparse attention put most of its weights on the
directive words, and that the highest attended words differ between the sparse
and dense mechanisms. However, in Figure 2 (a) $\operatorname{entmax}$ better
distributes its weights for cmo. We show more attention examples in Appendix
B. On the hierarchical transformer the sentence level sparsemax is
consistently too aggressive in selecting weights. For example, in the sentence
from Fig. 2 (b) the model only assigns attention to 8 out of 174 sentences.
Moreover, the attention weights at the word encoder level shifted away from
the directive words compared to the local attention.

##  4 Discussion and Future Work

In this paper, we compared the standard dense attention with sparse attention
mechanisms on two setups of local and global dependencies across
representations for mortality prediction with clinical notes. The sparsemax
attention mechanism shows competitive predictive performance compared to the
dense attention weights on the local self-attention model. However, the global
model based on sparse attention produces under-confident predictions.

The drop in performance in the hierarchical transformer is due to the sparse
attention on the sentence encoder only assigning weights to few sentences,
where most ICU stays contain hundreds of sentences.

Future work meriting investigation includes combining sparse and dense
mechanisms on the hierarchical transformer. In addition to showing examples of
the attention heatmaps, one may also analyse the predicted weights by
measuring their correlation to feature importance metrics (Vashishth et al.,
2019; Serrano and Smith, 2019).

## References

  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Caicedo-Torres and Gutiérrez (2020) William Caicedo-Torres and Jairo A. Gutiérrez. 2020.  Iseeu2: Visually interpretable ICU mortality prediction using deep learning and free-text medical notes.  _CoRR_ , abs/2005.09284.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019.  What does BERT look at? an analysis of BERT’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286, Florence, Italy. Association for Computational Linguistics.
  * Correia et al. (2019) Gonçalo M. Correia, Vlad Niculae, and André F. T. Martins. 2019.  Adaptively sparse transformers.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2174–2184, Hong Kong, China. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Grnarova et al. (2016) Paulina Grnarova, Florian Schmidt, Stephanie L. Hyland, and Carsten Eickhoff. 2016\.  Neural document embeddings for intensive care patient mortality prediction.  _CoRR_ , abs/1612.00467.
  * Harutyunyan et al. (2019) Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. 2019.  Multitask learning and benchmarking with clinical time series data.  _Scientific Data_ , 6(1).
  * Huang et al. (2019) Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019.  Clinicalbert: Modeling clinical notes and predicting hospital readmission.  _arXiv:1904.05342_.
  * Jain and Wallace (2019) Sarthak Jain and Byron C. Wallace. 2019.  Attention is not Explanation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3543–3556, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3:160035.
  * Kemp et al. (2019) J. Kemp, Alvin Rajkomar, and Andrew M. Dai. 2019.  Improved patient classification with language model pretraining over clinical notes.  _ArXiv_ , abs/1909.03039.
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  Cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
  * Lovelace et al. (2019) Justin R. Lovelace, Nathan C. Hurley, Adrian D. Haimovich, and Bobak J. Mortazavi. 2019.  Explainable prediction of adverse outcomes using clinical notes.  _CoRR_ , abs/1910.14095.
  * Martins and Astudillo (2016) Andre Martins and Ramon Astudillo. 2016.  From softmax to sparsemax: A sparse model of attention and multi-label classification.  volume 48 of _Proceedings of Machine Learning Research_ , pages 1614–1623, New York, New York, USA. PMLR.
  * Niculae and Blondel (2017) Vlad Niculae and Mathieu Blondel. 2017.  A regularized framework for sparse and structured neural attention.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Pappagari et al. (2019) Raghavendra Pappagari, Piotr Zelasko, Jesús Villalba, Yishay Carmiel, and Najim Dehak. 2019.  Hierarchical transformers for long document classification.  _CoRR_ , abs/1910.10781.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _EMNLP_ , volume 14, pages 1532–1543.
  * Peters et al. (2019) Ben Peters, Vlad Niculae, and André F. T. Martins. 2019.  Sparse sequence-to-sequence models.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1504–1519, Florence, Italy. Association for Computational Linguistics.
  * Rajkomar et al. (2018) Alvin Rishi Rajkomar, Eyal Oren, Kai Chen, Andrew Dai, Nissan Hajaj, Mila Hardt, Peter J. Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, Patrik Per Sundberg, Hector Yee, Kun Zhang, Yi Zhang, Gerardo Flores, Gavin Duggan, Jamie Irvine, Quoc Le, Kurt Litsch, Alex Mossin, Justin Jesada Tansuwan, De Wang, James Wexler, Jimbo Wilson, Dana Ludwig, Samuel Volchenboum, Kat Chou, Michael Pearson, Srinivasan Madabushi, Nigam Shah, Atul Butte, Michael Howell, Claire Cui, Greg Corrado, and Jeff Dean. 2018.  Scalable and accurate deep learning for electronic health records.  _npj Digital Medicine_.
  * Serrano and Smith (2019) Sofia Serrano and Noah A. Smith. 2019.  Is attention interpretable? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2931–2951, Florence, Italy. Association for Computational Linguistics.
  * Shickel et al. (2019) Benjamin Shickel, T. Loftus, Lasith Adhikari, T. Ozrazgat-Baslanti, A. Bihorac, and Parisa Rashidi. 2019.  Deepsofa: A continuous acuity score for critically ill patients using clinically interpretable deep learning.  _Scientific Reports_ , 9.
  * Si and Roberts (2019) Yuqi Si and Kirk Roberts. 2019.  Deep patient representation of clinical notes via multi-task learning for mortality prediction.  _AMIA Summits on Translational Science Proceedings_ , 2019:779.
  * Vashishth et al. (2019) Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019\.  Attention interpretability across NLP tasks.  _CoRR_ , abs/1909.11218.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 5998–6008.
  * Zhang et al. (2019) Xingxing Zhang, Furu Wei, and Ming Zhou. 2019.  HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5059–5069, Florence, Italy. Association for Computational Linguistics.

##  Appendix A Architectures

We define the following architectures given an input $x_{i,t}$ that consist of
clinical notes for an ICU stay represented into $i$ words and $t$ sentences.

#### Self-attention

Figure 3 shows the architecture for the self-attention model.

Figure 3: Self-attention architecture.

#### Hierarchical Transformer

Figure 4 shows the architecture for the hierarchical transformer model. The
architecture consists of pre-trained emebddings, position embeddings for words
$\omega_{\text{pos\\_i}}$, sentences $\omega_{\text{pos\\_t}}$, and
transformer layers with self-attention.

Figure 4: Hierarchical transformer architecture.

##  Appendix B Attention Heatmaps

(a)

(b)

(c)

(d)

Figure 5: Attention heatmaps for sentences in a positive instance (a) Att-
softmax, Att-entmax and Att-sparsemax respectively, and (b) Tr-softmax, Tr-
entmax and Tr-sparsemax. Sentences in negative instance (c) Att-softmax, Att-
entmax and Att-sparsemax and (d) Tr-softmax, Tr-entmax and Tr-sparsemax .


algorithmic

# AdaEmbed: Semi-supervised Domain Adaptation in the
Embedding Space

Ali Mottaghi<EMAIL_ADDRESS>
Stanford University, Stanford, CA Mohammad Abdullah Jamal
<EMAIL_ADDRESS>
Intuitive Surgical Inc., Sunnyvale, CA Serena Yeung †<EMAIL_ADDRESS>
Stanford University, Stanford, CA Omid Mohareri †<EMAIL_ADDRESS>
Intuitive Surgical Inc., Sunnyvale, CA

†These authors contributed equally to this work

###### Abstract

Semi-supervised domain adaptation (SSDA) presents a critical hurdle in
computer vision, especially given the frequent scarcity of labeled data in
real-world settings. This scarcity often causes foundation models, trained on
extensive datasets, to underperform when applied to new domains. AdaEmbed, our
newly proposed methodology for SSDA, offers a promising solution to these
challenges. Leveraging the potential of unlabeled data, AdaEmbed facilitates
the transfer of knowledge from a labeled source domain to an unlabeled target
domain by learning a shared embedding space. By generating accurate and
uniform pseudo-labels based on the established embedding space, the model
overcomes the limitations of conventional SSDA, thus enhancing performance
significantly. Our method’s effectiveness is validated through extensive
experiments on benchmark datasets such as DomainNet, Office-Home, and VisDA-C,
where AdaEmbed consistently outperforms all the baselines, setting a new state
of the art for SSDA. With its straightforward implementation and high data
efficiency, AdaEmbed stands out as a robust and pragmatic solution for real-
world scenarios, where labeled data is scarce. To foster further research and
application in this area, we are sharing the codebase of our unified framework
for semi-supervised domain adaptation.

##  1 Introduction

Foundation models, renowned for their extensive pre-training on vast datasets,
have established new standards in a variety of computer vision tasks. However,
despite their advanced architecture and training, these models often encounter
challenges in real-world scenarios characterized by a significant divergence
between the target domain and the source domain of their initial training.
This phenomenon, known as domain shift, frequently results in a noticeable
drop in the performance of these models, underscoring the need for robust
domain adaptation strategies Donahue et al. (2014); Tzeng et al. (2017). The
domain adaptation aims to transfer knowledge from a labeled source domain to a
new but related target domain. Acquiring labeled data for the target domain
can be challenging and resource-intensive, underscoring the need for semi-
supervised domain adaptation (SSDA). SSDA leverages a small amount of labeled
data in the target domain and a larger amount of unlabeled data to enhance the
model’s performance. Despite its wide range of real-world applications, SSDA
has received less attention than both unsupervised domain adaptation (UDA) and
semi-supervised learning (SSL).

Figure 1: AdaEmbed method for semi-supervised domain adaptation (SSDA). The
embedding space for a general SSDA problem is shown in (a), while (b)
illustrates how AdaEmbed estimates prototypes based on labeled and unlabeled
samples in the feature space, selects unlabeled samples in the embedding space
to generate pseudo-labels given their proximity to prototypes, and trains the
model on both labels and pseudo-labels. In addition, a contrastive feature
loss is incorporated during training to learn a more effective shared
embedding for both source and target domains.

Existing Semi-supervised Domain Adaptation (SSDA) research primarily focuses
on two strategies: learning domain-invariant embeddings and generating pseudo-
labels for unlabeled target examples. Notable works include Saito et al.
(2018), which emphasizes discrepancy maximization for alignment, and others
like Ganin et al. (2016); Sun & Saenko (2016), which propose matching feature
distributions in deep networks. Saito et al. (2019) introduced an adaptation
technique based on conditional entropy, while Berthelot et al. (2021)
developed AdaMatch for confident pseudo-label generation. Most current SSDA
methods enhance pseudo-labels through prediction refinements or utilize
contrastive learning for alignment at both inter-domain and instance levels.

In our paper, we present AdaEmbed, a new SSDA methodology that emphasizes
learning a shared embedding space and prototypes for generating precise and
balanced pseudo-labels for the unlabeled target domain (Figure 1). AdaEmbed
incorporates two primary forms of training supervision: cross-entropy loss for
actual and pseudo-labels, and contrastive loss for aligning instance features.
This approach fosters a robust embedding space that is resilient to
distribution shifts. Diverging from previous models, AdaEmbed uniquely
leverages the embedding space and prototypes to produce more evenly
distributed pseudo-labels, effectively addressing the imbalance issue
prevalent in earlier SSDA methods Singh (2021); Sahoo et al. (2021).

We conduct thorough experiments on major domain adaptation benchmarks to
evaluate the efficacy of AdaEmbed. Our primary contributions in this work are:

  * •

We have proposed AdaEmbed as a comprehensive approach for domain adaptation.
This method is versatile, applicable to both UDA and SSDA scenarios. It
effectively leverages a small number of labeled images in the target domain,
in addition to the larger set of unlabeled data.

  * •

A key feature of AdaEmbed is its innovative pseudo-labeling strategy, which
uses the embedding features to generate accurate and balanced pseudo-labels.
This approach addresses the common issue of imbalanced pseudo-label
distributions in SSDA.

  * •

Our experiments demonstrate AdaEmbed’s superiority over all baseline models. A
detailed ablation study further validates its efficacy, highlighting the
critical roles of pseudo-labeling, contrastive loss, and entropy loss in
enhancing the method’s effectiveness.

  * •

AdaEmbed is designed to be model-agnostic, allowing integration with any
standard classification backbone. This flexibility extends its applicability,
making it a versatile tool in various computer vision applications.

##  2 Related Works

###  2.1 Unsupervised Domain Adaptation (UDA)

Unsupervised Domain Adaptation (UDA) involves generalizing a model trained on
a source domain to a new but related, fully unlabeled target domain. A common
focus in UDA literature Liang et al. (2019); Long et al. (2015; 2016) is
reducing the discrepancy between source and target domain representations.
Some approaches align the final representations using maximum mean discrepancy
Tzeng et al. (2014); Gretton et al. (2006); Xiao & Zhang (2021), while others
match the distribution of intermediate features in deep networks Sun & Saenko
(2016); Peng et al. (2019); Chen et al. (2018). In Maximum Classifier
Discrepancy (MCD) Saito et al. (2018), two task classifiers are trained to
maximize the discrepancy on the target sample, then the feature generator is
trained to minimize this discrepancy.

Adversarial learning-based approaches Ganin & Lempitsky (2015); Kang et al.
(2018); Tzeng et al. (2017); Zhang et al. (2018a; b) use domain discriminators
and generative networks to introduce ambiguity between source and target
samples, learning domain-invariant features in the process. Several methods
also use pseudo-labels Bruzzone & Marconcini (2010); Chen et al. (2011); Xie
et al. (2018); Long et al. (2013); Saito et al. (2017) to incorporate
categorical information in the target domain. Test-time adaptation approaches
are another branch of research that utilizes the source model and unlabeled
target domain data, with notable methods including AdaContrast Chen et al.
(2022), which specifically refines pseudo-labels through soft voting among
nearest neighbors in the target feature space, SHOT Liang et al. (2020), and
TENT Wang et al. (2021).

###  2.2 Semi-supervised Domain Adaptation (SSDA)

Semi-supervised Domain Adaptation (SSDA) differs from UDA in that it utilizes
a labeled set in the target domain. The goal remains to achieve the highest
performance on the target domain using both labeled and unlabeled data from
the source and target domains. SSDA is less explored than UDA, with most
research focusing on image classification tasks Yao et al. (2015); Ao et al.
(2017); Saito et al. (2019). A key methodology in this area is the Minimax
Entropy (MME) approach Saito et al. (2019) which focuses on adaptation by
alternately maximizing the conditional entropy of unlabeled target data
concerning the classifier and minimizing it with respect to the feature
encoder. Another notable contribution is AdaMatch, proposed by Berthelot et
al. (2021), which extends the principles of FixMatch Sohn et al. (2020) to
SSDA setting. Additionally, recent advancements in SSDA include the
introduction of CLDA Singh (2021), a method utilizing contrastive learning for
reducing discrepancies both within and between domains, and ECACL Li et al.
(2021), which employs robust data augmentation to refine alignment strategies
in SSDA scenarios. Building on the core principles established by previous
research, our AdaEmbed framework capitalizes on the embedding space to propose
a comprehensive approach to domain adaptation.

###  2.3 Semi-Supervised Learning (SSL)

Semi-Supervised Learning (SSL) leverages both unlabeled and partially labeled
data to train models. Recent advances in self-supervised learning Caron et al.
(2020); Chen et al. (2020a; b); He et al. (2020); Grill et al. (2020) enable
model pre-training on large-scale datasets, followed by fine-tuning using a
small labeled dataset. Consistency regularization-based approaches Sajjadi et
al. (2016); Laine & Aila (2016); Tarvainen & Valpola (2017) are widely used in
SSL to enforce the consistency between different augmentations of the same
training sample. Augmentation-based methods like MixMatch Berthelot et al.
(2019b), ReMixMatch Berthelot et al. (2019a), and FixMatch Sohn et al. (2020)
utilize pseudo-labeling and consistency regularization. However, these methods
cannot be directly applied to the SSDA setting due to their assumption that
data is sampled from the same distribution.

###  2.4 Few-Shot Learning (FSL)

Few-Shot Learning (FSL) tasks require the model to generalize from limited
training samples. A rich line of work exists on meta-learning for few-shot
learning, with recent works focusing on optimization-based, metric-based, and
model-based approaches. Optimization-based methods Finn et al. (2017); Jamal &
Qi (2019); Ravi & Larochelle (2016); Li et al. (2017); Rusu et al. (2019); Sun
et al. (2019) aim to find the best model parameters for rapid adaptation to
unseen tasks. Metric-based methods Vinyals et al. (2016); Sung et al. (2017);
Snell et al. (2017); Oreshkin et al. (2018); Koch et al. (2015); Wang et al.
(2019); Satorras & Estrach (2018) learn an embedding function using a distance
function within the meta-learning paradigm. Transductive learning-based
approaches Dhillon et al. (2020); Bateni et al. (2020); Liu et al. (2019);
Shen et al. (2021) have also been proposed for FSL. Finally, model-based
approaches Santoro et al. (2016); Cai et al. (2018); Munkhdalai et al. (2017);
Mishra et al. (2018); Munkhdalai & Yu (2017); Edwards & Storkey (2017) involve
designing architectures specifically tailored for fast adaptation, for
instance, through the use of external memory. Our proposed approach, AdaEmbed,
draws inspiration from the metric-based methods.

##  3 Method

We propose AdaEmbed, a new domain adaptation approach for both unsupervised
domain adaptation (UDA) and semi-supervised domain adaptation (SSDA) settings
on any classification task. Our new method utilizes the embedding space to
generate pseudo-labels for the unlabeled target samples and with the help of
contrastive learning learns an embedding shared between source and target
domains. The main difference between UDA and SSDA is that in SSDA setting we
have a few labeled images (samples) in the target domain. Therefore in the
source domain we are given source images and their corresponding labels
$\mathcal{D}_{s}=\\{(x_{s}^{i},y_{s}^{i})\\}_{i=1}^{n_{s}}$. In the target
domain, we are also given a limited set of labeled target samples
$\mathcal{D}_{t}=\\{(x_{t}^{i},y_{t}^{i})\\}_{i=1}^{n_{t}}$. The rest of the
target is unlabeled and is denoted by
$\mathcal{D}_{u}=\\{x_{u}^{i}\\}_{i=1}^{n_{u}}$. Our goal is to train the
model on $\mathcal{D}_{s}$, $\mathcal{D}_{t}$, and $\mathcal{D}_{u}$ and
evaluate on the test set of the target domain. In UDA settings we only have
access to $\mathcal{D}_{s}$ and $\mathcal{D}_{u}$, therefore $\mathcal{D}_{t}$
will be empty. Our approach is not limited to only images and can take any
input type.

###  3.1 AdaEmbed

In this subsection we discuss different component of our method AdaEmbed. Data
points are first augmented and passed through the encoder and momentum
encoders. Then we generate pseudo labels for unlabeled target samples based on
the embedding features. The features are contrasted in the embedding space to
push the similar labels/pseudo-labels together and different class apart from
each other. Finally, the entropy of unlabeled target predictions is used to
update the prototypes in our model.

##### Model.

Our base model architecture, following Chen et al. (2019), consists of a
general encoder (feature extractor)
$F(.):\mathcal{X}_{s}\rightarrow\mathbb{R}^{d}$ and a single layer classifier
$C(.):\mathbb{R}^{d}\rightarrow\mathbb{R}^{c}$ where $d$ and $c$ are feature
dimension and number of classes. Any deep learning based neural architecture
can be used as the feature extractor for image or video inputs ($x$). The
extracted features $f=F(x)\in\mathbb{R}^{d}$ are first normalized with $l_{2}$
normalization i.e. $\frac{F(x)}{||F(x)||}$. The classifier consist of a weight
matrix $W=[w_{1},w_{2},\dots,w_{c}]\in\mathbb{R}^{d\times c}$ and it outputs
logits

$\displaystyle z=C(x)=\frac{W^{T}F(x)}{||F(x)||}$

The logits are then passed through a softmax to generate the probabilistic
output $p=softmax(z)$. The classifier uses cosine distances between the input
feature and the weight vector for each class that aims to reduce intra-class
variations. As discussed in Chen et al. (2019), the weight vectors can be
described as estimated prototypes for each class and the classification is
based on the distance of the input feature to these learned prototypes based
on cosine similarity.

Consider each sample and its augmented version as $x$ and $\tilde{x}$
respectively. For images, we use RandAugment Cubuk et al. (2020) and for
videos we augment them frame-wise with the same augmentation strategy. The
augmented data point $\tilde{x}$ is passed through the encoder to extract the
features $\tilde{f}=F(\tilde{x})$. We also keep the original version and pass
it though the momentum encoder to keep a queue of extracted features that will
be later used for generating pseudo-labels and computing the contrastive
losses. The momentum encoder $F^{\prime}$ parameters $\theta_{F^{\prime}}$ are
initialized with the encoder weights $\theta_{F}$ at the beginning of
training, and updated at each minibatch following He et al. (2020):

$\displaystyle\theta_{F^{\prime}}\leftarrow
m\theta_{F^{\prime}}+(1-m)\theta_{F}$ (1)

Then the momentum features and probabilities are computed as:

$\displaystyle f^{\prime}=F^{\prime}(x),\quad
p^{\prime}=\textit{softmax}(C(f^{\prime}))$

The momentum features and probabilities are stored in a queue $Q=\\{(f^{\prime
i},p^{\prime i})\\}_{i=1}^{M}$ of size $M$. This memory is used to generated
pseudo-labels for unlabeled samples and to computer contrastive loss for
features. Please note that we only use the features and probabilities
corresponding to the augmented data $(\tilde{f},\tilde{p})$ in back-
propagating gradients.

Figure 2: The framework for our semi-supervised domain adaptation method
(AdaEmbed). The left column visualises the types of data points and their
corresponding symbol in this figure. At the beginning of adaptation process
both the encoder and momentum encoder are initialized with the model trained
on labeled data. At each iteration we augment the data and send the augmented
version ($\tilde{x}$) to our encoder while the original version ($x$) is
passed through the momentum encoder. The features generated by the encoders
are then given to the classifier to generate the predictions ($p$ and
$\tilde{p}$). For labeled samples we compute the cross entropy loss between
the given label and the corresponding prediction ($\mathcal{L}_{s}$). For
unlabeled target samples we first generate pseudo-label ($\hat{y}$) for a few
samples based on their proximity to the class prototypes, and use the pseudo-
labels for the target loss ($\mathcal{L}_{t}$). We also unitize an instance
contrastive alignment loss ($\mathcal{L}_{c}$) to push the original and
augmented features ($f^{\prime}$ and $\tilde{f}$) together while pushing
features with different labels/pseudo-labels apart. Finally the prototypes are
updated based on the entropy of unlabeled target predictions ($\mathcal{H}$).
The training algorithm for AdaEmbed is outlined in Algorithm 1.

##### Supervised Loss.

For the labeled data, we compute the standard cross entropy loss which forms
the supervised term in our final loss function:

$\displaystyle\mathcal{L}_{s}$
$\displaystyle=\mathbb{E}_{(x,y)\in\mathcal{D}_{s}\cup\mathcal{D}_{t}}\mathcal{L}_{ce}(y,\tilde{p})$
(2)

##### Pseudo-Labeling.

In AdaEmbed, we propose a new pseudo-labeling strategy for generating accurate
and balanced pseudo-labels based on the embeddings. The proximity to the class
prototype (weights of the matrix $W$ in classifier $C$) is used as a measure
of accuracy of pseudo-labels. We also sample the same number of pseudo-labels
for each prototype to ensure a nearly uniform distribution of generated
pseudo-labels. After adding the new sample (or mini-batch of samples)
$(f^{\prime},p^{\prime})$ to the memory $Q$ we select the $k$ nearest
neighbours of each prototype, where $k$ is a hyper-parameter and lower $k$
results in more accurate pseudo-labels. If the sample feature $f^{\prime}$
falls inside one of the k nearest neighbours, we generate pseudo-labels for it
otherwise it will be ignored. More formally,

$\displaystyle mask=(f^{\prime}\in knn(Q_{f}))$ (3)

where $Q_{f}$ are the features in the memory queue. If selected, the generated
pseudo-labels will be $\hat{y}=\arg\max(p^{\prime})$ Selecting the $k$ nearest
neighbours for each class prototype ensures the most accurate pseudo-labels
are selected and the generated pseudo-labels overall have a uniform
distribution which is critical in datasets with long-tailed distribution.

Based on generated pseudo-labels, we add a new target loss term as follows:

$\displaystyle\mathcal{L}_{t}$
$\displaystyle=\mathbb{E}_{x\in\mathcal{D}_{u}}[\mathcal{L}_{ce}(\text{stopgrad}(\hat{y}),\tilde{p})\cdot
mask]$ (4)

Here, stopgrad stops the gradients from back-propagating into the momentum
encoder branch.

##### Instance Contrastive Alignment.

For unlabeled target samples, we use the InfoNCE loss to contrast the
generated features with the features in the memory bank Oord et al. (2018):

$\displaystyle\mathcal{L}_{c}$
$\displaystyle=-\mathbb{E}_{x\in\mathcal{D}_{u}}\log\frac{\exp(sim(\tilde{f}\cdot
f^{\prime})/\tau)}{\sum_{q\in\bar{Q}_{f}(\hat{y})}\exp(sim(\tilde{f}\cdot
q)/\tau)}$ (5)

However in contrast to Oord et al. (2018), we only use the features in the
feature bank where their corresponding pseudo-labels are not similar, i.e:

$\displaystyle\bar{Q}_{f}(\hat{y})=\\{q|(q,p)\in Q,\arg\max(p)\neq\hat{y}\\}$
(6)

In this way we are only pushing away target samples with different pseudo-
labels in the embedding space, therefore preserving the clusters for each
class in the embedding space while maximizing the distances between them.

##### Updating Prototypes.

The prototypes are learned based on adversarial minimax entropy training using
unlabeled target examples. The position of prototypes are updated by
maximizing the entropy between wight matrix $W$ and unlabeled target features
$\tilde{f}$:

$\displaystyle\mathcal{H}$
$\displaystyle=-\mathbb{E}_{x\in\mathcal{D}_{u}}\sum_{i=1}^{c}\log\tilde{p}_{i}\log(\tilde{p}_{i})$
(7)

where $\tilde{p}_{i}$ is the $i$th dimension of $\tilde{p}$, i.e. the
probability of prediction to class $i$. The higher entropy value means each
$w_{i}$ (prototypes) is similar to target features and therefore well
positioned to be domain-invariant features for each class.

##### Training Objectives.

The full overview of our domain adaptation method is depicted in Fig. 1. As
described in this section, in AdaEmbed we have three forms of training
objectives: 1) cross-entropy loss for the labels of the labeled samples 2)
masked cross-entropy loss for generated pseudo-labels of the unlabeled target
3) contrastive loss for the features of the unlabeled data.

To sum up the discussion, the model ($F$ and $C$ combined) is optimized to
minimize the supervised loss $\mathcal{L}_{s}$ for labeled data points, and
minimize the target pseudo-labeling loss $\mathcal{L}_{t}$ and contrastive
loss $\mathcal{L}_{c}$ for unlabeled samples. However, the task classifier $C$
is trained to maximize the entropy $\mathcal{H}$ while the feature extractor
$F$ is trained to minimize it. The overall adversarial learning objectives can
be formulated as:

$\displaystyle\hat{\theta}_{F}=\operatorname*{arg\,min}_{\theta_{F}}\mathcal{L}_{s}+\lambda_{t}\mathcal{L}_{t}+\lambda_{c}\mathcal{L}_{c}+\lambda\mathcal{H}$
(8)
$\displaystyle\hat{\theta}_{C}=\operatorname*{arg\,min}_{\theta_{C}}\mathcal{L}_{s}+\lambda_{t}\mathcal{L}_{t}+\lambda_{c}\mathcal{L}_{c}-\lambda\mathcal{H}$
(9)

As suggested by Saito et al. (2019), we use a gradient reversal layer Ganin &
Lempitsky (2015) to to flip the gradient between $F$ and $C$ with respect to
$\mathcal{H}$ and convert the minimax training with one forward and back-
propagation. We are ready to present Algorithm 1 for Semi-Supervised Domain
Adaptation.

Algorithm 1 Semi-Supervised Domain Adaptation in the Embedding Space

0: Source dataset $\mathcal{D}_{s}$, labeled target dataset $\mathcal{D}_{t}$,
Unlabeled target dataset $\mathcal{D}_{u}$.

0: Momentum encoder $F^{\prime}$ with encoder $F$ pretrained on labeled
samples. Memory $Q=\\{\\}$

1: repeat

2: Sample labeled source sample $(x_{s},y_{s})\in\mathcal{D}_{s}$, labeled
target sample $(x_{t},y_{t})\in\mathcal{D}_{t}$, and unlabeled target input
$x_{u}\in\mathcal{D}_{u}$.

3: Apply augmentation on inputs to get $\tilde{x}_{s}$, $\tilde{x}_{t}$, and
$\tilde{x}_{u}$

4: Compute supervised cross entropy loss for augmented labeled data using Eq.
2

5: Pass the original data points through the momentum encoder $F^{\prime}$ to
get features $f^{\prime}_{s}$, $f^{\prime}_{t}$, and $f^{\prime}_{u}$ and then
classifier $C$ to get predictions $p^{\prime}_{s}$, $p^{\prime}_{t}$, and
$p^{\prime}_{u}$.

6: Generate pseudo-label $\hat{y}_{u}=\arg\max(p^{\prime}_{u})$ if
$f^{\prime}_{u}$ falls inside the K-Nearest-Neighbours of prototypes and
feature memory $Q$

7: Compute target loss for generated pseudo-labels using Eq. 4

8: For unlabeled target samples, compute the contrastive loss between
$\tilde{f}$ and $f^{\prime}$ and the features in the memory bank $Q$ using Eq.
5 and 6.

9: Compute the entropy loss for unlabeled target samples to update the
prototypes using Eq. 7

10: Use a gradient reversal layer before the classifier layer to compute the
gradients and update the parameters for $\hat{\theta}_{F}$ and
$\hat{\theta}_{C}$ with Eq. 9

11: Update the momentum encoder parameters in Eq. 1

12: Add features and predictions to memory $Q$

13: until Training ends

14: return Trained encoder $F$ and classifier $C$

###  3.2 Discussion

A key difference between our approach and approaches such as Berthelot et al.
(2021) is the generation of pseudo-labels using prototypes based on the
embedding instead of the predictions. Recent approaches like AdaMatch
Berthelot et al. (2021) suffer from the imbalanced distribution during the
generation of pseudo-labels. AdaEmbed selects k nearest neighbors for each
prototype to generate a more uniform distribution of pseudo-labels.
Additionally, AdaEmbed utilizes the memory bank for the features instead of
only passing the gradient of strongly augmented version of the sample.

In addition to the pseudo-labeling strategy, AdaEmbed also incorporates
instance contrastive alignment, which has shown promising results in recent
domain adaptation literature. Like CLDA Singh (2021), AdaEmbed uses the
instance constrastive alignment, but it differs from CLDA by using the memory
bank for the features. Moreover, compare to ECACL Li et al. (2021) AdaEmbed
uses new categorical alignment loss and instance contrastive alignment.
Finally, the main difference between AdaContrast Chen et al. (2022) and
AdaEmbed is that the latter uses a new pseudo-labeling strategy based on
estimated prototype that are more accurate and more balanced compared to prior
works.

##  4 Experiments

###  4.1 Datasets

The performance of our proposed approach, AdaEmbed, is evaluated using several
benchmark datasets for domain adaptation. The primary dataset used in our
experiments is a subset of the original DomainNet dataset, referred to as
DomainNet-126 Peng et al. (2019). This subset includes $126$ classes across
$4$ domains, namely Real, Sketch, Clipart, and Painting. Our method is
evaluated on seven domain shifts, following the experimental settings outlined
by Saito et al. (2019). Additionally, we utilize the Office-Home dataset
Venkateswara et al. (2017) that includes $4$ domains (Real, Clipart, Art,
Product) encompassing $65$ classes and is widely regarded as a benchmark
dataset for unsupervised domain adaptation. Furthermore, we incorporate the
VisDA-C dataset Peng et al. (2017), in which we adapt synthetic images to real
images across $12$ object classes.

###  4.2 Implementation Details

In our experimental framework, the Swin Transformer V2 Liu et al. (2022)
serves as the backbone for feature extraction. We utilize the "tiny" variants
of the model, closely adhering to the hyperparameters outlined in their study.
Prior to the experiments, these Swin Transformer models are pre-trained on the
ImageNet-1K dataset Deng et al. (2009) ensuring a solid foundation for
subsequent adaptations. For classification tasks, we equip each model with a
singular linear head.

Throughout the experimentation process, we organize our data into three
distinct mini-batches. The first batch consists of labeled samples from the
source domain, the second batch is composed of labeled samples from the target
domain, and the third batch encompasses unlabeled samples from the target
domain. We incorporate a RandAugment Cubuk et al. (2020) as our data
augmentation strategy. Additionally, random erasing is employed to further
diversify the training dataset.

For the optimization process, we utilize a Stochastic Gradient Descent (SGD)
solver with a base learning rate of $0.05$, following a cosine learning rate
policy. The experiments are conducted over a maximum of $50$ epochs,
maintaining a momentum of $0.9$ and a weight decay of $10^{-4}$.

For hyperparameters in this method, $\lambda$ is set at $0.2$, $\lambda_{t}$
at 2.0, and $\lambda_{c}$ at $0.1$. The temperature parameter is set to
$0.05$, with an Exponential Moving Average (EMA) of $m=0.95$. We use $k=10$
neighbors and $\tau=0.9$, to effectively generate pseudo labels. Additionally,
a memory queue of size M=1000 is utilized to store features and predictions,
facilitating the generation of pseudo-labels and computation of contrastive
loss.

All experiments were conducted using the same source code and identical
augmentation and hyperparameter settings. This approach ensures consistency
across all tests, significantly improving the fairness and reliability of the
experiments. Our unified framework, designed for simplicity and
comprehensiveness, allows for direct comparisons between methods and a clear
understanding of their individual contributions to the overall performance.

###  4.3 Baselines

Our AdaEmbed method is compared with the following domain adaptation
strategies:

  * •

Supervised only: This approach forms our basic comparison baseline, relying
exclusively on labeled data during training.

  * •

Minimax Entropy (MME) Saito et al. (2019): Specially designed for SSDA
scenarios, MME employs adversarial training to align features and estimate
prototypes. It introduces a minimax game where the feature extractor and
classifier are trained in tandem to optimize domain-invariant feature
representations.

  * •

Contrastive Learning for Semi-Supervised Domain Adaptation (CLDA) Singh
(2021): CLDA applies contrastive learning to reduce both inter- and intra-
domain discrepancies, utilizing inter-domain and instance contrastive
alignment.

  * •

Enhanced Categorical Alignment and Consistency Learning (ECACL) Li et al.
(2021): ECACL leverages strong data augmentation to enhance alignment of
category-level features between source and target domains, targeting SSDA
scenarios.

  * •

AdaMatch Berthelot et al. (2021): AdaMatch represents a unified model that is
applicable to Semi-Supervised Learning (SSL), UDA, and SSDA. It accounts for
distributional shifts between source and target domains and combines
techniques from SSL and UDA to achieve effective domain adaptation.

  * •

AdaContrast Chen et al. (2022): Specifically designed for test-time domain
adaptation in UDA settings, AdaContrast refines pseudo-labels via soft voting
among nearest neighbors in the target feature space.

###  4.4 DomainNet Experiments

In our evaluation on the DomainNet dataset, AdaEmbed has showcased remarkable
performance in both UDA and SSDA settings, as detailed in Table 1. AdaEmbed
consistently outperformed all the baselines, demonstrating its robustness and
adaptability across varied domain adaptation tasks.

In UDA scenarios, AdaEmbed significantly surpassed the supervised baseline in
various domain pairs. For instance, in the Real to Clipart (R to C)
adaptation, AdaEmbed achieved an impressive accuracy of $79.14\%$, a marked
improvement over the source-only method’s $62.31\%$. Similarly, in the
Painting to Real (P to R) shift, AdaEmbed attained $82.69\%$ accuracy,
outperforming all baselines. In SSDA settings, particularly in one-shot and
three-shot scenarios, AdaEmbed’s effectiveness was further highlighted.
Notably, in the challenging Clipart to Sketch (C to S) task, AdaEmbed reached
accuracies of $72.29\%$ and $74.10\%$ for one-shot and three-shot settings,
respectively, outperforming all the baselines. These results illustrate
AdaEmbed’s comprehensive approach to domain adaptation.

Setting | Method | R $\rightarrow$ C | R $\rightarrow$ P | P $\rightarrow$ C | C $\rightarrow$ S | S $\rightarrow$ P | R $\rightarrow$ S | P $\rightarrow$ R | Avg.
---|---|---|---|---|---|---|---|---|---
UDA | Source | 62.31 | 66.43 | 60.38 | 57.68 | 56.79 | 51.63 | 75.93 | 61.59
MME | 67.14 | 70.14 | 62.28 | 57.55 | 61.63 | 59.00 | 75.55 | 63.46
CLDA | 70.18 | 70.92 | 67.10 | 63.67 | 65.28 | 63.26 | 77.91 | 68.33
ECACL | 78.30 | 78.54 | 74.04 | 35.02 | 72.68 | 73.03 | 78.78 | 70.05
AdaMatch | 77.97 | 78.33 | 74.27 | 69.44 | 73.97 | 73.78 | 80.20 | 75.42
| AdaContrast | 74.19 | 76.57 | 72.51 | 68.40 | 67.95 | 70.78 | 79.87 | 72.90
| AdaEmbed | 79.14 | 78.22 | 74.37 | 70.01 | 74.87 | 72.53 | 82.69 | 75.98
1-shot | S+T | 63.47 | 67.53 | 64.08 | 58.04 | 59.48 | 54.72 | 76.91 | 63.46
MME | 71.53 | 71.44 | 69.63 | 64.47 | 67.33 | 62.59 | 77.78 | 69.25
CLDA | 71.86 | 71.80 | 70.59 | 64.68 | 67.95 | 64.23 | 79.99 | 70.15
ECACL | 80.22 | 78.97 | 75.53 | 71.82 | 72.77 | 74.27 | 82.73 | 76.62
AdaMatch | 79.77 | 78.86 | 75.38 | 72.67 | 75.70 | 72.65 | 82.69 | 76.82
| AdaContrast | 78.18 | 77.65 | 76.94 | 71.95 | 74.26 | 73.30 | 84.25 | 76.65
| AdaEmbed | 80.08 | 78.82 | 76.16 | 72.29 | 75.23 | 74.26 | 84.51 | 77.34
3-shot | S+T | 70.09 | 70.35 | 71.93 | 66.77 | 68.63 | 61.29 | 80.34 | 69.92
MME | 74.51 | 73.62 | 73.36 | 67.77 | 70.80 | 67.23 | 79.91 | 72.46
CLDA | 74.43 | 73.08 | 74.09 | 68.01 | 70.79 | 66.41 | 81.57 | 72.63
ECACL | 82.37 | 79.81 | 79.24 | 73.46 | 77.20 | 75.74 | 84.74 | 78.94
AdaMatch | 80.68 | 79.11 | 77.30 | 73.76 | 77.07 | 75.11 | 84.95 | 78.28
| AdaContrast | 81.39 | 79.15 | 78.90 | 73.18 | 77.96 | 75.25 | 85.25 | 78.72
| AdaEmbed | 81.30 | 79.69 | 78.86 | 74.10 | 77.53 | 75.64 | 85.69 | 78.97

Table 1: Performance of AdaEmbed on the DomainNet-126 in UDA and SSDA (1-shot
and 3-shot) settings.

###  4.5 Office-Home Experiments

The effectiveness of AdaEmbed was rigorously evaluated on the Office-Home
dataset, a challenging benchmark for domain adaptation. This dataset,
encompassing diverse domains and a wide array of classes, offers a robust
platform for testing the versatility of domain adaptation techniques. As Table
2 illustrates, AdaEmbed showcased superior performance in both UDA and SSDA
settings.

In UDA, AdaEmbed notably outperformed the Supervised baseline. For instance,
in the Real to Clipart (R to C) adaptation, AdaEmbed attained an accuracy of
$64.51\%$, significantly higher than the baseline’s $49.22\%$. Similarly, in
the Art to Clipart (A to C) transition, AdaEmbed’s accuracy reached $56.34\%$,
exceeding the baseline’s $43.15\%$. In the SSDA context, particularly within
one-shot and three-shot frameworks, AdaEmbed’s proficiency in leveraging
limited labeled data in the target domain was evident. In the one-shot
scenario, AdaEmbed achieved $66.60\%$ accuracy in R to C split and $60.23\%$
in A to C split. Progressing to three-shot adaptation, the accuracies improved
to $69.92\%$ and $67.45\%$ respectively. This demonstrates AdaEmbed’s
effectiveness in utilizing sparse labeled data to enhance adaptability and
accuracy in varied domain adaptations.

Setting | Method | R $\rightarrow$ C | R $\rightarrow$ P | R $\rightarrow$ A | P $\rightarrow$ R | P $\rightarrow$ C | P $\rightarrow$ A | A $\rightarrow$ P | A $\rightarrow$ C | A $\rightarrow$ R | C $\rightarrow$ R | C $\rightarrow$ A | C $\rightarrow$ P | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
UDA | Source | 49.22 | 80.47 | 68.46 | 74.13 | 40.73 | 56.17 | 68.85 | 43.15 | 76.90 | 67.82 | 56.58 | 65.92 | 62.37
MME | 59.18 | 83.22 | 72.49 | 78.12 | 49.82 | 60.40 | 70.09 | 50.55 | 78.21 | 71.47 | 62.05 | 69.08 | 67.06
CLDA | 54.12 | 80.74 | 69.49 | 75.50 | 45.65 | 58.80 | 69.82 | 47.73 | 76.79 | 70.00 | 60.16 | 67.18 | 64.67
ECACL | 63.39 | 85.95 | 76.60 | 81.88 | 56.32 | 69.16 | 76.10 | 53.96 | 80.23 | 73.74 | 64.27 | 74.41 | 71.33
AdaMatch | 63.44 | 85.18 | 76.03 | 83.19 | 57.72 | 68.01 | 74.73 | 56.34 | 80.16 | 77.02 | 67.19 | 75.92 | 72.08
AdaContrast | 59.48 | 85.14 | 76.97 | 81.01 | 52.88 | 71.83 | 75.59 | 49.84 | 78.92 | 74.79 | 69.00 | 74.95 | 70.87
| AdaEmbed | 64.51 | 86.19 | 75.90 | 82.48 | 55.86 | 68.71 | 77.61 | 56.34 | 82.29 | 77.71 | 69.24 | 77.18 | 72.84
1-shot | S+T | 54.35 | 83.29 | 72.66 | 78.39 | 47.53 | 61.31 | 77.70 | 51.58 | 78.21 | 73.10 | 63.32 | 74.86 | 68.03
MME | 61.79 | 85.92 | 75.95 | 80.21 | 56.66 | 65.91 | 79.93 | 56.85 | 80.39 | 75.16 | 68.75 | 78.72 | 72.19
CLDA | 58.01 | 84.30 | 73.19 | 78.78 | 49.95 | 63.86 | 78.63 | 54.46 | 78.94 | 73.65 | 65.42 | 76.24 | 69.62
ECACL | 66.21 | 87.70 | 78.37 | 84.36 | 59.75 | 72.53 | 83.36 | 59.34 | 82.80 | 78.14 | 69.70 | 81.58 | 75.32
AdaMatch | 64.26 | 87.41 | 79.48 | 84.45 | 60.51 | 75.45 | 82.25 | 59.32 | 82.27 | 80.11 | 72.41 | 82.03 | 75.83
AdaContrast | 63.35 | 86.73 | 79.03 | 83.72 | 61.33 | 75.99 | 82.45 | 59.94 | 82.18 | 78.56 | 72.33 | 81.71 | 75.61
| AdaEmbed | 66.60 | 87.77 | 77.92 | 84.31 | 59.66 | 74.42 | 83.36 | 60.23 | 83.99 | 79.70 | 72.33 | 82.97 | 76.11
3-shot | S+T | 63.58 | 86.04 | 76.56 | 80.07 | 58.26 | 69.04 | 82.64 | 60.37 | 81.51 | 78.10 | 70.07 | 80.02 | 73.85
MME | 67.83 | 86.89 | 78.78 | 82.57 | 63.55 | 73.07 | 83.85 | 63.67 | 82.36 | 79.84 | 72.78 | 84.08 | 76.61
CLDA | 63.85 | 86.08 | 75.70 | 80.99 | 58.38 | 69.49 | 82.12 | 61.54 | 80.92 | 78.12 | 71.46 | 80.65 | 74.11
ECACL | 70.51 | 89.50 | 81.46 | 85.09 | 66.60 | 77.14 | 86.22 | 64.58 | 83.94 | 81.56 | 74.75 | 86.01 | 78.95
AdaMatch | 69.89 | 88.15 | 80.59 | 85.87 | 65.29 | 77.59 | 85.74 | 64.77 | 84.56 | 82.16 | 77.01 | 85.34 | 78.91
AdaContrast | 68.32 | 88.11 | 80.43 | 83.74 | 66.55 | 78.00 | 85.25 | 62.93 | 83.46 | 83.12 | 76.11 | 85.77 | 78.48
| AdaEmbed | 69.92 | 89.26 | 79.81 | 85.92 | 66.41 | 75.78 | 84.95 | 67.45 | 84.77 | 82.80 | 76.69 | 85.77 | 79.13

Table 2: Experiment results on Office-Home dataset across both UDA and SSDA
settings.

###  4.6 VisDA-C Experiments

AdaEmbed’s capabilities were rigorously evaluated on the VisDA-C dataset, a
key benchmark for domain adaptation that concentrates on the transition from
synthetic to real-world imagery. This dataset’s scope, covering 12 distinct
object classes, provided a comprehensive platform for detailed analysis,
including class-wise accuracies, average accuracy, and overall performance
metrics. The results, detailed in Table 3, demonstrate AdaEmbed’s remarkable
proficiency in both UDA and SSDA frameworks.

In UDA scenarios, AdaEmbed’s performance was particularly notable. It
consistently outstripped the achievements of other methods. For example, in
the challenging category of trucks, AdaEmbed achieved an accuracy of $58\%$,
which was a significant improvement over the supervised baseline’s $12\%$ and
higher than AdaContrast’s $56\%$. Similarly, in the skateboard category,
AdaEmbed’s accuracy of $86\%$ was notably higher than the supervised
baseline’s $29\%$ and exceeded AdaContrast’s $43\%$. These results illustrate
AdaEmbed’s capability to effectively utilize unlabeled data in the target
domain, underlining its consistent superiority across various classes.

Setting | Method | plane | bcycl | bus | car | horse | knife | mcycl | person | plant | sktbrd | train | truck | Avg. | Acc.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
UDA | Source | 90 | 50 | 64 | 76 | 90 | 10 | 94 | 32 | 85 | 29 | 93 | 12 | 60.42 | 65.57
MME | 92 | 29 | 33 | 65 | 75 | 4 | 66 | 16 | 98 | 8 | 81 | 2 | 47.42 | 52.37
CLDA | 96 | 72 | 85 | 73 | 97 | 19 | 95 | 35 | 94 | 49 | 91 | 14 | 68.33 | 71.08
ECACL | 97 | 80 | 91 | 60 | 98 | 18 | 97 | 54 | 96 | 55 | 91 | 14 | 70.92 | 72.46
AdaMatch | 98 | 88 | 90 | 75 | 98 | 95 | 95 | 65 | 96 | 84 | 93 | 51 | 85.67 | 83.74
AdaContrast | 97 | 90 | 85 | 74 | 97 | 96 | 93 | 84 | 94 | 43 | 92 | 56 | 83.42 | 82.92
| AdaEmbed | 98 | 86 | 88 | 74 | 97 | 94 | 95 | 82 | 96 | 86 | 91 | 58 | 87.08 | 85.17
1-shot | S+T | 92 | 76 | 93 | 63 | 96 | 31 | 91 | 54 | 95 | 54 | 80 | 6 | 69.25 | 69.88
MME | 90 | 69 | 89 | 77 | 93 | 17 | 92 | 78 | 97 | 32 | 69 | 1 | 67.0 | 70.68
CLDA | 95 | 79 | 92 | 68 | 97 | 50 | 93 | 56 | 96 | 57 | 85 | 9 | 73.08 | 73.39
ECACL | 97 | 82 | 91 | 76 | 98 | 71 | 96 | 69 | 95 | 42 | 90 | 14 | 76.75 | 77.78
AdaMatch | 97 | 85 | 90 | 72 | 98 | 94 | 96 | 60 | 97 | 87 | 93 | 50 | 84.92 | 82.76
AdaContrast | 98 | 90 | 84 | 69 | 97 | 96 | 92 | 50 | 96 | 1 | 92 | 62 | 77.25 | 78.39
| AdaEmbed | 97 | 89 | 85 | 75 | 97 | 95 | 94 | 84 | 94 | 86 | 92 | 54 | 86.83 | 85.05
3-shot | S+T | 95 | 80 | 77 | 74 | 94 | 74 | 88 | 40 | 90 | 50 | 87 | 39 | 74.0 | 74.68
MME | 95 | 71 | 74 | 84 | 93 | 74 | 95 | 69 | 94 | 41 | 88 | 22 | 75.0 | 76.64
CLDA | 96 | 84 | 83 | 73 | 96 | 77 | 92 | 39 | 93 | 60 | 90 | 42 | 77.08 | 76.98
ECACL | 97 | 82 | 89 | 80 | 98 | 90 | 96 | 75 | 96 | 82 | 92 | 40 | 84.75 | 83.80
AdaMatch | 98 | 86 | 90 | 77 | 98 | 94 | 96 | 59 | 96 | 89 | 94 | 52 | 85.75 | 84.04
AdaContrast | 98 | 90 | 86 | 68 | 96 | 95 | 92 | 61 | 94 | 88 | 92 | 60 | 85.0 | 82.34
| AdaEmbed | 98 | 88 | 87 | 82 | 98 | 93 | 96 | 70 | 96 | 89 | 94 | 56 | 87.25 | 85.88

Table 3: UDA and SSDA Results on VisDA-C Dataset

###  4.7 Ablation Studies

The ablation study shown in Table 4 investigates the contributions of
different components in the AdaEmbed method, specifically focusing on their
impact on the VisDA-C experiments in UDA setting. The first scenario in our
study, "No pseudo-labeling," omits the use of pseudo-labels during training.
The results, showing a drop in accuracy to $70.62\%$, underscore the
significance of pseudo-labeling in enhancing the model’s performance for
unsupervised domain adaptation. When the contrastive loss component is
removed, as shown in the second row, there is a noticeable but relatively
moderate decrease in accuracy to $85.04\%$. This result suggests that while
the contrastive loss contributes to the overall efficacy of AdaEmbed, its
impact may not be as critical as that of the other components. The third row,
"No entropy loss," highlights the substantial role of entropy loss in
AdaEmbed. Omitting this component leads to a significant reduction in accuracy
to $83.43\%$. This observation indicates that entropy loss is essential in
AdaEmbed’s methodology, particularly in ensuring the effective alignment of
domain-specific features. Finally, the complete AdaEmbed method, integrating
all three losses ($\mathcal{L}_{t}$, $\mathcal{L}_{c}$, and $\mathcal{H}$),
achieves the best performance, with an average accuracy of $87.08\%$ and
overall accuracy of $85.17\%$ on the VisDA-C dataset. This outcome
demonstrates the collective strength of these components in AdaEmbed,
validating their synergistic impact on enhancing the model’s domain adaptation
capabilities.

Ablation | $\mathcal{L}_{t}$ | $\mathcal{L}_{c}$ | $\mathcal{H}$ | Avg. | Acc.
---|---|---|---|---|---
No pseudo-labeling |  | ✓ | ✓ | 66.92 | 70.62
No contrastive loss | ✓ |  | ✓ | 86.41 | 85.04
No entropy loss | ✓ | ✓ |  | 86.34 | 83.43
AdaEmbed | ✓ | ✓ | ✓ | 87.08 | 85.17
Table 4: Ablation Studies on VisDA-C Dataset

###  4.8 Varying Number of Labels

In the context of semi-supervised domain adaptation, we analyzed the
performance impact of a varying number of labeled examples on the OfficeHome
dataset Real to Clipart split, as depicted in Figure 3. Our AdaEmbed method
consistently outperformed the Supervised Learning and AdaContrast approaches
across different quantities of labeled data, notably excelling with fewer
labels. The trend suggests a significant initial gain in accuracy with an
increase in labeled examples up to 5 per class, beyond which the rate of
improvement moderates. While the Supervised method shows a rapid ascent
initially, it plateaus, implying a limited benefit from additional data. In
contrast, AdaEmbed demonstrates its robustness and efficiency in utilizing
sparse labeled data, a critical advantage when such data is a scarce resource,
thereby endorsing its adaptability and effectiveness in domain adaptation
tasks.

Figure 3: The impact of varying the number of labeled examples on the accuracy
of domain adaptation methods in the OfficeHome dataset’s Real to Clipart
split. The plot demonstrates the performance of three strategies: Supervised
Learning, AdaContrast, and AdaEmbed. Each method’s accuracy improves with more
labeled data, with AdaEmbed showing the highest efficiency and performance
across all data regimes, especially in settings with fewer labeled examples.

###  4.9 Feature Visualization

In the unsupervised domain adaptation (UDA) setting on the DomainNet dataset’s
Real to Clipart split, t-SNE visualizations of feature embeddings from four
domain adaptation strategies—Supervised Learning (Source Only), AdaMatch,
AdaContrast, and AdaEmbed—are analyzed. The Supervised method demonstrates
distinct clusters with some overlap, indicating limited domain adaptation.
AdaMatch shows tighter clustering, yet with notable inter-class mixing.
AdaContrast’s approach yields clearer class demarcation, signifying better
domain invariance. Our proposed AdaEmbed method outshines the others,
presenting the most compact and well-separated clusters, indicative of
superior domain adaptation by effectively aligning domain-invariant features
and minimizing within-class variance. These embeddings, as visualized,
validate AdaEmbed’s robustness in learning transferable features across
domains.

(a) Supervised

(b) AdaMatch

(c) AdaContrast

(d) AdaEmbed

Figure 4: Comparative t-SNE visualizations of feature embeddings from
different domain adaptation methods on the DomainNet Real to Clipart split
under unsupervised domain adaptation setting. Green and orange circles
represent the source and target domains, respectively, while purple crosses
mark the prototypes. The Supervised Learning method exhibits basic clustering
with overlaps, reflecting a foundational level of domain adaptation. AdaMatch
and AdaContrast display progressively tighter and more distinct clusters,
signaling advancements in domain invariance. However, AdaEmbed stands out
distinctly, showcasing highly compact and distinctly separated clusters. These
embeddings visually encapsulate the superior domain adaptation proficiency of
AdaEmbed.

##  5 Conclusion

In this study, we introduced AdaEmbed, a novel and efficient approach for
semi-supervised domain adaptation (SSDA) that successfully navigates the
challenges posed by domain shift in computer vision tasks. AdaEmbed is
designed to leverage the synergy between a shared embedding space and
prototype generation, facilitating the creation of accurate and balanced
pseudo-labels for unlabeled target domains. This approach effectively
harnesses cross-entropy loss and contrastive loss as dual sources of
supervision, addressing the imbalance in pseudo-label distribution—a common
challenge in SSDA.

Our extensive evaluation of AdaEmbed across prominent domain adaptation
benchmarks, such as DomainNet-126, Office-Home, and VisDA-C, has demonstrated
its remarkable efficacy in both SSDA and UDA settings. The results
consistently indicate AdaEmbed’s superiority over existing state-of-the-art
methods, establishing new benchmarks across a range of evaluation metrics.
Notably, AdaEmbed’s innovative pseudo-labeling strategy and its robustness
against distribution shifts in the embedding space set it apart from existing
methods. AdaEmbed emerges as a robust and practical solution for domain
adaptation in real-world scenarios, where labeled data is often scarce or
unevenly distributed. By addressing both theoretical and practical aspects of
SSDA, AdaEmbed paves the way for future research in semi-supervised domain
adaptation.

#### Broader Impact Statement

This research on AdaEmbed for semi-supervised domain adaptation primarily
offers positive advancements in computer vision, particularly in efficiently
handling domain shifts with limited data. Its model-agnostic design enhances
its applicability across various sectors and promotes resource efficiency,
especially in settings with limited computational resources. However, it is
crucial to be mindful of potential negative impacts, such as the misuse of the
technology in privacy-sensitive applications like surveillance, the risk of
reduced human oversight in critical decision-making processes, and the
perpetuation of inherent data biases. As such, while AdaEmbed presents
significant benefits, its ethical use, continuous evaluation for fairness, and
maintenance of human oversight are essential to ensure its responsible and
beneficial application in society.

## References

  * Ao et al. (2017) Shuang Ao, Xiang Li, and Charles X. Ling.  Fast generalized distillation for semi-supervised domain adaptation.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , AAAI’17, pp. 1719–1725. AAAI Press, 2017.
  * Bateni et al. (2020) Peyman Bateni, Jarred Barber, Jan-Willem van de Meent, and Frank Wood.  Improving few-shot visual classification with unlabelled examples.  _CoRR_ , 2020.
  * Berthelot et al. (2019a) David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring.  _CoRR_ , 2019a.
  * Berthelot et al. (2019b) David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  _CoRR_ , 2019b.
  * Berthelot et al. (2021) David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin.  Adamatch: A unified approach to semi-supervised learning and domain adaptation.  _arXiv preprint arXiv:2106.04732_ , 2021.
  * Bruzzone & Marconcini (2010) Lorenzo Bruzzone and Mattia Marconcini.  Domain adaptation problems: A dasvm classification technique and a circular validation strategy.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 2010.
  * Cai et al. (2018) Qi Cai, Yingwei Pan, Ting Yao, Chenggang Yan, and Tao Mei.  Memory matching networks for one-shot image recognition.  _CoRR_ , 2018.
  * Caron et al. (2020) Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _CoRR_ , 2020.
  * Chen et al. (2018) Chaoqi Chen, Weiping Xie, Tingyang Xu, Wen-bing Huang, Yu Rong, Xinghao Ding, Yue Huang, and Junzhou Huang.  Progressive feature alignment for unsupervised domain adaptation.  _CoRR_ , 2018.
  * Chen et al. (2022) Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi.  Contrastive test-time adaptation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 295–305, 2022.
  * Chen et al. (2011) Minmin Chen, Kilian Q Weinberger, and John Blitzer.  Co-training for domain adaptation.  In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), _Advances in Neural Information Processing Systems_ , volume 24. Curran Associates, Inc., 2011.
  * Chen et al. (2020a) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020a.
  * Chen et al. (2020b) Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton.  Big self-supervised models are strong semi-supervised learners.  _CoRR_ , 2020b.
  * Chen et al. (2019) Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  In _International Conference on Learning Representations_ , 2019.
  * Cubuk et al. (2020) Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pp. 702–703, 2020.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Dhillon et al. (2020) Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto.  A baseline for few-shot image classification.  In _International Conference on Learning Representations_ , 2020.
  * Donahue et al. (2014) Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell.  Decaf: A deep convolutional activation feature for generic visual recognition.  In _Proceedings of the 31st International Conference on Machine Learning_ , 2014.
  * Edwards & Storkey (2017) Harrison Edwards and Amos Storkey.  Towards a neural statistician.  In _International Conference on Learning Representations_ , 2017.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , 2017.
  * Ganin & Lempitsky (2015) Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In _International conference on machine learning_ , pp. 1180–1189. PMLR, 2015.
  * Ganin et al. (2016) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  _The journal of machine learning research_ , 17(1):2096–2030, 2016.
  * Gretton et al. (2006) Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola.  A kernel method for the two-sample-problem.  _Advances in neural information processing systems_ , 19, 2006.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko.  Bootstrap your own latent: A new approach to self-supervised learning.  _CoRR_ , 2020.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 9729–9738, 2020.
  * Jamal & Qi (2019) Muhammad Abdullah Jamal and Guo-Jun Qi.  Task agnostic meta-learning for few-shot learning.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Kang et al. (2018) Guoliang Kang, Liang Zheng, Yan Yan, and Yi Yang.  Deep adversarial attention alignment for unsupervised domain adaptation: the benefit of target expectation maximization.  In _Proceedings of the European conference on computer vision (ECCV)_ , pp. 401–416, 2018.
  * Koch et al. (2015) Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al.  Siamese neural networks for one-shot image recognition.  In _ICML deep learning workshop_ , volume 2. Lille, 2015.
  * Laine & Aila (2016) Samuli Laine and Timo Aila.  Temporal ensembling for semi-supervised learning.  _CoRR_ , 2016.
  * Li et al. (2021) Kai Li, Chang Liu, Handong Zhao, Yulun Zhang, and Yun Fu.  Ecacl: A holistic framework for semi-supervised domain adaptation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 8578–8587, 2021.
  * Li et al. (2017) Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li.  Meta-sgd: Learning to learn quickly for few-shot learning, 2017.
  * Liang et al. (2019) Jian Liang, Ran He, Zhenan Sun, and Tieniu Tan.  Aggregating randomized clustering-promoting invariant projections for domain adaptation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2019.
  * Liang et al. (2020) Jian Liang, Dapeng Hu, and Jiashi Feng.  Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.  _CoRR_ , 2020.
  * Liu et al. (2019) Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sungju Hwang, and Yi Yang.  Learning to propagate labels: Transductive propagation network for few-shot learning.  In _International Conference on Learning Representations_ , 2019.
  * Liu et al. (2022) Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al.  Swin transformer v2: Scaling up capacity and resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 12009–12019, 2022.
  * Long et al. (2013) Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S. Yu.  Transfer feature learning with joint distribution adaptation.  In _2013 IEEE International Conference on Computer Vision_ , 2013.
  * Long et al. (2015) Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan.  Learning transferable features with deep adaptation networks.  In _International conference on machine learning_ , pp. 97–105. PMLR, 2015.
  * Long et al. (2016) Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.  Unsupervised domain adaptation with residual transfer networks.  _Advances in neural information processing systems_ , 29, 2016.
  * Mishra et al. (2018) Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel.  A simple neural attentive meta-learner.  In _International Conference on Learning Representations_ , 2018.
  * Munkhdalai & Yu (2017) Tsendsuren Munkhdalai and Hong Yu.  Meta networks.  _CoRR_ , abs/1703.00837, 2017.
  * Munkhdalai et al. (2017) Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, Tong Wang, and Adam Trischler.  Learning rapid-temporal adaptations.  _CoRR_ , 2017.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Oreshkin et al. (2018) Boris N. Oreshkin, Pau Rodriguez, and Alexandre Lacoste.  Tadam: Task dependent adaptive metric for improved few-shot learning, 2018.
  * Peng et al. (2017) Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko.  Visda: The visual domain adaptation challenge.  _arXiv preprint arXiv:1710.06924_ , 2017.
  * Peng et al. (2019) Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 1406–1415, 2019.
  * Ravi & Larochelle (2016) Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  In _International conference on learning representations_ , 2016.
  * Rusu et al. (2019) Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell.  Meta-learning with latent embedding optimization.  In _International Conference on Learning Representations_ , 2019.
  * Sahoo et al. (2021) Aadarsh Sahoo, Rutav Shah, Rameswar Panda, Kate Saenko, and Abir Das.  Contrast and mix: Temporal contrastive video domain adaptation with background mixing.  _Advances in Neural Information Processing Systems_ , 34:23386–23400, 2021.
  * Saito et al. (2017) Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada.  Asymmetric tri-training for unsupervised domain adaptation.  _CoRR_ , 2017.
  * Saito et al. (2018) Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3723–3732, 2018.
  * Saito et al. (2019) Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko.  Semi-supervised domain adaptation via minimax entropy.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 8050–8058, 2019.
  * Sajjadi et al. (2016) Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen.  Regularization with stochastic transformations and perturbations for deep semi-supervised learning.  _Advances in neural information processing systems_ , 29, 2016.
  * Santoro et al. (2016) Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap.  Meta-learning with memory-augmented neural networks.  In _International conference on machine learning_ , pp. 1842–1850, 2016.
  * Satorras & Estrach (2018) Victor Garcia Satorras and Joan Bruna Estrach.  Few-shot learning with graph neural networks.  In _International Conference on Learning Representations_ , 2018.
  * Shen et al. (2021) Xi Shen, Yang Xiao, Shell Xu Hu, Othman Sbai, and Mathieu Aubry.  Re-ranking for image retrieval and transductive few-shot classification.  _Advances in Neural Information Processing Systems_ , 34:25932–25943, 2021.
  * Singh (2021) Ankit Singh.  Clda: Contrastive learning for semi-supervised domain adaptation.  _Advances in Neural Information Processing Systems_ , 34:5089–5101, 2021.
  * Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard S. Zemel.  Prototypical networks for few-shot learning, 2017.
  * Sohn et al. (2020) Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _Advances in Neural Information Processing Systems_ , 33:596–608, 2020.
  * Sun & Saenko (2016) Baochen Sun and Kate Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In _European conference on computer vision_ , pp. 443–450. Springer, 2016.
  * Sun et al. (2019) Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele.  Meta-transfer learning for few-shot learning.  In _CVPR_ , pp. 403–412, 2019.
  * Sung et al. (2017) Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales.  Learning to compare: Relation network for few-shot learning, 2017.
  * Tarvainen & Valpola (2017) Antti Tarvainen and Harri Valpola.  Weight-averaged consistency targets improve semi-supervised deep learning results.  _CoRR_ , 2017.
  * Tzeng et al. (2017) E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell.  Adversarial discriminative domain adaptation.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Tzeng et al. (2014) Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell.  Deep domain confusion: Maximizing for domain invariance.  _arXiv preprint arXiv:1412.3474_ , 2014.
  * Venkateswara et al. (2017) Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 5018–5027, 2017.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  In _Advances in neural information processing systems_ , pp. 3630–3638, 2016.
  * Wang et al. (2021) Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell.  Tent: Fully test-time adaptation by entropy minimization.  In _International Conference on Learning Representations_ , 2021.
  * Wang et al. (2019) Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens Van Der Maaten.  Simpleshot: Revisiting nearest-neighbor classification for few-shot learning.  _arXiv preprint arXiv:1911.04623_ , 2019.
  * Xiao & Zhang (2021) Ni Xiao and Lei Zhang.  Dynamic weighted learning for unsupervised domain adaptation.  _CoRR_ , 2021.
  * Xie et al. (2018) Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen.  Learning semantic representations for unsupervised domain adaptation.  In _Proceedings of the 35th International Conference on Machine Learning_. PMLR, 2018.
  * Yao et al. (2015) Ting Yao, Yingwei Pan, Chong-Wah Ngo, Houqiang Li, and Tao Mei.  Semi-supervised domain adaptation with subspace learning for visual recognition.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pp. 2142–2150, 2015.
  * Zhang et al. (2018a) Jing Zhang, Zewei Ding, Wanqing Li, and Philip Ogunbona.  Importance weighted adversarial nets for partial domain adaptation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 8156–8164, 2018a.
  * Zhang et al. (2018b) Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu.  Collaborative and adversarial network for unsupervised domain adaptation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018b.


# An Affective Robot Companion for Assisting
the Elderly in a Cognitive Game Scenario

Nikhil Churamani, Alexander Sutherland and Pablo Barros  Knowledge Technology,
Department of Informatics
University of Hamburg, Hamburg, Germany
Email:{5churama, sutherland<EMAIL_ADDRESS>
###### Abstract

Being able to recognize emotions in human users is considered a highly
desirable trait in Human-Robot Interaction (HRI) scenarios. However, most
contemporary approaches rarely attempt to apply recognized emotional features
in an active manner to modulate robot decision-making and dialogue for the
benefit of the user. In this position paper, we propose a method of
incorporating recognized emotions into a Reinforcement Learning (RL) based
dialogue management module that adapts its dialogue responses in order to
attempt to make cognitive training tasks, like the 2048 Puzzle Game, more
enjoyable for the users.

##  I Introduction

Decreased physical, mental and social activity may lead to several cognitive
disorders in the elderly. As an individual retires from work, the risk of
cognitive-decline increases with age due to decreased mental and physical
activity [1]. Engaging in cognitively stimulating activities such as playing
board games or solving puzzles may reduce the risk of severe cognitive
deterioration and protect against the development of dementia [2].
Furthermore, coping with the decreased social activity with the inclusion of
robot companions may lead to positive long-term benefits as a result of
modelling engaging human-robot interactions [3]. Such robot companions can
assist the users in solving cognitive tasks such as puzzles, keeping them
mentally stimulated and at the same time engaging them in meaningful affective
interactions helping them to cope with the stress of their day-to-day life.
The goal of such a robot thus becomes to improve the performance of the user
in the task and at the same time reduce their stress, improving the overall
quality of their daily life.

Accounting for the user’s affective state becomes a critical part of such an
interaction design. Learning to adapt to the user’s mood allows the robot to
improve their interaction experience by changing its strategy, given the user
is feeling positive or negative while solving the task. Such an affective
understanding in the robot forms the basis for learning different emotional
expressions in order to empathise with the user [4]. Additionally, this
affective appraisal in the robot can also be used to model an interaction with
the user, directed towards solving a specific task.

In this position paper, we attempt to model a pedagogical approach towards
teaching the user to solve a cognitive task that requires logical and spacial
awareness. The robot shall actively monitor the emotional state of the user
and use it, along with the performance of the user in the task, to adapt its
strategy towards modelling an interaction with the user.

Figure 1: Set-up: The NAO robot as an advisory companion using visual,
auditory and linguistic cues to infer the emotional state of the user,
offering assistance in the $2048$ Puzzle Game. Figure 2: User asking the Nao
Robot for assistance in solving the 2048 Puzzle Game scenario.

##  II Interaction Scenario

The interaction scenario shall involve the subjects playing several rounds of
the $2048$ Puzzle111https://gabrielecirulli.github.io/2048/ [Accessed
01.04.2018]  Game [5] with the Nao robot (see Fig. 1). The robot shall be pre-
trained (using Reinforcement Learning (RL) techniques [6]) on solving the game
and shall provide suggestions to the users, depending upon their emotional
state, for the next best move to succeed in the game. If the user is feeling
positive and performing well in the game, the robot shall focus on further
improving their performance, actively giving feedback (verbal and gestural)
about their moves in the game. In case the user is feeling negative or
frustrated, the robot shall focus more on the user and try to improve their
emotional state. It shall adapt its behaviour to motivate the user towards the
task by offering them assistance and help them cope with the frustration. The
overall goal of the robot, in either situation, shall be to complete the game
with the user and at the same time achieve an overall improvement in the mood
of the user. The robot should also learn when to give advice to the user,
making sure it does not trivialise the task for the user and allows them to
make their own decisions. This shall be achieved by monitoring both the
performance and the emotional state of the user.

The scenario shall be realised using an emotion-based dialogue management
system that shall learn to interact with the user depending upon their
emotional state. The emotion perception and intrinsic emotion models shall be
based on the works of _Churamani et al._ [4] realising multi-modal sensory
capabilities, adding linguistic information to the model, to infer the
emotional state of the user. This affective understanding, along with the
performance of the user in the task, shall be used to model an emotional
dialogue with the user.

To measure the performance of the model and to study the impact of using the
affective understanding of the robot to adapt its ability to assist the user,
the model shall be compared to a baseline condition where the robot offers
assistance to the users whenever they ask for it. In such a scenario, the
robot does not model the emotional state of the user. Any time the users asks
for assistance (a hint), the robot provides the same. The proposed scenario
can be seen in Fig 2.

##  III Expected results

It is expected that the affective robot companion, that can adapt its ability
to model an interaction with the user based on their emotional state and
performance in the task, is perceived as more enjoyable and mindful than a
static robot that passively waits for user queries. This is expected to lead
to users playing the game longer, therefore having a positive impact on their
cognitive abilities by training longer. Furthermore, this work intends to show
that users who play with the affective robot shall, on average, perform better
over time than baseline users as the robot‘s mixed initiative approach towards
the interaction, intervening and offering assistance in the task, helps users
to faster develop a functional game strategy.

Experiment evaluation criteria shall be based on the works of Ho and MacDorman
[7] and Gray et al. [8], as well as objective performance evaluations based on
emotion recognition and the improvement in the performance of the users, over
time.

##  IV Conclusion

In this position paper, an emotion-based dialogue management model is
envisaged that uses reinforcement learning techniques to learn optimal
dialogue states based on the emotional state of the user. The model is
proposed to be able to infer the emotional state of the user, based on visual,
auditory and linguistic information, and use this affective understanding to
assist the users towards solving the $2048$ Puzzle Game.

## Acknowledgement

The authors gratefully acknowledge partial support from the German Research
Foundation DFG under project CML (TRR 169) and the European Union under
project SOCRATES (No. 721619).

MLP

    Multilayer Perceptron
CNN

    Convolutional Neural Network
HRI

    Human-Robot Interaction
RL

    Reinforcement Learning

## References

  * [1] S. Rohwedder and R. J. Willis, “Mental retirement,” _J Econ Perspect_ , vol. 24, no. 1, pp. 119–138, 2010.
  * [2] D. A. Glei, D. A. Landau, N. Goldman, Y.-L. Chuang, G. Rodríguez, and M. Weinstein, “Participating in social activities helps preserve cognitive function: an analysis of a longitudinal, population-based study of the elderly,” _International Journal of Epidemiology_ , vol. 34, no. 4, pp. 864–871, 2005.
  * [3] I. Leite, C. Martinho, and A. Paiva, “Social robots for long-term interaction: a survey,” _International Journal of Social Robotics_ , vol. 5, no. 2, pp. 291–308, 2013.
  * [4] N. Churamani, P. Barros, E. Strahl, and S. Wermter, “Learning empathy-driven emotion expression using affective modulations,” in _Proceedings of the International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2018, pp. 1400–1407.
  * [5] T. W. Neller, “Pedagogical possibilities for the 2048 puzzle game,” _J. Comput. Sci. Coll._ , vol. 30, no. 3, pp. 38–46, Jan. 2015.
  * [6] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, Feb. 2015.
  * [7] C.-C. Ho and K. F. MacDorman, “Revisiting the uncanny valley theory: Developing and validating an alternative to the godspeed indices,” _Computers in Human Behavior_ , vol. 26, no. 6, pp. 1508–1518, 2010.
  * [8] H. M. Gray, K. Gray, and D. M. Wegner, “Dimensions of mind perception,” _science_ , vol. 315, no. 5812, pp. 619–619, 2007.

  *[HRI]: Human-Robot Interaction
  *[RL]: Reinforcement Learning