(cvpr) Package cvpr Warning: Package ‘hyperref’ is not loaded, but highly
recommended for camera-ready version

# Multi-modal Domain Adaptation for REG via Relation Transfer

Yifan Ding Liqiang Wang
Department of Computer Science, University of Central Florida
<EMAIL_ADDRESS><EMAIL_ADDRESS>Boqing Gong
Google
<EMAIL_ADDRESS>
###### Abstract

Domain adaptation, which aims to transfer knowledge between domains, has been
well studied in many areas such as image classification and object detection.
However, for multi-modal tasks, conventional approaches rely on large-scale
pre-training. But due to the difficulty of acquiring multi-modal data, large-
scale pre-training is often impractical. Therefore, domain adaptation, which
can efficiently utilize the knowledge from different datasets (domains), is
crucial for multi-modal tasks. In this paper, we focus on the Referring
Expression Grounding (REG) task, which is to localize an image region
described by a natural language expression. Specifically, we propose a novel
approach to effectively transfer multi-modal knowledge through a specially
relation-tailored approach for the REG problem. Our approach tackles the
multi-modal domain adaptation problem by simultaneously enriching inter-domain
relations and transferring relations between domains. Experiments show that
our proposed approach significantly improves the transferability of multi-
modal domains and enhances adaptation performance in the REG problem.

##  1 Introduction

Domain adaptation aims to mitigate the discrepancy between domains such that a
model learned from the source domain can be well generalized to the target
domain. Recent methods in domain adaptation mainly study single-modality
tasks, such as image classification [You et al.(2019)You, Long, Cao, Wang and
Jordan, Long et al.(2017)Long, Cao, Wang and Jordan], semantic segmentation
[Zhang et al.(2017)Zhang, David and Gong, Du et al.(2019)Du, Tan, Yang, Feng,
Xue, Zheng, Ye and Zhang], object detection [Inoue et al.(2018)Inoue, Furuta,
Yamasaki and Aizawa, Hsu et al.(2020)Hsu, Yao, Tsai, Hung, Tseng, Singh and
Yang], and natural language processing [Ramponi and Plank(2020), Laparra et
al.(2020)Laparra, Bethard and Miller]. However, when it comes to multi-modal
data, the current approaches mainly rely on pre-training on large-scale multi-
modal datasets followed by fine-tuning on downstream tasks [Chen et
al.(2020)Chen, Li, Yu, El Kholy, Ahmed, Gan, Cheng and Liu, Lu et al.(2019)Lu,
Batra, Parikh and Lee, Su et al.(2019)Su, Zhu, Cao, Li, Lu, Wei and Dai]. Such
plain transfer learning methods are time-consuming and resource-consuming.

Figure 1: An example of relation transfer. Relations learned from the source
domain are shown in blue, and relations learned from the target domain are
shown in orange in both the language and visual modals. Relations shared
between two domains are denoted with brown color. New relations created after
transferring from source domain to target domain are shown in green dash
lines.

In this paper, we investigate multi-modal knowledge transfer and propose a new
effective approach in a domain adaptation manner. Specifically, we focus on
the referring expression grounding (REG) [Kazemzadeh et al.(2014)Kazemzadeh,
Ordonez, Matten and Berg] problem. REG aims to localize an image region
described by a natural language expression. As a multi-modal problem, REG
requires high-level language comprehension and object localization, which make
collecting and labeling REG data be expensive and time-consuming. To make up
for the limited training data, pretrain-then-transfer learning approaches are
widely used in recent research studies [Chen et al.(2020)Chen, Li, Yu, El
Kholy, Ahmed, Gan, Cheng and Liu, Lu et al.(2019)Lu, Batra, Parikh and Lee, Su
et al.(2019)Su, Zhu, Cao, Li, Lu, Wei and Dai]. It mainly relies on a “vision-
and-language” feature extractor, which is often pre-trained with large-scale
and usually roughly annotated “vision-and-language” datasets (_e.g_. the
Conceptual Captions dataset [Sharma et al.(2018)Sharma, Ding, Goodman and
Soricut]) and then fine-tuned together with the downstream classifier.
However, there are several limitations in such an approach. First, there is a
gap between the general “vision-and-language” feature extractor and the REG
task, especially when the most important comprehension part is missing in pre-
training due to the annotation limits. Secondly, the feature extractor (_e.g_.
transformer-based) is usually heavyweight, which results in large model size,
big pre-training dataset, and long pre-training time. It also causes a
burdened and slow fine-tuning process.

The goal of this paper is to effectively transfer knowledge between REG
datasets to avoid the above problems. However, it is not an easy task. First,
to capture the variance among different types of referring expressions, a
widely-used approach is modular networks, which contain different visual
modules processing different information such as name, attributes, location,
or relations to other objects in the referring expression [Yu et al.(2018)Yu,
Lin, Shen, Yang, Lu, Bansal and Berg]. Such modular networks lack a joint
vision-language embedding [Yu et al.(2018)Yu, Lin, Shen, Yang, Lu, Bansal and
Berg, Liu et al.(2019)Liu, Wang, Shao, Wang and Li, Gupta et al.(2020)Gupta,
Vahdat, Chechik, Yang, Kautz and Hoiem]. Thus regular domain adaptation
methods that focus on aligning features in two domains [Ganin and
Lempitsky(2015), Long et al.(2017)Long, Cao, Wang and Jordan, Cai et
al.(2019)Cai, Wang and He] cannot be applied . Moreover, enforcing domain-
invariance for each modality (language or vision) individually fails to
capture their interplay. Besides, the domain gap between two REG domains stems
from multiple factors, such as visual styles, biases in the bounding box
annotations, and textual mismatches in styles and dictionaries, making it
difficult to straightforwardly extend existing single modality adaptation
methods to the REG task.

To address the above problems, we propose to extract and transfer
language/objects features in a relation-based manner rather than aligning the
features. Figure 1 shows an example. In this example, relations between
words/objects in the source domain are learned in the pre-training. These
learned relations are universal and transferable (_e.g_. computer always sit
on table in both domains). Besides, they also help uncovering underlying
relations in the target task. For instance, in the language modal, from source
domain we learned a relation (“computer”, “table”); in the target domain,
“mouse” is only connected with “table” but an underlying connection (“mouse”,
“computer”) could be discovered. In the visual modal, we have similar cases.
Although our relation-based domain adaptation approach is specially tailored
for the REG problem, it could be extended to other multi-modal tasks. To the
best of our knowledge, we are the first to solve the problem from this angle.
Our approach aims to discover and maintain the relations between
language/objects and mitigate the multi-modal discrepancy between the source
and target domains.

From the language aspect, the discrepancy pivots on the difference between
source and target vocabularies. Since the source and target usually share
parts of their vocabularies and each has its own domain-specific words, our
task is to transfer learned embeddings from source-private and shared
vocabularies to target-private vocabulary. To achieve it, we design the cross-
attention embedding strategy, through which the two domains share common
knowledge in the shared vocabulary, meanwhile maintaining sufficient
expressive power to learn domain-specific knowledge.

Among the visual side, in the REG model, the relation module that handles
surrounding objects of a candidate object plays an important role in
comprehension. Through the relation module, the REG model recognizes objects
in the same category but have different surroundings (_e.g_. “man in the
middle of three men”). Previous approaches use geographical distance to define
the surrounding objects and pick up one with the highest matching score to
represent the environment of the candidate objects [Yu et al.(2018)Yu, Lin,
Shen, Yang, Lu, Bansal and Berg, Liu et al.(2019)Liu, Wang, Shao, Wang and
Li]. In our approach, however, we propose to re-build the relation modular
with Graph Neural Networks (GNN) [Scarselli et al.(2008)Scarselli, Gori, Tsoi,
Hagenbuchner and Monfardini]. GNN could model the relations between objects in
terms of graphs, which better describe the surrounding environment of a
candidate object. Besides, GNNs could utilize the co-occurrences between
objects in the source dataset. That also improves the features while
transferring to the target dataset. In addition, we extend scene graph [Xu et
al.(2017)Xu, Zhu, Choy and Fei-Fei] to enrich the graphs with semantic
relations between objects.

Our main contributions are as follows. First, we formulate domain adaption for
the REG problem as a multi-modal transfer learning task, which enriches the
transfer learning techniques in the vision-language research. Secondly, we
design a new relation-based adaptation method for the REG problem, which
enables better knowledge transfer to new REG dataset. Our method results in a
fast and lightweight transfer compared with traditional expensive pretrain-
then-transfer learning schema. Thirdly, the proposed multi-modal relation-
based domain adaption method significantly outperforms other approaches for
the REG problem.

##  2 Related Work

Our work is at the intersection of several subareas in the framework of deep
learning. In this section, we discuss related work on REG, domain adaptation,
and multi-modal adaptation for REG.

Figure 2: The architecture of our proposed approach. The model is composed of
a language module in which the cross-attention embedding is applied and three
visual modules (_i.e_., location, subject, and relation modules). The final
matching score $S$ between expression and visual modules is the weighted sum
of three modules $S_{loc}$, $S_{sub}$ and $S_{rel}$.

Referring expression grounding. Like most vision-language tasks [Antol et
al.(2015)Antol, Agrawal, Lu, Mitchell, Batra, Zitnick and Parikh, Wang et
al.(2016)Wang, Li and Lazebnik], REG also relies on multi-modal embeddings to
bridge the semantic gap between visual and textual contents. However, when
data distributions embody complex multi-modal structures, general domain
adaptation methods may fail to capture such multi-modal structures for a
discriminative alignment of distributions. Therefore, in most cases, either
training from scratch [Yu et al.(2018)Yu, Lin, Shen, Yang, Lu, Bansal and
Berg, Liu et al.(2019)Liu, Wang, Shao, Wang and Li] or large-scale pre-trained
models are used to transfer knowledge from large-scale vision-language dataset
to specific vision-language tasks [Chen et al.(2020)Chen, Li, Yu, El Kholy,
Ahmed, Gan, Cheng and Liu, Lu et al.(2019)Lu, Batra, Parikh and Lee, Su et
al.(2019)Su, Zhu, Cao, Li, Lu, Wei and Dai]. In this paper, we adopt a modular
structure proposed in MAttNet [Yu et al.(2018)Yu, Lin, Shen, Yang, Lu, Bansal
and Berg]. It has a language module and three visual modules that process the
visual feature, location feature and the surrounding object feature of the
candidate object, respectively. We reformulate the language module to process
the relations between words and the visual relation module to model the visual
object relations.

Domain adaptation. Domain adaptation aims to mitigate the discrepancy of
distributions between source domain and target domain data. The main methods
in domain adaptation are to minimize the statistical distances between source
and target feature distributions. These methods usually take advantage of
adversarial training [Mirza and Osindero(2014)] and include a domain
classifier that discriminates between the source and the target domains during
training [Ganin and Lempitsky(2015), Tzeng et al.(2017)Tzeng, Hoffman, Saenko
and Darrell]. In that direction, some recent methods condition the domain
discriminator to facilitate more accurate feature adaptation within each
category [Long et al.(2017)Long, Cao, Wang and Jordan, Hu et al.(2020)Hu, Kan,
Shan and Chen]. However, the adversarial training procedure tends to be
unstable and may cause performance drop. Strengthening Lipschitz continuity in
the target distribution is a method that avoids adversarial training yet still
supports domain adaptation [Shu et al.(2018)Shu, Bui, Narui and Ermon, Mao et
al.(2019)Mao, Ma, Yang, Chen and Li, Cai et al.(2019)Cai, Wang and He].

Domain adaptation for referring expression grounding. Since general domain
adaptation methods cannot be applied to the multi-modal problem directly,
several algorithms have been proposed to align distributions of features and
classes via separate domain discriminators [Hoffman et al.(2016)Hoffman, Wang,
Yu and Darrell, Chen et al.(2017)Chen, Chen, Chen, Tsai, Frank Wang and Sun,
Tsai et al.(2018)Tsai, Hung, Schulter, Sohn, Yang and Chandraker]. In
particular for the REG task, Liu et al. propose to transfer concept from
auxiliary classification data (images from new categories) and context
inheritance from REG data to ground new objects [Liu et al.(2020)Liu, Li,
Wang, Zha, Meng and Huang]. In our approach, Graph Convolutional Network (GCN)
[Kipf and Welling(2016)] is used to transfer knowledge from source to target
domains. Our approach is different from the transfer learning of graph network
[Lee et al.(2017)Lee, Kim, Lee and Yoon, Zhu et al.(2020)Zhu, Xu, Wang, Zhang,
Han and Yang], as our main task is to transfer the relations, not the graph
itself. Besides, pivot-based methods [Blitzer et al.(2006)Blitzer, McDonald
and Pereira, Ziser and Reichart(2016), Ziser and Reichart(2018)], which
recognize the frequent features in the source and target domains, also inspire
our cross-attention dictionary method.

##  3 Multi-modal Domain Adaptation for REG

Our main approach is described in this section. The framework is within
supervised domain adaptation, where the model has access to a small amount of
labeled data from the source and target domains. This allows us to pre-train
the model in the source domain and then fine-tune it in the target domain.

Different from previous REG models [Yu et al.(2018)Yu, Lin, Shen, Yang, Lu,
Bansal and Berg, Liu et al.(2019)Liu, Wang, Shao, Wang and Li, Gupta et
al.(2020)Gupta, Vahdat, Chechik, Yang, Kautz and Hoiem], our model transfers
language and visual features based on the word relations and visual object
relations. Our proposed model results in less discrepancy and preserves more
target domain oriented information during the transfer.

###  3.1 Cross-attention Word Embedding

In the language module, the first step is to encode words in the expression
sentence to fixed-length feature vectors. Under the domain adaptation
scenario, the source and target domains are likely to have different
vocabularies. For example, the RefCOCO dataset [Yu et al.(2016)Yu, Poirson,
Yang, Berg and Berg] has 1999 words while the RefCOCOg dataset [Mao et
al.(2016)Mao, Huang, Toshev, Camburu, Yuille and Murphy] has 3349 words. We
denote the vocabulary that appears in both the source and target dataset as
“shared vocabulary”, and the vocabulary that only appears in the source or
target datasets as “source-private” and “target-private” vocabulary,
respectively. Our goal is to transfer as much information from the source
domain to the target domain as possible, for which the shared words are
pivotal, it is not sufficient. We conjecture that not only the shared words
can be transferred between domains, the correlation among words also
transfers. We propose an attention mechanism to model the correlations between
words, in which the embeddings of domain private vocabularies are learned from
both the training data and the attended shared vocabulary embeddings. By the
attention machinism, we build the relationship between shared vocabulary and
private vocabularies, which also connects the source and target private words
indirectly. Two benefits are brought by the relation between shared and
private vocabularies : 1) words in the shared vocabulary are usually more
widely used and point to more general objects and represent more common
semantics which enrich the representation of private words with more general
and recognizable features , and 2) the embeddings of shared vocabulary turn to
be more domain-invariant to represent privates from two domains.

In the cross-attention embedding, we denote the embedding matrix by
$D\in\mathbb{R}^{d\times n}$ for a dictionary of $n$ words, each of which is
embedded to a $d$-dimensional vector. We denote the source and target
vocabulary embeddings by $D_{S}$ and $D_{T}$, the shared and two private
vocabulary embeddings by $D_{S\cap T},D_{S\backslash T}$ and $D_{T\backslash
S}$, respectively. Thus, $D=D_{S\cap T}\cup D_{S\backslash T}\cup
D_{T\backslash S}$. We also have $D_{S}=D_{S\cap T}\cup D_{S\backslash T}$ and
$D_{T}=D_{S\cap T}\cup D_{T\backslash S}$. $D_{S\cap T}$ is learnt from the
dataset, while $D_{S\backslash T}$ and $D_{T\backslash S}$ are attended by
$D_{S\cap T}$ with learnable matrices $W_{S\cap T\rightarrow S\backslash T}$
and $W_{S\cap T\rightarrow T\backslash S}$, which model the relations between
the shared vocabulary and the source and target private vocabulary,
respectively. These embedding matrices $D$ and attention matrices $W$ are
trained together with the language attention module in the REG model, as shown
in Figure 2.

Figure 3: Illustration of the workflow of cross-attention word embedding.
During the pre-training, the attention (denoted by $\oplus$) is learnt from
shared vocabulary $D_{S\cap T}$ to source-private vocabulary $D_{S\backslash
T}$. In the fine-tuning, the attention is learnt from shared vocabulary
$D_{S\cap T}$ to target-private vocabulary $D_{T\backslash S}$.

During pretraining, the update during forward pass is:

$\displaystyle\begin{aligned} \overline{D_{S\backslash T}}&=D_{S\backslash
T}+D_{S\cap T}\cdot\textrm{softmax}(W_{S\cap T\rightarrow S\backslash T}),\\\
\overline{D_{S\cap T}}&=D_{S\cap T},\end{aligned}$ (1)

where $\cdot$ is matrix multiplication. $W_{S\cap T\rightarrow S\backslash T}$
is the learnable weights of shape $n_{S\cap T}\times n_{S\backslash T}$. In
every forward pass, the private vocabulary embedding $D_{S\backslash T}$ is
updated (denoted as $\overline{D_{S\backslash T}}$) with the sum of original
embedding $D_{S\backslash T}$ and $D_{S\cap T}$ weighted by a softmax version
of $W_{S\cap T\rightarrow S\backslash T}$. After back-propagation, we update
$D$ which are trainable weights with learning rate $\eta$ :

$\displaystyle D_{S\backslash T}\leftarrow D_{S\backslash
T}-\eta\frac{\partial Loss}{\partial D_{S\backslash T}},\leavevmode\nobreak\
\leavevmode\nobreak\ D_{S\cap T}\leftarrow D_{S\cap T}-\eta\frac{\partial
Loss}{\partial D_{S\cap T}}$

During finetuning, $D_{T\backslash S}$ is updated with the target data and the
learned embeddings $D_{S\cap T}$ from the source domain:

$\displaystyle\begin{aligned} \overline{D_{T\backslash S}}&=D_{T\backslash
S}+D_{S\cap T}\cdot\textrm{softmax}({W_{S\cap T\rightarrow T\backslash
S}}),\\\ \overline{D_{S\cap T}}&=D_{S\cap T},\end{aligned}$ (2)

$W_{S\cap T\rightarrow T\backslash S}$ is of shape $n_{S\cap T}\times
n_{T\backslash S}$. $D_{S\backslash T}$ and $D_{T\backslash S}$ are also
updated in propagation.

Since private words are usually less common and have less training samples,
this mechanism improves the embedding quality in both $D_{S\backslash T}$ and
$D_{S\backslash T}$.. At the same time, the embeddings in shared vocabulary
also become more domain invariant by learning to represent private words from
the source and target domains.

With the embedding matrix $D$, we get domain invariant word embedding in pre-
training and fine-tuning. After that, we apply a bi-directional LSTM [Graves
et al.(2005)Graves, Fernández and Schmidhuber] to encode the whole expression
$R=\\{u_{t}\\}^{T}_{t=1}$:

$\displaystyle\begin{aligned} e_{t}&=D(u_{t}),\\\ h_{t}&=\textrm{Bi-
LSTM}(e_{t},h_{t-1}),\\\ \end{aligned}$ (3)

$H=\\{h_{t}\\}^{T}_{t=1}$ is the expression feature.

###  3.2 Object Relation Transfer via GNNs

In visual module, we also conduct domain adaptation by discovering
transferable relationships. For REG models, pairwise data are used in the
training, where each pair of data is consisted of a sentence of expression
(processed in the language module) and a candidate object. Usually a
Convolutional Neural Network [Krizhevsky et al.(2012)Krizhevsky, Sutskever and
Hinton] is implemented to process the visual feature (subject module) while a
multi-layer perceptron without convolutions is used to process the location
feature (location module) of the object. However, these two modules are not
enough to location a target object. For example, in the expression “first
giraffe on left”, we have to locate the surrounding objects of each giraffe
and only the one with no other giraffe on its left is the correct one. Here,
the co-occurrence of objects also helps locate the target object, even when
the expression didn’t mention them.

Based on the above observations, we propose to describe the relations between
objects in an image with graphs. Furthermore, we merge graphs from source and
target domains into one during learning via Graph Neural Networks (GNN)
[Scarselli et al.(2008)Scarselli, Gori, Tsoi, Hagenbuchner and Monfardini].
GNNs typically aggregate information from neighbor nodes to embedded
information into a hidden space regardless of which domain the node is from.
It utilizes the co-occurrence information between objects. Therefore when
transferring to a new dataset, the model still benefits from the learned
connections between objects in the source dataset. At the same time, the GNNs
are also updated with the new data coming from the target dataset.

Figure 4: An example of relation graph generalization.

####  3.2.1 Relation Graph Generalization

Our first step is to create the relation graph of candidate object $v$ by
$\mathcal{G}_{v}=(\mathcal{V}_{v},\mathcal{A}_{v})$, where $\mathcal{V}_{v}$
is the set of nodes and $\mathcal{A}_{v}$ denotes the set of edges. Since
geographical distance is one of the essential factors to locate neighbor
objects, we first create the geometrical relation graph
$\mathcal{G}_{v}^{geo}$ from images by setting edges between an object $v$ and
its $M$ nearest objects via
$\mathcal{G}_{v}^{geo}=(\mathcal{V}_{v}^{geo},\mathcal{A}_{v}^{geo})$,
$\mathcal{V}_{v}^{geo}=\\{v_{1},...,v_{M}\\}$ and
$\mathcal{A}_{v}^{geo}=\\{(v,u)|u\in\mathcal{V}_{v}^{geo},u\neq v\\}$. Figure
4 (b) shows an example of the geometrical graph.

Besides the geographical distance, we further explore the relationships
between objects via scene graph [Johnson et al.(2015)Johnson, Krishna, Stark,
Li, Shamma, Bernstein and Fei-Fei]. Scene graph constitutes the semantic
relationships between objects. It is a visually-grounded graph over the
objects in an image. The nodes denote visual objects and the directed edges
depict their pairwise relations. Through the relations excavated via scene
graphs, we are able to discover semantic relations between objects in an image
that might be physically far apart from each other. It is a necessary
complement of the geographical distance. Scene graph uses triplets to
represent the relations between two nodes. We denote the relation as a
directed edge $a_{s\rightarrow o}=<v_{s},r,v_{o}>$, where $v_{s}$ denotes the
subject object, $v_{o}$ denotes the target object and $r$ is the relation
between them. For example, in the edge $<``man",``playing",``Frisbee">$,
$v_{s}=``man"$, $r=``playing"$ and $v_{o}=``Frisbee"$. Figure 4 (a) shows an
example of the scene graph generated from an image.

We use scene graph to complement the geographical distance graph with semantic
relations. Specifically, we first use a scene graph extractor [Tang et
al.(2020)Tang, Niu, Huang, Shi and Zhang] to generate the scene graph
$\mathcal{G}_{I}=(\mathcal{V}_{I},\mathcal{A}_{I})$ of image $I$, which
contains all semantic relationships within the image. Then we generate the
semantic graph $\mathcal{G}_{v}^{sem}$ via $\mathcal{G}_{I}$ by selecting the
edges and nodes from scene graph that are directly or indirectly connected to
$v$ as shown in Figure 4 (a)-(c). Note that the edges in a semantic graph do
not have directions. The resulted semantic graph is denoted as
$\mathcal{G}_{v}^{sem}=(\mathcal{V}_{v}^{sem},\mathcal{A}_{v}^{sem})$

Finally, the relation graph is the union of geographical distance graph and
scene graph, which is demonstrated in Figure 4 (d).

$\displaystyle\mathcal{G}_{v}=(\mathcal{V}_{v},\mathcal{A}_{v})=(\mathcal{V}_{v}^{geo}\cup\mathcal{V}_{v}^{sem},\mathcal{A}_{v}^{geo}\cup\mathcal{A}_{v}^{sem})$
(4)

####  3.2.2 Relation Transfer via GNNs

Figure 5: Object relation transfer from source domain to target domain via
GNNs. Blue denotes the source domain, orange denotes the target domain and
brown denotes that the weights update is affected by both source and target
domains. In the GNN, features of each node are updated by aggregating
information from connected neighbor nodes. node $D$ and node $E$ are shared by
two domains thus the relations between $D$, $E$ and other objects could be
transferred to the target domain.

By aggregating information from neighbor nodes in both source and target
domains, Graph Neural Networks (GNN)s can jointly learn an embedding by
combing information from the two domains. As shown in Figure 5, the relations
between source objects are learned in pre-training, and they are then
transferred to the target domain in fine-tuning.

In the GNN model, each node is naturally defined by its features and the
connected nodes. GNN can learn a state embedding $h_{v}\in\mathbb{R}^{d}$ that
contains the information of neighborhood for each node. The state embedding
$h_{v}$ is a $d$-dimension vector of node $v$. $\overline{h_{v}}$ (the next
state of $h_{v}$) is computed via:

$\displaystyle\overline{h_{v}}$ $\displaystyle=\sigma(\frac{1}{\sum_{u\in
N_{v}}{w_{u\rightarrow v}}}\sum_{u\in N_{v}}w_{u\rightarrow v}h_{u}\theta+b),$
(5)

where $N_{v}$ is the neighborhood set of node $v$, $h_{u}$ is the state
embedding of node $u$. $w_{u\rightarrow v}$ is the weighted edge from node $u$
to node $v$. For un-directed graphs, $w_{u\rightarrow v}=w_{v\rightarrow u}$.
$\theta$ and $b$ are the weight matrix and bias of the GNN layer,
respectively. $\sigma$ is the activation function.

We use both the geometrical distance and semantic relation to define the
weight matrix $W$. Suppose $D$ denotes the geometrical distance metric, we
have:

$\displaystyle\begin{aligned} &W=[w_{v\rightarrow u}],\ \
(v,u)\in\mathcal{A}_{v}\\\ &w_{v\rightarrow u}=\left\\{\begin{aligned}
&1,(v,u)\in\mathcal{A}_{v}^{sem}\\\ &\frac{\min
D(\mathcal{A}_{v}^{geo})}{D(v,u)},otherwise\end{aligned}\right.\end{aligned}$
(6)

where the edges in $\mathcal{A}_{v}^{sem}$ are weighted by $1$ and edges in
$\mathcal{A}_{v}^{geo}$ are weighted by their geometrical distance. If an edge
exists in both semantic graph and geometrical graph, its weight is calculated
based on the semantic graph.

In the training, we use the C4 feature (last convolutional output of 4th-
stage) of Faster R-CNN [Ren et al.(2015)Ren, He, Girshick and Sun] as the node
feature. Besides, we also encode the location of each object in the relation
graph with a 5-d vector
$l_{u}=[\frac{x_{tl}}{W},\frac{y_{tl}}{H},\frac{x_{br}}{W},\frac{y_{br}}{H},\frac{w\cdot
h}{W\cdot H}]$. $W,H$ are the image weight and height, respectively. Therefore
the final feature of node $u$ is the concatenation of the GNN feature $h_{u}$
and location feature $l_{u}$. We then apply a fully connected layer to the
concatenated feature:

$\displaystyle r_{u}^{rel}=W_{r}[h_{u};l_{u}]+b_{r}$ (7)

$r_{u}^{rel}$ is the relation feature of node $u$ in graph $\mathcal{G}_{v}$.

###  3.3 Domain Invariant Modular Learning

The subject and location modules process the candidate object features, which
can be easier transferred between domains. In this paper, we adopt the same
structures used in MAttNet [Yu et al.(2018)Yu, Lin, Shen, Yang, Lu, Bansal and
Berg] for the two modules.

##### Subject module

The subject module processes the visual feature of the candidate object.
Specifically, it takes the C3 and C4 features from Faster R-CNN as input. C3
feature includes rich space information therefore an attentional pooling [Yu
et al.(2018)Yu, Lin, Shen, Yang, Lu, Bansal and Berg] is applied to the C3
feature thus the salient feature is emphasized. We denote the subject feature
of object $v$ as $r_{v}^{sub}$.

##### Location module

Similar to the relation module, a 5-d vector is used to encode the location of
the candidate object feature. Then a FC layer is used to encode the 5-d
vector. The final location feature of object $v$ is $r_{v}^{loc}$.

##### Attended language embeddings for visual modules

For each visual module (subject, location and relation), a separate attention
is applied to the language expression feature $H$ (see Section 3.1) for an
attended embedding:

$\displaystyle\begin{aligned}
q^{m}=\sum_{t=1}^{T}{\frac{\exp(f_{m}^{T}h_{t})e_{t}}{\sum_{k=1}^{T}{\exp(f_{m}^{T}h_{k})}}},\end{aligned}$
(8)

where $m=\\{sub,loc,rel\\}$ represents three visual modules. $f_{m}$ is the
learnable weight for each module. $q^{m}$ are used to attend the feature in
visual modules. Besides, the visual modular weights are also learned through a
FC layer:

##### Matching score

For subject and location modules, matching scores $S^{sub}$ and $S^{loc}$ are
calculated using cosine similarity (denoted by $M$) between the visual and the
expression feature, _i.e_., $S_{v}^{sub}=M(r_{v}^{sub},q^{sub})$, and
$S_{v}^{loc}=M(r_{v}^{loc},q^{loc})$, where $v$ is a candidate object. For the
relation module, its matching score is the maximum among all nodes in the
Relation Graph:

$\displaystyle S_{v}^{rel}=max_{u\in\mathcal{G}_{v},u\neq
v}M(r_{u}^{rel},q^{rel})$ (9)

The overall matching score is the weighted average of three visual modules:

$\displaystyle S_{v}=\sum_{m\in\\{sub,loc,rel\\}}{w^{m}S_{v}^{m}}$ (10)

Finally, expression and candidate object pairs (positive and negative) are
sampled for a contrastive learning strategy. Hinge loss is applied for
optimization.

###  3.4 Revision of Domain Adaptation Approaches

Most of the existing domain adaptation algorithms cannot be directly applied
to the multi-modal REG problem. We modified several representative domain
adaptation methods for the REG problem, which conduct domain adaptation from
different perspectives. Notably, some of the recent works such as ensemble-
based approaches [Zhou et al.(2020)Zhou, Yang, Qiao and Xiang] are not
applicable thus are not included in this paper.

Table 1: The differences between REG datasets | RefClef | RefCOCO | RefCOCO+ | RefCOCOg
---|---|---|---|---
image from | ImageCLEF | MSCOCO | MSCOCO | MSCOCO
expression style | concise and simple | concise and simple | no location words | complex and long
expressions/objects/images | 130525/96654/19894 | 142209/50000/19994 | 141564/49856/19992 | 85474/54822/26711

##### Unsupervised Domain Adaptation by Backpropagation (DANN) [Ganin and
Lempitsky(2015)]

The idea of DANN is to minimize the loss of the label classifier and to
maximize the loss of the domain classifier. The latter encourages domain-
invariant features to emerge in the course of the optimization. Suppose
$\theta_{G}$ is the parameters in the feature extractor $G$, $\theta_{D}$ is
the parameters in the domain classifier $D$, and $\theta_{C}$ is the
parameters in the matching score module $C$. The overall optimization
functional is

$\displaystyle\mathcal{E}(\theta_{G},\theta_{C},\theta_{D})=\sum_{i=1..N,d_{i}=0}L_{y}^{i}(\theta_{G},\theta_{C})-\lambda\sum_{i=1..N}{L_{d}^{i}(\theta_{G},\theta_{D})},$
(15)

where $L_{y}^{i}$ and$L_{d}^{i}$ denote the classification and domain loss,
respectively, evaluated at the $i^{th}$ training example. During the training,
the parameters $\theta_{D}$ of the domain classifier and parameters
$\theta_{G}$ and $\theta_{C}$ are optimized alternately to minimize the hinge
loss and maximize the domain loss. In our experiments, the DANN algorithm is
applied to either/both the visual and language modules.

##### Conditional Domain Adversarial Network (CDAN) [Long et al.(2017)Long,
Cao, Wang and Jordan]

Like recent popular domain adaptation algorithms [You et al.(2019)You, Long,
Cao, Wang and Jordan, Hu et al.(2020)Hu, Kan, Shan and Chen], CDAN requests to
involve classification outputs while aligning the features between the source
and target domains. In MAttNet, there is an auxiliary task to classify the
target object called attribute module $A$. We then use the output from the
attribute module to help the domain classifier. The CDAN error terms are:

$\displaystyle\mathcal{E}(G,C)=$
$\displaystyle\sum_{i=1..N,d_{i}=0}L_{y}^{i}(C(G(x_{i}^{i})),y_{i}^{s})$ (16)
$\displaystyle\mathcal{E}(D,G,C)$ $\displaystyle=-\mathbb{E}_{x_{i}^{s}\sim
D_{s}}\log[D(h(G(x_{i}^{s}),A(G(x_{i}^{s}))))]-$
$\displaystyle\mathbb{E}_{x_{j}^{t}\sim
D_{t}}\log[1-D(h(G(x_{j}^{t}),A(G(x_{j}^{t}))))]$ (17)

where $L(.,.)$ is the cross-entropy loss, and $h=G(x)\otimes A(G(x))$ is the
joint variable of feature representation $G(x)$ and attribute classifier
prediction $A(G(x))$. $\otimes$ denotes the outer product. The method also
utilizes a minimax game as the optimization strategy similar to DANN.

##### Smooth Representation for unsupervised Domain Adaptation (SRDA) [Cai et
al.(2019)Cai, Wang and He]

Beyond reducing the divergence between source and target domains, it has been
studied that the satisfaction of Lipschitz continuity also guarantees an error
bound for the target distribution [Ben-David and Urner(2014)]. SRDA proposes
the local smooth discrepancy (LSD) term to measure the degree that a sample
$x$ breaks down the local Lipschitz property for Lipschitz continuity
[Grandvalet et al.(2005)Grandvalet, Bengio et al.]. LSD is defined as:

$\displaystyle LSD(x,\theta)=d(C(G(x)+r),C(G(x))),||r||\leq\epsilon,$ (18)

where $d(\textperiodcentered,\textperiodcentered)$ is a discrepancy function
that measures the divergence between two outputs. $r$ is the noise and
$\epsilon$ denotes the maximum norm of $r$. As for the choice of
$d(\textperiodcentered,\textperiodcentered)$, we employ MSE loss instead of
the cross-entropy loss function used in the original paper since our problem
is not a classification problem.

During the training, all networks are first pre-trained. Then sensitive
samples $G(x)+r$ are generated in the feature space of $G$. Finally, $G$ is
optimized to minimize LSD for target samples.

$\displaystyle\min_{G}$ $\displaystyle\sum_{i=1...N}LSD(x_{i}^{t},\theta_{G})$
(19)

Table 2: Performance comparison (Best results are bolded and the second best are underlined.) | Target dataset | RefCOCO+ | RefCOCOg | RefClef
---|---|---|---|---
| Split | Val | TestA | TestB | Val | Test | Val | TestA | TestB | TestC
1 | Train on target (full dataset) | 69.41 | 73.14 | 64.12 | 75.84 | 75.60 | 85.51 | 84.00 | 88.40 | 78.08
2 | Finetune on target (full dataset) | 69.45 | 72.98 | 63.88 | 78.06 | 78.21 | 85.69 | 85.79 | 89.44 | 79.35
3 | Train on target | 57.87 | 60.25 | 55.80 | 67.63 | 67.44 | 78.45 | 75.74 | 84.12 | 69.57
4 | Finetune on target | 59.66 | 64.41 | 54.24 | 69.20 | 70.56 | 78.68 | 77.96 | 83.60 | 70.16
5 | DANN - Visual | 60.29 | 63.08 | 56.68 | 69.83 | 69.94 | 72.94 | 73.11 | 79.54 | 62.52
6 | DANN - Language | 61.31 | 65.44 | 56.78 | 71.81 | 70.85 | 78.90 | 76.60 | 83.53 | 72.41
7 | DANN - Language&Visual | 61.77 | 66.68 | 57.31 | 69.24 | 68.77 | 73.76 | 73.53 | 79.47 | 66.44
8 | Adversarial-cat | 61.20 | 63.60 | 57.11 | 69.69 | 69.08 | 78.10 | 77.45 | 84.34 | 69.37
9 | CDAN | 62.20 | 66.36 | 56.88 | 71.45 | 70.37 | 78.29 | 77.11 | 83.68 | 67.32
10 | SRDA | 58.35 | 61.79 | 53.96 | 70.83 | 70.40 | 72.79 | 72.68 | 78.73 | 65.36
11 | Ours - Language | 64.53 | 67.90 | 58.58 | 73.53 | 73.15 | 79.79 | 79.66 | 83.83 | 73.97
12 | Ours - Visual | 64.98 | 68.44 | 57.66 | 73.73 | 73.47 | 80.05 | 80.26 | 84.05 | 75.34
13 | Ours - Relation transfer | 65.16 | 68.76 | 58.66 | 74.23 | 73.53 | 80.08 | 80.68 | 85.52 | 74.66
14 | Ours - Relation transfer + Bert | 66.56 | 69.86 | 60.50 | 74.49 | 74.10 | 81.12 | 82.30 | 86.63 | 74.36

##  4 Experiments

###  4.1 Datasets and Experiment Settings

The proposed approach is compared with the adapted models presented in Section
3.4 on four benchmark datasets: RefClef [Kazemzadeh et al.(2014)Kazemzadeh,
Ordonez, Matten and Berg], RefCOCO [Yu et al.(2016)Yu, Poirson, Yang, Berg and
Berg], RefCOO+ [Yu et al.(2016)Yu, Poirson, Yang, Berg and Berg] and RefCOCOg
[Mao et al.(2016)Mao, Huang, Toshev, Camburu, Yuille and Murphy]. The main
differences between the four datasets are depicted in Table 1. We use RefCOCO
as the source domain dataset because it has the most number of expressions,
and differs from other three datasets from multiple perspectives such as
annotating styles or image bases.

In the experiments, each of the models is first trained with the full RefCOCO
data and then finetuned with 10% of the target data. The evaluation is
conducted on the full test data. We implement two baseline models without
domain adaption algorithms, one is directly trained on the 10% target data,
and the other is trained on the full RefCOCO data and finetuned on 10% of the
target data. The two baselines act as the lower bounds (rows 3-4 in Table 2).
We also implement two upper-bounds (rows 1-2 in Table 2), in which we
train/finetune on the full target dataset rather than 10%. For the graph
embedding model, we adopt a two-layer GCN in [Kipf and Welling(2016)]. We use
the pretrained scene graph extractor in [Xu et al.(2017)Xu, Zhu, Choy and Fei-
Fei]. For parameters, we follow the official implementation of these
algorithms. An NVIDIA XP GPU is used for all experiments. Best results are
recorded among the three different finetuning learning rates, which are equal
to, half of, or 10% of the pre-training learning rate, respectively.

###  4.2 Results

Table 2 shows the results of domain adaptation from RefCOCO to RefCOCO+,
RefCOCOg and RefClef. Each of the datsets has different splits. Details about
each split can be found in [Kazemzadeh et al.(2014)Kazemzadeh, Ordonez, Matten
and Berg] and [Yu et al.(2016)Yu, Poirson, Yang, Berg and Berg]. The first two
rows show the upper-bounds. For DANN [Ganin and Lempitsky(2015)], we consider
three scenarios in which the domain classifier is only implemented on the
subject domain (-visual), on the language domain (-language) or on both of the
visual and language domain (-language&visual). We also implement one model
where we concatenate the features from the visual and language branches and
implement a domain classifier proposed in [Tzeng et al.(2017)Tzeng, Hoffman,
Saenko and Darrell] on top of the concatenation features (adversarial-cat).
For this model, a generative adversarial nets (GAN) [Mirza and Osindero(2014)]
training strategy is applied. Besides, we also include the results of
replacing the embedding layer with a pre-trained Bert model [Devlin et
al.(2018)Devlin, Chang, Lee and Toutanova] as the language feature extractor.
(row 14 in Table 2) and using the language and visual relation-based method
separately (Table 2, rows 11-12).

Our approach achieves the best results compared with all baselines. Among the
three target domains, we have more improvements on RefCOCO+ and RefCOCOg than
on RefClef. The main difference is that RefCOCO share the same image base with
RefCOCO+ and RefCOCOg but different images with RefClef. It shows that our
approch works better while the image base is consistent since our approach
focus on the relation-based transfer, which could be potentially combined with
image feature transferring method and boost the results further.

Bert embedding. Bert model [Devlin et al.(2018)Devlin, Chang, Lee and
Toutanova] has been widely used in natural language processing tasks. Since
Bert model is pre-trained with large language datasets, it improves the
quality of language embedding and brings the model more domain-invariant
features. Therefore, we investigate to replace the embedding layer in the REG
model with the pre-trained Bert model. The results show that it achieves
improvements in almost all splits, but not significantly. Besides, for RefClef
on split TestC the results with the Bert embedding is slightly worse than
without it. That may caused by that TestC contains objects sampled from images
that contain at least 2 objects of the same category. [Kazemzadeh et
al.(2014)Kazemzadeh, Ordonez, Matten and Berg]. Bert embedding hardly help
distinguish them.

Performance improvement in each modal. The three proposed relation-based REG
domain adaptation optimization methods could actually be applied separately.
We show these results in Table 2, rows 11-12. We find that while using these
methods alone, they all improve the transferability of the REG model and
achieve better results than baselines. But still, the best performance is
achieved by stacking them together. Besides, the visual-side relation transfer
method (GCN relation encoding) achieves slightly better performance compared
with the language-side optimization method (cross-attention word embedding)
while being used separately.

##  5 Conclusions

In this paper, we investigate the domain adaptation problem for the REG task,
which is still not well studied but widely needed in real-world applications.
We propose several optimization methods based on the relations on visual and
language modalities, which efficiently transfer knowledge learnt in source
domain to target domain for the REG problem. Experimental results show that
our proposed approach outperforms other domain adaptation methods. We expect
our approach from the relation perspective, along with the sound experimental
results, can further facilitate future research in this area.

##  6 Acknowledgement

This material is based upon work supported by the National Science Foundation
under Grant No 1952792.

## References

  * [Antol et al.(2015)Antol, Agrawal, Lu, Mitchell, Batra, Zitnick and Parikh] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D., 2015.  Vqa: Visual question answering, in: Proceedings of the IEEE international conference on computer vision, pp. 2425–2433.
  * [Ben-David and Urner(2014)] Ben-David, S., Urner, R., 2014\.  Domain adaptation–can quantity compensate for quality?  Annals of Mathematics and Artificial Intelligence 70, 185–202.
  * [Blitzer et al.(2006)Blitzer, McDonald and Pereira] Blitzer, J., McDonald, R., Pereira, F., 2006.  Domain adaptation with structural correspondence learning, in: Proceedings of the 2006 conference on empirical methods in natural language processing, pp. 120–128.
  * [Cai et al.(2019)Cai, Wang and He] Cai, G., Wang, Y., He, L., 2019.  Learning smooth representation for unsupervised domain adaptation.  arXiv preprint arXiv:1905.10748 .
  * [Chen et al.(2020)Chen, Li, Yu, El Kholy, Ahmed, Gan, Cheng and Liu] Chen, Y.C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., Liu, J., 2020.  Uniter: Universal image-text representation learning, in: European Conference on Computer Vision, Springer. pp. 104–120.
  * [Chen et al.(2017)Chen, Chen, Chen, Tsai, Frank Wang and Sun] Chen, Y.H., Chen, W.Y., Chen, Y.T., Tsai, B.C., Frank Wang, Y.C., Sun, M., 2017\.  No more discrimination: Cross city adaptation of road scene segmenters, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 1992–2001.
  * [Devlin et al.(2018)Devlin, Chang, Lee and Toutanova] Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805 .
  * [Du et al.(2019)Du, Tan, Yang, Feng, Xue, Zheng, Ye and Zhang] Du, L., Tan, J., Yang, H., Feng, J., Xue, X., Zheng, Q., Ye, X., Zhang, X., 2019.  Ssf-dan: Separated semantic feature based domain adaptation network for semantic segmentation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 982–991.
  * [Ganin and Lempitsky(2015)] Ganin, Y., Lempitsky, V., 2015\.  Unsupervised domain adaptation by backpropagation, in: International conference on machine learning, PMLR. pp. 1180–1189.
  * [Grandvalet et al.(2005)Grandvalet, Bengio et al.] Grandvalet, Y., Bengio, Y., et al., 2005\.  Semi-supervised learning by entropy minimization., in: CAP, pp. 281–296.
  * [Graves et al.(2005)Graves, Fernández and Schmidhuber] Graves, A., Fernández, S., Schmidhuber, J., 2005.  Bidirectional lstm networks for improved phoneme classification and recognition, in: International conference on artificial neural networks, Springer. pp. 799–804.
  * [Gupta et al.(2020)Gupta, Vahdat, Chechik, Yang, Kautz and Hoiem] Gupta, T., Vahdat, A., Chechik, G., Yang, X., Kautz, J., Hoiem, D., 2020\.  Contrastive learning for weakly supervised phrase grounding.  arXiv preprint arXiv:2006.09920 .
  * [Hoffman et al.(2016)Hoffman, Wang, Yu and Darrell] Hoffman, J., Wang, D., Yu, F., Darrell, T., 2016.  Fcns in the wild: Pixel-level adversarial and constraint-based adaptation.  arXiv preprint arXiv:1612.02649 .
  * [Hsu et al.(2020)Hsu, Yao, Tsai, Hung, Tseng, Singh and Yang] Hsu, H.K., Yao, C.H., Tsai, Y.H., Hung, W.C., Tseng, H.Y., Singh, M., Yang, M.H., 2020.  Progressive domain adaptation for object detection, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 749–757.
  * [Hu et al.(2020)Hu, Kan, Shan and Chen] Hu, L., Kan, M., Shan, S., Chen, X., 2020.  Unsupervised domain adaptation with hierarchical gradient synchronization, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4043–4052.
  * [Inoue et al.(2018)Inoue, Furuta, Yamasaki and Aizawa] Inoue, N., Furuta, R., Yamasaki, T., Aizawa, K., 2018\.  Cross-domain weakly-supervised object detection through progressive domain adaptation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5001–5009.
  * [Johnson et al.(2015)Johnson, Krishna, Stark, Li, Shamma, Bernstein and Fei-Fei] Johnson, J., Krishna, R., Stark, M., Li, L.J., Shamma, D., Bernstein, M., Fei-Fei, L., 2015.  Image retrieval using scene graphs, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3668–3678.
  * [Kazemzadeh et al.(2014)Kazemzadeh, Ordonez, Matten and Berg] Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T., 2014\.  Referitgame: Referring to objects in photographs of natural scenes, in: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787–798.
  * [Kipf and Welling(2016)] Kipf, T.N., Welling, M., 2016\.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907 .
  * [Krizhevsky et al.(2012)Krizhevsky, Sutskever and Hinton] Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems 25, 1097–1105.
  * [Laparra et al.(2020)Laparra, Bethard and Miller] Laparra, E., Bethard, S., Miller, T.A., 2020.  Rethinking domain adaptation for machine learning over clinical language.  JAMIA open 3, 146–150.
  * [Lee et al.(2017)Lee, Kim, Lee and Yoon] Lee, J., Kim, H., Lee, J., Yoon, S., 2017.  Transfer learning for deep learning on graph-structured data, in: Proceedings of the AAAI Conference on Artificial Intelligence.
  * [Liu et al.(2020)Liu, Li, Wang, Zha, Meng and Huang] Liu, X., Li, L., Wang, S., Zha, Z.J., Meng, D., Huang, Q., 2020.  Transferrable referring expression grounding with concept transfer and context inheritance, in: Proceedings of the 28th ACM International Conference on Multimedia, pp. 3938–3946.
  * [Liu et al.(2019)Liu, Wang, Shao, Wang and Li] Liu, X., Wang, Z., Shao, J., Wang, X., Li, H., 2019\.  Improving referring expression grounding with cross-modal attention-guided erasing, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1950–1959.
  * [Long et al.(2017)Long, Cao, Wang and Jordan] Long, M., Cao, Z., Wang, J., Jordan, M.I., 2017.  Conditional adversarial domain adaptation.  arXiv preprint arXiv:1705.10667 .
  * [Lu et al.(2019)Lu, Batra, Parikh and Lee] Lu, J., Batra, D., Parikh, D., Lee, S., 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  arXiv preprint arXiv:1908.02265 .
  * [Mao et al.(2016)Mao, Huang, Toshev, Camburu, Yuille and Murphy] Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A.L., Murphy, K., 2016\.  Generation and comprehension of unambiguous object descriptions, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 11–20.
  * [Mao et al.(2019)Mao, Ma, Yang, Chen and Li] Mao, X., Ma, Y., Yang, Z., Chen, Y., Li, Q., 2019\.  Virtual mixup training for unsupervised domain adaptation.  arXiv preprint arXiv:1905.04215 .
  * [Mirza and Osindero(2014)] Mirza, M., Osindero, S., 2014\.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784 .
  * [Ramponi and Plank(2020)] Ramponi, A., Plank, B., 2020\.  Neural unsupervised domain adaptation in nlp—a survey.  arXiv preprint arXiv:2006.00632 .
  * [Ren et al.(2015)Ren, He, Girshick and Sun] Ren, S., He, K., Girshick, R., Sun, J., 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems 28, 91–99.
  * [Scarselli et al.(2008)Scarselli, Gori, Tsoi, Hagenbuchner and Monfardini] Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G., 2008.  The graph neural network model.  IEEE transactions on neural networks 20, 61–80.
  * [Sharma et al.(2018)Sharma, Ding, Goodman and Soricut] Sharma, P., Ding, N., Goodman, S., Soricut, R., 2018\.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556–2565.
  * [Shu et al.(2018)Shu, Bui, Narui and Ermon] Shu, R., Bui, H.H., Narui, H., Ermon, S., 2018.  A dirt-t approach to unsupervised domain adaptation.  arXiv preprint arXiv:1802.08735 .
  * [Su et al.(2019)Su, Zhu, Cao, Li, Lu, Wei and Dai] Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., Dai, J., 2019.  Vl-bert: Pre-training of generic visual-linguistic representations.  arXiv preprint arXiv:1908.08530 .
  * [Tang et al.(2020)Tang, Niu, Huang, Shi and Zhang] Tang, K., Niu, Y., Huang, J., Shi, J., Zhang, H., 2020\.  Unbiased scene graph generation from biased training, in: Conference on Computer Vision and Pattern Recognition.
  * [Tsai et al.(2018)Tsai, Hung, Schulter, Sohn, Yang and Chandraker] Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., Chandraker, M., 2018\.  Learning to adapt structured output space for semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7472–7481.
  * [Tzeng et al.(2017)Tzeng, Hoffman, Saenko and Darrell] Tzeng, E., Hoffman, J., Saenko, K., Darrell, T., 2017\.  Adversarial discriminative domain adaptation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167–7176.
  * [Wang et al.(2016)Wang, Li and Lazebnik] Wang, L., Li, Y., Lazebnik, S., 2016.  Learning deep structure-preserving image-text embeddings, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5005–5013.
  * [Xu et al.(2017)Xu, Zhu, Choy and Fei-Fei] Xu, D., Zhu, Y., Choy, C.B., Fei-Fei, L., 2017.  Scene graph generation by iterative message passing, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5410–5419.
  * [You et al.(2019)You, Long, Cao, Wang and Jordan] You, K., Long, M., Cao, Z., Wang, J., Jordan, M.I., 2019\.  Universal domain adaptation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2720–2729.
  * [Yu et al.(2018)Yu, Lin, Shen, Yang, Lu, Bansal and Berg] Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., Berg, T.L., 2018\.  Mattnet: Modular attention network for referring expression comprehension, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1307–1315.
  * [Yu et al.(2016)Yu, Poirson, Yang, Berg and Berg] Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L., 2016\.  Modeling context in referring expressions, in: European Conference on Computer Vision, Springer. pp. 69–85.
  * [Zhang et al.(2017)Zhang, David and Gong] Zhang, Y., David, P., Gong, B., 2017.  Curriculum domain adaptation for semantic segmentation of urban scenes, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 2020–2030.
  * [Zhou et al.(2020)Zhou, Yang, Qiao and Xiang] Zhou, K., Yang, Y., Qiao, Y., Xiang, T., 2020.  Domain adaptive ensemble learning.  arXiv preprint arXiv:2003.07325 .
  * [Zhu et al.(2020)Zhu, Xu, Wang, Zhang, Han and Yang] Zhu, Q., Xu, Y., Wang, H., Zhang, C., Han, J., Yang, C., 2020.  Transfer learning of graph neural networks with ego-graph information maximization.  arXiv preprint arXiv:2009.05204 .
  * [Ziser and Reichart(2016)] Ziser, Y., Reichart, R., 2016\.  Neural structural correspondence learning for domain adaptation.  arXiv preprint arXiv:1610.01588 .
  * [Ziser and Reichart(2018)] Ziser, Y., Reichart, R., 2018\.  Pivot based language modeling for improved neural domain adaptation, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1241–1251.


# A new approach for Trading based on Long-Short Term memory technique

Zineb Lanbouri$~{}^{\\#1}$, Saaid Achchab$~{}^{*2}$
# National School for Computer Science and Systems analysis,
Mohamed V University,
Rabat, Morocco
1<EMAIL_ADDRESS>
2<EMAIL_ADDRESS>

###### Abstract

Stock market prediction has always been crucial for stakeholders, traders and
investors. We developed an ensemble Long Short Term Memory (LSTM) model that
includes two time frequencies (annual and daily parameters) in order to
predict next day Closing price (one step ahead). Based on a four-step
approach, this methodology is a serial combination of two LSTM algorithms. The
empirical experiment is applied to 417 NY stock exchange companies. Based on
Open High Low Close metrics and other financial ratios , this approach prooves
that the stock market prediction can be improved.

##  I Introduction

Financial Times Series Forecasting is a crucial topic in finance and its
prediction is an extensive ongoing research issue where computer science
algorithms are very relevant. ”It was estimated that, in 2012, approximately
85% of trades within the US Stock markets were performed by algorithms” Glantz
et al. (2013). To date, there has been numerous works linking Machine Learning
to financial decisions and specially to trading strategies. Dixon et al.
(2016) shows that ”Back Propagation and Gradient Descent have been the
preferred method for training finance structures due to the ease of
implementation and their tendancy to converge” Fehrer et al. (2015); Teiseira
et al. (2010).
Kodia et al. (2010) focuses on social interactions using a multi-agent based
simulation. Batres et al. (2015)’s paper introduces financial time series in a
comparison between Logistic regression, MLP and Naive Bayes.
Deep Learning outperforms in finance as well, and Long-Short Term Memory
(LSTM) is very promising in forecasting field because of its ability to
memorize data. ”Capturing spatio-temporal dependencies, based on regularities
in the observations, is therefore viewed as a fundamental goal for Deep
Learning systems.” Arel et al. (2010)
We aim to use a Long Short Term Memory ensemble method with two input
sequences, a sequence of daily features and a second sequence of annual
features, in order to predict the next day closing price and make a better
decision in trading. LSTM is the most convenient for the following reasons:

  * •

LSTM is considered as an improvment of Reccurrent Neural Network which comes
as a solution to vanishing and exploding gradient, see figure (Fig 1) Bao et
al. (2017); Werbos et al. (1988); Schmidhuber et al. (1997);

  * •

LSTM is suitable for sequences Sutskever et al. (2014);

  * •

LSTM can store and retrieve information using its gates Bengio et al. (2009);

  * •

LSTM doesn’t flow in a single way (unlike Neural Networks);

  * •

LSTM technique can distinguish between recent and early examples Nelson et al.
(2017).

New studies focuses on Ensemble methods where the weakness of a method is
balanced out by the strength of another to produce high quality Chang et al.
(2009). Ensemble learning can be either parallel or serial. A parallel
ensemble results from different learners which are combined according to the
schemes of Majority Voting, Weighted Majority Voting, Min, Max, etc.. Serial
learning arranges different base learners in sequence and selects the result
of one learner as the final output. Our study relies on heteregenous features
with two temporal frequencies, it is then more appropriate to use two LSTM
learners to form one ensemble method.
The main contributions of this study are the following:
(1) a new ensemble method where two time frequency inputs can be combined to
predict the next day closing stock price; (2) evaluation of the method by
comparing it to other Machine Learning techniques and financial strategies.
The paper is divided into four sections. In section II we present some of the
related work to this subject that can be found in the literature. Section III
details the research methodology, describes the data pre-processing and
develops the experiment steps. Section IV analyses the results. Finally, in
section V, we draw some conclusions and future work. The objective of this
project is to study the effectivness of an LSTM ensemble method on the stock
market prediction based on Open, High, Low, Close and Volume indicators as
well as some financial metrics used by traders like MACD and RSI and other
financial ratios, evaluate its performance in terms of RMSE and other measures
through experiments on real NYSE Stock Exchange data and analyze and compare
our results to other machine learning techniques.

Figure 1: LSTM as an improvment of RNN

##  II Litterature Review

Times series prediction is a field that has been related for years by
statistics and Machine Learning techniques.

###  II-A Pure single learner for Stock market prediction

####  II-A1 Autoregressive integrated moving average (ARIMA)

Researchers have exploited statistical techniques to predict and forecast
financial times series like ARIMA which requires strict assumption like
stationarity while finance forecasting is complex, noisy and non-stationary
Bao et al. (2017).

####  II-A2 Artificial Neural Network

Kuremoto et al. (2014)’s paper shows that Artificial Neural Network is used
for Times Series Forecasting since 1980s, more than 5000 publications has been
released on ANN for Forecasting Crone et al. (2007). However, ANN presents
issues like overfitting, impact of initial values and $alpha$.
In order to tackle these issues, Deep Neural Network could be considered as a
solution.

####  II-A3 Deep Neural Network

DNN is effectively more robust toward overfitting and can model complex non
linear relationship between dependant and independant variables Arel et al.
(2010). Howerver, Schmidhuber et al. (2015) explains some of the drawbacks of
Deep Neural Network as:
-Slow Convergence time;
-Vanishing or exploding gradient;
-Expensive computation.

####  II-A4 Deep Learning

Hinton et al. (2006) used a Deep Belief Network and gets 1,25% test error.
Deep Learning has the following advantages:
-Ability to learn complexity;
-Aptitude to learn with little human input with low level/intermediate/high level of abstraction;
-Strong unsupervised learning.

###  II-B Ensemble and hybrid learning models for Stock market prediction

Zhang et al. (2003) apply a hybrid model based on ARIMA and Neural Network to
raise the prediction precision, Werbos et al. (1974) shows that ARIMA-ANN
model outperfoms statistical methods.
Ding et al. (2015) chooses to combine Neural tensor and Convolutional Neural
Network to predict short and long term influences. Bao et al. (2017) use
Stacked Auto-encoders and Long Short Term memory.

##  III Methodology

Our model combines two feature categories in order to transmit the different
aspects of an equity to the network.
The first aspect considers historical daily stock price that reflects its
trend. OHLC (Open, High, Low and Close stock prices), volume and trading
indicators MACD, RSI and Signal show a share evolution based mainly on supply
and demand.
The second aspect consists of financial metrics that are part of the financial
analysis. The objective is to take advantage of both technical analysis and
fundamental-financial analysis. We adopt an Ensemble LSTM approach as
explained in the figure below (Fig 2):

Figure 2: Ensemble LSTM approach

###  III-A Data exploration

The first step in this research is Data exploration. The database used in this
research contains New York Stock Exchange S&P500 data, publicly available at
https://www.kaggle.com/dgawlik/nyse, for the period between 04/01/2010 and
30/12/2016. It is splitted into 2 Datasets, one for prices : Open - Low - High
- Close - Volume and daily indicators : MACD - Signal - RSI.
Moving Average Convergence Divergence (MACD) is a trend-following momentum
indicator and when MACD crosses its signal, it can function as a buy and sell
signal. Relative Strength indicator (RSI) indicates the internal strength of
an equity, it also reflects the vitality of increases over decreases.
The second Dataset contains 75 financial annual ratios that shows informations
about the company itself. The following ratios are some of them:

  * •

EPS (Earning Per Share)

  * •

ROE (Return On Equity)

  * •

Payout ratio

  * •

Dividend yield

  * •

PER (Price Earning Ratio)

  * •

PBR (Price to Book ratio)

Data collected contains 417 companies, the training set consists of
consecutive observations from 04/01/2010 to 07/08/2015 while the test set
consists of observations from 08/08/2015 to 31/12/2016 which represents
sequences of 1408 time series for the training set and 352 for the test set.
Before using Data as input for our Ensemble LSTM method, missing values are
handled using Mean and Logistic regression functions in Matlab, then z-score
normalization is applied (see equation 1).

$z=\frac{(x-\mu)}{\sigma}$ (1)

MACD,the Signal and RSI are defined by the following formulas ( 2) and ( 3) :

$\left\\{\begin{array}[]{r c l}MACD=\underset{12}{EMA}-\underset{26}{EMA}\\\
EMA=(\underset{today}{Value}*\frac{smoothing}{(1+days})+(\underset{yesterday}{Value}*(1-\frac{smoothing}{(1+days}))\\\
smoothing=\frac{2}{(selectedtimeperiod+1)}\end{array}\right.$ (2)

$\begin{array}[]{r c
l}RSI(stepone)=100-\frac{100}{1+\frac{AverageGain}{AverageLoss}}\\\
RSI=100-\frac{100}{1+\frac{PreviousAverageGain*13+CurrentGain}{PreviousAverageLoss*13+CurrentLoss}}\end{array}$
(3)

###  III-B Hierarchical LSTM technique

The chosen Hierachical LSTM technique is based on the assumption that the use
of Ensemble method is a way such that the weakness of a method will be
balanced out by the strength of another. The second step in our model is to
construct the first LSTM learner that can also be considered as the first weak
learner (see fig 2).
As a reminder, Long Short Term Memory (LSTM) is a special type of Recurrent
Neural Network (RNN) with the capability of learning long term and short term
despendencies. Relying on gates, this Deep Learning algorithm has the ability
to memorize. It is then well suited for times series prediction (see fig 1)
and solve the problem of Vanishing gradient.
This model contains an LSTM algorithm modeling prediction based on annual
features (financial ratios) (see fig 3) for the sequence between 2012 and
2016. The input of the first learner consists on 75 features over a 5-sequence
(2012 - 2013 -2014 - 2015 - 2016). The output result is a prediction for the
closing price that will be considered as one of the inputs of the second LSTM
learner.

Figure 3: LSTM number 1

Then, the serial ensemble learning is the result of the combination of the
first learner’s output (prediction) and the other daily features as an input
for the second learner as follows:

  * •

Open

  * •

Highest price for the day

  * •

Lowest price for the day

  * •

Close

  * •

Volume

  * •

MACD

  * •

Signal

  * •

RSI

  * •

the first learner’s prediction

The training is then executed over two stages (sequentially).

After training, the model is capable of forecasting and predicting the Closing
price one step ahead (see fig 2).

##  IV Results and evaluation

Experiments were carried out to predict the next day Closing price following
the approach described above. As explained, the Ensemble LSTM approach is
composed of one main dataset splitted into two datasets sequentially different
since one is for annual features and the other for daily features.
The empirical experiment is applied on 417 New York Stock Exchange companies
using financial ratios for the first learner. Then the predicted value is
included into the input of the second LSTM learner. This article aims to
analyze the hierachical LSTM technique in terms of Root Mean Square Error
(RMSE) and Forecasting.

###  IV-A Results

The Ensemble LSTM to be used is a Regression algorithm that allows to predict
the next day Closing price. To fix the architecture of our method, several
experiences were done to choose the number of hidden units and the learning
rate in order to get the best results. The topology chosen is 20 hidden nodes
for the first LSTM and an initial learning rate of 0.005, the output is then
included into the input of the second LSTM, with a total of 9 features and 200
hidden nodes.
Figures 5 presents the Ensemble LSTM model performance in terms of Mean
Squared Error and Forecasting. Our model shows a great performance since RMSE
is equal to 0.0119 which is more interesting than LSTM standalone for the
first Databse alone with 0.0124.

Figure 4: Forecast vs real values when LSTM is applied to DB number 1 alone

Figure 5: Forecast vs real values when Ensemble LSTM approach is used

###  IV-B Analysis

In order to determine if there is an improvement while applying an ensemble
model, we executed the LSTM algorithm to the first Database (with daily
features) based on a sequence to determine the last Clossing price, we then
got an RMSE equal 0.0124. Then we applied LSTM standalone to the annual
Database and got RMSE equals to 0.08. And finally, to get sure that LSTM is
more appropriate in case of Times series, we chose to compare it to Neural
Network. We used Neural Network for times series with Matlab, Levenberg-
Marquardt as a training algorithm and 10 hidden nodes. We got RMSE equals to
0.067 which is better than LSTM applied to the second Database 6. Table I
compare our approach to LSTM standalone, using each dataset apart and then to
Neural Network.

Figure 6: Neural Network response TABLE I: RMSE for the LSTM standalone, LSTM ensemble and Neural Network LSTM 1 | LSTM 2 | Ensemble LSTM | NN
---|---|---|---
0.0124 | 0.08 | 0.0119 | 0.07

LSTM’s ability to store historic data for a long period and retrieve valuable
data when required makes this technique a good technique for time series
forecasting.

##  V Conclusion and future work

In this study, we established an ensemble method based on LSTM algorithm to
predict the next day Closing price. This model takes into account two
frequencies with two types of variables, daily and annual variables. It
confirms then our hypothesis that the combination of these two databases gives
a better performance. This ensemble method can be a useful tool for traders
and stakeholders to determine their trading strategy. For future work, we
attend to achieve a benchmark between this hierarchical technique and
traditional finance methods. Furthermore, Deep Learning and LSTM are very
promising, we can then expect to improve this model and achieve online
prediction for forecasting.

## References

  * Malhotra et al. (2015) Malhotra, Pankaj and Vig, Lovekesh and Shroff, Gautam and Agarwal, Puneet. (2015) ’Long short term memory networks for anomaly detection in time series’, Presses universitaires de Louvain.
  * Selvin et al. (2017) Selvin, Sreelekshmy and Vinayakumar, R and Gopalakrishnan, EA and Menon, Vijay Krishna and Soman, KP. (2017). ’Stock price prediction using LSTM, RNN and CNN-sliding window model’ International Conference on Advances in Computing, Communications and Informatics (ICACCI), 1643-1647.
  * Schmidhuber et al. (1997) Hochreiter, S., Schmidhuber, Jürgen (1997) ’ Long short-term memory’, Neural computation, 9(8), 1735-1780.
  * Dixon et al. (2016) Dixon, M., Klabjan, D., Bang, J. H. (2016) ’Classification-based financial markets prediction using deep neural networks’, Algorithmic Finance, 1-11.
  * Glantz et al. (2013) Glantz, Morton and Kissell, Robert. (2016) ’Multi-asset risk modeling: techniques for a global economy in an electronic and algorithmic trading era’, Academic Press.
  * Fehrer et al. (2015) Fehrer, R., Feuerriegel, S. (2015) ’Improving Decision Analytics with Deep Learning: The Case of Financial Disclosures’, arXiv preprint, arXiv:1508.01993.
  * Teiseira et al. (2010) Vieira, P. C., Teixeira, A. A. (2010) ’Are finance, management, and marketing autonomous fields of scientific research? An analysis based on journal citations’, Scientometrics, 85(3), 627-646.
  * Kodia et al. (2010) Kodia, Z., Said, L. B., Ghedira, K. (2010) ’ A study of stock market trading behavior and social interactions through a multi agent based simulation’, KES International Symposium on Agent and Multi-Agent Systems: Technologies and Applications, (pp. 302-311). Springer, Berlin, Heidelberg.
  * Arel et al. (2010) Arel, I., Rose, D. C., Karnowski, T. P. (2010) ’ Deep machine learning-a new frontier in artificial intelligence research’, IEEE computational intelligence magazine, 5(4), 13-18.
  * Batres et al. (2015) Batres-Estrada, B. (2015) ’ Deep learning for multivariate financial time series’.
  * Bao et al. (2017) Deng, Y., Bao, F., Kong, Y., Ren, Z., Dai, Q. (2017) ’ Deep direct reinforcement learning for financial signal representation and trading’, IEEE transactions on neural networks and learning systems, 28(3), 653-664.
  * Werbos et al. (1988) Werbos P. J. (1988) ’ Generalization of back-propagation with application to a recurrent gas market model’, Neural Networks, Vol. 1, pp. 339–356.
  * Sutskever et al. (2014) Sutskever, I., Vinyals, O., Le, Q. V. (2014) ’ Sequence to sequence learning with neural networks’, Advances in neural information processing systems, (pp. 3104-3112).
  * Bengio et al. (2009)
  * Nelson et al. (2017) Nelson, D. M., Pereira, A. C., de Oliveira, R. A. (2017) ’ Stock market’s price movement prediction with LSTM neural networks’, Neural Networks (IJCNN), 2017 International Joint Conference, (pp. 1419-1426).
  * Kuremoto et al. (2014) Kuremoto, T., Kimura, S., Kobayashi, K., Obayashi, M. (2014) ’ Time series forecasting using a deep belief network with restricted Boltzmann machines’, Neurocomputing, 137, 47-56.
  * Chang et al. (2009) Chang, P. C., Liu, C. H., Fan, C. Y., Lin, J. L., Lai, C. M. (2009) ’ An ensemble of neural networks for stock trading decision making’, International Conference on Intelligent Computing, (pp. 1-10).
  * Crone et al. (2007) Crone, S., Nikolopoulos, K. (2007) ’ Results of the NN3 neural network forecasting competition’, The 27th International Symposium on Forecasting, (pp. 129)
  * Schmidhuber et al. (2015) Schmidhuber, J. (2015) ’ Deep learning in neural networks: An overview’, Neural networks, 61, 85-117.
  * Hinton et al. (2006) Hinton, G. E., Osindero, S., Teh, Y. W. (2006) ’A fast learning algorithm for deep belief nets’, Neural computation, 61, 18(7), 1527-1554.
  * Zhang et al. (2003) Zhang, G. P. (2006) ’Time series forecasting using a hybrid ARIMA and neural network model’, Neurocomputing, 50, 159-175.
  * Werbos et al. (1974) Werbos, P. (1974) ’Beyond Regression:” New Tools for Prediction and Analysis in the Behavioral Sciences’, Ph. D. dissertation, Harvard University.
  * Ding et al. (2015) Ding, X., Zhang, Y., Liu, T., Duan, J.. (2015) ’Deep learning for event-driven stock prediction’, Twenty-Fourth International Joint Conference on Artificial Intelligence.
  * Bao et al. (2017) Bao, W., Yue, J., Rao, Y. (2017) ’A deep learning framework for financial time series using stacked autoencoders and long-short term memory’, PloS one, 12(7), e0180944. +++++++
  * SHarif et al. (2017) Sharif, Abu-Ghazaleh, Saifan (2017) ’Investigating Algorithmic Stock Market Trading using Ensemble Machine Learning Methods’.