# Context-based Transformer Models for Answer Sentence Selection

Ivano Lauriola
1 University of Padova, Dept of Mathematics
2 Fondazione Bruno Kessler
<EMAIL_ADDRESS>
Alessandro Moschitti
Amazon Alexa
email@domain

###### Abstract

An important task for the design of Question Answering systems is the
selection of the sentence containing (or constituting) the answer from
documents relevant to the asked question. Most previous work has only used the
target sentence to compute its score with the question as the models were not
powerful enough to also effectively encode additional contextual information.
In this paper, we analyze the role of the contextual information in the
sentence selection task, proposing a Transformer based architecture that
leverages two types of contexts, local and global. The former describes the
paragraph containing the sentence, aiming at solving implicit references,
whereas the latter describes the entire document containing the candidate
sentence, providing content based information. The results on three different
benchmarks show that the combination of local and global contexts in a
Transformer models significantly improves the accuracy in Answer Sentence
Selection.

##  1 Introduction

Answer Sentence Selection (AS2) is the task of identifying sentences that
contain the answer to a given question in documents relevant to the question,
e.g., retrieved by a search engine. Neural models have highly improved the
accuracy on such task and recent approaches, e.g., using Transformer
architectures as shown in [Garg et al., 2019], have shown an impressive
accuracy. Although AS2 models seem hard to improve, we note that most previous
work does not exploit contextual information in addition to the candidate
sentence. For example, [Tan et al., 2017] used context in a hierarchical gated
recurrent networks but their accuracy is 10-12 points below the state of the
art [Garg et al., 2019] (as measured on the same exact dataset). Thus we
cannot infer that context is really useful in state-of-the-art models for AS2.
However, we do know that some sentences contain ambiguities that cannot be
solved without using references to the context outside of the target sentence.

Question |  When was Lady Gaga born?
---|---
Prev. |  Lady Gaga is an American singer, songwriter, and actress.
Target |  She was born in 1986.
Next |  Both of her parents have Italian ancestry, and…
Table 1: An example of correct answer sentence requiring larger local context to be selected. Question |  Which role did Bradley Cooper play with Lady Gaga?
---|---
doc. title |  Avengers: endgame - Movie plot
sentence |  Rocket Raccoon was voiced by Bradley Cooper.
doc. title |  A star is born - Movie plot
sentence |  Jackson ”Jack” Maine (Bradley Cooper), a famous country rock singer…
doc. title |  American sniper - Movie plot
sentence |  Chris Kyle, the leading actor, was played by Bradley Cooper.
Table 2: Each of the three sentences can be a correct answer. Only the global
document information, e.g., the title and the link between document concepts,
allows us to select the correct sentence.

For example, Table 1 shows a simple question asking for the birthdate of _Lady
Gaga_. The answer is the middle sentence contained in a paragraph of three
sentences. Clearly, an AS2 classifier cannot select the middle sentence with
high reliability since the sentence does not reveal that _she_ refers to _Lady
Gaga_. On the other hand, AS2 is effective as it targets just one sentence at
a time: selecting an entire paragraph to be sent to the users, often provides
them with too much irrelevant information111Of course, a solution based on a
summarization approach would be optimal but poses complicated challenges,
which have prevented to obtain better solutions than AS2 (to our knowledge)..
A further example is described in Table 2, where the question asks for the
role of _Bradley Cooper_ in a specific movie. Each of the three sentences may
reasonably be a correct answer. Also, the title of the movie is not enough to
select the right answer and can be too far from the local context window.
However, _“A star is born - Movie plot”_ is the only document that contains
references to _Lady Gaga_. This related information allows us to recognize the
correct answer. Thus, to improve AS2, we would need local context, e.g., to
solve the pronoun _she_ , and global information from the whole document to
connect multiple concepts, e.g., the movie title with Lady Gaga.

In this paper, we propose to model local and global contexts for AS2 by
ensembling multiple sentences in Transformer networks [Vaswani et al., 2017]
and Bag-of-Word (BOW) features, respectively. More specifically, we consider
candidates as a triplet ($s_{i-1}$, $s_{i}$, $s_{i+1}$), where $s_{i}$ is the
target answer sentence and $s_{i-1}$ and $s_{i+1}$ are the preceding and the
next sentence of $s_{i}$, respectively. We integrate this triplet in
Transformer architectures by using one single RoBERTa [Liu et al., 2019b]
model encoding the three sentences in three embeddings. Then, we add document-
level BOW representation is the classification layer. We tested our models on
three different datasets, Google NQ and SQuAD adapted for the AS2 task, as
well as the well-known WikiQA, comparing with the very recent state of the art
in AS2 [Garg et al., 2019]. The results clearly show that local and global
contexts can improve AS2 models.

##  2 Related Work

Question Answering (QA) research manly regards two tasks: (i) AS2, which,
given a question and a set of answer sentence candidates, consists in
selecting sentences (e.g., retrieved by a search engine) that correctly answer
the question; and (ii) Machine Reading (MR) or reading comprehension [Chen et
al., 2017], which, given a question and a reference text, involves finding an
exact text span answering it. AS2 research originated from the TREC
competitions [Wang et al., 2007], which target large databases of unstructured
text.

Neural models have significantly contributed to both areas with new
techniques, e.g., [Wang and Jiang, 2016, Qiao et al., 2019, Nogueira and Cho,
2019]. In particular, recent approaches to neural language models, e.g., ELMO
[Peters et al., 2018], BERT [Devlin et al., 2018], RoBERTa [Liu et al.,
2019a], XLNet [Dai et al., 2019] have led to major advancements in several NLP
subfields. These methods capture dependencies between words and their
compounds by pre-training neural networks on large amounts of data.
Interestingly, the resulting models can be easily applied to different tasks
by fine-tuning them on the target training data. The impact of such methods on
AS2, also thanks to transfer learning, is impressive. For example, [Garg et
al., 2019] exceeded the state of the art by 50% (relative error reduction) on
WikiQA [Yang et al., 2015] and TREC-QA [Wang et al., 2007] datasets. However,
the proposed Transformer methods only focus on the similarity between the
question and the candidate sentence pairs, without taking any additional
information into account.

Contextual information was already introduced in neural networks for solving
AS2, e.g., [Tan et al., 2017], by combining question/answer pairs with context
information, selected by applying a similarity between question and document
sentences. Our model is built with state-of-the-art Transformer models for
AS2, which we improve. We also improve the results from [Tan et al., 2017] by
a huge margin (+12% on WikiQA and +5% on SQuAD). Finally, our approach is more
modular and can be easily extended with additional context definitions.

###  2.1 Transformer model for AS2

The Transformer is a popular neural network designed to learn language models,
e.g., dependencies between words, in a context. Transformer models have
recently shown to have remarkable impact on AS2 when used as ranker [Shao et
al., 2019, Garg et al., 2019, Kumar et al., 2019]. Besides architectural
definitions, Transformer models take advantage for extensive pre-training the
language model (i.e., the q/a pair representation) on large-scale corpora by
using the masked language model and next sentence prediction [Devlin et al.,
2018]. In those works, the question answer candidate pairs are codified as a
joint sequence of tokens with specialized delimiters and separators, i.e.,
_[CLS] $q^{1}\dots q^{n}$ [SEP] $s^{1}\dots s^{m}$ [EOS]_, where $x^{j}$
defines the $j$-th token of the sequence $x$. _[CLS]_ , _[SEP]_ , and _[EOS]_
are special tokens used to mark the beginning of the sequence, the separation
between question and candidate answer tokens, and the end of the text. Several
Transformer blocks are applied and then the representation associated with
_[CLS]_ is used in a linear fully-connected layer to to compute the final
score associated with the question/answer pair. The same concepts can be
applied to RoBERTa or other pre-trained Transformer models.

##  3 Contextual Transformer for AS2

To our knowledge no Transformer model for AS2 models use context, except for
the information on the sentences. This is critical as a sentence may contain
references to other part of the text and to external entities (see the example
in Table 1). We enhance the standard Transformer model for AS2 considering two
types of contexts: local and a global. The former aims at solving coreferences
between the answer constituents by relating the candidate sentence to its
neighborhood (typically corresponding to the paragraph containing the
sentence). The latter introduces information concerning the topics and
concepts of the document containing the target sentence.

###  3.1 Local context

Given the target answer sentence candidate, $s_{i}$, we extend the AS2 model
using the preceding, $s_{i-1}$, and the following, $s_{i+i}$, sentences. The
(local) contextual ranker $r_{\textsc{l}}$ takes four elements as input and
provides the following answer:
$a_{\textsc{l}}=\arg\max_{s_{i}\in\mathcal{S}(q)}{r_{\textsc{l}}}(q,s_{i-1},s_{i},s_{i+1}),$
where $\mathcal{S}(q)$ is the set of relevant sentences for the question $q$
and $r_{\textsc{l}}$ is our ranking function, To implement ${r_{\textsc{l}}}$
in the RoBERTa model, the input sequence becomes _[CLS] $q$ [SEP] $s_{i-1}$
[SEP] $s_{i}$ [SEP] $s_{i+1}$ [EOS]_. Additionally, RoBERTa encodes each input
word by using three pieces of information: the token, the sentence, and the
positional embeddings. The first is a standard word-embedding. The positional
embedding describes a token as a function of its position into the sequence.
Finally, the sentence embedding defines a token as a function of the sentence
that contains it. The sentence embedding helps the model to distinguish
between different input sentences: it can be seen as a particular word
embedding of size four, one entry for each element of the input tuple,
$(q,s_{i-1},s_{i},s_{i+1})$. This embedding plays a crucial role in our model
to learn that the instance label is exclusively associated with the middle
sentence. The three embeddings are then summed to produce the final
representation of the sentences to be fed as input to the Transformer. This
process is visually described in Figure 1 (see dashed squares).

Figure 1: BERT/RoBERTa input sequences.

###  3.2 Global context

Our global context describes the document content rather than the structure of
the paragraph containing the answer. We define a global ranker,
$r_{\textsc{g}}$ as
$a_{\textsc{g}}=\arg\max_{s_{i}\in\mathcal{S}(q)}{r_{\textsc{g}}}(q,s_{i},{d}(s_{i}))$,
where $d(s_{i})$ is the document containing $s_{i}$. A simple representation
of ${d}(s_{i})$ is the _bag-of-words_ (BOW) model, for which, we use the same
dictionary used by RoBERTa model. We combine ${d}(s_{i})$ with the _[CLS]_
representation (vector concatenation) before applying our ranking function. To
make the two vectors comparable (in terms of dimensions), we apply a random
projection to the original BOW representation, thus mapping the document into
a vectorial space with the same dimension222We also normalize the projection
to prevent scaling issues. of _[CLS]_ (i.e., 768).

###  3.3 Combined context

Local and global contexts contain different information, thus we also combined
them together in the model, Dual-CTX. This is a RoBERTa model that receives
the question and the candidate sentence with local context. The output of the
Transformer is then combined with the global representation. The architecture
is modular and extensible, local and global feature extraction modules can be
easily exchanged. However. the extensive evaluation of different context
combinations and strategies is beyond the scope of this paper. A general
schema of the architecture is shown in Figure 2.

Figure 2: The proposed architecture that combines global and local contextual
information.

##  4 Empirical assessment

We carried out comparative experiments to evaluate the local and global
contexts and their combination.

###  4.1 Corpora

We used three AS2 corpora, ASNQ, SQuAD, and WikiQA, to empirically assess the
proposed contextual architecture.

ASNQ, Answer Sentence Natural Question [Garg et al., 2019] is a large-scale
open domain corpus for AS2. The corpus is built by transforming the recently
proposed Natural Question (NQ) dataset [et al., 2019] corpus from MR into AS2.
In short, the corpus consists of 57,242 distinct natural questions for
training, and 2,672 for development. For each question, candidate answers have
been extracted from a single Wikipedia page. The corpus contains 21,307,630
question/answer pairs, with an average of 356 answer candidates per question.

WikiQA [Yang et al., 2015] is an open-domain corpus containing queries sampled
from Bing logs. Based on the user clicks, the questions have been associated
with a Wikipedia page (only the summaries were used). We used the clean
setting for which only questions having at least one good and one wrong
answers are considered. The resulting corpus consists of 2,118 training, 126
development, and 243 test questions, with about 10 candidate answers per
question on average. We merged the dev. and test sets as they are too small to
derive reliable results from each of them individually. Overall, we have 2117
questions and 20374 question/answer pairs.

SQuAD 1.1, Stanford Question Answering Dataset [Rajpurkar et al., 2018], is a
large-scale corpus consisting of questions crowdsourced on a set of 20,000
Wikipedia articles. The dataset was designed for MR. We transformed it into a
corpus for AS2 task, by applying the same procedure described by [Garg et al.,
2019]. In short, we split each input paragraph in sentences and labelled those
containing the annotated answers as correct candidates, and all the others as
negative candidates. After this preprocessing, our corpus contains 87,355
questions and 448,108 question/answer pairs. Please note that the results
presented in this paper are not directly comparable to the SQuAD
leaderboard333https://rajpurkar.github.io/SQuAD-explorer/.

Figure 3: Our three approaches to encode local context: a simple Transformer
with question/answer pairs (left), the contextual multi-sentence architecture
(center), and the ensemble of Transformers (right).

###  4.2 Models

We implemented our methods with RoBERTa pre-trained models, using the shared
checkpoint [Wolf et al., 2019]. We fine-tuned the checkpoint on our data by
using (i) the Adam optimizer set with the warmup linear scheduler and a
learning rate peak of $1e-6$; (ii) the binary cross-entropy loss; (iii) a
batch size of 64 examples on a single GPU to train on WikiQA and SQuAD; and
(iii) a batch size of 512 examples distributed on 8 GPUs to train on the ASNQ
corpus (which is much larger). We used the official dev. set to derive the
results, thus we set the parameters, i.e., learning rate, scheduler, and batch
size, on a small portion of the training set (as our dev. set). We train and
test our models on SQuAD and WikiQA four times and take the average results to
account for their variability. Finally, we also used the models generated with
TANDA (transfer and adapt) approach [Garg et al., 2019] for WikiQA. The
authors apply a first fine-tuning on ASNQ and then a second fine-tuning on the
target data. TANDA is the current state of the art, 7-10 points better than
any other approach on WikiQA.

We encode local context as depicted in Figure 3:

$\bullet$ Transformer: the Transformer model for AS2 introduced in Sec. 2.1.
It receives the question/answer pair as input without any context.

$\bullet$ Local Triplet (Loc_t): the proposed Transformer-based method
described in Sec. 3.1, which relies on three different sentences, i.e., the
previous, the target, and the next;

$\bullet$ Local Ensemble (Loc_e): an ensemble of three Transformer models
encoding the three pairs, q/$s_{i-1}$, q/$s_{i}$, and q/$s_{i+1}$ and a final
linear layer fed with the concatenation of the _[CLS]_ embeddings of the three
models. The latter do not share their weights except those from _[CLS]_. The
ensemble is a more expensive approach.

The baseline models for encoding global context are:

$\bullet$ Global BOW (Glob_b): the global context described in Sec. 3.2
consisting of a simple Transformer model with a (compressed) BOW feature set
on the top;

$\bullet$ Global Embedding (Glob_e): a document embedding constituted by the
average of the embeddings derived from all document sentences. We extract the
sentence embedding using RoBERTa fine-tuned on ASNQ. We concatenate the
average with the _[CLS]_ representation output by the AS2 Transformer model.

TransformerLoc_TLoc_EMR

$1$$2$$3$$4$$5$$0.66$$0.68$$0.7$$0.72$MAPASNQ$0$$10$$20$$30$$40$$0.8$$0.82$$0.84$$0.86$$0.88$WikiQA
(TANDA)$0$$5$$10$$15$$20$$0.95$$0.96$$0.97$SQuAD

$1$$2$$3$$4$$5$$0.55$$0.57$$0.59$$0.61$$0.63$$0.65$#
<EMAIL_ADDRESS>epochs$0$$5$$10$$15$$20$$0.94$$0.95$$0.96$# epochs

Figure 4: Local context results (including standard deviation) computed on the
dev. sets.

##  5 Results

We tested different context models on three different datasets using the state
of the art in AS2 as our baseline, i.e., the transformer model made available
in [Garg et al., 2019]. The latter improves 7-10 points all previous AS2
models on WikiQA and TREC-QA datasets.

###  5.1 Local context

Figure 4 shows the Mean Average Precision (MAP) and the Precision at 1 (P@1)
for each epoch for the Transformer, Loc_t, and Loc_e models. The plots show
two main results: first, the superior accuracy of Loc_t is evident on all
corpora, demonstrating that the local context has a positive impact on the AS2
model accuracy. Additionally, the performance of Loc_e method shows that the
mere use of more information is not sufficient: its arrangement into the model
is fundamental. Indeed, the simple aggregation of the three summarized context
vectors seems not able to capture sentence dependencies: disarranged
information produces noise, with a consequent drop in performance.

Next, we used an MR Transformer [Wolf et al., 2019] to implement a sentence
selector model. Our MR approach achieves 0.881 of F1 score on MR task (showing
competitive results on the SQuAD leaderboard with respect to single models).
Then, we simply select the sentence from which the MR extracts the answer span
to solve the AS2 task on SQuAD: the model achieves a P@1 of 0.952. Figure 4
shows that such model (straight line) is comparable to our baseline (single
Transformer models), whereas Loc_t achieves better performance. This is a
loose comparison but it suggests that our approach may be applied to develop
new MR methods.

TransformerLoc_TGlob_BGlob_EDual-CTX

$0$$10$$20$$30$$40$$0.6$$0.65$$0.7$$0.75$$0.8$MAPWikiQA$0$$10$$20$$30$$40$$0.8$$0.82$$0.84$$0.86$$0.88$WikiQA
(TANDA)$5$$10$$15$$20$$0.95$$0.96$$0.97$SQuAD

$0$$10$$20$$30$$40$$0.55$$0.6$$0.65$$0.7$#
<EMAIL_ADDRESS>epochs$0$$5$$10$$15$$20$$0.94$$0.95$$0.96$# epochs

Figure 5: Global context - empirical results computed on the development sets.
The standard deviation is not always exposed to improve the readability.

###  5.2 Global context

Figure 5 shows the MAP and the P@1 achieved by the simple Transformer and the
two global models, i.e., Glob_b and Glob_e. We also report the results of the
combined model, which includes local and global contexts. Finally, we
evaluated the models when applied to WikiQA without the TANDA approach,
showing their behavior in a scenario, when such approach cannot be used, i.e.,
there is no a large and general data for a first fine-tuning step.

The figure shows that both global methods, i.e., BOW and document embedding,
improve the standard model both on WikiQA and SQuAD. We did not apply Glob_B
and Glob_E to ASNQ as the training has a very large computational cost. This
means that we cannot apply TANDA to WikiQA with such context. In any case, the
global context produces accuracy increase on WikiQA and SQuAD (w/o TANDA).
Concerning the combined model, Dual-CTX improves the overall performance on
WikiQA (w/o TANDA) and SQuAD. It does not improve the MAP of Loc_t on WikiQA
when TANDA is used, but P@1 receives an interesting boost. This result
provides evidence that global and local features describe different (and
potentially orthogonal) information444We used BOW in the Dual-CTX rather than
the document embedding for computational reasons..

##  6 Conclusion

AS2 is an important sub-task of Question Answering, which provides an
effective solution for the design of automated QA systems. Usually, models for
AS2 consider only the question and the candidate answer sentence, without
taking the context into account. In this paper, we define two types context,
local and global. The former tries to solve implicit references in a candidate
sentence, and it consists of the previous and successive sentence of a
candidate answer. Conversely, the global context injects document related
information, such as the main content and topics. We proposed Transformer
based architectures that leverages the different context for AS2. Our
empirical assessment shows a remarkable improvement of the proposed approach
on three different AS2 datasets, i.e., ASNQ, WikiQA, and SQuAD 1.1, adapted
for AS2. We will release the contextualized checkpoints and the SQuAD adaption
for AS2555The link will be available after the reviewing process.

Interesting future extensions of our work regard the extraction of features
from the entire rank of documents retrieved for a question. Clearly, learning
to rank features can also improve the selection of answer sentences.

## References

  * [Chen et al., 2017]  Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes.  2017\.  Reading wikipedia to answer open-domain questions.  CoRR, abs/1704.00051.
  * [Dai et al., 2019]  Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov.  2019\.  Transformer-xl: Attentive language models beyond a fixed-length context.  CoRR, abs/1901.02860.
  * [Devlin et al., 2018]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2018\.  BERT: pre-training of deep bidirectional transformers for language understanding.  CoRR, abs/1810.04805.
  * [et al., 2019]  Tom Kwiatkowski et al.  2019\.  Natural questions: a benchmark for question answering research.  Transactions of the Association of Computational Linguistics.
  * [Garg et al., 2019]  Siddhant Garg, Thuy Vu, and Alessandro Moschitti.  2019\.  Tanda: Transfer and adapt pre-trained transformer models for answer sentence selection.
  * [Kumar et al., 2019]  Sawan Kumar, Kartik Mehta, Nikhil Rasiwasia, et al.  2019\.  Improving answer selection and answer triggering using hard negatives.  In EMNLP-IJCNLP.
  * [Liu et al., 2019a]  Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  2019a.  Roberta: A robustly optimized BERT pretraining approach.  CoRR, abs/1907.11692.
  * [Liu et al., 2019b]  Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov.  2019b.  Roberta: A robustly optimized bert pretraining approach.  ArXiv, abs/1907.11692.
  * [Nogueira and Cho, 2019]  Rodrigo Nogueira and Kyunghyun Cho.  2019\.  Passage re-ranking with BERT.  CoRR, abs/1901.04085.
  * [Peters et al., 2018]  Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  2018\.  Deep contextualized word representations.  CoRR, abs/1802.05365.
  * [Qiao et al., 2019]  Yifan Qiao, Chenyan Xiong, Zheng-Hao Liu, and Zhiyuan Liu.  2019\.  Understanding the behaviors of BERT in ranking.  CoRR, abs/1904.07531.
  * [Rajpurkar et al., 2018]  Pranav Rajpurkar, Robin Jia, and Percy Liang.  2018\.  Know what you don’t know: Unanswerable questions for squad.  arXiv preprint arXiv:1806.03822.
  * [Shao et al., 2019]  Taihua Shao, Yupu Guo, Honghui Chen, and Zepeng Hao.  2019\.  Transformer-based neural network for answer selection in question answering.
  * [Tan et al., 2017]  Chuanqi Tan, Furu Wei, Qingyu Zhou, Nan Yang, Bowen Du, Weifeng Lv, and Ming Zhou.  2017\.  Context-aware answer sentence selection with hierarchical gated recurrent neural networks.  IEEE/ACM Transactions on Audio, Speech, and Language Processing.
  * [Vaswani et al., 2017]  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008.
  * [Wang and Jiang, 2016]  Shuohang Wang and Jing Jiang.  2016\.  A compare-aggregate model for matching text sequences.  CoRR, abs/1611.01747.
  * [Wang et al., 2007]  Mengqiu Wang, Noah A. Smith, and Teruko Mitamura.  2007\.  What is the Jeopardy model? a quasi-synchronous grammar for QA.  In EMNLP-CoNLL, pages 22–32, Prague, Czech Republic, June. Association for Computational Linguistics.
  * [Wolf et al., 2019]  Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  2019\.  Transformers: State-of-the-art natural language processing.  arXiv preprint arXiv:1910.03771.
  * [Yang et al., 2015]  Yi Yang, Wen-tau Yih, and Christopher Meek.  2015\.  Wikiqa: A challenge dataset for open-domain question answering.  In EMNLP, pages 2013–2018.


# Semi-Supervised Image-to-Image Translation

Manan Oza  Department of Computer Engineering
D. J. Sanghvi College of Engineering
Mumbai, India
<EMAIL_ADDRESS>Himanshu Vaghela  Department of Computer Engineering
D. J. Sanghvi College of Engineering
Mumbai, India
<EMAIL_ADDRESS>Prof. Sudhir Bagul  Department of Computer
Engineering
D. J. Sanghvi College of Engineering
Mumbai, India
<EMAIL_ADDRESS>
###### Abstract

Image-to-image translation is a long-established and a difficult problem in
computer vision. In this paper we propose an adversarial based model for
image-to-image translation. The regular deep neural-network based methods
perform the task of image-to-image translation by comparing gram matrices and
using image segmentation which requires human intervention. Our generative
adversarial network based model works on a conditional probability approach.
This approach makes the image translation independent of any local, global and
content or style features. In our approach we use a bidirectional
reconstruction model appended with the affine transform factor that helps in
conserving the content and photorealism as compared to other models. The
advantage of using such an approach is that the image-to-image translation is
semi-supervised, independant of image segmentation and inherits the properties
of generative adversarial networks tending to produce realistic. This method
has proven to produce better results than Multimodal Unsupervised Image-to-
image translation.

###### Index Terms:

GANs, image-to-image translation, style transfer

##  I Introduction

Image-to-image transfer has established itself as an important domain in
computer vision since the first paper published by Gatys et al. [1]. Also
known as Neural Style Transfer, it has had many variations over the years,
image colorization [14], style transfer [1], image-to-image transfer [3] and
so on. For which generally deep neural networks have been used with
architectural variances. For instance, we can make a day time image (also
known as the content image) of a city look like a night time image by
selecting the appropriate style (reference) image. Likewise we can have
diverse types of features transfered from one image to another which include
time, color, seasonal translations as well.

Image-to-image translation is the process of translating one image onto
another while preserving the content and photorealism of the original content
image. Deep-learning techniques have proved excellent in faithful and
photorealistic style translation [1, 2, 3, 7]. Our approach is built upon the
idea of generative adversarial networks introduced by Goodfelow et al. [8].
The underlying concept of such a neural network architecture is that a GAN
consists of a generator and a discriminator. The discriminator is trained to
identify real images while the generator tries to fool the discriminator by
creating counterfiet images from noise and passes them on to the
discriminator. Which then returns a verdict on how close the counterfiet
images are to a real one. Based on this feedback the generator improves itself
and creates another image and the cycle repeats.

Here in our paper we make use of an improvised GAN architecture appended with
an Affine Loss factor calculated from a Matting Laplacian matrix [6] in the
final loss function. This additionl factor helps in maintaining spatial
integrity and preserve photorealism in the content image. Since generative
adversarial networks create images from noise they are prone to distortions
and noisy images but provide with the biggest advantage, they do not form the
basis of simple color and style mapping. They recreate the content image with
the style variations.

##  II Related Work

Image-to-image style transfer has reached state-of-the-art [2, 3, 7] results.
The current existing algorithms work in either of the two broadly divided
classes: local translation and global translation. But neither of the
algorithms excel in both photorealism and faithful style translation at the
same time and for all test cases. One or the other factor gets compromised.
Global stylization methods work by matching statistical factors of the pixel
values [11] whereas local stylization is achieved by algorithms that find
close and consistent relations between pixel values of the content and style
images. Another classification is based on the algorithm’s ability to
translate low-level and (or) high-level features. Low-level features
translation involves preservation of the intricacies in the content image
while modifying the color or position with respect to the style image. Whereas
high-level feature translation is the mapping of broader features which by
example means day to night, summer to winter translations.

The best works proposed by Luan et al. [2] and Li et al. [7] are based on the
paradigm of matching the gram matrices and makes use of semantic segmentaions
of the content and style images. Which take in only the content and style
images as the inputs for the network. These algorithms perform post-processing
like affine smoothing techniques thereby drastically improving the quality of
the resultant images. Such methodologies make use of segmented images derived
from the content and style images and then perform style translations from one
segment to another by comparing the gram matrices of the input images. Other
such algorithms based on a similar paradigm are proposed by Gatys et al. [1],
Huang et al. [2] and many others, [1, 7, 12].

Promising results have been showcased by various GAN architectures namely
Pix2pix by Isola et al. [13], Unsupervised Image-to-image Translation by Liu
et al. [4], CycleGAN and BicycleGAn by Zhu et al. [5]. All of which take in a
dataset consisting of multiple images similar to the content and the style
domain. Multimodal Unsupervised Image-to-Image Translation by Huang et al. [3]
provides an approach to the problem by narrowing down the content domain to
only one image and a number of style images which constitute the style latent
code [3]. They have proposed that to make the translation unsupervised the
syle images are decomposd into a common style latent space. The content space
is sampled from this style space based on a conditional distribution to perfom
the translation.

In our proposal we narrow down our method to one content and one style image
which does not make it completely unsupervised as there is only one target
style image. We use the same architecture as proposed by Huang et al. [3] with
an additional affine loss factor added to the loss function which adds to the
smoothness and faithful style transfer which are combined with the properties
of generative adversarial networks.

##  III Methodology

In addition to the model proposed by Huang et al. [3] we add the local affine
transfrom $\mathcal{L}_{m}$ also known as the photorealism factor of the
content image calculated from the Matting Laplacian matrix proposed by Levin
et al. [6].

###  III-A Assumptions

All assumptions are exactly the same as that made in the paper Multimodal
Unsupervised Image-to-Image Translation by Huang et al. [3] which are as
follows. The model assumes that the content and style images are composed of
distinct image spaces $x_{i}\in$ $\mathcal{X}_{i}$ where $x_{i}$ is the
$i^{th}$ image and $\mathcal{X}_{i}$ is its corresponding image space. Here
our goal is to estimate the conditional distributions $p(x_{1}|x_{2})$ and
$p(x_{2}|x_{1})$ leading to the learned translation models
$p(x_{1\longrightarrow 2}|x_{2})$ and $p(x_{2\longrightarrow 1}|x_{1})$
respectively given that $p(x_{1})$ and $p(x_{2})$ are the marginal
distributions of $x_{1}$ and $x_{2}$ respectively.

We make another assumption that $x_{i}\in$ $\mathcal{X}_{i}$ is composed of a
content latent space $c\in$ $\mathcal{C}$ and a style latent space $s_{i}\in$
$\mathcal{S}_{i}$ corresponding to every image from the dataset. Thus two
images $(x_{1},x_{2})$ are generated from the individual generators by
$x_{1}=G_{1}^{*}(c,s_{2})$ and $x_{2}=G_{2}^{*}(c,s_{1})$. $G_{1}^{*}$ and
$G_{2}^{*}$ are generator functions with $E_{1}^{*}$ and $E_{2}^{*}$ being
their inverse encoders where $E_{1}^{*}=(G_{1}^{*})^{-1}$ and
$E_{2}^{*}=(G_{2}^{*})^{-1}$. Hence our aim is to train the encoder and
generator functions using neural networks.

###  III-B Matting Laplacian

Image matting is the process of extracting the foreground and the background
from an image with minimal possible user intervention. The Matting Laplacian
[6] process produces an alpha matte which is the segmented image with the
foreground object in white and the background in black or vice versa as per
the requirements. Using this matting laplacian matrix we calculate the local
affine transform factor $\mathcal{L}_{m}$ also known as the photorealism
factor.

$\displaystyle\mathcal{L}_{m}=\sum_{c=1}^{3}V_{c}[O]^{T}\mathcal{M}_{I}V_{c}[O]$
(1)

It is a summation of the affine losses of all the three channels of the image.
$\mathcal{M}_{I}$ is the least-squares penalty function that is dependant on
the input image I. The dimensions of the $\mathcal{M}_{I}$ matrix are (N
$\times$ N) and $V_{c}[O]$ is the vectorized format of the input image O in
the channel c having dimensions (N $\times$ 1). Thus this factor proves
crucial in preserving the photorealism and the content image in our proposal.

(a) Single domain reconstruction of $x_{1}$

(b) Single domain reconstruction of $x_{2}$

Figure 1: The above given images are the representations of how the self
domain reconstruction of our model works. The images $x_{1}$ and $x_{2}$ are
encoded into their respective content and style latent codes $c_{i}$ and
$s_{i}$. The reconstructed images $x_{1}{{}^{\prime}}$ and
$x_{2}{{}^{\prime}}$ are not equal to their corresponding input imags because
of $\mathcal{L}_{1}$ loss. Figure 2: This image represents the cross-domain
translation of $x_{1\longrightarrow 2}$ and $x_{2\longrightarrow 1}$. The ’*’
represents the Gaussian prior. We encounter $\mathcal{L}_{1}$ losses when
reconstructing images from $s_{i}$ to $s_{i}{{}^{\prime}}$ and $c_{i}$ to
$c_{i}{{}^{\prime}}$ thereby fulfilling the bidirectional reconstruction
properties of our model. GAN loss is encountered when the translation of
$x_{1}$ $\rightarrow$ $x_{2\longrightarrow 1}$ and $x_{2}$ $\rightarrow$
$x_{1\longrightarrow 2}$ takes place.  Figure 3: The auto-encoder
architecture. It consists of a content encoder, style encoder and a decoder.
The content encoder comprises of three convolutional blocks which perform
downsampling, followed by four residual blocks. The style encoder comprises of
five convolutional layers followed by a global average pooling layer followed
by a fully connected layer at the end. The decoder makes use of a multi-layer
perceptron that generates AdaIN [15] parameters from the style code. The
content code along with AdaIN parameters is processed by four residual blocks.
The output of the residual blocks is passed on to three upsampling layers that
generate the final image.

###  III-C Model

Our model given in figure 3 constitutes an encoder and a decoder $E_{i}^{*}$
and $G_{i}^{*}$ respctively for every domain $\mathcal{X}_{i}$, in our case
$i$ = 1, 2. The encoder is factorized from the content and style latent codes
$c_{i}$ and $s_{i}$.

$\displaystyle(\textit{$c_{i}$},\textit{$s_{i}$})=(E_{i}^{c}(\textit{$x_{i}$}),E_{i}^{s}(\textit{$x_{i}$}))=E_{i}(\textit{$x_{i}$})$
(2)

Thus for image-to-image translation we interchange the encoders and decoders
i.e. for translation $x_{1\longrightarrow 2}$ we make use of the content code
$c_{1}$ = $E_{1}^{c}(x_{1})$ and a randomly drawn style latent code from
$s_{2}$. Subsequently we use the decoder $G_{2}$ to generate the image.

$\displaystyle\textit{$x_{1\longrightarrow 2}$}=\textit{$G_{2}(c_{1},s_{2})$}$
(3)

The loss function is composed of two factors, the bidirectional reconstruction
loss and the adversarial loss. The bidirectional reconstruction loss is added
to make sure that there is a two way reconstruction of images in the
directions, image $\rightarrow$ latent $\rightarrow$ image and latent
$\rightarrow$ image $\rightarrow$ latent. The image reconstruction loss is
computed as the difference between the image reconstructed from the latent
spaces $c_{1}$ and $s_{1}$ of image $x_{1}$ and the image $x_{1}$ which is
given by (it is similar to $\mathcal{L}_{recon}^{x_{2}}$ for the image
$x_{2}$):

$\displaystyle\mathcal{L}_{recon}^{x_{1}}=\mathbb{E}_{x_{1}\sim
p(x_{1})}[\|G_{1}(E_{1}(x_{1}))-x_{1}\|_{1}]$ (4)

The latent reconstruction loss $\mathcal{L}_{recon}^{c_{1}}$ is the difference
between the content encoding of the generated image $G_{2}(c_{1},s_{2})$ and
the content encoding $c_{1}$ of the image $x_{1}$ and
$\mathcal{L}_{recon}^{s_{2}}$ is the difference between the style encoding of
the generated image $G_{2}(c_{1},s_{2})$ and the style encoding $s_{2}$ of the
image $x_{2}$ they are given by the equations (which are similar for and
$\mathcal{L}_{recon}^{s_{1}}$):

$\displaystyle\mathcal{L}_{recon}^{c_{1}}=\mathbb{E}_{c_{1}\sim
p(c_{1}),s_{2}\sim p(s_{2})}[\|E_{2}^{c}(G_{2}(c_{1},s_{2}))-c_{1}\|_{1}]$ (5)
$\displaystyle\mathcal{L}_{recon}^{s_{2}}=\mathbb{E}_{c_{1}\sim
p(c_{1}),s_{2}\sim p(s_{2})}[\|E_{2}^{s}(G_{2}(c_{1},s_{2}))-s_{2}\|_{1}]$ (6)

Here $q(s_{2})$ is defined as the prior $\mathcal{N}$(0, I) and $p(c_{1})$ is
defined as $c_{1}=E_{1}^{c}(x_{1})$ where $x_{1}\sim p(x_{1})$.

Since we use a GAN framework we encounter an adversarial loss which is
supposed to be minimised so that the generated images are as identical as
possible to the original images. This loss is given by:

$\displaystyle\mathcal{L}_{GAN}^{x_{2}}=\mathbb{E}_{c_{1}\sim
p(c_{1}),s_{2}\sim p(s_{2})}[log(1-D_{2}(G_{2}(c_{1},s_{2})))]+$ (7)
$\displaystyle\mathbb{E}_{x_{2}\sim p(x_{2})}[logD_{2}(x_{2})]$

Here $D_{2}$ is the discriminator function that distinguishes between the real
image $x_{2}$ and the translated images. The discriminator function $D_{1}$
and loss $\mathcal{L}_{GAN}^{x_{2}}$ are defined in a similar way.

As mentioned earlier the architecture we use is essentially the same as that
was proposed by Huang et al. [3]. The only difference being that in our
approach we use only one style image and add the affine transform loss in the
overall loss function. We assume [2] that the input images are photorealistic
and we do not have to lose this property. Thus we penalize the loss fuction
with the photorealism factor so as not to lose this property while minimizing
the reconstruction losses from the image, content and style latent spaces. The
overall loss fuction proposed by us is given by:

$\displaystyle\textstyle\underset{E_{1},E_{2},G_{1},G_{2}}{\min_{\vphantom{p}}}\underset{D_{1},D_{2}}{\max_{\vphantom{p}}}\mathcal{L}(E_{1},E_{2},G_{1},G_{2},D_{1},D_{2})=$
(8)
$\displaystyle\mathcal{L}_{GAN}^{x_{1}}+\mathcal{L}_{GAN}^{x_{2}}+\lambda_{x}(\mathcal{L}_{recon}^{x_{1}}+\mathcal{L}_{recon}^{x_{2}})+$
$\displaystyle\lambda_{c}(\mathcal{L}_{recon}^{c_{1}}+\mathcal{L}_{recon}^{c_{2}})+\lambda_{s}(\mathcal{L}_{recon}^{s_{1}}+\mathcal{L}_{recon}^{s_{2}})+$
$\displaystyle\lambda_{A}(\mathcal{L}_{m}^{x_{1}}+\mathcal{L}_{m}^{x_{2}})$

Where $\lambda_{x}$, $\lambda_{c}$, $\lambda_{s}$ are the weights that control
the reconstruction, and $\lambda_{A}$ is the photorealism regularization
weight [2].

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

Figure 4: First two images in each row i.e. [4a and 4b], [4g and 4h] and [4m
and 4n] are the constituent datasets. Image 4c and 4i and 4o are the results
obtained from Multimodal Unsupervised Image-to-image Translation [3] with 4a,
4g and 4m as the content images respectively and 4b, 4h and 4n as the style
images respectively. While 4d, 4j and 4p are the results produced by the same
method with 4b, 4h and 4n as the content images respectively and 4a, 4g and 4m
as the style images respectively. Image 4e is our result with 4a as the
content image and 4b as the style image. Image 4f is our result with 4b as the
content image and 4a as the style image. Image 4k is our result with 4g as the
content image and 4h as the style image. Image 4l is our result with 4h as the
content image and 4g as the style image. Image 4q is our result with 4m as the
content image and 4n as the style image. Image 4r is our result with 4n as the
content image and 4m as the style image.

###  III-D Analysis

Our goal is to minimize the loss function defined in equation (8). This minima
is the optimal state of of our model and at this point the following states
are achieved:

$\displaystyle p(c_{1})=p(c_{2})$ (9) $\displaystyle p(s_{1})=q(s_{1})$ (10)
$\displaystyle p(s_{2})=q(s_{2})$ (11) $\displaystyle
p(x_{1},x_{1\longrightarrow 2})+p(c_{1})=p(x_{2\longrightarrow
1},x_{2})+p(c_{2})$ (12)

The equation (12) is different from the one proposed by Huang et al. [3]
because our model adds the local affine loss of the content images. Our model
is constructed in such a way that when $x_{1}$ is the content image $x_{2}$ is
taken as the style image and vice versa. Which is why the local affine loss of
both the images is taken into consideration in equation (8) and also the
content marginal distributions are added and taken into account when comparing
the joint distributions $p(x_{1},x_{1\longrightarrow 2})$ and
$p(x_{2\longrightarrow 1},x_{2})1$. At this state the content marginal
distributions $p(c_{1})$ and $p(c_{2})$ also become equal. Also at this
optimal state the style marginal distributions $p(s_{i})$ are equal to their
prior distributions $q(s_{i})$. The fact that we use one one-to-one image
mapping makes our process sound like it follows the supervised learning
paradigm, but it does not. Even though we have only one image in the content
and style domain the images are encoded into a content and style latent space
and translated on the basis of conditional probability. Thus our method is
free from any deterministic translations as performed by the methods [1, 2, 7,
12, 14] which make use of image segmentation that helps in mapping regions of
interest in both the content and style images.

##  IV Implementation Details

We have adapted the publicly available pytorch implementation of Multimodal
Unsupervised Image-to-image Translation [3]. The architecture consists of an
auto-encoder (generator) and a discriminator. The auto-encoder comprises of a
separate content and style encoder and a combined decoder. The auto-encoder
architecture consists of the following layers:

  * •

The content encoder whose content makes up the content latent space (in the
listed order):

    * –

7 $\times$ 7 convolutional block with stride 1 and 64 filters.

    * –

4 $\times$ 4 convolutional block with stride 2 and 128 filters.

    * –

4 $\times$ 4 convolutional block with stride 2 and 256 filters.

    * –

4 residual blocks each consisting of two 3 $\times$ 3 convolutional blocks
with 256 filters.

  * •

The style encoder whose output is added to the style latent space (in the
listed order):

    * –

7 $\times$ 7 convolutional block with stride 1 and 64 filters.

    * –

4 $\times$ 4 convolutional block with stride 2 and 128 filters.

    * –

3 4 $\times$ 4 convolutional block with stride 2 and 256 filters.

    * –

Global average pooling layer.

    * –

Fully connected layer with 8 filters.

  * •

The decoder which reconstructs an image from the content and style latent code
(in the listed order):

    * –

4 residual blocks each consisting of two 3 $\times$ 3 convolutional blocks
with 256 filters.

    * –

2 $\times$ 2 nearest-neighbour upsampling layer followed by a 5 $\times$ 5
convolutional layer with stride 1 and 128 filters.

    * –

2 $\times$ 2 nearest-neighbour upsampling layer followed by a 5 $\times$ 5
convolutional layer with stride 1 and 64 filters.

    * –

7 $\times$ 7 convolutional block with stride 1 and 3 filters.

The discriminator used is a multi-scale discriminator proposed by Wang et al.
[9] which makes use of the LSGAN objective function proposed by Mao et al.
[10]. This helps to pilot the generator towards producing realistic and perfom
effective translation while preserving the content. The architecture consists
of the following layers in the listed order:

  * •

4 $\times$ 4 convolutional block with stride 2 and 64 filters.

  * •

4 $\times$ 4 convolutional block with stride 2 and 128 filters.

  * •

4 $\times$ 4 convolutional block with stride 2 and 256 filters.

  * •

4 $\times$ 4 convolutional block with stride 2 and 512 filters.

We use the python implementation to compute the Matting Laplacian matrix [16]
from the tensorflow implementation of Deep Photo Style Transfer [2]. The
image, content and style reconstruction weights and the photorealism
regularization weight are experimentally set to $\lambda_{x}$ = 10,
$\lambda_{c}$ = 1, $\lambda_{s}$ = 1 and $\lambda_{A}$ = $10^{4}$ [2]
respectively. Our implementation is available on
$https://github.com/ozamanan/semisit$.

##  V Results

Our dataset is composed of only two 3 channel images with resolution 256
$\times$ 256\. Thus we use a batch size of 1. Furthermore for every iteration
both the images from the dataset are used once to train the respective parts
of the network. At once when one image is used as the content image the other
one is used as the style image and vice versa thereby completing the
bidirectional reconstruction process. All images used for experimental
purposes are taken from the implementation of Deep Photo Style Transfer [2].

The images fig. 4e, 4f, 4k, 4l, 4q and 4r shown in fig. 4 are the results
generated from our code whereas the images 4c, 4d, 4i, 4j, 4o and 4p are the
results generated using the code of Huang et al. [3]. The results shown by us
are the optimal results beyond which the images tend to converge to their
respective style images. The optimal solution is the one where the resultant
image holds the properties of both the content and style images while still
being recognised by the discriminator as a constituent image of the dataset.
This optimal state is mentioned in eq. (12).

This optimal state clearly shows an improvement in content preservation and
image smoothness over the proposal of Huang et al. [3]. It is achieved due to
the addition of the affine transform factors $\mathcal{L}_{m}^{x_{1}}$ and
$\mathcal{L}_{m}^{x_{2}}$. Thus our proposed methodology generates results
that are better in comparison to the results from the method used by Huang et
al. [3].

##  VI Conclusion

We have proposed an architecture that performs the task of unsupervised image-
to-image translation with better accuracy and results. The future work
includes reducing the noise and making the results more accurate even for low
resolutions. Another future scope lies in broadening this architecture for the
generation of music, text and videos.

## References

  * [1] L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2414–2423, 2016.
  * [2] Luan, F., Paris, S., Shechtman, E., Bala, K.: Deep photo style transfer. In: CVPR. (2017)
  * [3] Huang, X., Liu, M.Y., Belongie, S., Kautz, J.: Multimodal unsupervised image-toimage translation. In: ECCV. (2018)
  * [4] Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: NIPS. (2017)
  * [5] Almahairi, A., Rajeswar, S., Sordoni, A., Bachman, P., Courville, A.: Augmented cyclegan: Learning many-to-many mappings from unpaired data. arXiv preprint arXiv:1802.10151 (2018)
  * [6] A. Levin, D. Lischinski, and Y. Weiss. A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2):228–242, 2008.
  * [7] Y. Li, M.-Y. Liu, X. Li, M.-H. Yang, and J. Kautz, “A closed-form solution to photorealistic image stylization,” arXiv:1802.06474, 2018.
  * [8] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: NIPS. (2014)
  * [9] Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High resolution image synthesis and semantic manipulation with conditional gans. In: CVPR. (2018)
  * [10] Mao, X., Li, Q., Xie, H., Lau, Y.R., Wang, Z., Smolley, S.P.: Least squares generative adversarial networks. In: ICCV. (2017)
  * [11] Erik Reinhard, Michael Ashikhmin, Bruce Gooch, and Peter Shirley. Color transfer between images. IEEE Computer Graphics and Applications, 21(5):34–41, 2001.
  * [12] Ulyanov, V. Lebedev, A. Vedaldi, and V. Lempitsky. Texture networks: Feedforward synthesis of textures and stylized images. In International Conference on Machine Learning (ICML), 2016.
  * [13] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. 2016.
  * [14] Paulina Hensman and Kiyoharu Aizawa. cgan-based manga colorization using a single training image. arXiv:1706.06918, 2017.
  * [15] Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV. (2017)
  * [16] Martin Benson. [Online]. Available: $https://github.com/$
$martinbenson/deep-photo-styletransfer/blob/master/$
$deep\\_photo.py$