# Diformer: Directional Transformer for Neural Machine Translation

Minghan Wang1, Jiaxin Guo1, Yuxia Wang2, Daimeng Wei1, Hengchao Shang1,
Chang Su1, Yimeng Chen1, Yinglu Li1, Min Zhang1, Shimin Tao1, Hao Yang1
1Huawei Translation Services Center, Beijing, China
2The University of Melbourne, Melbourne, Australia
{wangminghan,guojiaxin1,weidaimeng,shanghengchao,
suchang8,chenyimeng,liyinglu,zhangmin186,
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Autoregressive (AR) and Non-autoregressive (NAR) models have their own
superiority on the performance and latency, combining them into one model may
take advantage of both. Current combination frameworks focus more on the
integration of multiple decoding paradigms with a unified generative model,
e.g. Masked Language Model. However, the generalization can be harmful on the
performance due to the gap between training objective and inference. In this
paper, we aim to close the gap by preserving the original objective of AR and
NAR under a unified framework. Specifically, we propose the Directional
Transformer (Diformer) by jointly modelling AR and NAR into three generation
directions (left-to-right, right-to-left and straight) with a newly introduced
direction variable, which works by controlling the prediction of each token to
have specific dependencies under that direction. The unification achieved by
direction successfully preserves the original dependency assumption used in AR
and NAR, retaining both generalization and performance. Experiments on 4 WMT
benchmarks demonstrate that Diformer outperforms current united-modelling
works with more than 1.5 BLEU points for both AR and NAR decoding, and is also
competitive to the state-of-the-art independent AR and NAR models.

##  1 Introduction

Machine translation can be considered as a conditional generation task, which
has been dominated by neural networks, especially after Transformer (Vaswani
et al., 2017). Conventional autoregressive (AR) NMT models obtain the
impressive performance, but it’s time-consuming to decode token one by one
sequentially (Sutskever et al., 2014; Bahdanau et al., 2015). Aiming at fast
inference, non-autoregressive (NAR) NMT models enhance the parallelizability
by reducing or removing the sequential dependency on the translation preﬁx
inside the decoder, but suffering from performance degradation owing to the
multi-modality problem, which is still an open-question (Gu et al., 2018; Shu
et al., 2020; Ghazvininejad et al., 2020; Lee et al., 2018; Ghazvininejad et
al., 2019; Stern et al., 2019; Welleck et al., 2019; Gu et al., 2019a, b).

It’s always non-trivial to balance high performance and low latency in a
single model perfectly. Therefore, another branch focuses on the unified-
modeling of multiple decoding paradigms so that decoding with AR or NAR in
different scenarios (AR for quality-first and NAR for speed-first) with one
model can be achieved (Mansimov et al., 2020; Tian et al., 2020; Qi et al.,
2021), making the performance and speed can be pursued more practically.

Whereas, challenges still exist. For example, a generalized conditional
language model is often required to support the generation with customized
orders or positions (Mansimov et al., 2020; Tian et al., 2020), which actually
prevents the model from being fully trained on specific decoding method,
leading to the declines in overall performance. In addition, in some works, AR
and NAR decoding may needs to be trained separately in the stage of
pretraining or fine-tuning (Qi et al., 2021), making the training more
expensive.

To ameliorate these issues, we propose Directional Transformer (Diformer)
which resolve the unification of AR and NAR in a more practical way. First of
all, we abandon the compatible of multiple flexible decoding strategies, but
focusing on the modeling of some commonly used strategies that have good
performance. For the AR decoding, it has been proved that monotonic linear
generation is still considered as the best strategy (Mansimov et al., 2020;
Tian et al., 2020), so we choose to only model the left-to-right (L2R) and
right-to-left (R2L) generation. For the NAR decoding, we choose to follow the
stream of masked-language model, like mask-predict in CMLM (Ghazvininejad et
al., 2019) or parallel easy-first in Disco (Kasai et al., 2020), since they
are simpler than insertion-based method but still being effective.

To this end, we unify two decoding paradigms into three generation directions
— L2R, R2L and straight, and formulate it through a new objective named as
Directional Language Model (DLM), making the prediction of tokens conditioned
on contexts controlled by a newly introduced direction variable. It ties AR
and NAR into a unified generation framework while still preserving the
original dependency assumptions of AR and NAR, retaining both generalization
and performance. Meanwhile, all directions can be trained simultaneously with
the time spent equally to the training of an independent NAR model, which
greatly reduces the training cost compared to two-stages methods.

Experimental results on the WMT14 En$\leftrightarrow$De and WMT16
En$\leftrightarrow$Ro datasets for all three directions indicate that Diformer
performs better than previous unification-based works by more than 1.5 BLEU
points. Comparing to other state-of-the-art independent AR and NAR models,
Diformer is also competitive when decoding in the same mode. We summarize
contributions of our work as:

  * •

We unify the AR and NAR decoding into three generation direction and formulate
it with the Directional Language Model.

  * •

We propose the Diformer, a Transformer-based model that can be trained with
DLM, where all direction can be trained simultaneously.

  * •

Experiments on WMT14 En$\leftrightarrow$De and WMT16 En$\leftrightarrow$Ro
demonstrate the ability of Diformer with competitive results compared to
unified or independent models.

###  1.1 Related Work

Mansimov et al. (2020) unifies decoding in directed and undirected models by a
generalized framework, in which the generating process is factorized as the
position selection and the symbol replacement, where the first step is
achieved by Gibbs sampling or learned adaptive strategies, the second step can
be handled by a masked language model pretrained on monolingual corpora and
fine-tuned on the NMT task. Their model supports at least 5 decoding
strategies including hand-crafted and learned, all of them can be used for
both linear time decoding (AR) and constant time decoding (NAR).

Similarly, Tian et al. (2020) unified AR and NAR by adapting permutation
language modeling objective of XLNet to conditional generation, making it
possible to generate a sentence in any order. The model is evaluated to decode
in monotonic and non-monotonic AR, semi-AR and NAR with at least 8 position
selection strategies including pre-defined and adaptive.

Both of them achieves the compatible to customized decoding through position
selection and applying the selected positions/orders on a generalized
generative model, which leads to the gap between training and inference. In
contrast to the position selection, we directly model the decoding process
with three generation directions in a task-specific manner, thereby without
introducing additional complexity to the task and close the gap between
training objective and inference strategy. We consider it is worthwhile to
obtain performance improvements by abandon some flexibility.

##  2 Method

###  2.1 Background

Before the description of Diformer, the conventional AR model and the
iterative mask prediction based NAR model that applied in Diformer will be
introduced first.

The likelihood of an AR model is a factorization following the product rule,
assuming each token is conditioned on all previous generated context. Taking
the L2R and R2L AR model as examples:

$\displaystyle\mathcal{L}_{\text{L2R}}$ $\displaystyle=\sum_{i=1}^{N}\log
P(y_{i}|y_{1:i-1},X;\theta)$ (1) $\displaystyle\mathcal{L}_{\text{R2L}}$
$\displaystyle=\sum_{i=1}^{N}\log P(y_{i}|y_{i+1:N},X;\theta)$ (2)

where $X$ is the source text, $y_{1:i-1}$ and $y_{i+1:N}$ are previous outputs
in opposite direction, $\theta$ is the learnable parameters, $N$ is the target
length.

In the iterative-refinement based NAR model like CMLM (Ghazvininejad et al.,
2019), the conditional dependency is loosed, assuming the prediction of target
token can be independent with each other, but conditioned on the output tokens
(context) from last iteration:

$\mathcal{L}_{\text{CMLM}}=\sum_{y_{i}\in
Y_{\text{mask}}^{(t)}}logP(y_{i}|X,Y_{\text{obs}}^{(t)};\theta).$ (3)

where $t$ is the iteration step $t=\\{1,...,T\\}$, $Y_{\text{obs}}$ are
observable tokens (context), $Y_{\text{mask}}=Y\setminus Y_{\text{obs}}$ are
masked tokens for predicting. In each iteration, $N\frac{T-t}{T}$ of predicted
tokens with low confidence will be re-masked and predicted again in the next
iteration, conditioned on remaining high-confidence predictions as observable
context until the last iteration. At the initial iteration, the model
determines the target length $N$ based on the source text $P(N|X)$ and makes
the first step prediction with $N-2$ mask symbols as well as [BOS] and [EOS]
input to the decoder, equivalent to merely conditioned on the source.

Instead of using the global context, in DisCo (Kasai et al., 2020), the target
token at each position is predicted with different context, namely, the
disentangled context. In such case, all tokens can be used for training and
updated at each iteration during inference:

$\mathcal{L}_{\text{DisCo}}=\sum_{i=1}^{N}logP(y_{i}|X,Y_{\text{obs}}^{i,t};\theta),$
(4)

where $Y_{\text{obs}}^{i,t}$ is the context only for $y_{i}$. The parallel
easy-first decoding strategy is proposed (we call it easy first in following
sections for simplicity) to improve the decoding efficiency, where the context
of each token is composed by predictions at easier positions determined in the
first iteration:

$Y_{\text{obs}}^{i,t}=\\{y_{j}^{t-1}|z(j)<z(i)\\},$ (5)

where $z(i)$ denotes the descending ordered rank of the probability $P_{i}$
computed in the first iteration. During the training of CMLM and DisCo, a
subset of tokens are selected as the context, CMLM updates parameters only
with the loss on masked tokens while DisCO uses all tokens for updating.

In the Diformer, we aim to unify the two exclusive dependency assumptions
(Yang et al., 2019) of AR and NAR essentially by proposing a new training
objective and model architecture that can make them trained jointly.

###  2.2 Directional Language Model

We aim to unify the AR and NAR decoding into three generation directions —
L2R, R2L and straight, i.e. making prediction on the target token at the
rightward, leftward and the original position. How to realize this goal is an
open-question. In this work, we achieve it by explicitly providing a direction
instruction and corresponded contexts to the model. Taking an example on the
target sequence $Y=[A,B,C,D,E]$, the probability of $y_{3}=C$ generated from
three directions can be expressed as:

$P_{3}=\begin{cases}P(y_{3}=C|X,\\{A,B\\})&\text{L2R}\\\
P(y_{3}=C|X,\\{D,E\\})&\text{R2L}\\\
P(y_{3}=C|X,\\{A,B,?,D,E\\})&\text{straight}\end{cases}$

where $?$ can be a mask symbol performing like a placeholder.

Formally, given the target sequence $Y=[y_{1},...,y_{N}]$, token $y_{i}$ can
be generated from direction $z_{i}\in\mathcal{Z}=\\{R,S,L\\}$ (i.e. L2R,
straight and R2L) given the context $Y_{z_{i}}$ and X:

$P(y_{z_{i}}|X,Y_{z_{i}}),$

where $Y_{z_{i}}$ is determined by the direction $z_{i}$:

$Y_{z_{i}}=\begin{cases}y_{1:i-1}&\ \ z_{i}=R\\\ y_{i+1:N}&\ \ z_{i}=L\\\
Y_{\text{obs}}^{i}&\ \ z_{i}=S\end{cases}$

When $z_{i}=R$ or $L$, the model works exactly same to the conventional AR
model by conditioning on previously generated tokens at leftwards or
rightwards. When $z_{i}=S$, the model works in an iterative-refinement manner
(e.g. mask-predict in CMLM or parallel easy-first in DisCO) by conditioning on
a partially observed sequence $Y_{\text{obs}}^{i}$ with multiple tokens being
masked including $y_{i}$, same as the disentangled context in DisCo.

We can thereby formulate the objective of directional language model as the
expectation over all possible generation directions on each token:

$P(Y|X)=\mathbb{E}_{z_{i}\in\mathcal{Z}}\big{[}\prod_{i=1}^{N}P(y_{z_{i}}|X,Y_{z_{i}})\big{]}$
(6)

The expectation can be approximated with sampling, similar to the permutation
language model in (Yang et al., 2019; Tian et al., 2020), where a permutation
of the sequence is sampled during training, we, instead, sample the direction
for each token. In this way, the factorization of DLM incorporates both
conditional dependency assumption of AR, and conditional independence
assumption of NAR, thereby makes the training objective closely related to the
decoding methods.

#### Training

The sampling of direction in DLM allows us to train the generation of all
directions simultaneously, we introduce the detailed method in this section.

As we all know that the training of Transformer (Vaswani et al., 2017) can be
paralleled with teacher forcing, achieved by feeding $y_{1:N-1}$ (context) to
the model at once and computing the loss on $y_{2:N}$ (target). The context
and target sequence can be easily created by a shifting operation that aligns
$y_{i-1}$ to $y_{i}$.

Diformer can also be trained in a similar way, but before that, we have to
make a slight change when implementing the computation of the likelihood in Eq
6 due to the difficulty of creating the context sequence $Y_{z_{i}}$ with
complicated dependencies. The original equation aims to compute the likelihood
on the ground-truth sequence $Y$ where each token is conditioned on a
customized context determined by the sampled direction, meaning that the
context sequence cannot be shared as Transformer does. Creating specialized
context for every token is non-trivial especially when encountered with
position changing caused by the shifting when $z_{i}=R\ \text{or}\ L$.

For the convenience of the implementation, we fix the input sequence $y_{1:N}$
and create a new target sequence $Y^{*}$ where tokens are accordingly shifted
with the sampled directions:

$P(Y^{*}|X)=\prod_{i=1}^{N}P(y_{j}|X,Y_{z_{i}}),$ (7)

where $j=i+1$ for $z_{i}=R$, $j=i-1$ for $z_{i}=L$ and $j=i$ for $z_{i}=S$.
When training on large corpus with random sampling on directions, we can say
that $P(Y^{*}|X)\approx P(Y|X)$ theoretically.

Formally, let the source and target sequence as $X=[x_{1},...,x_{|X|}]$ and
$Y=[y_{1},...,y_{N}]$ where $N$ is the target length. Then, we uniformly
sample a direction instruction sequence $Z=[z_{1},...,z_{N}]$ with $N$
elements, where $z_{1}$ and $z_{N}$ are fixed to be $R$ and $L$ as they are
[BOS] and [EOS], which can only be used to predict tokens inside the sequence
for the AR setting, and can never be masked in the NAR setting.

The input sequence $Y_{\text{in}}$ is created by directly copying from ground-
truth $Y$, which will be masked accordingly in the decoder to create the
disentangled context.

According to the sampled direction sequence $Z$, we can now create the
modified target sequence $Y^{*}$ by shifting tokens in $Y$ based on $z_{i}$,
which is shown in Figure 1.

To be compatible with the NAR decoding, we also predict the target length
$P(N|X)$ with the same way as (Ghazvininejad et al., 2019). Note that the
predicted length is only used for NAR decoding, the AR decoding still
terminates when [EOS] or [BOS] is generated for L2R and R2L setting.

Finally, the cross-entropy loss is used for both generation
($\mathcal{L}_{\text{DLM}}$) and length prediction
($\mathcal{L}_{\text{LEN}}$) task, the overall loss can be obtained by adding
them together:

$\mathcal{L}_{\text{Diformer}}=\mathcal{L}_{\text{DLM}}+\lambda\mathcal{L}_{\text{LEN}},$
(8)

where $\lambda$ is the factor on which the best performance can be obtained
with the value of 0.1, after searched from 0.1 to 1.0 in the experiment.

###  2.3 Directional Transformer

Figure 1: An example of training Diformer with DLM, where values in grids are
the relative distance of $K,V$ w.r.t $Q$, attention masks are indicated by
dark grids.

Diformer is mainly built upon the Transformer (Vaswani et al., 2017)
architecture but with several modifications for the compatible of the multi-
directional generation, especially for avoiding the information leakage during
training.

Specifically, we directly use the standard Transformer encoder in the
Diformer, except that an additional MLP layer is added on top of it for length
prediction. For the decoder, several modifications are performed: 1) We
introduce an additional embedding matrix to encode the direction instruction.
2) The original uni-directed positional embedding is expended to a bi-directed
positional embedding. 3) We follow the work in DisCo to disentangle the
context by de-contextualizing $K,V$ only with word embedding, and replacing
the input of $Q$ in the first layer only with direction and position signal.
4) To compensate the removed positional information in $K,V$, we integrate the
relative positional embedding in the self-attention, successfully resolved the
problem on information leakage and the compatible of bi-directional
generation. (see Appendix)

#### Directional Embeddings

An embedding matrix is used to map the categorical variable $z_{i}$ into the
hidden space, denoted as $\delta$,
$\delta(z_{i})\in\mathbb{R}^{d_{\text{model}}}$ where $d_{\text{model}}$ is
hidden size of the model. For simplicity, we directly use $z_{i}$ to represent
the embedded direction at position $i$ in following sections.

The joint training of L2R and R2L can be problematic with the positional
embedding of the original Transformer since the index is counted in a uni-
directed order, which can be used for cheating under the bi-directional
scenario since future positional index can leak information of the sentence
length.

To solve this, we propose to make the positional embedding directed, achieved
by encoding the position index counted oppositely based on the direction with
separate parameters:

$p_{z_{i}}=\begin{cases}\overrightarrow{Pe}(\overrightarrow{i})&\ \ z_{i}=R\ \
\text{or}\ \ z_{i}=S\\\ \overleftarrow{Pe}(\overleftarrow{i})&\ \
z_{i}=L\end{cases}$

where $\overrightarrow{Pe}$ and $\overleftarrow{Pe}$ are different embedding
matrics to encode position indices counting from L2R ($\overrightarrow{i}$) or
R2L ($\overleftarrow{i}$) accordingly. More detailed description can be found
in Figure 1.

Finally, we add encoded position and direction embeddings together as the
initial hidden-state $h_{i}^{l=0}$ for the computation of $Q^{l=0}$ in the
first self-attention layer $h_{i}^{0}=p_{z_{i}}+z_{i}$:

#### Directional Self-Attention

In DisCo, to prevent the information leakage from the disentangled context,
the input representation for computing $K,V$ is de-contextualized by directly
reusing the projection of input embeddings
$k_{i},v_{i}=\text{Proj}(w_{i}+p_{i})$. In Diformer, we have to further remove
the positional information since the directed positional embedding can still
be used for cheating in the computation of self-attention across layers.

Completely removing the the positional information on $K,V$ and only using the
word-embedding $w_{i}$ can be harmful to the performance. Therefore, we
propose an alternative solution by replacing the removed absolute positional
embedding with the relative positional embedding proposed in (Shaw et al.,
2018) for two reasons: 1) The relative position is computed in a 2 dimensional
space, meaning that $p_{ij}$ and $p_{kj}$ for token $y_{j}$ is not shared
between $y_{i}$ and $y_{k}$, which satisfies our requirements that each token
in the context should have the position information only used for $y_{i}$ but
not shared for $y_{k}$. 2) The position information is only injected during
the computation of self-attention without affecting the original word
embedding used in $K,V$.

Formally, we directly use the method in (Shaw et al., 2018) but replace the
hidden representation for computing $K,V$ with word embeddings:

$\displaystyle h_{i}^{l^{\prime}}$
$\displaystyle=\sum_{j=1}^{N}\alpha_{ij}(w_{j}W^{V}+p_{ij}^{V})$ (9)
$\displaystyle\alpha_{ij}$
$\displaystyle=\frac{\exp{e_{ij}}}{\sum_{k=1}^{N}\exp{e_{ik}}}$ (10)
$\displaystyle e_{ij}$
$\displaystyle=\frac{h_{i}^{(l-1)}W^{Q}(w_{j}W^{K}+p_{ij}^{K})^{\top}}{\sqrt{d_{\text{head}}}}$
(11)

where $h_{i}^{l^{\prime}}$ is the output of the self-attention in current
layer, $w_{j}$ is the word embedding, $p_{ij}^{V},p_{ij}^{K}$ are embedded
relative positions, $W^{Q},W^{K},W^{V}$ are parameters for $Q,K$ and $V$,
$h_{i}^{l-1}$ is the last layer’s hidden state, $d_{\text{head}}$ is the
hidden size of a single head. Two parameter matrics are used as embeddings —
$Re^{K}$ and $Re^{V}$, with the shape of $[2k+1,d_{\text{head}}]$, where $k$
is the max length. $p_{ij}$ is obtained by embedding the distance between $i$
and $j$ clipped by the maximum length $k$.

Finally, a customized attention mask (see Figure 1) is created during training
to simulate specific dependencies based on the sampled direction sequence $Z$
with following rules:

  * •

If $z_{i}=R$, all tokens for $j>i$ will be masked.

  * •

If $z_{i}=L$, all tokens for $j<i$ will be masked.

  * •

If $z_{i}=S$, $y_{i}$ and a subset of randomly selected tokens will be masked
following the method in (Ghazvininejad et al., 2020), excluding [BOS] and
[EOS].

#### Inference

Diformer can generate a sequence with 4 modes including L2R and R2L for AR
decoding, mask-predict and parallel easy-first for NAR decoding.

For the AR decoding, the model works exactly same as the conventional
Transformer, except that for each step, a fixed direction $z_{i}=R$ or $L$
should also be also be given, together with previously generated tokens,
making it a pure-linear autoregressive generation. Beam search can be directly
used in both L2R and R2L decoding. For the NAR decoding, the model uses mask-
predict or easy-first by applying specific masking operation during each
iteration, where all tokens are assigned with $z=S$. Length beam can be used
to further improve the performance. Detailed examples are shown in the
Appendix.

More importantly, we find that the multi-directional property of Diformer can
be used for reranking, which is quite beneficial for the NAR decoding.
Specifically, compared to other NAR models that uses an external AR model for
reranking, Diformer can do it all by its own without introducing additional
computational costs. For example, it first refines 5 candidates with 8
iterations and performs reranking with the rest of 2 iterations by re-using
the encoder states and scoring candidates with L2R and R2L modes, which is
equivalent to the computational cost of a 10-stepped refinement reported in
CMLM. The scores computed in two directions are averaged to obtain the final
rank. Experimental results show that 8 steps of refinement + 2 steps of
reranking obtains significant performance improvements compared to 10 steps of
refinement without re-ranking. It can also be used for AR decoding, where all
tokens are scored under the reversed direction, e.g. generating with L2R and
scoring with R2L. We name this method as self-reranking.

##  3 Experiments

| En-De | De-En | En-Ro | Ro-En
---|---|---|---|---
AR | NAR | AR | NAR | AR | NAR | AR | NAR
AR Models
T-big (Vaswani et al., 2017) | 28.4 | - | - | - | - | - | - | -
T-base (Vaswani et al., 2017) | 27.3 | - | - | - | - | - | - | -
T-big (our impl, En$\leftrightarrow$De teacher) | 28.52 | - | 32.10 | - | - | - | - | -
T-base (our impl, En$\leftrightarrow$Ro teacher) | 27.67 | - | 31.12 | - | 35.29 | - | 34.02 | -
T-base + distill | 28.41 | - | 31.69 | - | 35.21 | - | 33.87 | -
NAR models
NAT (Gu et al., 2018) | - | 19.17 | - | 23.20 | - | 29.79 | - | 31.44
iNAT (Lee et al., 2018) | - | 21.61 | - | 25.48 | - | 29.32 | - | 30.19
InsT (Stern et al., 2019) | 27.29 | 27.41 | - | - | - | - | - | -
CMLM (Ghazvininejad et al., 2019) | - | 27.03 | - | 30.53 | - | 33.08 | - | 33.31
LevT (Gu et al., 2019b) | - | 27.27 | - | - | - | - | - | 33.26
DisCO (Kasai et al., 2020) | - | 27.34 | - | 31.31 | - | 33.22 | - | 33.25
Unified models
Mansimov et al. (2020) | 25.66 | 24.53 | 30.58 | 28.63 | - | - | - | -
Tian et al. (2020) | 27.23 | 26.35 | - | - | - | - | - | -
Diformer (ours) |  |  |  |  |  |  |  |
\- L2R | 28.35/28.68 | - | 31.58/31.76 | - | 35.06/35.16 | - | 33.84/33.92 | -
\- R2L | 28.58/28.50 | - |  32.00/31.78 | - |  35.17/35.13 | - | 33.90/33.90 | -
\- mask-predict | - | 27.51/27.99 | - | 31.05/31.35 | - | 33.62/34.37 | - | 32.68/33.11
\- easy-first | - | 27.35/27.84 | - | 31.21/31.68 | - | 33.58/34.23 | - | 32.97/33.34

Table 1: This table shows the overall performance of Diformer compared to the
AR, NAR and unified models when decoding with AR or NAR strategies.
T-big/-base is the abbreviation of Transformer-big/-base. The BLEU score using
self-rerank (right) or not (left) is separated by /.

###  3.1 Experimental Setup

#### Data

We evaluate Diformer on 4 benchmarks including WMT14 En$\leftrightarrow$De
(4.5M sentence pairs) and WMT16 En$\leftrightarrow$Ro (610k sentence pairs).
The data is preprocessed in the same way with (Vaswani et al., 2017; Lee et
al., 2018), where each sentence is tokenized with Moses toolkit (Koehn et al.,
2007) and encoded into subwords using BPE (Sennrich et al., 2016). We follow
Gu et al. (2018); Ghazvininejad et al. (2019); Zhou et al. (2020) to create
the knowledge distilled (KD) data with L2R Transformer-big and Transformer-
base for En$\leftrightarrow$De and En$\leftrightarrow$Ro, the reported
performance in the overall results are all obtained by training on the KD
data.

#### Configuration

We follow the same configurations with previous works (Vaswani et al., 2017;
Ghazvininejad et al., 2019, 2020) on hyperparameters:
$n_{(encoder+decoder)\\_layers}$ = 6 + 6, $n_{heads}$ = 8, $d_{hidden}$ = 512,
$d_{FFN}$ = 2048. For customized components in Diformer, we tune the max
relative distance $k$ in [1,8,16,256] and find that $k=256$ obtains best
performance. Adam (Kingma and Ba, 2015) is used for optimization with 128k
tokens per batch on 8 V100 GPUs. The learning rate warms up for 10k steps to
5e-4 and decays with inversed-sqrt. Models for En$\leftrightarrow$De and
En$\leftrightarrow$Ro are trained for 300k and 100k steps, last 5 checkpoints
are averaged for final evaluation. We set beam size as 4 and 5 for AR and NAR
decoding. When decoding in NAR mode, we set the max iteration for mask-predict
and easy-fist decoding as 10 without using any early-stopping strategy. For
fair comparison, we reduce the max iteration to 8 when decoding with self-
reranking in NAR model. Our model is implemented with pytorch (Paszke et al.,
2019) and fairseq (Ott et al., 2019). BLEU (Papineni et al., 2002) is used for
evaluation.

###  3.2 Results & Analysis

We perform experiments on Diformer to evaluate its performance on three
generation directions with four decoding strategies. We mainly compare
Diformer to three types of models: 1) the unified-models that is able to
decode with multiple strategies, 2) pure AR model, i.e. standard Transformer,
3) pure NAR models. (see Table 1)

#### Comparison with unified models

For the comparison to unified-models (Mansimov et al., 2020; Tian et al.,
2020), Diformer outperforms others in all generation directions, by obtaining
more than 1.5 BLEU.

As discussed in the section 1, their support on multiple generation strategies
is achieved by applying certain position selection strategy on the masked
language model or generating with certain permutation with the permutation
language model. This creates the gap between the training and inference since
a specific decoding strategy might not be fully trained with the generalized
objective as analyzed in (Mansimov et al., 2020). So, compared to both, we use
the task-specific modelling in exchange for better performance by abandon
certain flexibility, thus makes the learned distribution to be same with the
one used in decoding, which answers why Diformer performs better.

#### Comparison with AR models

For the En$\leftrightarrow$De dataset, since we use a larger teacher model
(Transformer-big), therefore, we only compare Diformer with same sized
Transformer-base trained on the raw and distilled data. The Diformer
outperforms Transformer trained on the raw data with a large margin and
reaches the same level to the one trained on distilled data. Interesting, the
best performance of Diformer are usually obtained by the R2L decoding and the
reranked results on L2R, the reason of it will be further discussed in
ablation study sections. For the En$\leftrightarrow$Ro dataset, Diformer can
also obtain similar performance compared to the same sized Transformer trained
on the distilled data produced by a same sized teacher.

#### Comparison with NAR models

Diformer is also competitive to a series of NAR models including iterative-
refinement based and fully NAR models. We speculate the strong performance of
Diformer comes from the joint training of AR and NAR, since it is similar to
the multi-task scenario, where tasks are closely correlated but not same. This
could be beneficial for the task that is more difficult i.e. NAR, because the
learned common knowledge on AR tasks could be directly used in it. By applying
the self-reranking method, Diformer could obtain additional 0.5 BLEU over the
strong baseline.

###  3.3 Ablation Study

In this section, we perform extra experiments to investigate factors that
could influence the performance of Diformer and the mechanism behind it. All
experiments of ablation study are performed on the WMT14 En$\rightarrow$De
dataset.

#### The influence of Knowledge Distillation

Data Condition | R | L | mask-predict | easy-first
---|---|---|---|---
T-base (our impl) |  |  |  |
Raw data | 27.67 | - | - | -
Distilled data | 28.41 | - | - | -
Diformer |  |  |  |
Raw data | 27.21 | 27.08 | 24.12 | 24.18
Raw data (fixed right) | 27.63 | - | - | -
Distilled data | 28.35 | 28.55 | 27.51 | 27.35

Table 2: This table shows the performance of Transformer and Diformer trained
on raw and distilled data where T-base represents for Transformer-base. An
additional experiment with fixed $z_{i}=R$ for all tokens is also presented.

We train Diformer not only with distilled data but also with raw data as shown
in table 2. The degradation of NAR decoding when training on raw data is not
surprising which is a common problem faced by all NAR models. However, the
performance of AR decoding also degrades. We speculate that on the raw data,
the difficulty of learning to generate from straight and R2L increased
significantly, making the model to allocate more capacity to fit them,
resulting in the negative influence on the performance of L2R. We verify this
by fixing $z_{i}=R$ for all tokens and train the model on raw data. The result
confirms it because the performance recovers to its original level. On the
contrary, the knowledge distilled data is cleaner and more monotonous (Zhou et
al., 2020), making it easier to learn for all directions, and allows the model
to allocate balanced capacity on each direction. As for the better performance
obtained by R2L decoding, we consider the reason is that, the R2L is able to
learn the distilled data generated by the L2R teacher in a complementary
manner, making it more efficient to learn the knowledge that cannot be learned
by L2R due to the same modeling method.

#### The Importance of Relative Position

max $k$ | R | L | mask-predict | easy-first
---|---|---|---|---
256 | 28.35 | 28.58 | 27.51 | 27.35
16 | 28.51 | 28.48 | 27.25 | 27.32
8 | 28.13 | 28.25 | 26.58 | 26.71
1 | 26.81 | 26.85 | 18.78 | 19.53

Table 3: This table shows the performance of Diformer with different max $k$.

We also demonstrate the importance of the relative positional embedding by
evaluating the model with different maximum relative distance $k$ and obtain
the same conclusion (Shaw et al., 2018) — the distance should be at least 8.
Meanwhile, we observe that NAR is more sensitive to the positional
information, which is reasonable, since the decoding of NAR is conditioned on
the bi-directional context, where the positional information contains both
distance and direction thereby is more important compared to that in AR.

#### Improvements of Self-Reranking

Figure 2: The heatmap shows the BLEU score decoded with mask-predict when
using self-reranking or not under different max iteration number and length
beam size.

As shown in the overall results, self-reranking is a useful method to improve
the performance especially for NAR decoding. For the AR decoding, the
improvements is not that significant since the outputs are already good enough
for L2R or R2L, the tiny gap between reranking and generation direction cannot
provide enough help, which indicates that using self-reranking for AR is not
that profitable compared to NAR.

We further investigate its ability on NAR decoding (mask-predict) given
different max iteration number and length beam size, as shown in Figure 2. It
clearly shows that without reranking, the incorrect selection on beam
candidates may even reduce the performance with larger beam size. The use of
self-reranking actually lets the performance and beam-size positively
correlated, meaning that exchanging 2 steps of iteration with self-reranking
can be profitable with larger beam size. In practical usage of self-reranking,
it is critical to find the optimal combination by balancing the beam size and
max iteration number so that both high performance and low latency can be
obtained.

##  4 Conclusion

In this paper, we present Directional Transformer which is able to model the
autoregressive and non-autoregressive generation with a unified framework
named Directional Language Model which essentially links two types of
conditional language model with three generation directions. Compared to
previous works, Diformer exchanges the generalization on decoding strategies
for better performance and thereby only support 4 decoding strategies.
Experimental results on WMT14 En$\leftrightarrow$De and WMT16
En$\leftrightarrow$Ro demonstrate that the unification of AR and NAR can be
achieved by Diformer without losing any performance. The bi-directional
property of Diformer allows it to perform self-reranking which is especially
useful for NAR decoding to improve performance with no additional
computational cost.

Except from machine translation, Diformer can be easily extended to other
tasks like language modeling by removing the dependency on $X$. It has the
potential to unify the representation learning and generation with a single
model, which is actually our ongoing work.

## References

  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Ghazvininejad et al. (2020) Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020\.  Aligned cross entropy for non-autoregressive machine translation.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 3515–3523. PMLR.
  * Ghazvininejad et al. (2019) Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019.  Mask-predict: Parallel decoding of conditional masked language models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 6111–6120. Association for Computational Linguistics.
  * Gu et al. (2018) Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018\.  Non-autoregressive neural machine translation.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Gu et al. (2019a) Jiatao Gu, Qi Liu, and Kyunghyun Cho. 2019a.  Insertion-based decoding with automatically inferred generation order.  _Trans. Assoc. Comput. Linguistics_ , 7:661–676.
  * Gu et al. (2019b) Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019b.  Levenshtein transformer.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 11179–11189.
  * Kasai et al. (2020) Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. 2020.  Non-autoregressive machine translation with disentangled context transformer.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 5144–5155. PMLR.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007.  Moses: Open source toolkit for statistical machine translation.  In _ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic_. The Association for Computational Linguistics.
  * Lee et al. (2018) Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018.  Deterministic non-autoregressive neural sequence modeling by iterative refinement.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , pages 1173–1182. Association for Computational Linguistics.
  * Mansimov et al. (2020) Elman Mansimov, Alex Wang, Sean Welleck, and Kyunghyun Cho. 2020.  A generalized framework of sequence generation with application to undirected sequence models.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Demonstrations_ , pages 48–53. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA_ , pages 311–318. ACL.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc.
  * Qi et al. (2021) Weizhen Qi, Yeyun Gong, Jian Jiao, Yu Yan, Weizhu Chen, Dayiheng Liu, Kewen Tang, Houqiang Li, Jiusheng Chen, Ruofei Zhang, Ming Zhou, and Nan Duan. 2021\.  BANG: bridging autoregressive and non-autoregressive generation with large scale pretraining.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8630–8639. PMLR.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics.
  * Shaw et al. (2018) Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018.  Self-attention with relative position representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers)_ , pages 464–468. Association for Computational Linguistics.
  * Shu et al. (2020) Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. 2020.  Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8846–8853. AAAI Press.
  * Stern et al. (2019) Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019.  Insertion transformer: Flexible sequence generation via insertion operations.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 5976–5985. PMLR.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_ , pages 3104–3112.
  * Tian et al. (2020) Chao Tian, Yifei Wang, Hao Cheng, Yijiang Lian, and Zhihua Zhang. 2020.  Train once, and decode as you like.  In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_ , pages 280–293. International Committee on Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Welleck et al. (2019) Sean Welleck, Kianté Brantley, Hal Daumé III, and Kyunghyun Cho. 2019\.  Non-monotonic sequential text generation.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 6716–6726. PMLR.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 5754–5764.
  * Zhou et al. (2020) Chunting Zhou, Jiatao Gu, and Graham Neubig. 2020.  Understanding knowledge distillation in non-autoregressive machine translation.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.

##  Appendix A Appendices

Figure 3: The architecture of the model (left) with the example of
corresponded operations in key components during training (right).

Step | L2R | R2L | Mask-Predict | Easy-First
---|---|---|---|---
1 |  | A
---
[B]
R
| E
---
[E]
L
| [B] - - B - - [E]
---
[B] - - - - - [E]
S S S S S
| [B] A A C C D [E]
---
[B] - - - - - [E]
S S S S S
2 |  | A B
---
[B] A
R R
| D E
---
E [E]
L L
| [B] A B - - - [E]
---
[B] - - B - - [E]
S S S S S
| [B] A B C C E [E]
---
[B] A A C C D [E]
S S S S S
3 |  | A B C
---
[B] A B
R R R
| C D E
---
D E [E]
L L L
| [B] A B - - E [E]
---
[B] A B - - - [E]
S S S S S
| [B] A B C D E [E]
---
[B] A B C C E [E]
S S S S S
4 |  | A B C D
---
[B] A B C
R R R R
| B C D E
---
C D E [E]
L L L L
| [B] A B C - E [E]
---
[B] A B - - E [E]
S S S S S
-
5 |  | A B C D E
---
[B] A B C D
R R R R R
| A B C D E
---
B C D E [E]
L L L L L
| [B] A B C D E [E]
---
[B] A B C - E [E]
S S S S S
-
6 |  | A B C D E [E]
---
[B] A B C D E
R R R R R R
| [B] A B C D E
---
A B C D E [E]
L L L L L L
- | -

Table 4: An example of 4 decoding strategies with the output, context input
and direction input placed in three sub-rows, where [B] and [E] represent for
[BOS] and [EOS] token, - represents for mask.


11institutetext: Korea University 11email:
<EMAIL_ADDRESS>22institutetext:
Hong Kong University of Science and Technology
22email<EMAIL_ADDRESS><EMAIL_ADDRESS>33institutetext: Clova AI
Research, NAVER Corp.

# Coloring with Words: Guiding Image Colorization Through Text-based Palette
Generation

Hyojin Bahng These authors contributed equally.11 Seungjoo Yoo* 11 Wonwoong
Cho* 11 David Keetae Park 1133 Ziming Wu 22 Xiaojuan Ma 22 Jaegul Choo 1133

###### Abstract

This paper proposes a novel approach to generate multiple color palettes that
reflect the semantics of input text and then colorize a given grayscale image
according to the generated color palette. In contrast to existing approaches,
our model can understand rich text, whether it is a single word, a phrase, or
a sentence, and generate multiple possible palettes from it. For this task, we
introduce our manually curated dataset called Palette-and-Text (PAT). Our
proposed model called Text2Colors consists of two conditional generative
adversarial networks: the text-to-palette generation networks and the palette-
based colorization networks. The former captures the semantics of the text
input and produce relevant color palettes. The latter colorizes a grayscale
image using the generated color palette. Our evaluation results show that
people preferred our generated palettes over ground truth palettes and that
our model can effectively reflect the given palette when colorizing an image.

###### Keywords:

Color Palette Generation Image Colorization Conditional Generative Adversarial
Networks.

Figure 1: Colorization results of Text2Colors given text inputs. The text
input is shown above the input grayscale image, and the generated palettes are
on the right of the grayscale image. The color palette is well-reflected in
the colorized image when compared to the ground truth image. Our model is
applicable to a wide variety of images ranging from photos to patterns (top
right). Figure 2: How Text2Colors works. Our model can produce a diverse
selection of palettes when given a text input. Users can optionally choose
which palette to be applied to the final colorization output.

##  1 Introduction

Humans can associate certain words with certain colors. The real question is,
can machines effectively learn the relationship between color and text? Using
text to express colors can allow ample room for creativity, and it would be
useful to visualize the colors of a certain semantic concept. For instance,
since colors can leave a strong impression on people [19], corporations often
decide upon the season‘s color theme from marketing concepts such as
‘passion.’ Through text input, even people without artistic backgrounds can
easily create color palettes that convey high-level concepts. Since our model
uses text to visualize aesthetic concepts, its range of future applications
can encompass text to even speech.

Previous methods have a limited range of applications as they only take a
single word as input and can recommend only a single color or a color palette
in pre-existing datasets [12, 8, 15, 25]. Other studies have further attempted
to link a single word with a multi-color palette [21, 36] since multi-color
palettes are highly expressive in conveying semantics [18]. Compared to these
previous studies, our model can generate multiple plausible color palettes
when given rich text input, including both single- and multi-word
descriptions, greatly increasing the boundary of creative expression through
words.

In this paper, we propose a novel method to generate multiple color palettes
that convey the semantics of rich text and then colorize a given grayscale
image according to the generated color palette. Perception of color is
inherently multimodal [4], meaning that a particular text input can be mapped
to multiple possible color palettes. To incorporate such multimodality into
our model, our palette generation networks are designed to generate multiple
palettes from a single text input. We further apply our generated color
palette to the colorization task. Motivated from previous user-guided
colorizations that utilize color hints given by users [42, 45], we design our
colorization networks to utilize color palettes during the colorization
process. Our evaluation demonstrates that the colorized outputs do not only
reflect the colors in the palette but also convey the semantics of the text
input.

The contribution of this paper includes:

(1) We propose a novel deep neural network architecture that can generate
multiple color palettes based on natural-language text input.

(2) Our model is able to use the generated palette to produce plausible
colorizations of a grayscale image.

(3) We introduce our manually curated dataset called Palette-and-Text (PAT),
which includes 10,183 pairs of a multi-word text and a multi-color palette.
111Dataset and codes are publicly available at https://github.com/awesome-
davian/Text2Colors/

##  2 Related Work

####  2.0.1 Color Semantics

Meanings associated with a color are both innate and learned [9]. For
instance, red can make us instinctively feel alert [9]. Since color has a
strong association with high-level semantic concepts [10], producing palettes
from text input is useful in aiding artists and designers [18] and allows
automatic colorization from palettes [42, 5]. A downside to using text to
choose a filter is that filter names do not usually convey the filter’s colors
[21], thus making it difficult for users to find the filter that matches their
taste just by looking at filter names. To bridge this discrepancy between
color palettes and their names, palette recommendation based on user text
input has long been studied. Query-based methods [21, 36] use text inputs to
query an image from an image dictionary where colors are extracted from the
queried image to make an associated palette. This method is problematic in
that the text input is mapped to the image content of the queried image rather
than the color that the text implies. Instead of looking for a target
directly, learning-based approaches [14, 27, 23] match or generate color
palettes to their linguistic descriptions by learning their semantic
association from large-scale data. However, our model is the only generative
model that supports phrase-level text input.

####  2.0.2 Conditional GANs

Conditional generative adversarial networks (cGAN) are GAN models that use
conditional information for the discriminator and the generator [24]. cGANs
have drawn promising results for image generation from text [32, 31, 43] and
image-to-image translation [16, 13, 7]. StackGAN [43] is the first model to
use conditional loss for text to image synthesis. Our model is the first to
utilize the conditioning augmentation technique from StackGAN to output
diverse palettes even when given the same input text.

####  2.0.3 Interactive Colorization

Colorization is a multimodal task and desired colorization results for the
same object may vary from person to person [4]. A number of studies introduce
interactive methods that allow users to control the final colorization output
[45, 20]. In these models, users directly interact with the model by
pinpointing where to color. Even though these methods achieve satisfactory
results, a limitation is that users need to have a certain level of artistic
skill. Thus instead of making the user directly color an image, other studies
take a more indirect approach by utilizing color palettes to recolor an image
[3, 5]. Palette-based filters of our model are an effective way for non-
experts to recolor an image [3].

####  2.0.4 Sequence-to-Sequence with Attention

Recurrent Neural Networks (RNNs) are a popular tool due to their superior
ability to learn from sequential data. RNNs are used in various tasks
including sentence classification [39], text generation [37], and sequence-to-
sequence prediction [38]. Incorporating attention into a sequence-to-sequence
model is known to improve the model performance [22] as networks learn to
selectively focus on parts of a source sentence. This allows a model to learn
relations between different modalities as is done by our model (e.g., text -
colors, text - action [1], and English - French [40]).

##  3 Palette-and-Text (PAT) Dataset

This section introduces our manually curated dataset named Palette-and-Text
(PAT). PAT contains 10,183 text and five-color palette pairs, where the set of
five colors in a palette is associated with its corresponding text description
as shown in Figs. 3(b)-(d). Words vary with respect to their relationships
with colors; some words are direct color words (e.g., pink, blue, etc.) while
others evoke a particular set of colors (e.g., autumn or vibrant). To the best
of our knowledge, there has been no dataset that matches a multi-word text and
its corresponding 5-color palette. This dataset allows us to train our models
for predicting semantically consistent color palettes with textual inputs.

####  3.0.1 Other Color Datasets

Munroe‘s color survey [26] is a widely used large-scale color corpus. Based on
crowd-sourced user judgment, it matches a text to a single color. Another
dataset, Kobayashi‘s Color Image Scale [18], is a well-established multi-color
dataset. Kobayashi only uses 180 adjectives to express 1170 three-color
palettes, which greatly limits its range of expression. In contrast, our
dataset is made up of 4,312 unique words. This includes much more text that
was not traditionally used to express colors. Our task requires a more
sophisticated dataset like PAT, that matches a text to multiple colors and is
large enough for a deep learning model to learn from.

Figure 3: Our Palette-and-Text (PAT) dataset. On the left are diverse text-
palette pairs included in PAT. PAT has a very wide range of expression,
especially when compared to existing datasets. Our dataset is designed to
address rich text and multimodality, where the same word can be mapped to a
wide range of possible colors.

####  3.0.2 Data Collection

We generated our PAT dataset by refining user-named palette data crawled from
a community website called color-hex.com. Thousands of users upload custom-
made color palettes on color-hex, and thus our dataset was able to incorporate
a wide pool of opinions. We crawled 47,665 palette-text pairs and removed non-
alphanumerical and non-English words. Among them, we found that users
sometimes assign palette names in an arbitrary manner, missing their semantic
consistency with their corresponding color palettes. Some names are a
collection of random words (e.g., ‘mehmeh’ and ‘i spilled tea all over my
laptop rip’), or are riddled with typos (e.g., ‘cause iiiiii see right through
you boyyyyy’ and ‘greene gardn’). Thus, using unrefined raw palette names
would hinder model performances significantly.

To refine the noisy raw data, four annotators voted whether the text paired
with the color palette properly matches its semantic meanings. We then used
only the text-palette pairs in which at least three annotators out of four
agreed that semantic matching exists between the text and color palette.
Including text-palette pairs in the dataset only when all four annotators
agree was found to be unnecessarily strict, leaving not much room for personal
subjectivity. Annotator’s perception is inherently subjective, meaning that a
text-palette pair perfectly plausible to one person may not be agreeable to
another. We wanted to incorporate such subjectivity by allowing a diverse
selection of text-palette pairs. Mis-spelling and punctuation errors were
manually corrected after the annotators finished sorting out the data.

Figure 4: Overview of our Text2Colors architecture. During training, generator
$G_{0}$ learns to produce a color palette $\hat{y}$ given a set of conditional
variables $\hat{c}$ processed from input text $x=\\{x_{1},\cdots,x_{T}\\}$.
Generator $G_{1}$ learns to predict a colorized output of a grayscale image
$L$ given a palette $p$ extracted from the ground truth image. At test time,
the trained generators $G_{0}$ and $G_{1}$ are used to produce a color palette
from given text and then colorize a grayscale image reflecting the generated
palette.

##  4 Text2Colors: Text-Driven Colorization

Text2Colors consists of two networks: Text-to-Palette Generation Networks
(TPN) and Palette-based Colorization Networks (PCN). We train the first
networks to generate color palettes given a multi-word text and then train the
second networks to predict reasonable colorizations given a grayscale image
and the generated palettes. We utilize conditional GANs (cGAN) for both
networks.

###  4.1 Text-to-Palette Generation Networks (TPN)

####  4.1.1 Objective Function

In this section, we illustrate the Text-to-Palette Generation Networks shown
in Figs. 4 and 5. TPN produces reasonable color palettes associated with the
text input. Let $x_{i}\in\mathbb{R}^{300}$ be word vectors initialized by
300-dimensional pre-trained vectors from GloVe [29]. Words not included in the
pre-trained set are initialized randomly. Using the CIE _Lab_ space for our
task, $y\in\mathbb{R}^{15}$ represents a 15-dimensional color palette
consisting of five colors with _Lab_ values. After a GRU encoder encodes $x$
into hidden states $h=\\{h_{1},\cdots,h_{T}\\}$, we add random noise to the
encoded representation of text by sampling latent variables $\hat{c}$ from a
Gaussian distribution $\mathcal{N}(\mu(h),\Sigma(h))$. The sequence of
conditioning vectors $\hat{c}=\\{\hat{c}_{1},\cdots,\hat{c}_{T}\\}$ is given
as condition for the generator to output a palette $\hat{y}$, while its mean
vector $\bar{c}=\frac{1}{T}{\sum_{i=1}^{T}\hat{c}}$ is given as the condition
for the discriminator. Our objective function of the first cGAN can be
expressed as

$L_{D_{0}}=\mathbb{E_{\mathit{y\sim P_{data}}}}[\log
D_{0}(\bar{c},y)]+\mathbb{E_{\mathrm{\mathit{x\sim
P_{data}}}}\mathrm{[\log(1-\mathit{D_{0}}(\mathit{\bar{c},\hat{y}}))],}}$ (1)
$\displaystyle L_{G_{0}}=\mathbb{E_{\mathit{x\sim
P_{data}}}}[\log(1-\mathit{D}_{0}(\mathit{\bar{c},\hat{y}}))],$ (2)

where discriminator ${D}_{0}$ tries to maximize $L_{D_{0}}$ against generator
$G_{0}$ that tries to minimize $L_{G_{0}}$. The pre-trained word vectors $x$
and the real color palette $y$ is sampled from true data distribution
$P_{data}$.

Previous approaches have benefited from mixing the GAN objective with $L_{2}$
distance [28] or $L_{1}$ distance [13]. We have explored previous loss options
and found the Huber (or smooth $L_{1}$) loss to be the most effective in
increasing diversity among colors in generated palettes. The Huber loss is
given by

$\displaystyle
L_{H}(\hat{y},y)=\begin{cases}\frac{1}{2}(\hat{y}-y)^{2}\;\;\mathrm{for}\;\;\left|\hat{y}-y\right|\leq\delta\\\
\delta\left|\hat{y}-y\right|-\frac{1}{2}\delta^{2}\;\;\mathrm{otherwise}.\end{cases}$
(3)

This loss term is added to the generator’s objective function to force the
generated palette to be close to the ground truth palette. We also adopted the
Kullback-Leibler (KL) divergence regularization term [43], i.e.,

$D_{KL}(\mathcal{N}(\mu(h),\Sigma(h))\parallel\mathcal{N}(0,I)),$ (4)

which is added to the generator’s objective function to further enforce the
smoothness over the conditioning manifold. Our final objective function is

$L_{D_{0}}=\mathbb{E_{\mathit{y\sim P_{data}}}}[\log
D_{0}(\bar{c},y)]+\mathbb{E_{\mathrm{\mathit{x\sim
P_{data}}}}\mathrm{[\log(1-\mathit{D_{0}}(\mathit{\bar{c},\hat{y}}))],}}$ (5)
$\displaystyle L_{G_{0}}=\mathbb{E_{\mathit{x\sim
P_{data}}}}[\log(1-\mathit{D}_{0}(\mathit{\bar{c},\hat{y}}))]+\lambda_{H}L_{H}(\hat{y},y)$
(6)
$\displaystyle+\lambda_{KL}D_{KL}(\mathcal{N}(\mu(h),\Sigma(h))\parallel\mathcal{N}(0,I)),$

$\lambda_{H}$ and $\lambda_{KL}$ are the hyperparameters to balance the three
terms in Eq. 6. We set $\delta=1,\lambda_{H}=100,\lambda_{KL}=0.5$ in our
model.

Figure 5: Model architecture of a generator $G_{0}$ that produces the $t$-th
color in the palette given a sequence of conditioning variables
$\hat{c}=\\{\hat{c}_{1},\cdots,\hat{c}_{T}\\}$ processed from an input text
$x=\\{x_{1},\cdots,x_{T}\\}$. Note that randomness is added to the encoded
representation of text before it is passed to the generator.

####  4.1.2 Networks Architecture

##### Encoding Text through Conditioning Augmentation.

Learning a mapping from text to color is inherently multimodal. For instance,
a text ‘autumn’ can be mapped to a variety of plausible color palettes. As
text becomes longer, such as ‘midsummer to autumn’ or ‘autumn breeze and
falling leaves’, the scope of possible matching palettes becomes more broad
and diverse. To appropriately model the multimodality of our problem, we
utilize the conditioning augmentation (CA) [43] technique. Rather than using
the fixed sequence of encoded text as input to our generator, we randomly
sample latent vector $\hat{c}$ from a Gaussian distribution
$\mathcal{N}(\mu(h),\Sigma(h))$ as shown in Fig. 5. This randomness allows our
model to generate multiple plausible palettes given same text input.

To obtain the conditioning variable
$\hat{c}=\\{\hat{c}_{1},\cdots,\hat{c}_{T}\\}$, the pre-trained word vectors
$x=\\{x_{1},\cdots,x_{T}\\}$ are first fed into a GRU encoder to compute
hidden states $h=\\{h_{1},\cdots,h_{T}\\}$. This text representation is fed
into a fully-connected layer to generate $\mu$ and $\sigma$ (the values in the
diagonal of $\Sigma$) for the Gaussian distribution
$\mathcal{N}(\mu(h),\Sigma(h))$. Conditioning variable $\hat{c}$ is computed
by $\hat{c}=\mu+\sigma\odot\epsilon$, where $\odot$ is the element-wise
multiplication and $\epsilon\sim\mathcal{N}(0,I)$. The resulting set of
vectors $\hat{c}=\\{\hat{c_{1}},\cdots,\hat{c_{T}}\\}$ will be used as
_condition_ for our generator.

##### Generator.

We design our generator $G_{0}$ as a variant of a GRU decoder with attention
mechanism [22, 2, 6]. The $i$-th color of the palette $\hat{y}_{i}$ is
computed as

$\hat{y}_{i}=f(s_{i})\;\;\mathrm{where}\;\;s_{i}=g(\hat{y}_{i-1},c_{i},s_{i-1}).$
(7)

$s_{i}$ is a GRU hidden state vector for time $i$, having the previously
generated color $\hat{y}_{i-1}$, the context vector $c_{i}$, and the previous
hidden state $s_{i-1}$ as input. The GRU hidden state $s_{i}$ is given as
input to a fully-connected layer $f$ to output the $i$-th color of the palette
$\hat{y}_{i}\in\mathbb{R}^{3}$. The resulting five colors are combined to
produce a single palette output $\hat{y}$.

The context vector $c_{i}$ depends on a sequence of conditioning vectors
$\hat{c}=\\{\hat{c_{1}},\cdots,\hat{c_{T}}\\}$ and the previous hidden state
$s_{i-1}$. The context vector $c_{i}$ is computed as the weighted sum of these
conditions $\hat{c_{i}}$’s, i.e.,

$c_{i}=\sum_{j=1}^{T}\alpha_{ij}\hat{c}_{j}.$ (8)

The weight $\alpha_{ij}$ of each conditional variable $\hat{c}_{j}$ is
computed by

$\alpha_{ij}=\frac{\exp(e_{ij})}{\sum_{k=1}^{T}\exp(e_{ik})}\;\;\mathrm{where}\;\;e_{ij}=a\left(s_{i-1},\hat{c}_{j}\right).$
(9)
$a\left(s_{i-1},\hat{c}_{j}\right)=w^{T}\sigma(W_{s}s_{i-1}+W_{\hat{c}}\hat{c}_{j}),$
(10)

where $\sigma(\cdot)$ is a sigmoid activation function and $w$ is a weight
vector. The additive attention [2] $a\left(s_{i-1},\hat{c}_{j}\right)$
computes how well the $j$-th word of the text input matches the $i$-th color
of the palette output. The score $\alpha_{ij}$ is computed based on the GRU
hidden state $s_{i-1}$ and the $j$-th condition $\hat{c}_{j}$. The attention
mechanism enables the model to effectively map complex text input to the
palette output.

##### Discriminator.

For the discriminator $D_{0}$, the conditioning variable $\bar{c}$ and the
color palette are concatenated and fed into a series of fully-connected
layers. By jointly learning features across the encoded text and palette, the
discriminator classifies whether the palettes are real or fake.

###  4.2 Palette-based Colorization Networks (PCN)

####  4.2.1 Objective Function

The goal of the second networks is to automatically produce colorizations of a
grayscale image guided by the color palette as a conditioning variable. The
inputs are a grayscale image $L\in\mathbb{\mathbb{\mathbb{R^{\mathit{H\times
W\times 1}}}}}$ representing the lightness in CIE _Lab_ space and a color
palette $p\in\mathbb{R^{\mathrm{15}}}$ consisting of five colors in _Lab_
values. The output $\hat{I}\in\mathbb{R^{\mathit{H\times W\times 2}}}$
corresponds to the predicted _ab_ color channels of the image. The objective
function of the second model can be expressed as

$L_{D_{1}}=\mathbb{E_{\mathit{I\sim P_{data}}}}[\log
D_{1}(p,I)]+\mathbb{E_{\mathrm{\mathit{\hat{I}\sim
P_{G_{1}}}}}\mathrm{[\log(1-\mathit{D}_{1}(\mathit{p,\hat{I}}))],}}$ (11)
$L_{G_{1}}=\mathbb{E_{\mathit{\hat{I}\sim
P_{G_{1}}}}}[\log(1-\mathit{D}_{1}(\mathit{p,\hat{I}}))]+\lambda_{H}L_{H}(\hat{I},I).$
(12)

$D_{1}$ and $G_{1}$ included in the equation are shown in Fig.4. We have also
added the Huber loss to the generator’s objective function. In other words,
the generator learns to be close to the ground truth image with _plausible_
colorizations, while incorporating palette colors to the output image to fool
the discriminator. We set $\lambda_{H}=10$ in our model.

####  4.2.2 Networks Architecture

##### Generator.

The generator consists of two sub-networks: the main colorization networks and
the conditioning networks. Our main colorization networks adopts the U-Net
architecture [33], which has shown promising results in colorization tasks
[13, 45]. The skip connections help recover spatial information [33], as the
input and the output images share the location of prominent edges [13].

The role of the conditioning networks is to apply the palette colors to the
generated image. During training, the networks are given a palette
$p\in\mathbb{R}^{15}$ extracted from the ground truth image $I$. We utilize
the Color Thief 222http://lokeshdhakar.com/projects/color-thief/ function to
extract a palette consisting of five dominant colors of the ground truth
image. Similar to the previous work [45], the conditioning palette $p$ is fed
into a series of $1\times 1$ conv-relu layers as shown in Fig. 4. The feature
maps in layers 1, 2, and 4 are duplicated spatially to match the spatial
dimension of the $conv9$, $conv8$, and $conv4$ features in the main
colorization networks and added in an element-wise manner. The palette $p$ is
fed into upsampling layers with skip connections as well as the middle of the
main networks. This allows the generator to detect prominent edges and apply
palette colors to suitable locations of the image. During test time, we use
the generated palette $\hat{y}$ from the first networks (TPN) as the
conditioning variable, colorizing the grayscale image with the predicted
palette colors.

##### Discriminator.

As our discriminator $D_{1}$, we use a variant of the DCGAN architecture [30].
The image and conditioning variable $p$ are concatenated and fed into a series
of conv-leaky relu layers to jointly learn features across the image and the
palette. Afterwards, it is fed into a fully-connected layer to classify
whether the image is real or fake.

###  4.3 Implementation Details

We first train $D_{0}$ and $G_{0}$ of TPN for 500 epochs using the PAT
dataset. We then train $D_{1}$ and $G_{1}$ of the PCN for 100 epochs, using
the extracted palette from a ground truth image. Finally, we use the trained
generators $G_{0}$ and $G_{1}$ during test time to colorize a grayscale image
with generated palette $\hat{y}$ from a text input $x$. All networks are
trained using Adam optimizer [17] with a learning rate of 0.0002. Weights were
initialized from a Gaussian distribution with zero mean and standard deviation
of 0.05. We set other hyper parameters as $\delta=1,\lambda_{H}=100$, and
$\lambda_{KL}=0.5$.

##  5 Experimental Results

This section presents both quantitative and qualitative analyses of our
proposed model. We evaluate the TPN (Section 4.1) based on our PAT dataset.
For the training of the PCN (Section 4.2), we use two different datasets,
CUB-200-2011 (CUB) [41] and ImageNet ILSVRC Object Detection (ImageNet
dataset) [34].

Figure 6: Comparison to baselines and qualitative analysis on multimodality:
Our TPN generates appealing color palettes that reflect all details of the
text input. Also our model can generate multiple palettes with the same text
input(three rows from bottom). In comparison, Heer and Stone [12]‘s model
frequently generates unrelated colors and has deterministic outputs.

###  5.1 Analysis on Multimodality and Diversity of Generated Palettes

This section discusses the evaluation on multimodality and diversity of our
generated palettes. Multimodality refers to how many different color palettes
a single text input can be mapped to. In other words, if a single text can be
expressed with more color palettes, the more multimodal it is. As shown in
Fig. 6, our model is multimodal, while previous approaches are deterministic,
meaning that it generates only a particular color palette when given a text
input. Diversity within a palette refers to how diverse the colors included in
a single palette are. Following the current standard for perceptual color
distance measurement, we use the CIEDE2000 [35] on CIE _Lab_ space to compute
a model’s multimodality and diversity. To measure multimodality, we compute
the average minimum distances between colors from different palettes. To
measure diversity of a color palette, we measure the average pairwise distance
between the five colors within a palette. All measurements are computed based
on the test dataset.

##### Results.

Table 1 shows the multimodality and diversity measurement among the variants
of our model. The CA module (Section 4.1.2) enables our networks to suggest
multiple color palettes when given the same text input. The model variant
without CA (the first row in Table 1) results in zero multimodality,
indicating that the networks generate identical palettes for the same text
input. Another palette generation model by Heer and Stone [12] also has zero
multimodality. This shows that TPN is the only existing model that can
adequately express multimodality, which is crucial in the domain of colors.
Although Heer and Stone’s model has higher diversity than TPN, Fig. 6 shows
that their palettes contain irrelevant colors that may increase diversity but
decrease palette quality. On the other hand, TPN creates those palettes
containing colors that well match each other. Results on the fooling rate will
be further illustrated in Section 5.3.

###  5.2 Analysis on Attention Outputs

The attention module (Section 4.1.2) plays a role of attending to particular
words in text input to predict the most suitable colors for the text input.
Fig. 7 illustrates how the predicted colors are influenced by attention
scores. The green-colored boxes show attention scores computed for each word
token when predicting each corresponding color in the palette. Higher scores
are indicated by dashed-line boxes. We observe that three colors generated by
attending to ghoul are all dark and gloomy, while the other two colors
attending to fun are bright. This attention mechanism enables our model to
thoroughly reflect the semantics included in text inputs of varying lengths.

Figure 7: Attention analysis. Attention scores measured by the TPN for two
text input samples. Each box color (in green) denotes the attention score
computed in producing the corresponding color shown on top. The dashed-line
boxes indicate the word that each color output attended to. Figure 8:
Qualitative analysis on semantic context. Our model reflects subtle nuance
differences in the semantic context of a given text input in the color palette
outputs. Except for the first column, all the text combinations shown here are
unseen data. Table 1:  Quantitative analysis results

|  |  | Palette Evaluation |  | User Study: Part I
---|---|---|---|---|---
Model Variations |  | Diversity |  | Multimodality |  | Fooling Rate (%)
Objective Function | CA |  | Mean | Std |  | Mean | Std |  | Mean | Std |  | Max | Min
Ours (TPN) | X |  | $19.36$ | $8.74$ |  | $0.0$ | $0.0$ |  | - | - |  | - | -
Ours (TPN) | O |  | $20.82$ | $7.43$ |  | $5.43$ | $8.11$ |  | 56.2 | $12.7$ |  | 76.7 | $37.1$
Heer and Stone | - |  | $35.92$ | $12.66$ |  | $0.0$ | $0.0$ |  | $39.6$ | $10.8$ |  | $58.2$ | $25.8$
Ground truth palette | - |  | $32.60$ | $21.84$ |  | - | - |  | - | - |  | - | -

Figure 9: Colorization performance comparisons. Mean and standard deviation
values for each question are reported for the baseline [45] and our PCN. Our
PCN scores higher on all of the questions, showing that users are more
satisfied with PCN.

###  5.3 User Study

We conduct a user study to reflect universal user opinions on the outputs of
our model. Our user study is composed of two parts. The first part measures
how the generated palettes match the text inputs. The second part is a survey
that compares the performance of our palette-based colorization model to
another state-of-the-art colorization model. 53 participants took part in our
study.

####  5.3.1 Part I: Matching between Text and Generated Palettes

Our goal is to generate a palette with a strong semantic connection with the
given text input. A natural way to evaluate it is to quantify the degree of
connection between the text input and the generated palette, in comparison to
the same text input and its ground truth palette. Given a text input, its
generated palette, and the ground truth palette, we ask human observers to
select the palette that best suits the text input. A fooling rate (FR) in this
study indicates the relative number of generated palettes chosen over ground
truth palettes. More people choosing the generated palette results in a higher
FR. This measure has often been used to assess the quality of colorization
results [45, 11]. We will use this metric to measure how much a text input
matches its generated palette.

##### Study Procedure.

Users participate in the user study over TPN and Heer and Stone’s model [12].
Each consists of 30 evaluations. We randomly choose a single data item out of
992 test data and show the text input along with the generated palette and the
ground truth palette.

##### Results.

In Table 1, we measure the FR score for each person and compute the mean and
the standard deviation (std) of all of the scores from participants. Max and
min scores represent the highest and the lowest FR scores, respectively,
recorded by a single person. While Heer and Stone’s model [12] shows low FR of
39.6%, our TPN has the FR of 56.2% while maintaining a high level of diversity
and multimodality. The FR of 56.2% indicates that the generated palettes are
indistinguishable to human eyes and sometimes even match the input text better
than the ground truth palettes. Note that the standard deviation of 12.7%
implies diverse responses to the same data pairs.

Figure 10: We compare colorization results with previous work [45]. The five-
color palette used for colorization is shown next to the input grayscale
image. Note that our PCN performs better at applying various colors included
in the palette.

####  5.3.2 Part II: Colorization Comparisons

In this part of the user study, we conduct a survey on the performance of the
PCN given palette inputs. Users are asked to answer five questions based on
the given grayscale image, the color palette, and the colored image. For
quantitative comparison, we set a state-of-the-art colorization model [45] as
our baseline. This model originally contains local and global hint networks.
In our implementation of the baseline model, we utilize the global hint
networks to infuse our generated palette to the main colorization networks.
Note that we modified the baseline model to fit our task. Our novelty is the
ability to produce high-quality colorization with only five colors of a
palette while our baseline [45] needs 313 bins of _ab_ gamut. Our model is
able to colorize with limited information due to novel components such as the
conditional adversarial loss and feeding the palette into skip-connection
layers.

##### Study Procedure.

We show colorization results of our PCN and the baseline model one-by-one in a
random order. Then, we ask each participant to answer five different questions
(shown in Fig. 9) based on a five-point Likert scale. The focus of our
questions is to evaluate how well the palette was used in colorizing the given
grayscale image. The total number of data samples per test is 15.

##### Results.

The resulting statistics are reported in Fig. 9. Our PCN achieves higher
scores than the baseline model across all the questions. We can infer that the
palettes generated by our model are preferred over palettes created by a human
hand. Since our model learns consistent patterns from a large number of human-
generated palette-text pairs, our model may have generated color palettes that
more users could relate to.

##  6 Conclusions

We proposed a generative model that can produce multiple palettes from rich
text input and colorize grayscale images using the generated palettes.
Evaluation results confirm that our TPN can generate plausible color palettes
from text input and can incorporate the multimodal nature of colors.
Qualitative results on our PCN also show that the diverse colors in a palette
are effectively reflected in the colorization results. Future work includes
extending our model to a broader range of tasks requiring color recommendation
and conducting the detailed analysis of our dataset.

Acknowledgement. This work was partially supported by the National Research
Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No.
NRF2016R1C1B2015924). Jaegul Choo is the corresponding author.

## References

  * [1] Ahn, H., Ha, T., Choi, Y., Yoo, H., Oh, S.: Text2Action: Generative adversarial synthesis from language to action. In: Proc. the IEEE International Conference on Robotics and Automation (ICRA) (2018)
  * [2] Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: Proc. the International Conference on Learning Representations (ICLR) (2014)
  * [3] Chang, H., Fried, O., Liu, Y., DiVerdi, S., Finkelstein, A.: Palette-based photo recoloring. ACM Transactions on Graphics (TOG) 34(4) (2015)
  * [4] Charpiat, G., Hofmann, M., Schölkopf, B.: Automatic image colorization via multimodal predictions. In: Proc. the European Conference on Computer Vision (ECCV) (2008)
  * [5] Cho, J., Yun, S., Lee, K., Choi, J.Y.: PaletteNet: Image recolorization with given color palette. In: Proc. the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2017)
  * [6] Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014)
  * [7] Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In: Proc. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
  * [8] Chuang, J., Stone, M., Hanrahan, P.: A probabilistic model of the categorical association between colors. In: Proc. the IS&T Color and Imaging Conference (CIC). vol. 2008 (2008)
  * [9] Crozier, W.: The psychology of colour preferences. Coloration Technology 26(1) (1996)
  * [10] De Bortoli, M., Maroto, J.: Colours across cultures: Translating colours in interactive marketing communications. In: Proc. the European Languages and the Implementation of Communication and Information Technologies (ELICIT) (2001)
  * [11] Guadarrama, S., Dahl, R., Bieber, D., Norouzi, M., Shlens, J., Murphy, K.: Pixcolor: Pixel recursive colorization. In: Proc. the British Machine Vision Conference (BMVC) (2017)
  * [12] Heer, J., Stone, M.: Color naming models for color selection, image editing and palette design. In: Proc. the SIGCHI Conference on Human Factors in Computing Systems (SIGCHI) (2012)
  * [13] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proc. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
  * [14] Jahanian, A., Keshvari, S., Vishwanathan, S., Allebach, J.P.: Colors–messengers of concepts: Visual design mining for learning color semantics. ACM Transactions on Computer-Human Interaction (TOCHI) 24(1) (2017)
  * [15] Kawakami, K., Dyer, C., Routledge, B.R., Smith, N.A.: Character sequence models for colorful words. In: Proc. the Conference on Empirical Methods in Natural Language Processing (EMNLP) (2016)
  * [16] Kim, T., Cha, M., Kim, H., Lee, J., Kim, J.: Learning to discover cross-domain relations with generative adversarial networks. In: Proc. the International Conference on Machine Learning (ICML) (2017)
  * [17] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Proc. the International Conference on Learning Representations (ICLR) (2014)
  * [18] Kobayashi, S.: Color image scale. http://www.ncd-ri.co.jp/english/main_0104.html (2009)
  * [19] Labrecque, L.I., Milne, G.R.: Exciting red and competent blue: the importance of color in marketing. Journal of the Academy of Marketing Science 40(5) (2012)
  * [20] Li, X., Zhao, H., Nie, G., Huang, H.: Image recoloring using geodesic distance based color harmonization. Computational Visual Media 1(2) (2015)
  * [21] Liu, Y., Cohen, M., Uyttendaele, M., Rusinkiewicz, S.: Autostyle: Automatic style transfer from image collections to users’ images. Computer Graphics Forum (CGF) 33(4) (2014)
  * [22] Luong, M.T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. In: Proc. the Conference on Empirical Methods in Natural Language Processing (EMNLP) (2015)
  * [23] McMahan, B., Stone, M.: A bayesian model of grounded color semantics. Transactions of the Association of Computational Linguistics (TACL) 3(1) (2015)
  * [24] Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  * [25] Monroe, W., Hawkins, R.X., Goodman, N.D., Potts, C.: Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association of Computational Linguistics (ACL) (2017)
  * [26] Munroe, R.: Color survey results. Online at http://blog.xkcd.com/2010/05/03/color-surveyresults (2010)
  * [27] Murray, N., Skaff, S., Marchesotti, L., Perronnin, F.: Toward automatic and flexible concept transfer. Computers & Graphics 36(6) (2012)
  * [28] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proc. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [29] Pennington, J., Socher, R., Manning, C.: Glove: Global vectors for word representation. In: Proc. the Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014)
  * [30] Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: Proc. the International Conference on Learning Representations (ICLR) (2015)
  * [31] Reed, S., Akata, Z., Lee, H., Schiele, B.: Learning deep representations of fine-grained visual descriptions. In: Proc. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [32] Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H.: Generative adversarial text to image synthesis. In: Proc. the International Conference on Machine Learning (ICML) (2016)
  * [33] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Proc. the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (2015)
  * [34] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV) 115(3) (2015)
  * [35] Sharma, G., Wu, W., Dalal, E.N.: The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations. Color Research & Application 30(1) (2005)
  * [36] Solli, M., Lenz, R.: Color semantics for image indexing. In: Proc. the Conference on Colour in Graphics Imaging and Vision (CGIV) (2010)
  * [37] Sutskever, I., Martens, J., Hinton, G.E.: Generating text with recurrent neural networks. In: Proc. the International Conference on Machine Learning (ICML) (2011)
  * [38] Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems (NIPS) (2014)
  * [39] Tang, D., Qin, B., Liu, T.: Document modeling with gated recurrent neural network for sentiment classification. In: Proc. the Conference on Empirical Methods in Natural Language Processing (EMNLP) (2015)
  * [40] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems (NIPS) (2017)
  * [41] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The Caltech-UCSD Birds-200-2011 Dataset. Tech. Rep. CNS-TR-2011-001, California Institute of Technology (2011)
  * [42] Xiao, Y., Zhou, P., Zheng, Y.: Interactive deep colorization with simultaneous global and local inputs. arXiv preprint arXiv:1801.09083 (2018)
  * [43] Zhang, H., Xu, T., Li, H., Zhang, S., Huang, X., Wang, X., Metaxas, D.: StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In: Proc. the IEEE International Conference on Computer Vision (ICCV) (2017)
  * [44] Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Proc. the European Conference on Computer Vision (ECCV) (2016)
  * [45] Zhang, R., Zhu, J.Y., Isola, P., Geng, X., Lin, A.S., Yu, T., Efros, A.A.: Real-time user-guided image colorization with learned deep priors. ACM Transactions on Graphics (TOG) (2017)

## Supplementary Materials

##  7 User Study Samples

Our user study consists of two parts, one for evaluation of Text-to-palette
Generation Networks (TPN) and the other for evaluation of Palette-based
Colorization Networks (PCN). Fig. 11(a)-(b) illustrates how our data tuples
were shown to the participants in Part I and Part II, respectively.

Figure 11: UI design of our user study.

##  8 Text-to-Palette Generation Networks (TPN)

###  8.1 Model Comparisons for Learning Global Color Distributions

Fig. 12 shows comparisons of color distributions between ground truth palettes
of the training data and generated palettes from our test data. For each color
distribution, we quantize the _ab_ values of every palette color into 313
color bins [44] and visualize the probability distribution of _ab_ values. We
compare three model variants of different objective functions: cGAN+Huber
($\lambda_{H}$=100), Huber ($\lambda_{H}$=100), and cGAN ($\lambda_{H}$=0). We
also compute the Kullback-Leibler (KL) divergence between the ground truth
palette distribution of the training data and that of our model variants.

As shown in the bottommost plot of Fig. 12, the Huber loss plays a critical
role in producing proper colors close to the ground truth image. Without the
Huber loss, the model does not only fail to recover the color distribution
similar to the ground truth data but also exhibits the lowest fooling rate of
30.7% in user study results. On the other hand, the model with cGAN+Huber loss
($\lambda_{H}$=100) records the lowest KL divergence of 0.2299 as well as the
best fooling rate of 56.2%, while the model with only the Huber loss
($\lambda_{H}$=100) records the second best. This is due to the fact that only
using the Huber loss leads to blindly averaging over multiple ground truth
palettes, resulting in slightly desaturated palette results as shown in the
second row of Fig. 13. In contrast, the model with both cGAN+Huber loss learns
and preserves various ground truth colors rather than simply averaging them,
resulting in bright, highly saturated results as shown in the first row of
Fig. 13.

Figure 12: Color Distribution Comparisons. Red lines correspond to color
distributions of generated palettes from three model variants. The blue lines
denote the ground truth color distribution of the training data. The KL
divergence of the three distribution pairs are computed as 0.2299, 0.2375, and
0.5833 in order. Figure 13: Comparison of palette prediction results from
different model variations.

###  8.2 Additional Results

This section shows additional, diverse and detailed results from TPN.

Figure 14: Handling phrase-level inputs about ‘love’.

Fig. 14 shows how our model handles phrase-level inputs. To make comparison
easier, all the phrases contains the word ‘love.’ It is interesting to see how
our model chooses to express the subtle nuance differences included in the
input text. Notice how the output color palettes tend to be darker for text
inputs that are negative towards ‘love’ (e.g., ‘i thought i loved you’ and
‘where did our love go’). All input phrases included in this figure are unseen
data.

Figure 15: Palette predictions from test set.

Fig. 15 shows outputs of our model in comparison to ground truth palettes. If
an input word is seen at least once in the training data, our model is able to
output a color palette related to the input word. For instance, take a look at
the color palette named ‘mango and grapefruit’ on the top left. The word
‘grapefruit’ is included only once in the training set. Yet, the model
successfully outputs a color palette that matches the text input. Also, ground
truth palettes are included for a direct comparison with generated palettes.
Even if the predicted palette is not exactly identical to the ground truth
palette, both can be perceived as reasonable colors.

Figure 16: Failed results of TPN. Our model fails and outputs the same washed-
out grayish-brown color palettes for unknown tokens.

Even though our model can effectively produce semantically meaningful
colorizations, it struggles when unknown tokens are given as input. Unknown
tokens refer to words not included in the training set. it is not surprising
that our model fails and outputs the same washed-out grayish-brown palettes as
we can see in Fig. 16. On the other hand, our model can still produce
reasonable palettes in the case of unseen, new combinations of words found in
the training set. For example, ‘bright life’ in Fig. 15 was seen separately as
‘bright’ and ‘life’ in the training set but not together. Thus, ‘bright life’
is classified as unseen data, which our model has no problem in predicting
color palettes from.

##  9 Palette-Based Colorization Networks (PCN)

We present additional colorization results on datasets including CUB-200-2011
(CUB dataset) [41], ImageNet ILSVRC Object Detection (ImageNet dataset) [34],
and Graphical Pattern images (Pattern images) in Figs. 17-22. In these
figures, the leftmost columns are grayscale images. Text inputs are given
above the grayscale image. The vertical color palettes next to the grayscale
images are palettes generated from the text input. The output has been
colorized with the generated color palette. We would like to emphasize that
our model effectively utilizes the generated color palettes during the
colorization process. The colorized image may be different from its natural
colors because our networks incorporate additional color hints. We display the
original ground truth image on the right to compare how different an image
becomes after applying the palettes.

###  9.1 CUB-200-2011

Figs. 17 and 18 show additional colorization results on the CUB dataset.

Figure 17: Results on CUB dataset (1). Figure 18: Results on CUB dataset (2).

###  9.2 ImageNet ILSVRC Object Detection

Figs. 19 and 20 show additional colorization results on the ImageNet dataset.

Figure 19: Results on ImageNet dataset (1). Figure 20: Results on ImageNet
dataset (2).

###  9.3 Graphical Pattern Images

Our PCN model generalizes surprisingly well on other types of images. Our
model is trained on ImageNet dataset, which is mostly made up of natural
images. Instead of natural images, we used our colorization model to colorize
graphical pattern images. The graphical pattern images are crawled from Google
through searching keywords such as ‘pattern,’ ‘fabric pattern,’ or ‘beautiful
patterns.’ As seen in Figs. 21 and 22, graphical pattern images are
significantly different from natural images. The colorized outputs show that
our model can apply our generated color palettes on images of diverse shapes
and textures. The results qualitatively show that our palette-based
colorization model is transferable to other image domains.

Figure 21: Results on graphical pattern images (1). Figure 22: Results on
graphical pattern images (2).