We present the Power Law Graph Transformer, a transformer model with well defined deductive and inductive tasks for prediction and representation learning. The deductive task learns the dataset level (global) and instance level (local) graph structures in terms of learnable power law distribution parameters. The inductive task outputs the prediction probabilities using the deductive task output, similar to a transductive model. We trained our model with Turkish-English and Portuguese-English datasets from TED talk transcripts for machine translation and compared the model performance and characteristics to a transformer model with scaled dot product attention trained on the same experimental setup. We report BLEU scores of $17.79$ and $28.33$ on the Turkish-English and Portuguese-English translation tasks with our model, respectively. We also show how a duality between a quantization set and N-dimensional manifold representation can be leveraged to transform between local and global deductive-inductive outputs using successive application of linear and non-linear transformations end-to-end.

§ INTRODUCTION

Statistically distributed representations of language models[1, 2] and application of attention models [3, 4] resulted in breakthrough improvements in Natural Language Processing (NLP) tasks using deep neural networks. These approaches can also be used to design a graph transformer that has deductive and inductive components more clearly established than a transductive model. The deductive functionality can be achieved by expanding the data representation to learn generic representations for a vocabulary $\mathcal{V}$ (of tokens), which is a quantization set of $V$ discrete graph states ($x_i \in \mathcal{V}$) that is a superset of a sentence $\vx=\{x_1, x_2,...,x_S\}$. A sentence that is syntactically and semantically valid in a language model (LM) represents a graph instance of tokens from the quantization set, each represented with statistically distributed dense embedding vectors with N feature dimensions. A graph transformer model can be developed if we can learn metric tensor instances of language model manifold from graph instances and derive an accompanying energy-curvature tensor that can be used to propagate the language model vectors across the encoder-decoder network. A big challenge in defining such a model is the need for expert domain knowledge to define connections between graph states in terms of a weighted adjacency matrix or a more abstract metric tensor that can generalize to an N-dimensional manifold where N can be very large. In our previous work [5], we showed that it is possible to predict molecular properties in a simple one-hot encoding setting where metric tensor was a hand-engineered inverse-distance weighted adjacency matrix of size $W \times W$ with $W$ being a pre-set maximum number of nodes for each graph. The energy-curvature tensor was a matrix of same size derived as part of a learnable coulomb attention model applied on the adjacency matrix and hidden states. We also proposed in our previous work that this attention model can be improved and generalized by using distributed embedding representations and transformer architecture.

In this paper, we present a generalized form of our power law attention model that is scalable to any graph size for a given quantization base set $\mathcal{V}$ of size V and a non-linear manifold of N feature dimensions. Specifically, we develop an end-to-end deductive-inductive power law graph transformer (PLGT) model for machine translation task by using a set of linear embedding vectors from source and target languages. For deductive task, the model learns generalized power law coefficients, metric tensor and energy-curvature tensor instances for a language model manifold. For the inductive task, the attention model learned in deductive task is used to predict probabilities from source input autoregressively, producing same output as a transductive transformer.

In the next sections, we will briefly go over background work, and present the details of the power law attention model and the graph transformer architecture. Then, we will show our results for Turkish-English (TR-EN) and Portuguese-English (PT-EN) translation tasks from ted_hrlr_translate dataset [6, 7].

§ BACKGROUND

A key understanding in data representation that significantly improved the performance of neural machine translation (NMT) models was distributed representation of data first introduced in [1]. The distributed representations of statistical language models developed in [2] demonstrated that the joint probability distribution of discrete random variables can be used to represent each token (e.g. word, subword) in a sentence as a dense vector. These vectors can provide the model with information that grows exponentially within an embedding vector space to reduce the curse of dimensionality. Each embedding vector is composed of a representation with fixed number of feature dimensions for each token in a vocabulary. Then a joint probability distribution for a word sequence can be learned from these vectors which are conveniently called word embeddings. The ability to represent a language model statistically with word embeddings was further improved for large scale data in [8, 9] that introduced projection only training for CBOW and skip-gram word2vec models with additional optimizations for the objective function. A key achievement of word2vec was their efficient linear representation of syntactic and semantic relationships with embedding vectors demonstrating improved analogical reasoning [9]. Another word embedding model, GloVe used global corpus statistics to demonstrate similar analogical reasoning capabilities [10].

State of the art in statistical machine translation (SMT) was further improved by using recurrent neural networks (RNN) with Long-Short Term Memory (LSTM) cells [11] and Gated Recurrent Units (GRU) [12] in an encoder-decoder architecture. The RNN encoder-decoder architectures suffered from inability to translate longer sentences, where a fixed sized vector formed a bottleneck to represent all the data from source sentence into the decoder. The introduction of attention models that can attend to different parts of the source sentence by learning an additive alignment model improved these pioneering SMT models to predict longer sentences more accurately. An attention model provides a weighted context vector learned from source sentence to the decoder to predict the next target word [3] to overcome the bottleneck from a fixed-length vector. Efficient methods using dot-product attention with global and local approaches were explored and compared in [13]. The concept of self attention that used linear combination of hidden states to achieve representation of variable length sentences into a fixed sized embedding was demonstrated in [14]. The transformer model [4] that relies on dot product based self attention and encoder-decoder architecture without recurrent networks demonstrated results better than RNN based SMT models with reduced training cost. The transformer model forms the basis for advanced NLP architectures today  [15, 16]. These models utilize mainly a transductive learning approach [17].

§ MODEL ARCHITECTURE

Our model follows the general design principles in [4] that uses a scaled dot product attention (SDPA) based encoder-decoder architecture to represent and translate data within the model. A key difference is in the attention model which utilizes both linear self attention and power law attention together with deeply connected neural layers. The model is autoregressive, consisting of single layer encoder-decoder configuration that takes in a sequence of input sentences and target sentence is formed one token at a time, where earlier predicted tokens are fed as input to the decoder. The encoder and decoder first learn a metric tensor instance through a deep neural network accepting linear self attention of source and target inputs, respectively. This metric tensor is then used to learn the energy-curvature tensor that can facilitate localized linear transformations between source and target languages. The general layout of graph transformer is shown in fig. <ref>.

Single layer encoder-decoder architecture of the graph transformer

Functional diagram of the attention block

§.§ Encoder and Decoder Layers

Encoder first converts the tokenized input sentence $\vx=\{x_{1}, x_{2}, x_{3},\ldots, x_{S}\}$ into a learned embedding matrix $\mX$ with vector space dimension $d_{emb}$ for each token: $\mX \in \R^{S \times d_{emb}}$. A positional encoding is added to $\mX$ to inform the model of the sequence of tokens in input sentence after multiplying the embedding matrix with $\sqrt{d_{emb}}$ [4]. The encoder consists of a multi-head attention layer and deep fully connected dense layers with residual connections and layer normalization  [18] at their output. The power law attention design used in encoder is a combination of linear transformations and deep residual neural network layers that learn the source language model (SLM) representation from an ensemble of source input sentences.

The decoder takes as input an embedding matrix $\mY_{shifted}$ prepared in the same way as the input embedding to the encoder layer. The decoder encodes the right shifted target sentence into a target language model (TLM) representation in the first attention layer. The second attention layer takes as input the encoder output for the source language and output of the first attention layer to form the cross-language model (XLM) representation from projections of source and target language model representations of input and target sentence. The last stage of decoder is a fully-connected dense layer same as in encoder. Each attention layer and dense layer has residual connections followed by layer normalization at the output. The attention output in the decoder is masked to ensure that the prediction can only depend on known outputs that occur earlier in the sequence.

We use single encoder and decoder layer for the graph transformer implemented in this work, although it is possible to scale model with identical encoder and decoder stacks.

§.§ The Power Law Graph Attention (PLGA) Layer

The attention layer for graph transformer consists of three stages as shown in fig. <ref>. In the first stage, a metric tensor is inferred from an input graph which is a matrix formed by concatenating embedding vectors with feature dimension of $d_{emb}$ for $S$ tokens (graph nodes) in the input sentence. Metric tensor for a language model manifold is a generalized, abstract form of a weighted adjacency matrix learned through a deep neural network. For many types of unstructured data that can be represented as a graph with large number of dimensions as features and many connections between nodes, it is not straightforward to define a distance metric between each node. In our earlier study [5], the inverse of three dimensional eucledian norm between each node was used as a hand-engineered distance metric to define a weighted adjacency matrix for each graph (molecules) to demonstrate a reasonable level of prediction capacity for the graph attention model. The first stage learns the metric tensor in an end-to-end fashion using self attention and a deep residual network without the need to define a distance heuristic from domain knowledge.

The second stage uses the metric tensor as an input to learn power law relationships and coupling coefficients for a generalized energy-curvature (EC) tensor for the language model manifold. Thus, each element of an EC tensor is a superposition of exponentiated metric tensor elements weighted by a coupling coefficient. EC tensor corresponds to the generalized form of a language model represented  by a manifold with $d_{LM}$ dimensions. We refer to this tensor the Energy-Curvature tensor for two reasons: First, it is derived entirely from the metric tensor in a similar fashion the Ricci tensor that defines the curvature of a Riemannian manifold is derived. Secondly, imposing a power law relationship through metric tensor elements approximates a sum of abstract potentials that manipulates the curvature of the manifold.

The third stage is a linear transformation which evolves the embedding space representation of the input sentence to an instance of a language model representation as output of the attention layer.

The attention layer has multi-head support where input is split into $h$ heads with depth of each attention sublayer defined as $d_k=d_{LM}/h$. Each head learns its own subspace of metric tensor and energy-curvature tensor for a subset of language model dimensions. The attention layer architecture is shown in fig. <ref>.

§.§.§ Learning Metric Tensor from self-attention

The input to attention model is a dense matrix $\mX$ of size $S \times d_{emb}$ superposed with positional encoding. The language model dimension $d_{LM}$ is set to be equal to embedding dimension $d_{emb}$ in our implementation. We define a localized graph operator for a single input graph instance represented by $\mQ(=\mX)$ using self-attention:

\begin{equation}
\mD_{Q}=\mQ^{T}\mQ \equiv \Ket{\mQ^T} \Bra{\mQ^T} \label{eq1}
\end{equation}

$\mD_{Q}$ is a $d_{LM} \times d_{LM}$ density matrix operator for a graph (sentence) with mixed statistically distributed representations. We also introduce the bra-ket notation for $\mQ$ which is a concatenated, well-defined sequence of embedding vectors that carry linear syntactic and semantic relationships and distributed probabilistic representations of elements in a Vocabulary. To align the size defined in model implementation ($S \times d_{LM}$ for $\mQ$) with bra-ket notation, $\mQ^T$ is used as the bra-ket state. Thus $\Ket{\mQ^T}$ state has same size of the matrix $\mQ^T$ ($d_{LM} \times S$) and $\Bra{\mQ^T}$ is the transpose. Each element of $\mQ$ attends onto other elements of the same graph and this operator can be used to get the degree of $\mQ$-ness in another graph $\mV$ such that $\Ket{\mQ^T} \Braket{\mQ^T|\mV^T}$. The inner product $\Braket{\mQ^T|\mV^T}$ is a matrix where each entry is dot product of token vectors and is a measure of similarity between tokens.

The metric tensor $\mA$ is learned from the self-attention of training instances through a deep residual network where each residual unit is composed of two fully-connected layers of size A-dff with ReLU activation for each layer followed by a linear fully-connected layer of size $d_k$ and layer normalization as shown in fig. <ref>.

The generalized metric tensor $\mA$ is finally wrapped through a fully-connected layer with learnable weights $\mW$ and bias $\vb_{W}$:

\begin{equation}
\mA_{LM}=ReLU\left(\mW\mA+\vb_{W}\right)+\epsilon \label{eq2}
\end{equation}

The use of ReLU activation with a small value of $\epsilon=1 \times 10^{-9}$ ensures that $\mA_{LM}$ is a tensor with positive non-zero elements. We found that the model converges robustly with this configuration since we also randomly initialize the learnable elementwise power matrix $\mP$ defined in next section with glorot initialization [19].

Model architecture of attention block implemented within encoder and decoder of the power law graph transformer.

§.§.§ Learning Energy-Curvature Tensor for the Language Model

The energy-curvature tensor for the language model $\mG_{LM}$ is derived from metric tensor as:

\begin{equation}
\mG_{LM} = \va\mA_{LM}^{\odot{\mP}}+\vb_{a}  \label{eq3}
\end{equation}

$\va$ and $\vb_{a}$ are the learnable coupling weights and bias for potentials generated from metric tensor $\mA_{LM}$. $\mP$ is a learnable power matrix that is applied to $\mA_{LM}$ elementwise.

The deductive task infers a generalized metric tensor, energy-curvature tensor and learns coupling and power coefficients for the language model characterized with $d_{LM}$-dimensional manifold and a quantization set of the Vocabulary size $V$. To achieve the inductive task, it is necessary to obtain a localized instance projection of the energy-curvature tensor that can transform the representation of an input graph to a language model representation. The EC tensor is first projected onto the graph instance by finding the expected value of the EC tensor weighted over query and key inputs in eq. <ref>. The localized EC operator is scaled by $\sqrt{d_k}$ to avoid small gradient regions. The scaled EC operator is then run through a Leaky ReLU activation step followed by softmax. Masking is applied before softmax by setting values to be ignored close to $-\infty$. The resulting localized graph operator $\mE_{LM}$ is then applied onto input value vector $\mV$ as a linear transformation (eq. <ref>):

\begin{eqnarray}
\label{eq4}
E_{QK}[ \mG_{LM}] &=& \mQ \mG_{LM} \mK^T \equiv \Braket{\mQ^T|\mG_{LM}|\mK^T} \label{eq41}\\
\mE &=& LeakyReLU\left(E_{QK}[ \mG_{LM}]/\sqrt{d_k}\right) \label{eq42}\\
\mE_{LM} &=& softmax\left[mask(\mE)\right] \label{eq43}\\
\mV_{LM} &=& \mE_{LM} \mV \equiv \mE_{LM} \Ket{\mV} \label{eq44}
\end{eqnarray}

The inductive task output of the attention layer is the language model representation of input $\mV_{LM}$ and the deductive task outputs are:  $\mE_{LM}$, $\mP$, $\va$, $\vb_{a}$, $\mG_{LM}$, $\mA_{LM}$.

For the source language model encoder, there is a single stage attention layer where query, key and value entries are all equal to the source sentence embedding (SE) vector sequence, $\mQ_{SE}=\mK_{SE}=\mV_{SE}=\mX$. For the decoder, the first attention layer has its query, key and value as the target sentence embedding (TE) vector sequence shifted right, $\mQ_{TE}=\mK_{TE}=\mV_{TE}=\mY_{shifted}$. For the second attention layer (XLM) of decoder for cross-language model transformation, query is the output of the first attention layer, $\mQ_{XLM}=\mV_{TLM}$ and key and value are the output of source encoder layer, $\mK_{XLM}=\mV_{XLM}=\mV_{SLM}$.

§ THE DATASET

The dataset is used in this study is a parallel corpus created from TED Talk transcripts for two different language families: Portuguese-English (PT-EN) and Turkish-English (TR-EN) machine translation tasks [6]. The PT-EN dataset is composed of 51785 sentence pairs for train, 1193 sentence pairs for development and 1803 sentence pairs for test. The TR-EN dataset is composed of 182450 sentence pairs for train, 4045 sentence pairs for development and 5029 sentence pairs for test. We used the dataset as prepared in Tensorflow Datasets Catalog [7]. The training dataset was shuffled before each training run. The datasets were used to create a subword vocabulary of maximum 15k tokens using the wordpiece approach used in BERT implementation [20, 15]. The tokenizer used for the source and target were Bert Tokenizer implementation in Tensorflow-Text package [21]. Vocabulary is generated separately for Portuguese and English ($\sim8k$ tokens each), as well as Turkish and English ($\sim15k$ tokens each) from their respective paired train datasets.

§ EXPERIMENTAL SETUP

We trained graph transformers that have single encoder-decoder layer as shown in fig. <ref>. The embedding and language model manifolds have dimension sizes of $d_{emb}=d_{LM}=512$. We ran models with different number of attention heads and scaled dense connections for the metric tensor and pointwise feedforward connections accordingly to maintain same $\textit{A-dff}/d_k$ ratio for all models. We tried multi-head attention with 1,2,4,8 and 16 heads and scaled residual layer/residual dense/point-wise feed-forward parameters as shown in table <ref>. We also trained a transformer model with scaled dot-product attention (SDPA) that has 4 encoder-decoder layers, 8 heads and a drop-out rate of 0.1 for comparison. SDPA transformer has same $d_{LM}$ and $\textit{dff}$ values as the power law graph transformer models. The number of residual layers for graph transformers and the number of encoder-decoder layers for SDPA model were chosen to be maximum values a single GPU in our setup can handle for each model without running out of memory except for model #3 in table <ref>.

The training is performed by using Adam optimizer [22] at a custom scheduled learning rate [4] with warm-up steps of $15000$ for PLGA and $4000$ for SDPA transformer models and used a batch size of $64$ for both. Attention layer weights, fully connected layer weights and biases were initialized with glorot normal, glorot uniform and zeros, respectively. During training, we kept track of the training and validation cross-entropy loss (log perplexity), and accuracy at the end of every epoch. Outside the attention layer, drop-out [23] is applied to embedding inputs after positional encodings are added as well as before residual sum and layer normalization at the attention layer and encoder-decoder outputs. Outside drop-out rate was set at $0.4$. Inside the attention layer, a drop-out rate of $0.1$ is also applied at output of every residual unit before summing and layer normalization for metric tensor learning. The drop-out rate within attention layer for inputs $\mQ$, $\mK$ were kept at zero and dropout rate for $\mE_{LM}$ was set at $0.1$. We found these values to give good compromise to avoid overfit of loss curve quantified by log perplexity over $120$ epochs. We kept checkpoints for the model parameters at $10$ epochs after the minimum validation loss is observed [12], at highest validation accuracy and a number of checkpoints sampled over $120$ epochs for comparison. Training took $\sim10-36$ hours for each model based on hyperparameters and the dataset.  BLEU metric [24] was used to evaluate the test dataset. BLEU score was calculated using sacrebleu package [25]. For evaluation of our models, we run predictions using beam search with beam length of $1$ (greedy search) and $4$ with length normalization [26]. The maximum number of iterations carried out for evaluation was set at 50 iterations above input sentence length.  Model variations were implemented using Tensorflow [27]. Implementation of Power Law Graph Transformer can be found at <https://github.com/burcgokden/Power-Law-Graph-Transformer>.

Set of model hyperparameters used for training the dataset. The unfilled sections have the same value as model #1. Models #1-6 are power law graph transformers. SDPA is transformer with scaled dot product attention.

Model     # Layers     # Heads     A-dff      # Res. Dense Layers     # Res. Units     $d_{LM}$     dff

#1     1     16     128     2     10     512     2048

#2          8     256          9

#3          8     256          8

#4          4     512          5

#5          2     1024          2

#6          1     2048          1

SDPA     4     8     n/a     n/a     n/a     512     2048

§ RESULTS

Evaluation with beam length=1. The results for inductive task are shown in table <ref> for graph transformers and SDPA transformer model. We ran PT-EN translation task on all model variations for comparison. The model #2 with 8 heads and 9 residual layers gave the best BLEU score among graph transformer variations. Model #1 with 16 heads and 10 residual layers exhibited reduced BLEU scores for PT-EN task compared to model #2 with 8 heads. The reduction of head size and residual layer size have a big impact on the model capacity. Model #6 with single head and single residual unit had lowest BLEU score of $16.02$ even if the fully connected layers in residual units have the largest number of neurons per layer. This suggests that the number of deep residual connections that learn the metric tensor and number of heads exploring alternate versions of graph manifold are important to represent unstructured data such as language datasets, that usually employ ambiguous relationships between nodes. The SDPA model was also trained using PT-EN dataset and had a slightly better BLEU score of $27.97$ vs $27.79$ for model #2 variation of the graph transformer. The models #2 and #3 trained on larger TR-EN dataset gave similar results of $17.58$ and $17.61$ which was $\sim0.8$ higher than SDPA BLEU score of $16.82$ on the same dataset.

We also compared the loss and accuracy curves over 120 epochs for PLGA model #2 and SDPA model. The results are shown in fig. <ref> for models trained on PT-EN and TR-EN tasks. The SDPA transformer converges to a minimum loss earlier during training. The validation loss curve starts to overfit at longer training times, therefore an early stopping strategy is expected to give the best case BLEU score for SDPA model in this work. For the PLGA model, the overfit is much less and validation and training accuracy have a smaller gap. The best case results and highest accuracy points occur at later epochs for the PLGA model. This suggests that the PLGA and SDPA architectures explore the model space differently.

Loss and accuracy curves for model #2 (a,b) and SDPA (c,d) architectures trained on PT-EN and TR-EN datasets. Full (hollow) circles in red are for train set loss (accuracy) values and full (hollow) triangles in blue are validation set loss (accuracy) values. For clarity values at every 5 epochs are plotted.

Evaluation with beam length=4. We evaluated the model #2 and SDPA model with highest BLEU scores using beam search at beam length=4 to compare. The results are shown in table <ref>. The PLGA model results in better BLEU score than RNN model [3] with attention  evaluated in [6] for PT-EN and TR-EN tasks with standard (randomly initialized) embeddings. When evaluated at beam length=4, the SDPA model fared better in BLEU score than the PLGA model for PT-EN and TR-EN tasks.

Greedy search BLEU results for models trained using TR-EN and PT-EN datasets for 120 epochs and evaluated at various intervals. Maximum BLEU scores are shown. (HA: model was evaluated at highest validation accuracy.)

Model     Dataset     BLEU     Log Perplexity     Epoch

#1     PT-EN      26.96     2.74     110

#2     PT-EN     27.79     2.64     110(HA)

#2     TR-EN      17.58     2.66      118(HA)

#3     PT-EN     27.66     2.68     120

#3     TR-EN     17.61     2.67     120

#4     PT-EN      27.55     2.64     110

#5     PT-EN      16.74     3.42     80

#6     PT-EN      16.02     3.40     118(HA)

SDPA     PT-EN      27.97     2.15     17(HA)

SDPA     TR-EN      16.82     2.43     30

BLEU score comparison for PLGA transformer, SDPA transformer and the RNN model trained in ref. [6] using same datasets with standard and pre-trained embeddings. PLGA and SDPA transformers were evaluated using beam length of 4 and 1 (shown in parentheses).

3-4 Model     Dataset     $(std-std)$     $(pre-pre)$

#2     TR-EN     17.79(17.58)     –

SDPA     TR-EN     18.31(16.82)     –

from [6]     TR-EN     14.9     17.9

#2     PT-EN     28.33(27.79)     –

SDPA     PT-EN     29.57(27.97)     –

from [6]     PT-EN     26.2     30.8

For the deductive task of the model, we analyzed the 2D heatmap and histogram distributions of the set $ ( \mE_{LM}, \mP_{LM}, \va_{LM}, \vb_{a}, \mA_{LM}, \mG_{LM} )$. Out of these parameters, $(\mP_{LM}, \va_{LM}, \vb_{a})$ are learned for the entire dataset and are generalized for the language model. The rest of the outputs  $(\mE_{LM}, \mA_{LM}, \mG_{LM})$ are inferred instances for an input sentence (graph instance). We show in figs. <ref> and <ref>, the heatmaps and histograms from head $4$ of last attention stage (X-LM attention) of model #2 trained using PT-EN dataset and evaluated with greedy search. The outputs from all heads for X-LM, source LM and target LM attention models are included in the appendix. Following input sentence from PT-EN dataset was evaluated to generate the deductive task outputs:

Input          :    “este é um problema que temos que resolver ."

Prediction     :    “this is a problem that we have to solve ."

Ground truth   :    “this is a problem we have to solve ."

The heatmaps for $(\mP_{LM}, \va_{LM}, \vb_{a})$ show approximately gaussian distribution without any clear pattern visible. The histogram distribution for $\mP_{LM}$ (fig. <ref>) has slightly longer tail above zero. The coefficients $\va_{LM}$ and  bias $\vb_a$ are less skewed and more broadly distributed around zero value. $\vb_a$ profile indicates that the model will have a non-zero background distribution if the metric tensor for an input instance were all zero.

The metric tensor $\mA_{LM}$ (fig. <ref>) for input sentence is positive definite and heat map is intriguingly not approximately gaussian, where the dark regions close to zero and non-zero “active" regions are grouped and connected similar to a loosely knit straw basket pattern. The histogram shows a long tail distribution with a sharp peak at number of values close to zero.  This indicates that the metric tensor is still sparse for $d_{emb}=512$ chosen as embedding feature dimension in our models. The EC tensor $\mG_{LM}$ is shown in fig. <ref> and was derived using eq. <ref> in the attention model. Unlike $\mA_{LM}$, the EC tensor distribution has no clear pattern on the heatmap and is approximately gaussian and fairly centered around zero. Fig. <ref> shows the attention weights between source and predicted target sentences which is similar to attention weights observed from other attention based models [3, 4]. We observe that the range of attention weights for $\mE_{LM}$ varies on a scale from 0 to 1. The deductive output values are well behaved, given that the input is not normalized and learned parameters are randomly initialized. We suspect that the layer normalization and positive-definite $\mA_{LM}$ condition applied in eq. <ref> help improve the interpretability of the deductive task outputs.

Other head outputs show similar distributions for heatmap and histogram we presented here for X-LM attention outputs. For source and target attention blocks, the attention weights $\mE_{LM}$ show more varied patterns among heads.

Heatmap and histogram distributions of deductive task outputs for head 4 of XLM Attention from model #2. (a,b): $\mP_{LM}$, (c,d): $\va_{LM}$, (e,f): $\vb_{a}$. Dashed line in orange marks zero value.

Heatmap and histogram distribution of deductive task outputs for head 4 of XLM Attention from model #2. (a,b): $\mA_{LM}$, (c,d): $\mG_{LM}$, (e): $\mE_{LM}$. Dashed line in orange marks zero value.

§ DISCUSSION AND FURTHER WORK

We can make several key observations about the graph transformer architecture explained in this work. The model uses a quantization set (subword Vocabulary generated from dataset) and its dense vector representation for each element in the set (embedding space vectors with $d_{emb}$ feature dimensions) for linear transformations. A language model manifold (defined by $d_{LM}$ feature dimensions) is  obtained by non-linear transformations through a deep neural network that learns from a large ensemble of local instances of sentences (graph instances). The manifold and quantization set define a duality where we can statistically build global relationships from a large ensemble of local instances and similarly infer single local instances from global relationships. The deductive task builds the global relationships by learning $(\mP_{LM}, \va_{LM}, \vb_{a})$ which are parameters characterizing the relationships for the entire language model. These parameters are not associated with a single sentence but language defined by the entire dataset available to the model, unlike attention weights that are obtained from a SDPA transformer model for a single sentence. The inductive task builds the localized relationships by propagating through $(\mE_{LM}, \mA_{LM}, \mG_{LM})$ from a single instance of input sentence where the output is same as that of a transductive transformer model. The duality transformation between local and global relationships is highly non-linear, defined by a deep residual network. On the other hand, we take advantage of linear transformations whenever local inferences were made from an input instance.

The number of heads also have a large impact on locality condition besides model capacity. The multi-head configuration allows the model to consider multiple interpretations of an input sentence in sub-spaces. By splitting the input into smaller size heads, we also enforce the model to consider feature dimensions in the same head split to interact more closely. Therefore, a constraint on locality is also introduced within each head split.

Several possible configurations were not explored in this study due to limitations of scope and our experimental setup. The model architecture can be also explored for deeper residual networks with less head splits and multi-layer encoder-decoder networks. The number of embedding dimensions is another hyperparameter that could have an impact on both deductive and inductive task outputs. The hyperparameters we used in this study were optimizations that were reported to work well for the SDPA transformer model in the literature. A better hyperparameter set optimized for graph transformer architecture could improve the BLEU score of this model further. Other optimizations reported for large scale NLP systems [28, 29] could be used to scale graph transformer to larger datasets and multi-GPU setups. The deductive outputs provide a rich set of statistical information of the language model and neural network itself. A more in depth analysis of the deductive outputs can provide better understanding of the dataset domain and model architecture.

§ CONCLUSION

We presented a generalized power law graph transformer architecture with well defined deductive and inductive tasks. The deductive task learns the global characteristics of the dataset using a power law attention model. The inductive task uses the global characteristics to predict the output probabilities for an input instance through encoder-decoder architecture. We applied our model for TR-EN and PT-EN machine translation tasks and compared its performance and characteristics to a SDPA transformer model evaluated on same experimental setup. The graph transformer developed in this work used many of the optimizations that SDPA transformer was shown to benefit from in the literature and we believe that hyperparameters better optimized for graph transformer architecture can result in higher BLEU scores.

Our model empirically takes advantage of a duality between a subword Vocabulary represented by $d_{emb}$ embedding feature dimensions and a language model represented by $d_{LM}$ feature dimensions to define local and global statistics from a machine translation dataset. While a single instance of a sentence can be considered as a graph instance exploring a local region of the language model manifold, a large ensemble of such localized instances can be used to learn an abstract, statistical representation for the entire manifold.

In more general terms, graph samples generated from a linear quantization set are used to build a statistical representation for a non-linear manifold using deep residual networks and attention based on a power-law relationship. The power law relationship is inherently scale invariant and we expect that it will be particularly interesting to apply the model to datasets with varying scale and features from domains beyond NLP tasks such as graph databases, communication networks, and many-body problems in quantum mechanics and astronomy.

§.§.§ Acknowledgments

I thank my parents for their support and patience. I would like to acknowledge numerous contributions of machine learning research community on NLP tasks and graph networks in recent years. This research was conducted independently without support from a grant or corporation.

[1]
G. E. Hinton, J. L. Mcclelland, and D. E. Rumelhart, “Distributed
representations,” in Parallel Distributed Processing: Explorations in
the Microstructure of Cognition, Volume 1: Foundations (D. E. Rumelhart
and J. L. Mcclelland, eds.), pp. 77–109, Cambridge, MA: MIT Press, 1986.

[2]
Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin, “A neural probabilistic
language model,” J. Mach. Learn. Res., vol. 3, p. 1137–1155, Mar.

[3]
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly
learning to align and translate,” 2014.
cite arxiv:1409.0473Comment: Accepted at ICLR 2015 as oral

[4]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information
Processing Systems, NIPS'17, (Red Hook, NY, USA), pp. 6000–6010, Curran
Associates Inc., 2017.

[5]
B. Gokden, “Coulgat: An experiment on interpretability of graph attention
networks,” CoRR, vol. abs/1912.08409, 2019.

[6]
Y. Qi, D. Sachan, M. Felix, S. Padmanabhan, and G. Neubig, “When and why are
pre-trained word embeddings useful for neural machine translation?,” in Proceedings of the 2018 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies,
Volume 2 (Short Papers), (New Orleans, Louisiana), pp. 529–535, Association
for Computational Linguistics, June 2018.

[7]
“Tensorflow dataset for ted talk transcripts.”

[8]
T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word
representations in vector space,” CoRR, vol. abs/1301.3781, 2013.

[9]
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed
representations of words and phrases and their compositionality,” in NIPS, pp. 3111–3119, Curran Associates, Inc., 2013.

[10]
J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word
representation.,” in EMNLP, vol. 14, pp. 1532–1543, 2014.

[11]
I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with
neural networks,” in Advances in neural information processing
systems, pp. 3104–3112, 2014.

[12]
K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares,
H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN
encoder–decoder for statistical machine translation,” in Proceedings
of the 2014 Conference on Empirical Methods in Natural Language Processing
(EMNLP), (Doha, Qatar), pp. 1724–1734, Association for Computational
Linguistics, Oct. 2014.

[13]
T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based
neural machine translation,” in Proceedings of the 2015 Conference on
Empirical Methods in Natural Language Processing, (Lisbon, Portugal),
pp. 1412–1421, Association for Computational Linguistics, Sept. 2015.

[14]
Z. Lin, M. Feng, C. N. dos Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio, “A
structured self-attentive sentence embedding,” CoRR,
vol. abs/1703.03130, 2017.

[15]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of
deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies,
Volume 1 (Long and Short Papers), (Minneapolis, Minnesota), pp. 4171–4186,
Association for Computational Linguistics, June 2019.

[16]
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal,
A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss,
G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu,
C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess,
J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei,
“Language models are few-shot learners,” 2020.

[17]
V. N. Vapnik, The Nature of Statistical Learning Theory, ch. 9, p. 291.
Springer, 2nd ed., November 1999.

[18]
L. J. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” CoRR,
vol. abs/1607.06450, 2016.

[19]
X. Glorot and Y. Bengio, “Understanding the difficulty of training deep
feedforward neural networks,” in Proceedings of the Thirteenth
International Conference on Artificial Intelligence and Statistics (Y. W.
Teh and M. Titterington, eds.), vol. 9 of Proceedings of Machine
Learning Research, (Chia Laguna Resort, Sardinia, Italy), pp. 249–256,
PMLR, 13–15 May 2010.

[20]
M. Schuster and K. Nakajima, “Japanese and korean voice search,” in International Conference on Acoustics, Speech and Signal Processing,
pp. 5149–5152, 2012.

[21]
“Tensorflow text library.” <https://github.com/tensorflow/text/>.

[22]
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in
3rd International Conference on Learning Representations, ICLR 2015,
San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (Y. Bengio
and Y. LeCun, eds.), 2015.

[23]
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov,
“Dropout: A simple way to prevent neural networks from overfitting,” J. Mach. Learn. Res., vol. 15, pp. 1929–1958, Jan. 2014.

[24]
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for
automatic evaluation of machine translation,” in Proceedings of the
40th Annual Meeting of the Association for Computational Linguistics,
(Philadelphia, Pennsylvania, USA), pp. 311–318, Association for
Computational Linguistics, July 2002.

[25]
M. Post, “A call for clarity in reporting BLEU scores,” in Proceedings
of the Third Conference on Machine Translation: Research Papers, (Brussels,
Belgium), pp. 186–191, Association for Computational Linguistics, Oct. 2018.

[26]
Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun,
Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu,
L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian,
N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals,
G. Corrado, M. Hughes, and J. Dean, “Google's neural machine translation
system: Bridging the gap between human and machine translation,” CoRR,
vol. abs/1609.08144, 2016.

[27]
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado,
A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving,
M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg,
D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens,
B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan,
F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and
X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous
systems,” 2015.
Software available from tensorflow.org.

[28]
D. Britz, A. Goldie, M.-T. Luong, and Q. Le, “Massive exploration of neural
machine translation architectures,” in Proceedings of the 2017
Conference on Empirical Methods in Natural Language Processing, (Copenhagen,
Denmark), pp. 1442–1451, Association for Computational Linguistics, Sept.

[29]
Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun,
Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu,
Łukasz Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian,
N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals,
G. Corrado, M. Hughes, and J. Dean, “Google's neural machine translation
system: Bridging the gap between human and machine translation,” CoRR,
vol. abs/1609.08144, 2016.

$\mE_{LM}$ heatmap plots for all heads from XLM attention stage from graph transformer model #2.

$\mP_{LM}$ heatmap plots for all heads from XLM attention stage from graph transformer model #2.

$\mP_{LM}$ histogram plots for all heads from XLM attention stage from graph transformer model #2. Dashed line in orange marks zero value.

$\va_{LM}$ heatmap plots for all heads from XLM attention stage from Graph transformer model #2.

$\va_{LM}$ histogram plots for all heads from XLM attention stage from Graph transformer model #2. Dashed line in orange marks zero value.

$\vb_{a}$ heatmap plots for all heads from XLM attention stage from graph transformer model #2.

$\vb_{a}$ histogram plots for all heads from XLM attention stage from graph transformer model #2. Dashed line in orange marks zero value.

$\mA_{LM}$ heatmap plots for all heads from XLM attention stage from graph transformer model #2.

$\mA_{LM}$ histogram plots for all heads from XLM attention stage from graph transformer model #2.

$\mG_{LM}$ heatmap plots for all heads from XLM attention stage from graph transformer model #2.

$\mG_{LM}$ histogram plots for all heads from XLM attention stage from graph transformer model #2. Dashed line in orange marks zero value.

$\mE_{LM}$ heatmap plots for all heads from SLM attention stage from graph transformer model #2 for PT-EN translation task.

$\mP_{LM}$ heatmap plots for all heads from SLM attention stage from graph transformer model #2 for PT-EN translation task.

$\mP_{LM}$ histogram plots for all heads from SLM attention stage from graph transformer model #2 for PT-EN translation task. Dashed line in orange marks zero value.

$\va_{LM}$ heatmap plots for all heads from SLM attention stage from Graph transformer model #2 for PT-EN translation task.

$\va_{LM}$ histogram plots for all heads from SLM attention stage from Graph transformer model #2 for PT-EN translation task. Dashed line in orange marks zero value.

$\vb_{a}$ heatmap plots for all heads from SLM attention stage from graph transformer model #2 for PT-EN translation task.

$\vb_{a}$ histogram plots for all heads from SLM attention stage from graph transformer model #2 for PT-EN translation task. Dashed line in orange marks zero value.

$\mA_{LM}$ heatmap plots for all heads from SLM attention stage from graph transformer model #2 for PT-EN translation task.

$\mA_{LM}$ histogram plots for all heads from SLM attention stage from graph transformer model #2 for PT-EN translation task.

$\mG_{LM}$ heatmap plots for all heads from SLM attention stage from graph transformer model #2 for PT-EN translation task.

$\mG_{LM}$ histogram plots for all heads from SLM attention stage from graph transformer model #2 for PT-EN translation task. Dashed line in orange marks zero value.

$\mE_{LM}$ heatmap plots for all heads from TLM attention stage from graph transformer model #2 for PT-EN translation task.

$\mP_{LM}$ heatmap plots for all heads from TLM attention stage from graph transformer model #2 for PT-EN translation task.

$\mP_{LM}$ histogram plots for all heads from TLM attention stage from graph transformer model #2 for PT-EN translation task. Dashed line in orange marks zero value.

$\va_{LM}$ heatmap plots for all heads from TLM attention stage from Graph transformer model #2 for PT-EN translation task.

$\va_{LM}$ histogram plots for all heads from TLM attention stage from Graph transformer model #2 for PT-EN translation task. Dashed line in orange marks zero value.

$\vb_{a}$ heatmap plots for all heads from TLM attention stage from graph transformer model #2 for PT-EN translation task.

$\vb_{a}$ histogram plots for all heads from TLM attention stage from graph transformer model #2 for PT-EN translation task. Dashed line in orange marks zero value.

$\mA_{LM}$ heatmap plots for all heads from TLM attention stage from graph transformer model #2 for PT-EN translation task.

$\mA_{LM}$ histogram plots for all heads from TLM attention stage from graph transformer model #2 for PT-EN translation task.

$\mG_{LM}$ heatmap plots for all heads from TLM attention stage from graph transformer model #2 for PT-EN translation task.

$\mG_{LM}$ histogram plots for all heads from TLM attention stage from graph transformer model #2 for PT-EN translation task. Dashed line in orange marks zero value.


# Multi-Domain Image-to-Image Translation with Adaptive Inference Graph

The-Phuc Nguyen  Department of Computer Vision
Vingroup Big Data Institute
Hanoi, Vietnam
<EMAIL_ADDRESS>Stéphane Lathuilière  LTCI, Télécom Paris,
Institut Polytechnique de Paris
Palaiseau, France
<EMAIL_ADDRESS>Elisa Ricci  FBK and University of
Trento
Trento, Italy
<EMAIL_ADDRESS>
###### Abstract

In this work, we address the problem of multi-domain image-to-image
translation with particular attention paid to computational cost. In
particular, current state of the art models require a large and deep model in
order to handle the visual diversity of multiple domains. In a context of
limited computational resources, increasing the network size may not be
possible. Therefore, we propose to increase the network capacity by using an
adaptive graph structure. At inference time, the network estimates its own
graph by selecting specific sub-networks. Sub-network selection is implemented
using Gumbel-Softmax in order to allow end-to-end training. This approach
leads to an adjustable increase in number of parameters while preserving an
almost constant computational cost. Our evaluation on two publicly available
datasets of facial and painting images shows that our adaptive strategy
generates better images with fewer artifacts than literature methods.

##  I Introduction

The task of Image-to-image translation consists in learning a mapping between
two different visual domains [1, 2]. For instance, daylight images can be
changed to night images or the age of a person in a picture can be changed
while preserving identity and pose [3]. Image-to-image translation has
received a growing interest in the computer vision community since many
problems, such as style transfer [4], photo editing [3], image segmentation
[5] or depth estimation [6], can be addressed following an image-to-image
formulation.

Image-to-image translation has been also addressed in the multi-domain setting
where images are translated between multiple visual domains. This problem can
be addressed by considering either multiple models, as in ComboGAN [7] or with
a single model [8]. The major difficulty of this problem lies in the high
complexity of the models when the number of domains is large. In particular,
when the number of domain increases, training generator networks for every
pair of domains would lead to a cubic growth of the number of network
parameters. To tackle this issue, in ComboGAN [7], domain-specific encoders
and decoders are combined to allow translation between any pair of domains.
This approach leads to a linear growth of the number of parameters. To further
reduce complexity, Choi et al. proposed StarGAN [8] where a single generator
can map images between any pair of domain leading to constant network
complexity. In Table I, we report the network complexity of these two
approaches in terms of number of parameters and floating-point operation at
inference time. While both methods have the same computational cost, ComboGAN
has a prohibitively large number of parameters when the number of domain is
high and StarGAN has a constant number of parameters. However, assuming a
constant number of parameter may be problematic when the number of domains is
high.

In addition, in this paper, we address the problem of multi-domain translation
in the scenario of limited computation resources. In this case, only small
architectures with few convolution filters can be employed. In this
challenging scenario, existing approaches, such as the StarGAN, fail in
generating realistic images since a single small architecture cannot handle
the visual diversity of different domains. Therefore, in this paper, we
propose a novel generator architecture named Ada2Net. This architecture is
adaptive to two extents. First, the network topology is adapted on the fly
depending on the input. In this way, the number of parameters can be adjusted
while keeping a constant computational complexity in terms of floating-point
operations (See Tab. I). By increasing the number of parameters, our model can
handle a high number of visual domains. Second, we adopt the adaptive instance
normalization layer to adapt the parameters of normalization layer parameters
to the image style.

TABLE I: Complexity comparison. We consider a common backbone encoder-decoder
with C parameters and F floating point operations. In Ada2Net, we can adjust
the parameter K depending on number of domains or memory constraints while
keeping a constant computational cost.

| Nbr. Params. | FLOPs
---|---|---
ComboGAN [7] | CD | F
StarGAN [8] | C | F
Ada2Net (Proposed) | CK |  $\approx$F

In short, our contributions are the following: (i) We propose an encoder-
decoder network with an architecture that is dynamically chosen at inference
time. To the best of our knowledge, this work is the first that employs
dynamic network architectures for generation tasks. (ii) We impose a constant
computation complexity by selecting specific sub-networks at inference time.
Our framework is trained end-to-end via the use of Gumbel-Softmax for sub-
network selection. (iii) Evaluation is performed on two widely used datasets
of facial and painting images and shows that our method outperforms state-of-
the-art approaches. Our code is available at
https://github.com/thephucnguyen/Ada2Net.

##  II Related Work

Image-to-image translation consists in learning a mapping between two
different domains. In the seminal work of Isola et al. [1], a neural network
is trained using paired data from the two domains. In many applications,
paired data is not available or too expensive. To tackle this limitation,
cycle consistency is used in CycleGAN [2] to learn translation from unpaired
images. As an alternative, geometry-consistency [9, 10, 11] can be imposed to
obtain better translation results. Other recent works improve image quality by
learning style-content disentangled representations [12, 13] or including
object-specific prior-knowledge [3, 14, 15].

These works can be extended to the multiple domain setting. In this task,
assuming an input image, we generate an image with the appearance of a
specified target domain. To tackle this problem, ComboGAN [7] combines
multiple encoder-decoder that are trained using cycle consistency principle
introduced in CycleGAN[2]. This solution leads to a prohibitively large
architecture that cannot be used in our limited resources scenario.
Conversely, a single encoder-decoder is trained in StarGAN [8] to map any pair
of domains. However, StarGAN generates low-quality images when the network
architecture is reduced to satisfy our low resource constraints. In this work,
we argue that a single architecture, especially small, can hardly handle a
large number of domains since it requires learning a specific input image
representation for every domain. Consequently, we introduce an adaptive
network that can adjust its graph at inference time. Thus, we augment the
network capacity without its computation cost at inference time.

In the context of image generation, adaptive instance normalization [4]
recently has become a popular tool. This layer can be seen as an adaptive
layer since its parameters are not learned but predicted by another layer at
inference time. Nevertheless, the network topology remains fixed. In parallel,
several works proposed neural architectures with adaptive inference graphs[16,
17] in order to reduce computation cost at inference time. In these works,
network architecture is predicted on the fly with the aim of avoiding
unnecessary operations. The network is trained end-to-end using Gumbel-softmax
[18] to obtain a differentiable approximation of the argmax function. In this
work, we propose these two types of adaptive mechanisms. To the best of our
knowledge, this work is the first attempt to use an adaptive network with a
topology inferred on the fly in the context of generation tasks.

Figure 1: Ada2Net: Two encoders extract content and style representations of
the input. These two encodings are fed to a sequence of residual blocks with
an adaptive graph. The network topology is predicted on the fly. Adaptive
inference Normalization layers are used in each elementary block to obtain
better style-content disentanglement. Finally, the resulting tensor is then
provided to a small decoder network that generates the output image.

##  III Ada2Net

In this section, we detail the proposed architecture with an adaptive
inference graph for multi-domain image-to-image translation. First, we
describe the overall architecture. Then, we detail our gating mechanism for
adaptive inference. Finally, we provide details concerning the training
procedure.

###  III-A Overall architecture

Building on StarGAN framework [8], we perform image-to-image translation for
multiple domains using only a single network $G$. Considering an input image
$x$ from a source domain, the generator network translates $x$ into the domain
$d$ as follows $x_{d}=G(x,d)$. The target domain label $d$ is provided to the
generator network using one-hot encoding.

In this work, we further assume that each image $x$ contains content and style
information. Following [12], we employ two encoder networks $E^{c}$ and
$E^{s}$ that respectively extract a feature representation
$c=E^{c}(x,d)\in\mathbb{R}^{H,W,C}$ describing content and a style code
$s=E^{s}(x,d)\in\mathbb{R}^{S}$. Note that, in our low resource scenario, we
use very small encoder networks. In our multi-domain translation problem, this
choice is motivated by the idea that content lies in a shared latent space
between different domains while style is specific to each domain. In other
words, the content code $c$ encodes the shared information among all domains
that should be preserved during translation, while the specific style code
represents remaining variations from different domains that are not contained
in the input image.

The input image representations $c$ and $s$ are combined using a sequence of
adaptive style-based Residual blocks described in the following section. In
this way, we obtain a tensor $\tilde{x}=R(c,d)$ where $R$ denotes the sequence
of adaptive style-based Residual blocks. This tensor $\tilde{x}$ is then input
to a 2-layer decoder network $D$ that outputs the generated image.

Importantly, we disentangle content and style information thanks to the way
the style code is used. Following recent style-based architectures [4, 12],
our style code is used only in the instance normalization layers included in
the residual blocks. In instance normalization, the same affine transformation
is applied at every location. Therefore, the style encoder extracts
information that is not related to location and structure. Conversely, the
content encoder network encodes what cannot be reconstructed from the style
code.

###  III-B Adaptive style-based residual blocks

In this section, we detail how we combine the content and style representation
in order to obtain a single tensor. In place of commonly used residual blocks
[2, 19], we propose a novel adaptive style-based residual block. The
motivation for adaptive residual blocks is that a small fixed architecture can
hardly handle the visual diversity of different domains.

A standard Residual network [20] uses a ”identity shortcut connection” that
skips layers. Formally, assuming an input tensor
$x_{l-1}\in\mathbb{R}^{H,W,C}$, the output of a standard residual block can be
written as follows:

$x_{l}=x_{l-1}+F_{l}(x_{l-1})$ (1)

where $F_{l}$ is a two-layer convolutional layer with ReLu. In order to
augment the capacity of the network, we propose to replace $F_{l}$ by $K$
different residual functions $(F_{l}^{1}\ldots F_{l}^{K})$. Nevertheless, in
order to preserve a constant computational cost, only a single residual
function is employed. Following [16], the layer input $x_{l-1}$ is fed to a
small multi-layer perceptron $\Gamma_{l}$ that predicts which function should
be used. More precisely, we assume that $\Gamma_{l}$ ends by a layer of
dimension $K$ with softmax activation:
$\Gamma_{l}=(\gamma_{l}^{1}\ldots\gamma_{l}^{K})$. Then, the output of our
adaptive residual block is given by:

$\displaystyle x_{l}=x_{l-1}+F_{l}^{k^{*}}(x_{l-1})$ (2)
$\displaystyle\text{with
}k^{*}=\underset{k}{\mathrm{argmax}}(\gamma_{l}^{k}(x_{l-1}))$

The intuition here is that each function $F_{l}^{k}$ can be specialized in a
sub-region of the feature space. Therefore, we obtain a neural network with a
larger capacity with a marginal computational over-head. Indeed, the total
number of parameters of each block is multiplied by $K$ (ignoring the
parameters of $\Gamma_{l}$). Concerning computation, the additional
computation cost lies only in the gate network $\Gamma_{l}$. To reduce its
cost, we use a global averaging pooling layer as the first layer of
$\Gamma_{l}$. In this way, we obtain a computational over-head negligible with
respect to the cost of $F_{l}^{k^{*}}$.

The main difficulty in this approach is that the argmax function used in
Eq.(2) needs to be implemented in a differentiable manner in order to allow
end-to-end training of the whole network. Standard softmax cannot be used
since it would imply that all the $F_{l}^{k}$ are computed leading to a high
computational cost. In addition, replacing softmax by argmax at test time
would introduce differences between training and test that may affect the
results. To tackle this issue, we use the Gumbel-Max trick and its softmax
relaxation proposed in [18]. The key idea of this approach is that by sampling
noise according to $K$ Gumbel distributions, we can obtain samples that follow
the distribution corresponding to the probabilities provided by a softmax
activation. Based on this idea, during training, in the forward pass, we
employ discrete samples, while during the backward process, the gradient of
the softmax is propagated. At test time, the argmax function is employed on
the softmax predictions. For more details concerning the Gumbel-Max trick,
please refer to [18].

Our adaptive residual blocks, as described so far, do not use any style
information. Following recent trends in image generation, the style code is
used to parametrize the instance normalization layer of each residual function
$F_{l}$. This approach is usually referred to as adaptive instance
normalization [4]. We employ a small network to predict the shift and scale
parameter of each instance normalization layer. Formally, for a given layer
input $x$, the output of the adaptive instance normalization layer can be
written by:

$In(x,s)=\alpha(s)\Big{(}\frac{x-\mu(x)}{\sigma(x)}\Big{)}+\beta(s)$ (3)

where $\alpha$ and $\beta$ denote the two heads of a small multi-layer
perceptron.

###  III-C Training procedure

Inspired by [8] and [12], our network is trained in an end-to-end manner
combining cycle reconstruction losses and style-content disentanglement
losses.

Multi-domain Cycle reconstruction losses. We use the cycle consistency ensued
from CycleGAN [2]. First, we consider a reconstruction loss. Assuming an image
$x$ from a domain $d$ and a target domain $d^{\prime}$, the cycle
reconstruction loss can be written:

$\mathcal{L}_{rec}^{cyc}=\mathbb{E}_{x,d^{{}^{\prime}}}[||x-G(G(x,d^{\prime}),d)||_{1}]$
(4)

We also include a reconstruction loss that enforces that the network
reconstructs the input image when the target domain corresponds to the input
domain:

$\mathcal{L}_{rec}^{in}=\mathbb{E}_{x,d}[||x-G(x,d)||_{1}]$ (5)

Then, in order to generate realistic images, we introduce a discriminator
network and employ the least square adversarial framework from [21]. It leads
to two adversarial losses $\mathcal{L}_{adv}^{disc}$ and
$\mathcal{L}_{adv}^{gen}$ that are used to update the discriminator and
generator respectively. Finally, we employ a domain confusion loss. The goal
of this loss is to assess that the generated image looks similar to images of
the target domain. More specifically, we use a classifier $C_{cls}$ that is
trained to predict the domain $d$ of a real image $x$:

$\mathcal{L}^{dis}_{cls}=\mathbb{E}_{x,d}[-\log C_{cls}(d|x)]$ (6)

In contrast, this domain-classifier is used to train the generator network
using the following loss:

$\mathcal{L}^{gen}_{cls}=\mathbb{E}_{x,d^{{}^{\prime}}}[-\log
C_{cls}(d^{\prime}|G(x,d^{\prime}))]$ (7)

By optimizing this objective, the generator $G$ tries to generate images that
are classified as the target domain $d^{\prime}$ by the classifier. In this
way, we enforce that the generated images look similar to the real images of
the targeted domain. Practically, the domain-classifier is implemented as an
auxiliary head of the discriminator used to compute the adversarial losses
previously described.

Style-content disentanglement losses. In order to obtain better style-content
disentanglement, these reconstruction losses are completed with the style and
content reconstruction losses from in [12]. More precisely, for a given style
code $s$, if an image is generated from $s$ by the decoder (i.e. $D(R(c,s))$),
then, encoding this image with the style encoder should return a style code
similar to the initial style code $s$:

$\mathcal{L}_{rec}^{s}=\mathbb{E}_{c,s,d}[||E^{s}(D(R(c,s)),d)-s||_{1}]$ (8)

Similarly, content consistency is imposed as follows:

$\mathcal{L}_{rec}^{c}=\mathbb{E}_{c,s,d}[||E^{c}(D(R(c,s)),d)-c||_{1}]$ (9)

where $c=E^{c}(x)$ and s is sampled according to a unitary Gaussian prior
$\mathcal{N}(0,\textbf{I})$. For further mathematical discussion about these
style-content disentanglement losses, please refer to [12].

Total losses. Finally, we obtain two total objective functions for the
generator and the discriminator respectively that can be written as follows:

$\displaystyle\mathcal{L}^{gen}_{total}=$
$\displaystyle\lambda_{x}(\mathcal{L}_{rec}^{cyc}+\mathcal{L}_{rec}^{in})+\lambda_{s}\mathcal{L}^{s}_{rec}+\lambda_{c}\mathcal{L}^{c}_{rec}$
$\displaystyle+\lambda_{cls}\mathcal{L}^{gen}_{cls}+\mathcal{L}_{adv}^{gen}$
(10)
$\mathcal{L}^{dis}_{total}=\mathcal{L}_{adv}^{dis}+\lambda_{cls}L^{dis}_{cls}$
(11)

##  IV Experiments

###  IV-A Implementation details

#### Network Architecture

Let us introduce some notations to describe the network architecture. Let
`kSs-f } denote a $k \times k$ convolution block with stride {\verb s } and
{\verb f } filters. {RkSs-k} denotes residual blocks that with two $k \times
k$ convolution blocks. {\verb AbRkS1-f } denotes an adaptive residual blocks
{\verb b } branches with two $k \times k$ convolution blocks with {\verb f }
filters. {\verb U-f } denotes a nearest-neighbor up-sampling by 2 layer
followed by a $5 \times 5$ convolution with stride 1 and {\verb f } filters.
{\verb GAP } denotes a global average pooling layer and {\verb F`-f denotes a
fully connected layer with ` } `ilters.

Employing this notation, the generator architecture can be written:

  * •

Content encoder: `7S1-64 }, {\verb `4S2-128 ,

        4S2-256 }, {\verb R3S1-256 }, {\verb R3S1-R256 }, {\verb R3S1-R256 }, {\verb R3S1-R256 }
        \item Style encoder: {\verb

7S1-64 , `4S2-128 }, {\verb `4S2-256 , `4S2-256 }, {\verb `4S2-256 ,

        AP }, {\verb FC-50 }
        \item Decoder: {\verb A3R3S1-256 }, {\verb A3R3S1-256 }, {\verb A3R3S1-256 }, {\verb A3R3S1-256 }, {\verb U-128 }, {\verb U-64 }, {\verb C7S1-3 }
    \end{itemize}
    We employ a discriminator with multi-scale architecture with 3 scales using Leaky Relu as in \cite{zhu2017unpaired}. We employ the following discriminator architecture: {\verb C4S2-64 }, {\verb C4S2-128 }, {\verb C4S2-256 }, {\verb C4S2-512 }, {\verb C1S1-1+c(8:scale)s1-14 } (with {\verb scale }= 1,2,3 for 3 different scales)
    \paragraph{Training hyper-parameters}
    We train our model for $20,000$ iterations using the Adam optimizer \cite{kingma2014adam} with $\beta_{1} = 0.5$, $\beta_{2} = 0.999$ starting at learning rate of $0.0001$ and decaying by half after each $50,000$ iterations. In all experiments, we use a batch size of 16 and set the loss weights to $\lambda_{x} = 10$, $\lambda_{c} = 1$, $\lambda_{s} = 1$ and $\lambda_{cls} = 1$. Note that, these weighting parameters are taken from \cite{Huang_2018_ECCV} and \cite{choi2018stargan}. Despite the high number of loss terms, our model does not require further hyper-parameter tuning to converge properly.
    \subsection{Datasets}
    In our experiments, we employ two datasets:\\
    \noindent
    \textbf{Painting-14.} The Painting-14 dataset \cite{anoosheh2018combogan} contains approximately 10,000 paintings in total from 14 different artists from Wikiart.org. We follow the protocol used in \cite{anoosheh2018combogan}. We crop a square in all the images at the center and then resize them to $128 \times 128$. We randomly select 700 images from all artists as test set and use all remaining images for training.
    \noindent
    \textbf{CelebA dataset.} We use CelebFaces Attributes (CelebA) dataset \cite{liu2015faceattributes} which is a large scale face attributes data set with large diversities, large quantities, and rich annotations, including 202,599 face images from 10,177 identities with 40 binary attributes annotations per image. In our experiments, we obtain seven domains using the following attributes: hair color (black, blond, brown), gender (male or female), and age (young or old). The generated images have a single attribute transfer in either hair color (black,  blond,  brown), gender, or age. Concerning image pre-processing , we firstly crop the original $178 \times 218$ images at the center to the size of $178 \times 178$ before resizing them as $128 \times 128$ images. The test set contains 2000 randomly chosen images and all remaining images are used for training.
    \begin{figure*}[t]
    \begin{center}
    %\fbox{\rule{0pt}{1in} \rule{0.9\linewidth}{0pt}}
    \includegraphics[width=0.99\linewidth]{figures/Result_combo_star_our.png}
    \end{center}
    \caption{Qualitative comparison with state-of-the-art on the Painting-14 dataset. The network receives as input the painting in the first columns and outputs the other images using the style indicated in each column.}
    \label{fig:result_1}
    \end{figure*}
    \subsection{Metrics}
    For quantitative evaluation, we employ the Frechet Inception Distance (FID) score \cite{heusel2017gans} which is a popular metric for image generation tasks. The FID measures the similarity between the generated samples and the real ones in terms of statistics using a pre-trained Inception-v3 model\cite{szegedy2016rethinking}. The lower FID score is, the more similar two image distributions are. For evaluating the quality of images generated by generative adversarial networks, lower scores have been shown to correlate well with a higher quality of generated images \cite{heusel2017gans}.
    In order to calculate FID score, we firstly use the pre-trained Inception network to extract features from the 2048 dimensional activations of the Inception-v3 pool3 layer. After that, we assume

aussian feature distributions for both real and generated images. The FID can
be written:

$\displaystyle\mathcal{FID}=$ $\displaystyle\parallel
m_{r}-m_{g}\parallel^{2}_{2}+\mathcal{T}_{r}({C}_{r}+{C}_{g}-2({C}_{r}{C}_{g})^{\frac{1}{2}})$
(12)

where $\mathcal{T}_{r}$ sums up all the diagonal elements, $m_{r}$ an
${C}_{r}$ denote the mean and covariance of the real image features while
$m_{g}$ and ${C}_{g}$ the mean and covariance of generated image features.

###  IV-B Comparison with State-of-the-Art

For the qualitative comparison, we compare our Ada2Net with two literature
methods: ComboGAN and StarGAN. Recent approaches such as DRIT [26], MUNIT [12]
or MSGAN [27] are not included in the comparison since they address the
problem of translation between pairs of domains. On the contrary, in this
work, we address the problem of multi-domain translation in the low resource
regime. In this context, adapting these recent methods using multiple networks
would not be fair. Note that our architecture has a number of FLOPS and
parameters similar to StarGAN at inference time. Figure LABEL:fig:result_1
shows an example of style translation from a Picasso artwork to the remaining
styles. We can observe that both ComboGAN and StarGAN fail to preserve the
content of the input image. Most generated images by ComboGAN change the color
of people to unrealistic colors. For instance, ComboGAN predicts very dark
colors when translating in the styles of Boudin, Cezanne, Chagall, Corrot,
Gauguin, and Monet. StarGAN affects the output image quality with many visible
artifacts and abnormal bright regions. In addition, it also removes details
such as the face of the character in the original painting.

By using the model with content and style extracted separately, our Ada2Net
model produces images that are more diverse and realistic but also preserves
better the content information of the input image. By using our model with
content and style extracted separately, we obtain images that are more diverse
and realistic and also that respect better the content information of the
input image. The qualitative observations above are confirmed by our
quantitative evaluations.

As shown in Table II, the competitor methods obtain very high FID scores
indicating that they generate unrealistic images with low visual quality. More
precisely, the FID score of ComboGAN is quite high (154.3) due to unrealistic
coloring as we can see in Fig. LABEL:fig:result_1. StarGAN mitigates this
issue but, the presence of artifacts, that alter the content, leads to a high
FID score. Interestingly, Ada2Net approach obtains the lowest score for every
single artist’s style. It shows that our adaptive strategy succeeds in
generating images of higher quality than state-of-the-art approaches.

TABLE II: Quantitative comparison with state-of-the-art: we compare the
Frechet Inception Distance (FID) obtained with the different methods. The
lower the better.

Artist | ComboGAN | StarGAN | Ada2Net
---|---|---|---
Beksinski | 179.05 | 146.05 | 111.5
Boudin | 160.3 | 148.54 | 107.07
Burliuk | 142.84 | 136.57 | 105.16
Cezanne | 146.44 | 129.98 | 92.28
Chagall | 142.66 | 109.96 | 102.43
Corot | 173.73 | 165.66 | 106.37
Earle | 178.68 | 172.73 | 147.95
Gauguin | 149.87 | 141.56 | 98.51
Hassam | 140.98 | 141.24 | 98.49
Levitan | 151.83 | 186.92 | 112.44
Monet | 144.96 | 131.48 | 78.81
Picasso | 145.1 | 134.1 | 123.69
Ukiyo-e | 151.81 | 110.22 | 99.97
Van Gogh | 151.91 | 135.45 | 97.25
Average$\pm$Std | 154.3$\pm$12.91 | 142.18$\pm$20.72 | 105.85$\pm$15.38

###  IV-C Ablation study: adaptive graph

Figure 2: Qualitative ablation study: impact of our adaptive network on style
translation on the Painting-14 dataset. The network receives as input the
painting in the first row and outputs the other images using the style
indicated in each column.

Finally, we empirically demonstrate that our architecture can increase the
network performance without significant computational over-head and perform
well on limited resources. We propose to compare three variants of our Ada2Net
architecture:

    * –

No-AdaGraph: In this first baseline, we consider a standard architecture
without adaptive graph. In this case, we use standard Residual blocks with
adaptive inference Normalization layers.

    * –

Ada2Net-label: In this second model, we use a variant of our adaptive graph
strategy where the gates are predicted by the labels. In other words, we use
target labels as inputs of the gating units, instead of the image content.
Therefore, the network topology depends only on the target domain label but
not on the input image content.

    * –

Ada2Net: Finally, we evaluate our full Ada2Net model as described in Sec.III
that uses the image content $c$ to control the network architecture.

In order to simulate the limited resources environment, we reduce the size of
the model by reducing the number of filters in every convolution layer by 4
compared to the model used in the comparison with state of the art. This
network size reduction is applied to the three baselines of the ablation
study.

Qualitative comparison is reported in Figure 2. First, we observe that the
model without adaptive graph suffers from many artifacts. These defects are
especially visible in the Burliuk, Cezanne, Hassam and Van Gogh styles. When
we employ an adaptive network where gating is based on the domain labels, we
obtain smaller artifacts. This improvement is especially clear in the domains
Burliuk, Hassam, and Monet. When the graph is inferred from the content
feature we further reduce the visual artifacts in the output images. A
possible explanation for better performance of this content-based approach is
that, in the case of the Painting-14 dataset, the images are really diverse
both in terms of style and content. Indeed, the images depict a large
diversity of scenes such as portrait, see-side landscape, country-side
landscapes. In order to infer its graph, the network clusters the input space
into sub-regions associated with a specific network graph. A possible
interpretation of the better performance could be that clustering the input
data according to the content is more efficient than clustering according to
visual style. Still, this experiment shows that having a dynamic content-
specific architecture improves the quality of the generated images.

This qualitative analysis is completed by a quantitative analysis reported in
Table III. Indeed, the better qualitative performance of our adaptive approach
is confirmed since the highest FID scores are obtained by _No-AdaGraph_. The
use of an adaptive graph improves the performance for almost every style and
results in an average FID gain of $9.13$. Nevertheless, our full model that
infers the network topology from the content vector leads to the best
performance with an average FID of $134.35$. This quantitative analysis
illustrates the potential of our adaptive strategy for multi-domain image-to-
image translation.

TABLE III: Ablation Study: we evaluate the impact on the performance in term
of Frechet Inception Distance (FID) of the proposed adaptive graph model. The
lower the better.

| No-AdaGraph | Ada2Net-label | Ada2Net
---|---|---|---
Beksinski | 168.63 | 151.79 | 128.94
Boudin | 163.27 | 136.46 | 127.84
Burliuk | 146.14 | 158.01 | 146.17
Cezanne | 135.43 | 158.02 | 129.54
Chagall | 145.35 | 134.89 | 128.51
Corot | 189.43 | 145.1 | 126.47
Earle | 201.91 | 196.26 | 186.77
Gauguin | 164.79 | 145.57 | 131.67
Hassam | 146.69 | 131.7 | 121.51
Levitan | 148.12 | 140.86 | 128.94
Monet | 99.06 | 99.46 | 90.16
Picasso | 188.54 | 177.33 | 167.19
Ukiyo-e | 141.36 | 129.32 | 125.85
Van Gogh | 143.75 | 149.9 | 131.66
Average$\pm$Std | $155.89\pm 25.30$ | $146.76\pm 21.92$ | $\bf 134.35\pm 21.35$

TABLE IV: Ablation Study: we evaluate the impact of each loss term on the performance in term of Frechet Inception Distance (FID). The lower the better. Target Domain | w/o classifier losses | w/o reconstruction losses | w/o content loss | w/o style loss | Ada2Net
---|---|---|---|---|---
Black Hair | 33.46 | 27.72 | 24.10 | 24.94 | 23.51
Blond Hair | 53.02 | 53.32 | 42.09 | 44.14 | 41.82
Brown Hair | 39.77 | 34.51 | 28.09 | 27.50 | 27.47
Gender | 50.33 | 49.61 | 42.06 | 43.36 | 41.73
Age | 43.69 | 39.82 | 35.48 | 36.12 | 34.94
Average$\pm$Std | $44.05\pm 7.91$ | $40.99\pm 10.55$ | $34.37\pm 8.14$ | $35.21\pm 8.83$ | $\bf 33.89\pm 8.28$

We complete our ablation study by reporting additional results on CelebA
dataset for the attribute transfer task. Qualitative results are reported in
Figure 3. The top row shows the original input images while the next three
rows provide a qualitative comparison between the baselines _No-AdaGraph_ ,
_Ada 2Net-$label$_ and _Ada 2Net_. The adaptive approach of _Ada 2Net_ shows a
significant improvement since the generated images have fewer artifacts as
well as sharper details. The attributes are also better translated. For
instance, _No-AdaGraph_ does not succeed in transferring parts of the hair
from blonde to black. Besides, _Ada 2Net-$label$_ generates important
artifacts in the _Age_ column. We note that the benefit of our adaptive graph
model is smaller on this dataset than on the Painting-14 dataset. A possible
explanation is that face images have a small diversity than paintings, and
consequently, starGAN can handle all the face diversity pretty well.

Figure 3: Qualitative ablation study on the CelebA dataset for attribute
transfer. Figure 4: Qualitative ablation study on the CelebA dataset for the
impact of each loss term.

###  IV-D Ablation study: training losses

In this section, we perform an ablation study to show the impact of each loss
term. We compare four baselines of _Ada 2Net_ where we remove the classifier
losses (Eqs.(6) and (7)), the style loss (Eq.(8)), the content loss (Eq.(9))
and the reconstruction losses (Eqs.(4) and (5)), respectively. Qualitative
results are shown in Fig. 4. First, we observe that, when we do not use the
domain classifier (second row), the network generates realistic images but do
not transfer the attribute. For example, in the first two columns, the women’s
hair are similar to the input images. When removing the style or the content
loss, artifacts appear leading to poorer image quality. The artifacts are
clearly visible in the _Blond Hair_ and _Gender_ columns. The artifacts are
even stronger when the reconstruction losses are not used. These results are
confirmed by the quantitative evaluation reported in Table IV. The model
without domain-classifier reaches the highest FID showing that the generated
images do not follow the distribution of the target domain. The average FID is
also higher when the reconstruction losses are not used. Finally, the full
_Ada 2Net_ model leads to the best FID score for every attribute. These
consistent results show the positive impact of each loss term.

##  V Conclusions

In this work, we presented Ada2Net, an adaptive method for the problem of
multi-domain image-to-image translation. In our literature review, we observed
that state-of-the-art methods require a large and deep model in order to
handle the visual diversity of multiple domains. When computational resources
are limited, increasing the network size may be problematic. Therefore, we
introduced a novel network architecture with an adaptive graph structure. At
inference time, the network estimates its own topology on the fly by selecting
specific sub-network branches. The branch selection is implemented using
Gumbel-Softmax to allow end-to-end training. This approach leads to an
adjustable increase in number of parameters while preserving an almost
constant computational cost. Our evaluation shows that Ada2Net can generate
better images than literature methods. As future works, we plan to investigate
this adaptive network for other structured problems such as instance
segmentation or depth estimation [28].

## References

    * [1] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _CVPR_ , 2017.
    * [2] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _ICCV_ , 2017\.
    * [3] Z. Geng, C. Cao, and S. Tulyakov, “3d guided fine-grained face manipulation,” _CVPR_ , 2019.
    * [4] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _ICCV_ , 2017.
    * [5] C. Couprie, P. Luc, and J. Verbeek, “Joint future semantic and instance segmentation prediction,” in _ECCV_ , 2018.
    * [6] A. Pilzer, S. Lathuiliere, N. Sebe, and E. Ricci, “Refine and distill: Exploiting cycle-inconsistency and knowledge distillation for unsupervised monocular depth estimation,” in _CVPR_ , 2019.
    * [7] A. Anoosheh, E. Agustsson, R. Timofte, and L. Van Gool, “Combogan: Unrestrained scalability for image domain translation,” in _CVPR Workshops_ , 2018.
    * [8] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “Stargan: Unified generative adversarial networks for multi-domain image-to-image translation,” in _CVPR_ , 2018.
    * [9] H. Fu, M. Gong, C. Wang, K. Batmanghelich, K. Zhang, and D. Tao, “Geometry-consistent generative adversarial networks for one-sided unsupervised domain mapping,” in _CVPR_ , 2019.
    * [10] A. Siarohin, S. Lathuilière, S. Tulyakov, E. Ricci, and N. Sebe, “First order motion model for image animation,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 7137–7147.
    * [11] ——, “Animating arbitrary objects via deep motion transfer,” in _CVPR_ , 2019.
    * [12] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz, “Multimodal unsupervised image-to-image translation,” in _ECCV_ , 2018.
    * [13] H.-Y. Lee, H.-Y. Tseng, J.-B. Huang, M. K. Singh, and M.-H. Yang, “Diverse image-to-image translation via disentangled representations,” in _ECCV_ , 2018.
    * [14] A. Siarohin, S. Lathuilière, E. Sangineto, and N. Sebe, “Appearance and pose-conditioned human image generation using deformable gans,” _IEEE TPAMI_ , 2019.
    * [15] M. Tomei, M. Cornia, L. Baraldi, and R. Cucchiara, “Art2real: Unfolding the reality of artworks via semantically-aware image-to-image translation,” in _CVPR_ , 2019.
    * [16] A. Veit and S. Belongie, “Convolutional networks with adaptive inference graphs,” in _ECCV_ , 2018.
    * [17] D. Abati, J. M. Tomczak, T. Blankevoort, S. Calderara, R. Cucchiara, and B. E. Bejnordi, “Conditional channel gated networks for task-aware continual learning,” in _CVPR_ , 2020.
    * [18] E. Jang, S. Gu, and B. Poole, “Categorical reparametrization with gumbel-softmax,” in _ICLR_ , 2017.
    * [19] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-resolution image synthesis and semantic manipulation with conditional gans,” in _CVPR_ , 2018.
    * [20] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
    * [21] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in _ICCV_ , 2017.
    * [22] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _ICLR_ , 2015.
    * [23] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in _ICCV_ , 2015.
    * [24] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in _NeurIPS_ , 2017.
    * [25] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _CVPR_ , 2016.
    * [26] H.-Y. Lee, H.-Y. Tseng, J.-B. Huang, M. Singh, and M.-H. Yang, “Diverse image-to-image translation via disentangled representations,” in _ECCV_ , 2018.
    * [27] Q. Mao, H.-Y. Lee, H.-Y. Tseng, S. Ma, and M.-H. Yang, “Mode seeking generative adversarial networks for diverse image synthesis,” in _CVPR_ , 2019.
    * [28] D. Xu, E. Ricci, W. Ouyang, X. Wang, N. Sebe _et al._ , “Monocular depth estimation using multi-scale continuous crfs as sequential deep networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 6, pp. 1426–1440, 2018.