# InstaFormer: Instance-Aware Image-to-Image Translation with Transformer

Soohyun Kim Jongbeom Baek Jihye Park Gyeongnyeon Kim Seungryong Kim
Korea University, Seoul, Korea
<EMAIL_ADDRESS>Corresponding
author

###### Abstract

We present a novel Transformer-based network architecture for instance-aware
image-to-image translation, dubbed InstaFormer, to effectively integrate
global- and instance-level information. By considering extracted content
features from an image as tokens, our networks discover global consensus of
content features by considering context information through a self-attention
module in Transformers. By augmenting such tokens with an instance-level
feature extracted from the content feature with respect to bounding box
information, our framework is capable of learning an interaction between
object instances and the global image, thus boosting the instance-awareness.
We replace layer normalization (LayerNorm) in standard Transformers with
adaptive instance normalization (AdaIN) to enable a multi-modal translation
with style codes. In addition, to improve the instance-awareness and
translation quality at object regions, we present an instance-level content
contrastive loss defined between input and translated image. We conduct
experiments to demonstrate the effectiveness of our InstaFormer over the
latest methods and provide extensive ablation studies.

##  1 Introduction

For a decade, image-to-image translation (I2I), aiming at translating an image
in one domain (i.e., source) to another domain (i.e., target), has been
popularly studied, to the point of being deployed in numerous applications,
such as style transfer [15, 22], super-resolution [12, 32], inpainting [46,
25], or colorization [64, 65].

In particular, most recent works have focused on designing better disentangled
representation to learn a multi-modal translation from unpaired training data
[35, 23, 45]. While they have demonstrated promising results, most of these
methods only consider the translation on an _whole_ image, and do not account
for the fact that an image often contain many object instances of various
sizes, thus showing the limited performance at content-rich scene translation,
e.g., driving scene, which is critical for some downstream tasks, such as
domain adaptive object detection [3], that require well-translated object
instances.

To address the aforementioned issues, some methods [51, 3, 28] seek to
explicitly consider an object instance in an image within deep convolutional
neural networks (CNNs). This trend was initiated by instance-aware I2I (INIT)
[51], which treats the object instance and global image separately. Following
this [51], some variants were proposed, e.g., jointly learning translation
networks and object detection networks, called detection-based unsupervised
I2I (DUNIT) [3], or using an external memory module, called memory-guided
unsupervised I2I (MGUIT) [28]. While these methods improve an instance-
awareness to some extent, they inherit limitation of CNN-based architectures
[51, 3, 28], e.g., local receptive fields or limited encoding of relationships
or interactions between pixels or patches within an image which are critical
in differentiating an object instance from an whole image and boosting its
translation.

Input image (sunny)Translated image (rainy)

Figure 1:  Results of InstaFormer for instance-aware image-to-image
translation. Our InstaFormer effectively considers global- and instance-level
information with Transformers, which enables high quality instance-level
translation.

To tackle these limitations, for the first time, we present to utilize
Transformer [55] architecture within I2I networks that effectively integrates
global- and instance-level information present in an image, dubbed
InstaFormer. We follow common disentangled representation approaches [35, 23]
to extract both content and style vectors. By considering extracted content
features from an image as tokens, our Transformer-based aggregator mixes them
to discover global consensus by considering global context information through
a self-attention module, thus boosting the instance-awareness during
translation. In addition, by augmenting such tokens by an instance-level
feature extracted from the global content feature with respect to bounding box
information, our framework is able to learn an interaction between not only
object instance and global image, but also different instances, followed by a
position embedding technique to consider both global- and instance-level
patches at once, which helps the networks to better focus on the object
instance regions. We also replace layer normalization (LayerNorm) [1] in
Transformers with adaptive instance normalization (AdaIN) [22] to facilitate a
multi-modal translation with extracted or random style vectors. Since
aggregating raw content and style vectors directly with Transformers requires
extremely large computation [13, 31], we further propose to apply a
convolutional patch embedding and deconvolutional module at the beginning and
end of our Transformer-based aggregator. In addition, to improve the instance-
awareness and quality of translation images at object regions, we present an
instance-level content contrastive loss defined between input and translated
images.

In experiments, we demonstrate our framework on several benchmarks [16, 9, 51]
that contain content-rich scenes. Experimental results on various benchmarks
prove the effectiveness of the proposed model over the latest methods for
instance-aware I2I. We also provide an ablation study to validate and analyze
components in our model.

##  2 Related Work

#### Image-to-Image Translation.

While early efforts for I2I are based on supervised learning [27], most recent
state-of-the-arts focus on unpaired settings [63, 69, 2, 67, 14, 39]. CycleGAN
[70] attempts this by proposing a cycle-consistency loss which has been one of
standard losses for unpaired I2I. Inspired by CycleGAN, numerous methods
utilize cycle-consistency [33, 61, 7, 21, 35, 23], and they can be largely
divided into uni-modal models [38, 61, 63] and multi-modal models [23, 35, 7]
methods. Specifically, MUNIT[23] assumes that an image representation can be
disentangled into a domain-specific style and a domain-invariant content
representation and uses these disentangled latent features with cycle-
consistency to generate the translations. However, the content in translated
image can be easily distorted, and cycle mapping requires multiple generators
and discriminators. To address these, CUT [45]and F-LSeSim [67] propose novel
losses inspired by infoNCE [44] to directly compute distance between input and
translated images in an one-side framework without cycle-consistency. However,
they still have shown limited performance to encode an object-awareness at the
translated image.

#### Instance-Aware Image-to-Image Translation.

Some methods attempted to address the aforementioned issues [43, 51, 3, 28].
INIT [51] attempted to translate the whole image and object instances
independently. DUNIT [3] proposed to further train detection module and
adopted instance-consistency loss for object-awareness. MGUIT [28] utilizes
bounding box to read and write class-wise memory module, and has access to
class-aware features on memory at test-time. The aforementioned methods
inherit limitation of CNN-based architecture, e.g., local receptive fields or
limited encoding of relationships or interactions within an image [70, 35,
23].

#### Vision Transformers and Image Generation

Recently, Vision Transformers (ViT) have shown to attain highly competitive
performance for a wide range of vision applications, such as image
classification [13, 54, 57, 11], object detection [5, 71, 10], and semantic
segmentation [68, 60]. Inspired by ViT [13], some improvements are made to
improve the computational complexity [31, 57, 40, 56]. For example, Swin
Transformer [40] proposes relative position biases, and restricts self-
attentioncomputation within shifted windows. MLP-Mixer [52] suggests to
replace self-attention with an MLP, achieving memory efficiency and
competitive performance [41, 53, 37]. In this paper, we introduce ViT-based
aggregator to further enhance to learn instance-awareness by aggregating
information from local region as well as global image.

On the other hands, there exist several efforts to adapt Vision Transformers
to image generation tasks [30, 66, 6, 24, 36]. As seminal work, TransGAN [30]
first presents a GAN structure using pure Transformer, but has only validated
on low-resolution images. [66] has achieved success on generating high-
resolution images. [24] leverages Transformers to build the bipartite
structure to allow long-range interactions. To our best knowledge, our work is
the first attempt to adopt Transformers in instance-aware image translation.

(a) Architecture

(b) ViT Encoder Block

Figure 2: Network configuration: (a) overall architecture for image-to-image
translation, (b) ViT encoder block in details. Our networks consist of content
encoder, Transformer encoder, and generator. The gray background represents
the test phase, where we have no access on object instance bounding box (Best
viewed in color).

##  3 Methodology

###  3.1 Overview

Our approach aims to learn a multi-modal mapping between two domains
$\mathcal{X}\subset\mathbb{R}^{H\times W\times 3}$ and
$\mathcal{Y}\subset\mathbb{R}^{H\times W\times 3}$ without paired training
data, but with a dataset of unpaired instances
$\mathrm{X}=\\{\mathbf{x}\in\mathcal{X}\\}$ and
$\mathrm{Y}=\\{\mathbf{y}\in\mathcal{Y}\\}$. Especially, we wish to model such
a mapping function to have an ability that jointly accounts for whole image
and object instances. Unlike conventional I2I methods [35, 23, 17, 51, 3, 28]
that were formulated in a two-sided framework to exploit a cycle-consistency
constraint, which often generates some distortions on the translated images
and requires auxiliary networks for inverse mapping [70], we formulate our
approach in an one-sided framework [45].

In specific, as illustrated in Fig. 2, our framework, dubbed InstaFormer,
consists of content encoder $\mathcal{E}$ and generator $\mathcal{G}$, similar
to [45, 67], and additional encoder $\mathcal{T}$ with Transformers [55] to
improve the instance-awareness by considering global consensus between whole
image and object instances. To translate an image $\mathbf{x}$ in domain
$\mathcal{X}$ to domain $\mathcal{Y}$, our framework first extracts a content
feature map $\mathbf{c}=\mathcal{E}(\mathbf{x})\in{\mathbb{R}}^{h\times
w\times l_{c}}$ from $\mathbf{x}$, with height $h$, width $w$, and $l_{c}$
channels, and randomly draws a style latent code
$\mathbf{s}\in\mathbb{R}^{1\times 1\times{l_{s}}}$ from the prior distribution
$q(\mathbf{s})\sim\mathcal{N}(0,\mathbf{I})$ to achieve a multi-modal
translation. Instead of directly feeding $\mathbf{c}$ and $\mathbf{s}$ to the
generator $\mathcal{G}$, as done in the literature [35, 23], we aggregate
information in the content $\mathbf{c}$ to discover global consensus between
the global image and object instances in a manner that we first extract an
object instance content vector $\mathbf{c}^{\mathrm{ins}}_{i}$ for $i$-th
object bounding box with parameters $B_{i}=[x_{i},y_{i},h_{i},w_{i}]$, where
$(x_{i},y_{i})$ represent a center point, and $h_{i}$ and $w_{i}$ represent
height and width of the box and $i\in 1,...,N$ where $N$ is the number of
instance, and then mix
$\\{\mathbf{c},\\{\mathbf{c}^{\mathrm{ins}}_{i}\\}_{i},\mathbf{s}\\}$ through
the proposed Transformer module $\mathcal{T}$ to extract global embedding
$\mathbf{u}$ and instance embedding $\mathbf{u}^{\mathrm{ins}}_{i}$, which are
independently used to generate global-level translated image
$\hat{\mathbf{y}}=\mathcal{G}(\mathbf{u})\in\mathbb{R}^{h\times w\times 3}$
and instance-level translated images
$\hat{\mathbf{y}}^{\mathrm{ins}}_{i}=\mathcal{G}(\mathbf{u}^{\mathrm{ins}}_{i})\in\mathbb{R}^{h_{i}\times
w_{i}\times 3}$. In our framework, during training, we have access to the
ground-truth object bounding boxes, while we do not access them at test-time.

To train our networks, we first use an adversarial loss defined between a
translated image $\hat{\mathbf{y}}$ and a real image $\mathbf{y}$ from
$\mathcal{Y}$ with discriminators, and a global content contrastive loss
defined between $\mathbf{x}$ and $\hat{\mathbf{y}}$ to preserve the global
content. To improve the disentanglement ability for content and style,
following [23], we also use both image reconstruction loss and style
reconstruction loss by leveraging an additional style encoder for
$\mathcal{Y}$. To improve the instance-awareness and the quality of
translation images at object instance regions, we newly present an instance-
level content contrastive loss between $\mathbf{x}$ and $\hat{\mathbf{y}}$.

(a) content image

(b) translated image

(c) w/o $\mathcal{L}_{\mathrm{NCE}}^{\mathrm{ins}}$ for A

(d) w/o $\mathcal{L}_{\mathrm{NCE}}^{\mathrm{ins}}$ for B

(e) w/ $\mathcal{L}_{\mathrm{NCE}}^{\mathrm{ins}}$ for A

(f) w/ $\mathcal{L}_{\mathrm{NCE}}^{\mathrm{ins}}$ for B

Figure 3: Visualization of learned self-attention. For (a) content image
containing instances A and B, our networks generate (b) translated image,
considering attention maps (c,d) without
$\mathcal{L}_{\mathrm{NCE}}^{\mathrm{ins}}$ and (e,f) with
$\mathcal{L}_{\mathrm{NCE}}^{\mathrm{ins}}$ for instance A, B, respectively.

###  3.2 Content and Style Mixing with Transformers

Most existing I2I methods [35, 23, 51, 45, 3, 28] attempted to aggregate a
content feature map with deep CNNs with residual connections, which are often
called residual blocks, often inserted between encoder and generator networks.
They are thus limited in the sense that they inherit limitation of CNN-based
architecture, e.g., local receptive fields or limited encoding of
relationships or interactions between pixels and patches within an image [35,
23, 51]. In instance-aware I2I task, enlarging the receptive fields and
encoding an interaction between objects and global image may be of prime
importance. For instance, if an image contains a car object on the road, using
the context information of not only global background, e.g., road, but also
other instances, e.g., other cars or person, would definitely help to
translate the image more focusing on the instance, but existing CNN-based
methods [35, 23, 45] would limitedly handle this.

To overcome this, we present to utilize Transformer architecture [55] to
enlarge the receptive fields and encode the interaction between features for
instance-aware I2I. To this end, extracted content vector
$\mathbf{c}\in{\mathbb{R}}^{h\times w\times l_{c}}$ from $\mathbf{x}$ can be
flattened as a sequence $\mathbf{c}^{\prime}=\text{Reshpae}(\mathbf{c})$ with
the number of tokens $hw$ and channel $l_{c}$, which can be directly used as
input for Transformers. However, this requires extremely high computational
complexity due to the huge number of tokens $hw$, e.g., full HD translation.

#### Patch Embedding and Expanding.

To address this issue, inspired by a patch embedding in ViT [13], we first
apply sequential convolutional blocks to reduce the spatial resolutions.
Instead of applying a single convolution in ViT [13] for extracting non-
overlapping patches, we use sequential overlapped convolutional blocks to
improve the stability of training while reducing the number of parameters
involved [59]. We define this process as follows:

$\mathbf{p}=\mathrm{Conv}(\mathbf{c})\in{\mathbb{R}}^{(h/k)\times(w/k)\times{l^{\prime}_{c}}},$
(1)

where $k\times k$ is the stride size of convolutions, and $l^{\prime}_{c}$ is
a projected channel size. After feed-forwarding Transformer blocks such that
$\mathbf{z}=\mathcal{T}(\mathbf{p})\in{\mathbb{R}}^{(h/k)\times(w/k)\times{l^{\prime}_{c}}}$,
downsampled feature map $\mathbf{z}$ should be upsampled again with additional
deconvolutional blocks, which are symmetric architectures to the convolutions,
defined as follows:

$\mathbf{u}=\mathrm{DeConv}(\mathbf{z})\in\mathbb{R}^{\mathnormal{h\times
w\times{l_{c}}}}.$ (2)

In addition, for a multi-modal translation, we leverage a style code vector
$\mathbf{s}\in\mathbb{R}^{1\times 1\times{l_{s}}}$, and thus this should be
considered during mixing with Transformers [55]. Conventional methods [22, 35,
23] attempted to mix content and style vectors using either concatenation [35]
or AdaIN [22]. In our framework, by slightly changing the normalization module
in Transformers, we are capable of simultaneously mixing content and style
vectors such that $\mathcal{T}(\mathbf{p},\mathbf{s})$.

Any forms of Transformers [13, 62, 52, 40, 57] can be considered as a
candidate in our framework, and in experiments, ViT-like [13] architecture is
considered for $\mathcal{T}$. In the following, we explain the details of
Transformer modules.

#### Transformer Aggregator.

In order to utilize Transformer to process content patch embeddings
$\mathbf{p}$, our work is built upon the ViT encoder, which is composed of an
multi-head self-attention (MSA) layer and a feed-forward MLP with GELU [55],
where normalization layers are applied before both parts. Especially, for I2I,
we adopt AdaIN instead of LayerNorm [1] to control the style of the output
with the affine parameters from style vector $\mathbf{s}$ and to enable
multimodal outputs. In specific, content patch embedding $\mathbf{p}$ is first
reshaped, and position embedding is achieved such that

$\mathbf{z}_{0}=\mathrm{Reshape}({\mathbf{p}})+{\mathbf{E}\in{\mathbb{R}}^{(h/k\cdot
w/k)\times l_{c}^{\prime}}},$ (3)

where $\mathbf{E}$ represents a position embedding [55], which will be
discussed in the following.

These embedded tokens $\mathbf{z}_{0}$ are further processed by the sequential
Transformer encoder blocks as follows:

$\begin{split}&\mathbf{z}^{\prime}_{t}=\mathrm{MSA}\left({\mathrm{AdaIN}\left({\mathbf{z}}_{t-1},\mathbf{s}^{\prime}\right)}\right)+{\mathbf{z}}_{t-1},\\\
&{\mathbf{z}}_{t}=\mathrm{MLP}\left({\mathrm{AdaIN}\left(\mathbf{z}^{\prime}_{t},\mathbf{s}^{\prime}\right)}\right)+\mathbf{z}^{\prime}_{t},\end{split}$
(4)

where $\mathbf{z}^{\prime}_{t}$ and ${\mathbf{z}}_{t}$ denote the output of
MSA and MLP modules for $t$-th block respectively and $t\in 1,...,T$,
respectively, $\textbf{s}^{\prime}$ indicates AdaIN parameters extracted from
$S$. After $L$ Transformer modules, followed by reshaping to original
resolution, we finally achieve the output of Transformer block $\mathcal{T}$
such that $\mathbf{z}_{T}=\mathcal{T}(\mathbf{p},\mathbf{s}^{\prime})$. As
exemplified in Fig. 3, our learned self-attention well considers the
interaction between object instances and global image.

###  3.3 Instance-Aware Content and Style Mixing

So far we discussed a method for content and style mixing with Transformers
[13]. This framework can improve the translation quality especially at
instance regions to some extent, but the nature of irregular shape of object
instances may hinder the performance boosting of our framework. In particular,
global-level aggregation itself is limited to capture details of a tiny object
and it is not always guaranteed that an object is located in a single regular
patch. To overcome this, we present a novel technique to aggregate _instance_
-level content features and _global_ -level content features simultaneously,
which enables the model to pay more attention to the relationships between
global scenes and object instances.

In specific, given ground-truth bounding boxes with parameters $B_{i}$, we
extract instance-level content feature maps through through ROI Align [19]
module defined as follows:

${\mathbf{c}}^{\mathrm{ins}}_{i}=\mathrm{RoIAlign}({\mathbf{c}};{B}_{i})\in{\mathbb{R}}^{k\times
k\times{l_{c}}},$ (5)

where $k\times k$ is a fixed spatial resolution. This can be further processed
with the convolutional blocks as proposed above such that

${\mathbf{p}}^{\mathrm{ins}}_{i}=\mathrm{Conv}({\mathbf{c}}^{\mathrm{ins}}_{i})\in{\mathbb{R}}^{1\times
1\times{l^{\prime}_{c}}}.$ (6)

In our framework, by concatenating ${\mathbf{p}}$ and
${\mathbf{p}}^{\mathrm{ins}}_{i}$, we build a new input for Transformer
$\hat{\mathbf{z}}_{0}$ such that

$\hat{\mathbf{z}}_{0}=\mathrm{Reshape}(\mathrm{Cat}({\mathbf{p}},\\{{\bf{p}}^{\mathrm{ins}}_{i}\\}_{i}))+\hat{\mathbf{E}}\in{\mathbb{R}}^{(h/k\cdot
w/k+N)\times{l^{\prime}_{c}}},$ (7)

where $\mathrm{Cat}(\cdot,\cdot)$ denotes a concatenation operator and
$\hat{\mathbf{E}}$ is a corresponding positional embedding. Transformer blocks
are then used to process $\hat{\mathbf{z}}_{0}$ similarly to above to achieve
$\hat{\mathbf{z}}_{T}$, which is decomposed into ${\mathbf{z}}_{T}$ and
${\mathbf{z}}^{\mathrm{ins}}_{T,i}$.

Figure 4: Illustration of building position embedding for regular patches and
instance-level patches.

###  3.4 Instance-Aware Position Embedding

Since Transformer [55] block itself does not contain positional information,
we add positional embedding E as described above. To this end, our framework
basically utilizes existing technique [13], but the main difference is that
our proposed strategy enables simultaneously considering regularly-partitioned
patches ${\mathbf{p}}$ and instance patches ${\mathbf{p}}^{\mathrm{ins}}_{i}$
in terms of their spatial relationships.

The deep networks are often biased towards learning lower frequency functions
[47], so we use high frequency functions to alleviate such bias. We denote
$\gamma(\cdot)$ as a sinusoidal mapping into $\mathbb{R}^{2K}$ such that
$\gamma(\textit{a})=(\mathrm{sin}(2^{0}\pi\textit{a}),\mathrm{cos}(2^{0}\pi\textit{a}),...,\mathrm{sin}(2^{K-1}\pi\textit{a}),\mathrm{cos}(2^{K-1}\pi\textit{a}))$
for a scalar $a$.

In specific, as a global feature map is divided into regular girds, each
regular patch can be represented to have center coordinates $(x_{g},y_{g})$
with patch width $w_{g}$ and height $h_{g}$ of regular size for $g$-th patch
$\mathbf{p}(g)$. After embedding for each information through $\gamma(\cdot)$
and concatenating along the channel axis, it is further added to the patch
embedded tokens.

$\mathbf{E}=\mathrm{Cat}(\gamma(x_{g}),\gamma(y_{g}),\gamma(w_{g}),\gamma(h_{g}))$
(8)

Unlike regular patches, which have the same size of width and height for each,
instance patches contain positional information of corresponding bounding
boxes, which contain the centerpoint coordinates $(x_{i},y_{i})$ and width and
height ($w_{i}$, $h_{i}$). Instance-wise $\mathbf{E}$ is denoted as:

$\mathbf{E}^{\mathrm{ins}}=\mathrm{Cat}(\gamma(x_{i}),\gamma(y_{i}),\gamma(w_{i}),\gamma(h_{i})).$
(9)

Then $\hat{\mathbf{E}}=\mathrm{Cat}(\mathbf{E},\mathbf{E}^{\mathrm{ins}})$.
Fig. 4 illustrates the difference in how regular patch and instance patch are
handled.

Figure 5: Illustration of global content loss and instance-level content loss.
Blue box indicates a positive sample, while yellow box means a negative sample
(Best viewed in color).

###  3.5 Loss Functions

#### Adversarial Loss.

Adversarial loss aims to minimize the distribution discrepancy between two
different features [18, 42]. We adopt this to learn the translated image
$\hat{\mathbf{y}}$ to be similar to an image $\mathbf{y}$ from $\mathcal{Y}$
defined such that

$\begin{split}\mathcal{L}_{\mathrm{GAN}}=&\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}[\mathrm{log}(1-\mathcal{D}(\hat{\mathbf{y}}))]+\mathbb{E}_{\mathbf{y}\sim\mathcal{Y}}[\mathrm{log}\,\mathcal{D}(\mathbf{y})],\end{split}$
(10)

where $\mathcal{D}(\cdot)$ is the discriminator.

#### Global Content Loss.

To define the content loss between $\mathbf{x}$ and $\hat{\mathbf{y}}$, we
exploit infoNCE loss [44], defined as

$\begin{split}&\ell(\hat{\mathbf{v}},\mathbf{v}^{+},\mathbf{v}^{-})=\\\
&\mathrm{-log}\left[\frac{\mathrm{exp}(\hat{\mathbf{v}}\cdot\mathbf{v}^{+}/\tau)}{\mathrm{exp}(\hat{\mathbf{v}}\cdot\mathbf{v}^{+}/\tau)+\sum_{\mathrm{n}=1}^{\mathrm{N}}\mathrm{exp}(\hat{\mathbf{v}}\cdot\mathbf{v}^{-}_{\mathrm{n}}/\tau)}\right],\end{split}$
(11)

where $\tau$ is the temperature parameter, and $\mathbf{v^{+}}$ and
$\mathbf{v^{-}}$ represent positive and negative for $\mathbf{\hat{v}}$.

We set _pseudo_ positive samples between input image ${\mathbf{x}}$ and
translated image $\hat{\mathbf{y}}$. For the content feature from translated
image $\hat{\mathbf{c}}(s)$ = $\mathcal{E(\hat{\mathbf{y}})}$, we set positive
patches $\mathbf{c}(s)$, and negative patches $\mathbf{c}(S\setminus s)$ from
$\mathbf{x}$, where $S\setminus s$ represents indexes except for $s$,
following [45, 67]. Global content loss function is then defined as

$\begin{split}\mathcal{L}_{\mathrm{NCE}}^{\mathrm{global}}=\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}\sum_{l}\sum_{s}\ell(\hat{\mathbf{c}}_{l}(s),{\mathbf{c}}_{l}(s),{\mathbf{c}}_{l}({S\setminus
s})),\end{split}$ (12)

where $\mathbf{c}_{l}$ is feature at $l$-th level, $s\in\\{1,2,...,S_{l}\\}$
and $S_{l}$ is the number of patches in each $l$-th layer.

| CycleGAN [70] | UNIT [38] | MUNIT [23] | DRIT [35] | INIT [51] | DUNIT [3] | MGUIT [28] | InstaFormer
---|---|---|---|---|---|---|---|---
| CIS | IS | CIS | IS | CIS | IS | CIS | IS | CIS | IS | CIS | IS | CIS | IS | CIS | IS
sunny$\rightarrow$night | 0.014 | 1.026 | 0.082 | 1.030 | 1.159 | 1.278 | 1.058 | 1.224 | 1.060 | 1.118 | 1.166 | 1.259 | 1.176 | 1.271 | 1.200 | 1.404
night$\rightarrow$sunny | 0.012 | 1.023 | 0.027 | 1.024 | 1.036 | 1.051 | 1.024 | 1.099 | 1.045 | 1.080 | 1.083 | 1.108 | 1.115 | 1.130 | 1.115 | 1.127
sunny$\rightarrow$rainy | 0.011 | 1.073 | 0.097 | 1.075 | 1.012 | 1.146 | 1.007 | 1.207 | 1.036 | 1.152 | 1.029 | 1.225 | 1.092 | 1.213 | 1.158 | 1.394
sunny$\rightarrow$cloudy | 0.014 | 1.097 | 0.081 | 1.134 | 1.008 | 1.095 | 1.025 | 1.104 | 1.040 | 1.142 | 1.033 | 1.149 | 1.052 | 1.218 | 1.130 | 1.257
cloudy$\rightarrow$sunny | 0.090 | 1.033 | 0.219 | 1.046 | 1.026 | 1.321 | 1.046 | 1.249 | 1.016 | 1.460 | 1.077 | 1.472 | 1.136 | 1.489 | 1.141 | 1.585
Average | 0.025 | 1.057 | 0.087 | 1.055 | 1.032 | 1.166 | 1.031 | 1.164 | 1.043 | 1.179 | 1.079 | 1.223 | 1.112 | 1.254 | 1.149 | 1.353

Table 1: Quantitative evaluation on INIT dataset [51]. For evaluation, we
perform bidirectional translation for each domain pair. We measure CIS [23]
and IS [50](higher is better). Our results shows the best results in terms of
CIS and IS.

#### Instance-level Content Loss.

To improve the instance-awareness and the quality of translation images at
object regions, we newly present an instance-level content contrastive loss.

Our instance-level content loss is then defined such that

$\begin{split}\mathcal{L}_{\mathrm{NCE}}^{\mathrm{ins}}=\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}\sum_{i}\sum_{m}\ell(\hat{\mathbf{c}}_{i}^{\mathrm{ins}}(m),{\mathbf{c}}_{i}^{\mathrm{ins}}(m),{\mathbf{c}}_{i}^{\mathrm{ins}}({M\setminus
m})),\end{split}$ (13)

where $m\in\\{1,2,...,M_{i}\\}$ and $M_{i}$ is the number of patches at each
instance. Fig. 5 illustrates how our suggested content losses work, with the
procedure to define positive and negative samples.

#### Image Reconstruction Loss.

We additionally make use of image reconstruction loss to help disentanglement
between content and style. For regularization, we use a reconstruction loss to
ensure that our $\mathcal{G}$ can reconstruct an image for domain
$\mathcal{Y}$. To be specific, $\mathbf{y}$ is fed into $\mathcal{E}$ and
style encoder $\mathcal{S}$ to obtain a content feature map
$\mathbf{c^{\mathcal{Y}}}=\mathcal{E}(\mathbf{y})$ and a style code
$\mathbf{s^{\mathcal{Y}}}=\mathcal{S}(\mathbf{y})$. We then compare the
reconstructed image
${\mathcal{G}(\mathcal{T}(\mathbf{c^{\mathcal{Y}}},\mathbf{s^{\mathcal{Y}}}))}$
for domain $\mathcal{Y}$ with $\mathbf{y}$ as follows:

$\mathcal{L}_{\mathrm{recon}}^{\mathrm{img}}=\,\mathbb{E}_{\mathbf{y}\sim\mathcal{Y}}[\|{\mathcal{G}(\mathcal{T}(\mathbf{c^{\mathcal{Y}}},\mathbf{s^{\mathcal{Y}}}))-\,\mathbf{y}}\|_{1}].\vspace{-10pt}$
(14)

#### Style Reconstruction Loss.

In order to better learn disentangled representation, we compute L1 loss
between style code from the translated image and randomly generated style code
in order to enable mapping generated style features to Gaussian distribution
such that

$\begin{split}\mathcal{L}_{\mathrm{recon}}^{\mathrm{style}}=\mathbb{E}_{\mathbf{x}\sim\mathcal{X},\mathbf{y}\sim\mathcal{Y}}[\|{\mathcal{S}(\hat{\mathbf{y}})-\,{\mathbf{s}}}\|_{1}].\vspace{-10pt}\end{split}$
(15)

#### Total Loss.

The total loss function is as follows:

$\begin{split}\min\limits_{{\mathcal{E}},\mathcal{G},\mathcal{S}}\max\limits_{\mathcal{D}}\mathcal{L}(\mathcal{E},\mathcal{G},\mathcal{D})=&\mathcal{L}_{\mathrm{GAN}}+\lambda^{\mathrm{glob}}\mathcal{L}_{\mathrm{NCE}}^{\mathrm{global}}+\lambda^{\mathrm{ins}}\mathcal{L}_{\mathrm{NCE}}^{\mathrm{ins}}\\\
&+\lambda^{\mathrm{style}}\mathcal{L}_{\mathrm{recon}}^{\mathrm{style}}+\lambda^{\mathrm{img}}\mathcal{L}_{\mathrm{recon}}^{\mathrm{img}},\end{split}$
(16)

where $\lambda^{\mathrm{glob}}$, $\lambda^{\mathrm{ins}}$,
$\lambda^{\mathrm{style}}$, and $\lambda^{\mathrm{img}}$ are weights that
control the importance of each loss.

##  4 Experiments

###  4.1 Implementation Details

We first summarize implementation details in our framework. We conduct
experiments using a single 24GB RTX 3090 GPU. Training datasets are resized to
the size of 352×352. We employ an Adam optimizer for 200 epochs using a step
decay learning rate scheduler. A batch size of 8, an initial learning rate of
2e-4. The number of NCE layers $L$ is 3. For the loss weights, we set as
$\lambda^{\mathrm{glob}}=1$, $\lambda^{\mathrm{ins}}=1$,
$\lambda^{\mathrm{style}}=10$, and $\lambda^{\mathrm{img}}=5$. As described
above, we implement our framework with the most representative vision
Transformer-based, i.e., ViT[13], but we will show our framework works with
MLP-Mixer[52] in the following. We will make our code publicly available.

###  4.2 Experimental Setup

We conduct experiments on two standard datasets for instance-aware I2I, INIT
dataset [51] and KITTI-Cityscapes dataset [16, 9]. INIT dataset [51] provides
street scene images including 4 domain categories (sunny, night, rainy,
cloudy) with object bounding box annotations for car, person, and traffic
sign. We conduct translation experiments for sunny$\rightarrow$night,
night$\rightarrow$sunny, sunny$\rightarrow$rainy, sunny$\rightarrow$cloudy,
and cloudy$\rightarrow$sunny. KITTI object detection benchmark [16] and
Cityscapes [9] dataset are used to evaluate domain adaptation for object
detection on KITTI$\rightarrow$Cityscapes. KITTI contains 7,481 images for
training and 7,518 images for testing with the bounding box annotations for 6
object classes. Cityscapes dataset consists of 5,000 images with pixel-level
annotations for 30 classes.

In this section, we compared our InstaFormer with recent state-of-the-art
instance-aware I2I methods: INIT [51], DUNIT [3], MGUIT [28], and several
unsupervised image-to-image translation methods: CycleGAN [70], UNIT [38], CUT
[45], MUNIT [23], and DRIT [35].

(a) Input

(b) CycleGAN [70]

(c) UNIT [38]

(d) MUNIT [23]

(e) DRIT [35]

(f) MGUIT [28]

(g) InstaFormer

Figure 6: Qualitative comparison on INIT dataset [51]: (top to bottom)
sunny$\rightarrow$night, night$\rightarrow$sunny, and cloudy$\rightarrow$sunny
results. Among the methods, ours preserves object details well and show
realistic results.

###  4.3 Experimental Results

#### Qualitative Evaluation.

We first conduct qualitative comparisons of our method to CycleGAN [70], UNIT
[38], MUNIT [23], DRIT [35], and MGUIT [28] on sunny$\rightarrow$night,
night$\rightarrow$sunny, sunny$\rightarrow$cloudy, and sunny$\rightarrow$rainy
tasks in INIT dataset [51]. As shown in Fig. 6, our model generates higher
quality translated results, particularly at object instance regions.
Especially, as exemplified in the highlighted regions in Fig. 7, our model is
good at capturing local regions within multiple instances thanks to
Transformer-based architecture that simultaneously consider object instances
and global image, and proposed instance-level contrastive learning. Note that
our attention map visualization also proves this, well illustrated in Fig. 3.
Note that MGUIT [28] has access on their trained memory module during test-
time, which is additional burden.

Figure 7: Visual comparison with MGUIT [28]: (a) input, (b) MGUIT [28], and
(c) InstaFormer. We show the results for sunny$\rightarrow$rainy (left) and
sunny$\rightarrow$cloudy (right).

#### Quantitative Evaluation.

Following the common practice [51, 3, 28], we evaluate our InstaFormer with
inception score (IS) [50] and conditional inception score (CIS) [23]. Since
the metrics above are related to diversity of translated images, we also
evaluate our methods with fréchet inception score (FID) [20] and structural
similarity index measure (SSIM) [58] in terms of quality of translated images.
Note that we evaluate the results under the same settings for all the methods.
We adopt FID to measure the distance between distributions of real images and
synthesized images in a deep feature domain. In addition, since SSIM index is
an error measurement which is computed between the original content images and
synthesized images, we apply to measure instance-wise structural consistency.
It should be noted that for image translation tasks, there often exists some
discrepancy between quantitative evaluations and human perceptions [4], thus
the user study in the following would be a better precise metric.

As shown in Table 1, our InstaFormer outperforms the current state-of-the-art
methods in terms of diversity (CIS, IS). Furthermore, in terms of global
distribution, or instance-level similarity as shown in Table 2, FID and SSIM
score show our InstaFormer tends to outperform prior methods in almost all the
comparisons. In particular, results on SSIM demonstrate that our network is
faithfully designed to encode an instance-awareness. Our method improves the
FID score by a large margin compared to previous leading methods MGUIT [28] on
INIT dataset [51].

#### User Study.

We also conducted a user study on 110 participants to evaluate the quality of
synthesized images in the experiments with the following questions: “Which do
you think has better image quality in overall/ similar content to content
image / represent style similar to target domain?” on INIT dataset, summarized
in Fig. 8. Our method ranks the first in every case, especially on content
relevance and overall preference. Note that no standard evaluation metric has
been emerged yet, human evaluation has an effect as evaluation metrics in
image translation tasks.

Figure 8: User study results on INIT dataset [51]. Our method is most
preferred for overall quality, semantic consistency and style relevance,
compared to CUT [45], MUNIT [23], DRIT [35], and MGUIT [28].

Methods | sunny$\rightarrow$night | night$\rightarrow$sunny | Average
---|---|---|---
FID$\downarrow$ | SSIM$\uparrow$ | FID$\downarrow$ | SSIM$\uparrow$ | FID$\downarrow$ | SSIM$\uparrow$
CUT [45] | 75.28 | 0.698 | 80.72 | 0.634 | 78.00 | 0.666
MUNIT [23] | 100.32 | 0.703 | 98.04 | 0.631 | 99.18 | 0.680
DRIT [35] | 79.59 | 0.312 | 99.33 | 0.266 | 89.46 | 0.289
MGUIT [28] | 98.03 | 0.836 | 82.17 | 0.848 | 90.10 | 0.842
InstaFormer | 84.72 | 0.872 | 71.65 | 0.818 | 79.05 | 0.845

Table 2: Quantitative evaluation with FID [20] metric for data distribution
and SSIM [58] index measured at each instance.

(a) Content image

(b) InstaFormer

(c) MLP-Mixer [52]

(d) w/o $\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$

(e) w/o $\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$, $\mathcal{T}$

(f) CUT [45]

(g) w/o AdaIN

Figure 9: Ablation study on different settings: instance-level loss
($\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$), Transformer encoder
($\mathcal{T}$), normalization, and another backbone (MLP-Mixer). Note that
CUT equals to the setting w/o $\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$,
$\mathcal{T}$, and AdaIN.
Figure 10: User study results on ablation study.

###  4.4 Ablation Study

In order to validate the effectiveness of each component in our method, we
conduct a comprehensive ablation study. In particular, we analyze the
effectiveness of instance-level loss
($\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$), Transformer encoder
($\mathcal{T}$), and AdaIN, shown in Fig. 9. It should be noted that CUT [45]
can be regarded as the setting without
$\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$, $\mathcal{T}$, AdaIN from
InstaFormer. Without $\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$, our self-
attention module has limited capability to focus on objects, thus generating
images containing blurred objects, as evaluated in Fig. 3 as well. To validate
the effect of $\mathcal{T}$ in our model, we conduct ablation experiments by
replacing it with Resblocks (without
$\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$, $\mathcal{T}$). Without
Transformers, it fails to capture global relationship between features. It is
obvious that CUT [45] shows limited results containing artifacts, while
InstaFormer dramatically improves object-awareness and quality of the
generated image thanks to our architecture. Since AdaIN helps to understand
global style by leveraging affine parameters, the result without AdaIN, which
is replaced with LayerNorm, shows limited preservation on style with a single-
modal output. We also validate our ablation study results on human evaluation.
110 participants are asked to consider three aspects: overall quality,
semantic consistency and style consistency, summarized in Fig. 10, where we
also validate the superiority of each proposed component.

In addition, we conduct experiments using MLP-Mixer [52]-based aggregator that
replaces $\mathcal{T}$ consisted of ViT [13] blocks to justify robustness of
our framework. Fig. 9(c) shows result examples by MLP-Mixer [52]-based
aggregator. Although ViT-based model is slightly better on MLP-Mixer
[52]-based model in overall quality in Fig. 9(b), the object instance and
style representation are faithfully preserved, which indicates that our method
can be adopted in another Transformer backbone.

###  4.5 Domain Adaptive Object Detection

Additionally, we evaluate our method on the task of unsupervised domain
adaptation for object detection. We follow the experimental setup in DUNIT
[3]. We used Faster-RCNN [48] as baseline detector. In Table 3, we report the
per-class average precisions (AP) for the KITTI→Cityscapes case [16, 9].
Compared to DUNIT [3] and MGUIT [28], our model shows impressive results. It
should be noted that we do not access any information about bounding box
information on test-time, while DUNIT contains object detection network and
MGUIT has access to trained external memory by reading class-aware features.
In particular, our model significantly outperforms other methods in almost all
classes, which indicates that our suggested instance loss has strength on
instance-awareness.

Method | Pers | Car | Truc. | Bic | mAP
---|---|---|---|---|---
DT [26] | 28.5 | 40.7 | 25.9 | 29.7 | 31.2
DAF [23] | 39.2 | 40.2 | 25.7 | 48.9 | 38.5
DARL [34] | 46.4 | 58.7 | 27.0 | 49.1 | 45.3
DAOD [49] | 47.3 | 59.1 | 28.3 | 49.6 | 46.1
DUNIT [3] | 60.7 | 65.1 | 32.7 | 57.7 | 54.1
MGUIT [28] | 58.3 | 68.2 | 33.4 | 58.4 | 54.6
InstaFormer | 61.8 | 69.5 | 35.3 | 55.3 | 55.5

Table 3: Results for domain adaptive detection. We compare the per-class
Average Precision for KITTI $\rightarrow$ CityScape.

##  5 Conclusion

In this paper, we have presented Transformer-based networks, dubbed
InstaFormer, for instance-aware image-to-image translation, which enables
boosting the translation of object instances as well as global image. By
simultaneously considering instance-level features and global-level features
with Transformers, we learned an interaction between not only object instance
and global image, but also different instances. To improve the instance-
awareness during translation, we proposed an instance-level content
contrastive loss. Experiments on various datasets with evaluation metrics have
shown that our framework outperforms the existing solutions for instance-aware
I2I.

Acknowledgements. This research was supported by the MSIT, Korea
(IITP-2022-2020-0-01819, ICT Creative Consilience program), and National
Research Foundation of Korea (NRF-2021R1C1C1006897).

## References

  * [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [2] Kyungjune Baek, Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Hyunjung Shim.  Rethinking the truly unsupervised image-to-image translation.  In ICCV, pages 14154–14163, 2021.
  * [3] Deblina Bhattacharjee, Seungryong Kim, Guillaume Vizier, and Mathieu Salzmann.  Dunit: Detection-based unsupervised image-to-image translation.  In CVPR, pages 4787–4796, 2020.
  * [4] Ali Borji.  Pros and cons of gan evaluation measures.  Computer Vision and Image Understanding, 179:41–65, 2019.
  * [5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, pages 213–229. Springer, 2020.
  * [6] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao.  Pre-trained image processing transformer.  In CVPR, pages 12299–12310, 2021.
  * [7] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In CVPR, pages 8789–8797, 2018.
  * [8] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In CVPR, pages 8188–8197, 2020.
  * [9] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In CVPR, pages 3213–3223, 2016.
  * [10] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen.  Up-detr: Unsupervised pre-training for object detection with transformers.  In CVPR, pages 1601–1610, 2021.
  * [11] Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan.  Coatnet: Marrying convolution and attention for all data sizes.  arXiv preprint arXiv:2106.04803, 2021.
  * [12] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  TPAMI, 38(2):295–307, 2015.
  * [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [14] Aviv Gabbay and Yedid Hoshen.  Scaling-up disentanglement for image translation.  arXiv preprint arXiv:2103.14017, 2021.
  * [15] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In CVPR, pages 2414–2423, 2016.
  * [16] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In CVPR, pages 3354–3361. IEEE, 2012.
  * [17] Abel Gonzalez-Garcia, Joost Van De Weijer, and Yoshua Bengio.  Image-to-image translation for cross-domain disentanglement.  arXiv preprint arXiv:1805.09730, 2018.
  * [18] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NeurIPS, pages 2672–2680, 2014.
  * [19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, pages 2961–2969, 2017.
  * [20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In NeurIPS, pages 6626–6637, 2017.
  * [21] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In ICML, pages 1989–1998. PMLR, 2018.
  * [22] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In ICCV, pages 1501–1510, 2017.
  * [23] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz.  Multimodal unsupervised image-to-image translation.  In ECCV, pages 172–189, 2018.
  * [24] Drew A Hudson and C Lawrence Zitnick.  Generative adversarial transformers.  arXiv preprint arXiv:2103.01209, 2021.
  * [25] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.  Globally and locally consistent image completion.  ACM Transactions on Graphics (ToG), 36(4):1–14, 2017.
  * [26] Naoto Inoue, Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa.  Cross-domain weakly-supervised object detection through progressive domain adaptation.  In CVPR, pages 5001–5009, 2018.
  * [27] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In CVPR, pages 1125–1134, 2017.
  * [28] Somi Jeong, Youngjung Kim, Eungbean Lee, and Kwanghoon Sohn.  Memory-guided unsupervised image-to-image translation.  In CVPR, pages 6558–6567, 2021.
  * [29] Liming Jiang, Changxu Zhang, Mingyang Huang, Chunxiao Liu, Jianping Shi, and Chen Change Loy.  Tsit: A simple and versatile framework for image-to-image translation.  In European Conference on Computer Vision, pages 206–222. Springer, 2020.
  * [30] Yifan Jiang, Shiyu Chang, and Zhangyang Wang.  Transgan: Two transformers can make one strong gan.  arXiv preprint arXiv:2102.07074, 1(3), 2021.
  * [31] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.  Transformers are rnns: Fast autoregressive transformers with linear attention.  In ICML, pages 5156–5165. PMLR, 2020.
  * [32] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Accurate image super-resolution using very deep convolutional networks.  In CVPR, pages 1646–1654, 2016.
  * [33] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim.  Learning to discover cross-domain relations with generative adversarial networks.  arXiv preprint arXiv:1703.05192, 2017.
  * [34] Taekyung Kim, Minki Jeong, Seunghyeon Kim, Seokeon Choi, and Changick Kim.  Diversify and match: A domain adaptive representation learning paradigm for object detection.  In CVPR, pages 12456–12465, 2019.
  * [35] Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang.  Diverse image-to-image translation via disentangled representations.  In ECCV, pages 35–51, 2018.
  * [36] Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu.  Vitgan: Training gans with vision transformers.  arXiv preprint arXiv:2107.04589, 2021.
  * [37] Hanxiao Liu, Zihang Dai, David R So, and Quoc V Le.  Pay attention to mlps.  arXiv preprint arXiv:2105.08050, 2021.
  * [38] Ming-Yu Liu, Thomas Breuel, and Jan Kautz.  Unsupervised image-to-image translation networks.  In NeurIPS, pages 700–708, 2017.
  * [39] Yahui Liu, Enver Sangineto, Yajing Chen, Linchao Bao, Haoxian Zhang, Nicu Sebe, Bruno Lepri, Wei Wang, and Marco De Nadai.  Smoothing the disentangled latent style space for unsupervised image-to-image translation.  In CVPR, pages 10785–10794, 2021.
  * [40] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  arXiv preprint arXiv:2103.14030, 2021.
  * [41] Luke Melas-Kyriazi.  Do you even need attention? a stack of feed-forward layers does surprisingly well on imagenet.  arXiv preprint arXiv:2105.02723, 2021.
  * [42] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [43] Sangwoo Mo, Minsu Cho, and Jinwoo Shin.  Instagan: Instance-aware image-to-image translation.  arXiv preprint arXiv:1812.10889, 2018.
  * [44] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [45] Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu.  Contrastive learning for unpaired image-to-image translation.  arXiv preprint arXiv:2007.15651, 2020.
  * [46] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In CVPR, pages 2536–2544, 2016.
  * [47] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville.  On the spectral bias of neural networks.  In ICML, pages 5301–5310. PMLR, 2019.
  * [48] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  NeurIPS, 28:91–99, 2015.
  * [49] Adrian Lopez Rodriguez and Krystian Mikolajczyk.  Domain adaptation for object detection via style consistency.  arXiv preprint arXiv:1911.10033, 2019.
  * [50] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  NeurIPS, 29:2234–2242, 2016.
  * [51] Zhiqiang Shen, Mingyang Huang, Jianping Shi, Xiangyang Xue, and Thomas S Huang.  Towards instance-level image-to-image translation.  In CVPR, pages 3683–3692, 2019.
  * [52] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al.  Mlp-mixer: An all-mlp architecture for vision.  arXiv preprint arXiv:2105.01601, 2021.
  * [53] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al.  Resmlp: Feedforward networks for image classification with data-efficient training.  arXiv preprint arXiv:2105.03404, 2021.
  * [54] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In ICML, pages 10347–10357. PMLR, 2021.
  * [55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, pages 5998–6008, 2017.
  * [56] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pvtv2: Improved baselines with pyramid vision transformer.  arXiv preprint arXiv:2106.13797, 2021.
  * [57] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  arXiv preprint arXiv:2102.12122, 2021.
  * [58] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  TIP, 13(4):600–612, 2004.
  * [59] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick.  Early convolutions help transformers see better.  arXiv preprint arXiv:2106.14881, 2021.
  * [60] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  arXiv preprint arXiv:2105.15203, 2021.
  * [61] Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong.  Dualgan: Unsupervised dual learning for image-to-image translation.  In ICCV, pages 2849–2857, 2017.
  * [62] Dong Zhang, Hanwang Zhang, Jinhui Tang, Meng Wang, Xiansheng Hua, and Qianru Sun.  Feature pyramid transformer.  In ECCV, pages 323–339. Springer, 2020.
  * [63] Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen.  Cross-domain correspondence learning for exemplar-based image translation.  In CVPR, pages 5143–5153, 2020.
  * [64] Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful image colorization.  In ECCV, pages 649–666. Springer, 2016.
  * [65] Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros.  Real-time user-guided image colorization with learned deep priors.  arXiv preprint arXiv:1705.02999, 2017.
  * [66] Long Zhao, Zizhao Zhang, Ting Chen, Dimitris N Metaxas, and Han Zhang.  Improved transformer for high-resolution gans.  arXiv preprint arXiv:2106.07631, 2021.
  * [67] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai.  The spatially-correlative loss for various image translation tasks.  In CVPR, pages 16407–16417, 2021.
  * [68] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In CVPR, pages 6881–6890, 2021.
  * [69] Xingran Zhou, Bo Zhang, Ting Zhang, Pan Zhang, Jianmin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen.  Cocosnet v2: Full-resolution correspondence learning for image translation.  In CVPR, pages 11465–11475, 2021.
  * [70] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In ICCV, pages 2223–2232, 2017.
  * [71] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  arXiv preprint arXiv:2010.04159, 2020.

In this appendix, we describe detailed network architecture, PyTorch-like
pseudo-code, and additional results for “InstaFormer: Instance-Aware Image-to-
Image Translation with Transformer”.

##  Appendix A Experimental Details

###  A.1 Network Architecture of InstaFormer

We first summarize the detailed network architecture of our InstaFormer in
Table 4. We basically follow the content encoder and generator architecture
from CUT[45] with Transformer blocks and style encoder. The double-line inside
Encoder table indicates the end of content encoder, while style encoder has
same structure with content encoder but has additional Adaptive Avg. Pool. and
Conv-4 as shown below. $(.)$ in the convolution indicates the padding
technique, and the structures of Upsample , Downsample and discriminator are
the same as those of CUT[45].

|  Encoder |
---|---|---
Layer |  Parameters $(\mathtt{in},\mathtt{out},\mathtt{k},\mathtt{s},\mathtt{p})$ |  Output shape $(C\times H\times W)$
Conv-1 (Reflection) |  $(3,64,7,1,3)$ |  $(64,352,352)$
InstanceNorm |  - |  $(64,352,352)$
ReLU |  - |  $(64,352,352)$
Conv-2 (Zeros) |  $(64,128,3,1,1)$ |  $(128,352,352)$
InstanceNorm |  - |  $(128,352,352)$
ReLU |  - |  $(128,352,352)$
Downsample |  - |  $(128,176,176)$
Conv-3 (Zeros) |  $(128,256,3,1,1)$ |  $(256,176,176)$
InstanceNorm |  - |  $(256,176,176)$
ReLU |  - |  $(256,176,176)$
DownSample |  - |  $(256,88,88)$
AdaptiveAvgPool |  - |  $(256,1,1)$
Conv-4 |  $(256,8,1,1,0)$ |  $(8,1,1)$
|  |
|  Transformer Aggregator |
Layer |  Parameters $(\mathtt{in},\mathtt{out})$ |  Output shape $(C)$
AdaptiveInstanceNorm |  - |  $(1024)$
Linear-1 |  $(1024,3072)$ |  $(3072)$
Attention |  - |  $(1024)$
Linear-2 |  $(1024,1024)$ |  $(1024)$
AdaptiveInstanceNorm |  - |  $(1024)$
Linear-3 |  $(1024,4096)$ |  $(4096)$
GELU |  - |  $(4096)$
Linear-4 |  $(4096,1024)$ |  $(1024)$
|  |
|  Generator |
Layer |  Parameters $(\mathtt{in},\mathtt{out},\mathtt{k},\mathtt{s},\mathtt{p})$ |  Output shape $(C\times H\times W)$
UpSample |  - |  $(256,176,176)$
Conv-1 (Zeros) |  $(256,128,3,1,1)$ |  $(128,176,176)$
InstanceNorm |  - |  $(128,176,176)$
ReLU |  - |  $(128,176,176)$
UpSample |  - |  $(128,352,352)$
Conv-2 (Zeros) |  $(128,64,3,1,1)$ |  $(64,352,352)$
InstanceNorm |  - |  $(64,352,352)$
ReLU |  - |  $(64,352,352)$
Conv-3 (ReflectionPad) |  $(64,3,7,1,3)$ |  $(3,352,352)$
Tanh |  - |  $(3,352,352)$
|  |

Table 4: Network architecture of our InstaFormer.

###  A.2 PyTorch-like Pseudo-code

#### Pseudo-code for Instance-level Content Loss.

Here we provide the (PyTorch-like) pseudo-code for
$\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$ in InstaFormer. To re-emphasize,
our simple instance-level content loss learns the representations for a
translation task, effectively focusing on local object-region; this is unlike
classical PatchNCE loss that utilizes the regular-grid patches from features
in unconditional way.

#### Pseudo-code for Transformer Aggregator.

We also provide pseudo-code for the input of $\mathcal{T}$, in order to show
how our novel technique aggregates _instance_ -level content features and
_global_ -level content features simultaneously. This enables the model to pay
more attention to the relationships between global scenes and object
instances.

##  Appendix B Additional Results

###  B.1 Visualization of Multi-modal Image Translation

We visualize the multimodal translated results in Fig. 11. Our InstaFormer
generates not only high-quality visual results, but also produces results with
large diversity.

(a) Input image (Sunny)

(b) Translated image 1 (Night)

(c) Translated image 2 (Night)

(d) Translated image 3 (Night)

Figure 11: Results of multi-modal image translation. We use randomly sampled
style codes to generate night images from a sunny image.

###  B.2 Qualitative Results of Domain Adaptive Object Detection

In the main paper, we have evaluated our method on the task of unsupervised
domain adaptation for object detection providing quantitative results. In this
section, we also show the qualitative results of the task in Fig. 12. Our
model successfully works on complex domain adaptation tasks.

(a) Input image (KITTI)

(b) Translated image (CityScape)

Figure 12: Qualitative results for domain adaptive detection for KITTI
$\rightarrow$ CityScape.

###  B.3 Comparison with the State-of-the-Art I2I methods.

As shown in Fig. 13 and Table 5, we additionally compare InstaFormer with SoTA
I2I methods, such as TSIT [29], StarGANv2 [8] and Smoothing [39] on INIT
dataset [51] for sunny$\rightarrow$night.

(a) Input

(b) TSIT

(c) StarGAN2

(d) Smoothing

(e) Ours

Figure 13: Comparison with TSIT [29], StarGANv2 [8] and Smoothing [39].

Methods | sunny$\rightarrow$night
---|---
FID$\downarrow$ | SSIM$\uparrow$
TSIT | 90.28 | 0.822
StarGANv2 | 88.49 | 0.545
Smoothing | 85.28 | 0.667
InstaFormer | 84.72 | 0.872

Table 5: More quantitative evaluation.

###  B.4 Additional Examples on Ablation Study

In the main paper, we have examined the impacts of instance-level loss
($\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$), Transformer encoder
($\mathcal{T}$), normalization, and another backbone (MLP-Mixer). CUT equals
to the setting w/o $\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$, $\mathcal{T}$,
and AdaIN. We provide more examples on INIT dataset [51], depicted in Fig. 14.
Our InstaFormer produces better visual results. In particular, as shown in
Fig. 14 (d), tiny objects tend to disappear or be blurred without
$\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$.

(a) Content image

(b) InstaFormer

(c) MLP-Mixer [52]

(d) w/o $\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$

(e) w/o $\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$, $\mathcal{T}$

(f) CUT [45]

(g) w/o AdaIN

Figure 14: Ablation study on different settings: instance-level loss
($\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$), Transformer encoder
($\mathcal{T}$), normalization, and another backbone (MLP-Mixer). Note that
CUT equals to the setting w/o $\mathcal{L}^{\mathrm{ins}}_{\mathrm{NCE}}$,
$\mathcal{T}$, and AdaIN.

###  B.5 Additional Translation Results

In the main paper, we have shown some of our results of instance-aware image-
to-image translation. Here, we show additional results in Fig. 15 to
demonstrate the robustness of InstaFormer.

(a) Input image (Sunny)

(b) Translated image (Night)

(c) Translated image (Cloudy)

(d) Translated image (Rainy)

Figure 15: Qualitative comparison on INIT dataset [51]: (left to right) sunny,
sunny$\rightarrow$night, sunny$\rightarrow$cloudy, and sunny$\rightarrow$rainy
results. Our method achieves high-quality of realistic results while
preserving object details as well.

###  B.6 Variants of InstaFormer Architecture

#### Effects of the Number of Transformer Blocks.

We show qualitative and quantitative comparisons of the number of Transformers
blocks. As described in Table 6 and Fig. 16, the results using 6 and 9 blocks
show almost close FID metric [20] and SSIM metric [58] scores and visual
results, while the score of 3 blocks shows insufficient result. As smaller
models have a lower parameter count, and a faster throughput, we choose 6
blocks for our architecture.

#### Effects of the Number of Heads.

We analyze the effects of the number of heads in our model. We show the
quantitative results in Table 6 and visualization of self-attention maps in
Fig. 17 according to the number of heads. More heads tend to bring lower FID
metric score and higher SSIM metric score with better self-attention learning,
but the scores using 8 heads shows slight better results compared the scores
using 4 heads. Thus, we decide to use 4 heads considering memory-efficiency.

#### Effects of Transformers.

We additionally compare InstaFormer with CNN-based model in Table 6. While
single head version of InstaFormer has a lower parameter count, it shows
better performance in terms of FID metric compared to CNN-based model. This
demonstrates that our outstanding performance is not due to its complexity.

Variants | #blocks | #heads | #params | FID$\downarrow$ | SSIM$\uparrow$
---|---|---|---|---|---
Less blocks | 3 | 4 | 37.776M | 89.96 | 0.711
Ours | 6 | 4 | 75.552M | 84.72 | 0.872
More blocks | 9 | 4 | 113.329M | 85.28 | 0.879
Less heads | 6 | 1 | 4.732M | 89.17 | 0.738
Ours | 6 | 4 | 75.552M | 84.72 | 0.872
More heads | 6 | 8 | 302.100M | 81.92 | 0.873
CNN-based | 6 | - | 7.081M | 89.73 | 0.708

Table 6: Effects of the number of blocks and heads in our InstaFormer
architecture, providing quantitative evaluations with FID [20] and SSIM [58].
The only setting that vary across model is the number of Transformer blocks or
heads, and we keep the others constant for sunny$\rightarrow$night on INIT
dataset [51]. Larger models tend to have a higher parameter count, and better
FID [20] and SSIM [58] metric scores.

(a) Input image

(b) #blocks = 3

(c) #blocks = 6

(d) #blocks = 9

Figure 16: Visual results on variants of the number of Transformer blocks. (a)
input image and translated images (b,c,d) with different number of Transformer
blocks for sunny$\rightarrow$night.

(a) Input image with instances

(b) #heads = 1

(c) #heads = 4

(d) #heads = 8

Figure 17: Visualization of learned self-attention. (from top to bottom)
attention map for a car and a truc, respectively. (a) input image and
following self-attention maps (b,c,d) for different number of heads.