# ICAR: Image-based Complementary Auto Reasoning

Xijun Wang1, Anqi Liang2, Junbang Liang2, Ming Lin1,2, Yu Lou2, Shan Yang2

###### Abstract

Scene-aware Complementary Item Retrieval (CIR) is a challenging task which
requires to generate a set of compatible items across domains. Due to the
subjectivity, it is difficult to set up a rigorous standard for both data
collection and learning objectives. To address this challenging task, we
propose a visual compatibility concept, composed of similarity (resembling in
color, geometry, texture, and etc.) and complementarity (different items like
table vs chair completing a group). Based on this notion, we propose a
compatibility learning framework, a category-aware Flexible Bidirectional
Transformer (FBT), for visual “scene-based set compatibility reasoning” with
the cross-domain visual similarity input and auto-regressive complementary
item generation. We introduce a “Flexible Bidirectional Transformer (FBT),”
consisting of an encoder with flexible masking, a category prediction arm, and
an auto-regressive visual embedding prediction arm. And the inputs for FBT are
cross-domain visual similarity invariant embeddings, making this framework
quite generalizable. Furthermore, our proposed FBT model learns the inter-
object compatibility from a large set of scene images in a self-supervised
way. Compared with the SOTA methods, this approach achieves up to 5.3% and
9.6% in FITB score and 22.3% and 31.8% SFID improvement on fashion and
furniture, respectively.

Figure 1: Cover Image: We present a new self-supervised model for scene-aware,
visually compatible object retrieval tasks. In this example, given an
inspirational home scene image (sampled from STL-home(Kang et al. 2019) column
1, 3, 5) with a pool of objects (3D-FRONT(Fu et al. 2021)) from an unseen
domain, our model auto-regressively retrieves a set of stylistically
compatible items (column 2, 4, 6).

##  1 Introduction

Online shopping catalogs provide great convenience, such as searching and
comparing similar items. However, when customers can compare similar items,
they often miss the browsing of complementary items in the e-shopping
experience. Millions of online images offer a new opportunity to shop with
inspirational home decoration ideas or outfit matching. But, to retrieve
stylistically compatible products from these online images for set matching
can be an overwhelming process.The ability to recommend visually complementary
items becomes especially important, when shopping for home furniture and
clothings. The subjectivity makes the visual compatibility even more difficult
to model computationally.

In this work, we aim to address the visual scene-aware Complementary Item
Retrieval (CIR) (Sarkar et al. 2022; Kang et al. 2019) task. In this task (as
shown in Figure 2), we attempt to model human’s ability to select a set of
objects from cross-domain pools, given a scene image, objects in the scene,
and object categories. Therefore, we propose a visual compatibility concept,
consisting of two key elements: similarity and complementarity. Visual
similarity and complementarity, however, can contradict each other sometimes.
Items that look similar (color, geometry, texture, and etc.) may not be
complementary (different items like dinner table vs sofa) when putting them
into a set. Items that complement each other do not necessarily look similar
(e.g. an outfit set in contrasting colors). The ambiguous definition for
visual complementarity is a major challenge. This ambiguity makes it difficult
to rigorously define an objective and creates extra challenge for collecting
such datasets, when designing a data-driven method.

Figure 2: Scene-aware Complementary Item Retrieval Task Illustration. Given a
query scene image, (optional) scene objects and item categories, the task goal
is to generate a cross-domain set of stylistically compatible items.

To address these issues, we first propose a “compatibility leaning” framework
to model the visual similarity and complementarity. To the best of our
knowledge, we are among the first to show qualitatively that our model based
on this framework can generalize to unseen domains (where the model is not
trained with, as shown in Figure 1 and Figure 6). For the scene-based CIR
task, it’s complex to learn both the cross-domain similarity and
complementarity. Therefore, we use cross-domain visual similarity invariant
embeddings in our framework. Many previous CIR works (Han et al. 2017; Kang et
al. 2019) also start from some types of learned embedding. But failing to
model the visual similarity creates extra complexity for the complementary
learning. Secondly, we propose to use self-supervised learning for visual
complementarity reasoning by introducing an auto-regressive transformer based
architecture. Given the difficulty to define style complementarity
mathematically, we propose a solution based on the assumption that the items
exist in the inspirational scene images are compatible with each other.

Built upon the aforementioned premises, we present a novel self-supervised
transformer-based learning framework (overview shown in Fig. 3). Our model
effectively learns both the similarity and complementarity between a set of
items. Our model does not require extra complementary labels. In addition,
compared to the prior work that models complementary items as pairs or a
sequence of items, we model them as unordered sets. We carefully design our
compatibility leaning model. First, we ensure that the learned embedding both
contains and extracts all the necessary information for the compatibility
learning. Second, we make full use of Transformer’s ability in reasoning about
the interactions between these learned embeddings. To model flexible-length
unordered set generation with cross-domain retrieval, we propose a new
Flexible Bidirectional Transformer (FBT). In this FBT model, we model the
unordered set generation using random shuffle and masking technique. In
addition, we introduce a category prediction arm and a cross-domain retrieval
arm to the transformer encoder. The added category prediction branch helps the
model to reason about the complementary item types. As pointed out in
(Vasileva et al. 2018), the category embedding representation of each item
carries the notion of both similarity and complementarity. Our proposed model,
compared with prior work that also models the multi-item compatibility using
neural networks, such as works by (Li et al. 2017), (Han et al. 2017) and
(Sarkar et al. 2022), does not require the partial sets to be given.

We validate our method on the CIR benchmark datasets include, Shop The Look
(STL) (Kang et al. 2019), Exact Street2Shop (Hadi Kiapour et al. 2015) and
DeepRoom (Gadde, Feng, and Martinez 2021). Our method consistently outperforms
the state-of-the-art methods. More importantly, we notice that most of the CIR
prior work are evaluated via the Fill-In-The-Blank (FITB) (Han et al. 2017)
metric or human in the loop. The FITB metric can reflect the model’s ability
in cross-domain retrieval but it does not measure the complementarity as a
set. Human in the loop evaluation, however, is both limited to scale and
biases, if not conducted thoughtfully. To address these issues, we propose a
new CIR evaluation metric: “Style Frechet Inception Distance” (SFID) (see
supplementary for details).

In summary, the key contributions of this work include:

  * •

Visual compatibility is defined based on similarity and complementarity for
the Scene-aware Complementary Item Retrieval (CIR) task and a new
compatibility learning framework is designed to solve this task.

  * •

For the compatibility learning framework, a category-aware Flexible
Bidirectional Transformer (FBT) is introduced for visual scene-based set
compatibility reasoning with the cross-domain visual similarity input and
auto-regressive complementary item generation.

##  2 Related Work

##### Visual Similarity Learning

Visual similarity learning has been a main computer vision topic. The goal is
to mimic human’s ability in finding visually similar objects or scenes. This
is particularly studied in image retrieval (El-Nouby et al. 2021; Radenović,
Tolias, and Chum 2018; Teh, DeVries, and Taylor 2020; Cheng et al. 2021) \-
finding images with a certain definition of similarity and so on. Prior to
retrieving similar clothing, researchers also studied how to detect and
segment clothing from real-life images (Yamaguchi et al. 2012; Yang and Yu
2011; Gallagher and Chen 2008). With clothing detection or segmentation,
similar clothing retrieval is explored via style analysis (Hsiao and Grauman
2017; Kiapour et al. 2014; Simo-Serra and Ishikawa 2016; Yu et al. 2012;
Yamaguchi, Hadi Kiapour, and Berg 2013; Kiapour et al. 2014; Di et al. 2013).
In the meantime, Liu (Liu et al. 2012b, a) pioneer ways to do cross-domain
retrieval which retrieves similar clothing from real-life images and targets
images from a different domain, such as well-staged product images. Recent
fashion retrieval tasks can be further categorized based on the input
information, such as images (Kalantidis, Kennedy, and Li 2013; Liu et al.
2016; Simo-Serra and Ishikawa 2016; Zhai et al. 2017; Hadi Kiapour et al.
2015; Tran et al. 2019), clothing attributes (Ak et al. 2018; Di et al. 2013),
videos (Cheng et al. 2017).

Visual complementarity Learning Visual complementarity learning, unlike visual
similarity learning, is much more ambiguous. There are a couple of research
directions: pairwise complementary item retrieval (Veit et al. 2015; Song et
al. 2017a; Vasileva et al. 2018; McAuley et al. 2015; Lin, Tran, and Davis
2020; Taraviya et al. 2021), set complementary prediction (no cross domain
retrieval) (Tangseng, Yamaguchi, and Okatani 2017; Li et al. 2017; Han et al.
2017; Hsiao and Grauman 2018; Tangseng, Yamaguchi, and Okatani 2017; Shih et
al. 2018; Li et al. 2020), set complementary item retrieval (Hu, Yi, and Davis
2015; Huang et al. 2015; Liu et al. 2012a), personalized set complementary
item prediction (requires user input) (Taraviya et al. 2021; Chen et al. 2019;
Li et al. 2020; Su et al. 2021; Zheng et al. 2021; Guan et al. 2022b, a) and
multi-modal complementary item prediction (Guan et al. 2021). All these prior
work focus on feature representation learning. Another line of works (Chen et
al. 2015; Song et al. 2017b; Tan et al. 2019; Lin, Tran, and Davis 2020) focus
on learning multiple sub-embedding based on different properties for both
similarity and compatibility. More recently, Transformer (Vaswani et al. 2017)
has demonstrated strong performance across various natural language processing
and computer vision tasks. Kang (Kang et al. 2019) proposes to use CNN visual
classification features and attention mechanism. Later on, Sarkar (Sarkar et
al. 2022) uses Transformer and CNN-based image classification features for
compatibility learning. Similarly, Chen (Chen et al. 2019) applies Transformer
together with CNN image classification feature to learn a mapping between user
picked item pool and a set of most compatible items from that pool. Unlike all
the work above, we build our visual compatibility model which focuses on both
similarity and complementarity.

Learning Framework Many researchers have studied and explored building a
cascaded learning framework. The cascaded method here means learning how to
encode the data then modeling the statistics of this encoding. Many of the
methods proposed for CIR task can also be categorized as two-stage models. But
almost all of them use the image classification training target as the first-
stage feature extractor (Han et al. 2017; Sarkar et al. 2022; Kang et al.
2019; Chen et al. 2019). Taraviya (Taraviya et al. 2021) propose a two-stage
model for personalized pairwise complementary item recommendation where they
learn a feature embedding specially designed for customer preferences in their
first stage. In our compatibility learning, we set cross-domain visual
similarity embedding as input, and design FTB for complementary set
generation. We show empirically that the visual similarity feature, compared
to image classification learned features, are better suited for CIR. Our
design makes our model surpass the prior work in scene aware CIR task.

##  3 Method

Figure 3: ICAR Model Overview. In similarity learning, we apply a CNN-based
model(Jun et al. 2019) to learn the visual similarity features across two
domains. The learned features are required for both complementary reasoning in
the complementarity learning and the cross-domain retrieval. With the learned
features, in the complementarity learning, we propose a Flexible Bidirectional
Transformer (FBT) model to learn the multi-object visual compatibility.

##### Problem Statement

Given a scene image $\mathcal{I}$, a set of unordered objects
$\mathcal{O}=\\{o_{i}\\}_{i=0}^{N},o_{i}\in\mathcal{D}_{A}$ in the scene and a
set of unordered object categories $\mathcal{C}=\\{c_{i}\\}_{i=0}^{L}$, the
problem is to retrieve cross-domain a set of complementary objects
$\mathcal{X}=\\{x_{i}\\}_{i=0}^{L},x_{i}\in\mathcal{D}_{B}$. This generated
set of objects needs to be both visually compatible with each other and of
visually similar style to the input scene image $\mathcal{I}$. Here we use
$\mathcal{D}_{A}$ and $\mathcal{D}_{B}$ to denominate the two different visual
domains, $L$ to represent the number of objects to retrieve during inference
and $N$ is the number of scene objects. The difference between the two domains
$\mathcal{D}_{A}$ and $\mathcal{D}_{B}$ can be quantified as the Fréchet
distance $\mathcal{F}$ larger than a certain threshold $\theta$.

###  3.1 Conditional Compatibility Auto Reasoning

We formulate the problem of generating a set of objects
$\mathcal{X}=\\{x_{i}\\}_{i=0}^{L}$ conditioned on the scene image
$\mathcal{I}$ and a specified set of categories as how to compute likelihood
(Eq. 1) of creating the object set $\mathcal{X}$ given the scene image
$\mathcal{I}$, objects in the scene $\mathcal{O}$, and set of categories
$\mathcal{C}$. We model the probability of generating the unordered set
$\mathcal{X}$ as the sum of generating the set in any permutation
$\hat{\mathcal{X}}$:

$p(\mathcal{X}_{i}|\mathcal{I},\mathcal{O},\mathcal{C})=\sum_{\hat{\mathcal{X}}\in\Phi(\mathcal{X}_{i})}p(x_{i}|x_{0},\dots,x_{i-1},\mathcal{I},\mathcal{O},\mathcal{C}),i\leq
L$ (1)

where $\Phi(\mathcal{X})$ includes all the permutation of the target object
set $\mathcal{X}$ given all the permutation of the categories $\mathcal{C}$,
and $L$ is the maximum number of items to compose a set. For each permutation
of $\mathcal{X}$, the set generation becomes a sequence generation problem. We
model the sequence generation as an auto-regressive process. In the auto-
regressive process, the next item in the set is generated conditioned on the
prior items. This auto-regressive process statistically formulated as the
multiplication of the probabilities:

$p(x_{i}|x_{0},\dots,x_{i-1})=\prod_{j}^{j<(i-1)}p(x_{j}|x_{0},\dots,x_{j-1}).\vspace{-2mm}$
(2)

To learn to conditionally generate the best set of objects, our model learns
to maximize the log likelihood of the probability,
$p(\mathcal{X}|\mathcal{I},\mathcal{O},\mathcal{C})$,

$\log
p(\mathcal{X}_{i}|\mathcal{I},\mathcal{O},\mathcal{C})=\sum_{\hat{\mathcal{X}}\in\Phi(\mathcal{X}_{i})}\sum_{j}^{j<i}\log
p(x_{i}|x_{<j},\mathcal{I},\mathcal{O},\mathcal{C})$ (3)

To approximate the log likelihood Eq. 3, we propose a two-stage learning
framework.

###  3.2 Compatibility Learning Framework

Visual Similarity Learning

Figure 4: VSIM: Visual Similarity Model.

To relieve the complexity of learning both the visual complementarity and
similarity directly from pixel domain, we propose to separate them into two
stages. In the first stage, our model focuses on the visual similarity
learning. As shown in Figure 4, we apply a CNN-based (ResNet50) visual
similarity model (Jun et al. 2019) with normalized softmax loss (Zhai and Wu
2018) and soft-margin triplet loss (Hermans, Beyer, and Leibe 2017) (refer to
Supplementary for more details). With this model, we project the scene image
$\mathcal{I}$, objects in the image $\mathcal{O}$, and the item images in
retrieval pool $\mathcal{X}$ onto this embedding:

$\\{\mathbf{I},\mathbf{O},\mathbf{X}\\}=g(\\{\mathcal{I},\mathcal{O},\mathcal{X}\\}),\mathcal{I}\in\mathbb{R}^{3},\mathcal{O}\in\mathbb{R}^{3},\mathcal{X}\in\mathbb{R}^{3}$
(4)

where $g$ is our visual similarity model. This projection helps our second
stage model to converge faster, similar in spirit to sequential optimization.
We also show empirically (refer to Sec. Similarity Learning Results for
details) that the visual similarity embedding is best suited for learning
cross-domain visual compatibility.

Complementarity Reasoning with Flexible Bidirectional Transformer

Figure 5: FBT: Flexible Bidirectional Transformer. We randomly sample
$M\in[0,N]$ items from the total item number $N$ of items (in a scene) as
input set, and the $(M+1)_{th}$ item not in the input set as output target. We
put the scene embedding at the beginning of input set, and take the scene
embedding as the start token $EI$. We set a zero vector as the end token
$E_{e}$.

At the second stage, we propose a new Flexible Bidirectional Transformer (FBT)
(see Figure 5 for conditional cross-domain unordered set generation). We
choose Transformer model (Vaswani et al. 2017) as the core architecture to
learn the inter-object compatibility. The vanilla Transformer model (Vaswani
et al. 2017), as it is originally proposed for modeling ordered sequence
structured data, such as languages and images, is insufficient for our task.

We introduce: (1) random shuffling together with random length sequence
masking for set generation; (2) category prediction arm to better model the
category distribution for a set of objects; and (3) visual embedding
prediction arm for visual compatibility modeling. During inference, our FBT
model generates an unordered set auto-regressively (Eq. 2 and shown in the
green part of Figure 3). Inspired from the CLS token proposed in the Vision
Transformer (ViT) (Dosovitskiy et al. 2020), we also use a trainable variable
denoted as $\mathbf{q}$ to extract inter-token relation,

$\displaystyle\begin{split}\mathbf{q^{\prime}}&=e(\mathbf{EI},\Phi(\mathbf{E}\mathbf{O});\mathbf{E}\mathbf{q})\\\
&=\text{MLP}(\text{MSA}(\mathbf{EI},\Phi(\mathbf{E}\mathbf{O}),\mathbf{E}\mathbf{e},\mathbf{E}\mathbf{q}))\\\
\mathbf{E}\mathbf{O}&=[\mathbf{E}\mathbf{o}_{1},\mathbf{E}\mathbf{o}_{2},\dots,\mathbf{E}\mathbf{o}_{M},\text{MASK}],\end{split}$
(5)

where $\mathbf{q}^{\prime}$ denotes the corresponding output of the trainable
input token $\mathbf{q}$, $\mathbf{e}$ is the end token, $\Phi$ is the masking
operation, $e()$ represents the Transformer encoder with MSAs (the Multi-
headed Self-Attention layers), MLPs (Multi-Layer Perception), $\mathbf{E}$ is
the linear projection and $M$ is the unmasked sequence length. The output
$\mathbf{q}^{\prime}$ is then be used for predicting both the category
$\mathbf{c}_{M+1}$ (Eq. 6) and the visual embedding of the next item
$\mathbf{x}_{M+1}$ (Eq. 7).

$\displaystyle\mathbf{\hat{c}}$
$\displaystyle=\text{MLP}(\mathbf{q}^{\prime})$ (6)
$\displaystyle\mathbf{\hat{x}_{M+1}}$
$\displaystyle=\text{MLP}[\mathbf{q}^{\prime},\mathbf{\hat{c}}]$ (7)

The output category embedding $\mathbf{\hat{c}}$ is supervised via the Cross-
Entropy loss. And the visual feature embedding $\mathbf{\hat{x}}$ is
supervised using a triplet loss (Yang et al. 2019). To form a triplet, the
anchor is the predicted embedding $\mathbf{\hat{x}_{M+1}}$ with the target
item’s embedding $\mathbf{x_{M+1}}$ as the positive and randomly selected same
category object’s embedding as the negative. One challenge in the feature
learning is the space collapsing, where points in the embedding space get too
close. This space collapse can lower the representation capacity. To avoid
such an issue, we apply a differential entropy regularizer (Girdhar et al.
2019) to maximize the distance between each point and its closest neighbors in
the embedding space. The regularizer is defined as follows:

$\vspace{-2mm}\text{Reg}=-\frac{1}{N}\sum_{i}^{N}log(D_{min_{i\neq
j}}(z_{i},z_{j})),$ (8)

where we divide the L2 distance between sample i, j by 4 as $D_{i,j}$ to make
the distance between 0 and 1. $z$ is the input feature maps.

##  4 Experiments

###  4.1 Setup

Benchmark Datasets:  In the following experiments, we evaluate our proposed
ICAR using four datasets. DeepRooms (Gadde, Feng, and Martinez 2021) is a
large-scale (1.4 million), high-quality human annotated object detection
dataset covering a total of 81 fine-grained furniture and home product
categories with 210K room-scene images. STL-Home (Kang et al. 2019) includes
24,022 interior home design images and 41,306 home decor items, which can
generate 93,274 scene-product pairs. STL-Fashion (Kang et al. 2019) contains
72,189 fashion-product pairs from its 47,739 fashion images and 38,111 product
images. And Exact Street2Shop (Hadi Kiapour et al. 2015) provides fashion-
product 10,608 pairs from its 10,482 fashion images and 5,238 product images
with the bounding box of products in scene images.

Implementation: All the models in our experiments are trained using AdamW
(Loshchilov and Hutter 2017) with cosine learning scheduler from an initial
learning rate of $2e-4$ to 0. Models are trained for 500 epochs with a batch
size of 256. We choose 1 negative sample in the triplet loss. And we use 1.0,
1.0, and 0.05 as the weights for cross-entropy loss, triplet loss, and
regularizer loss respectively.

Figure 6: Scene-aware Cross-Domain CIR Qualitative Results. We show
qualitatively our model is capable of retrieving stylistically compatible
items from both seen (Row 2 and 4) and unseen domains (Row 1 and 3), given a
home (Row 1-3) or fashion (Row 4) scene image. Column 1, 5, 9 are the input
scene images. See supplementary materials for more examples.

###  4.2 Evaluation Metrics

To evaluate our proposed model’s performance for scene-aware cross-domain CIR,
we benchmark our method using previously proposed Fill-In-The-Blank (FITB)
(Han et al. 2017) and our newly proposed SFID metrics. FITB accuracy is
measured via counting percentage of times the model correctly picked the
ground-truth object from the candidate pool. Following (Kang et al. 2019), we
set the number of candidate to 2 for STL and Street2Shop and 3 for DeepRooms.
For fair comparison, we apply the same method in setting up the FITB candidate
pool across all the experiments. While FITB estimates the model’s ability to
retrieve the best object, it fails to capture the compatibility among a set of
items. To compensate, we propose a new distribution distance-based metric:
Style FID (SFID). It is difficult to define visual style, as it covers various
aspects, including color, texture, shape and so on. Instead, we use the visual
similarity distribution between the generated set and the designed sets of
items as the measurement of stylistic compatibility. Similar to the FID
(Heusel et al. 2017), we apply the Fréchet distance $\mathcal{F}$ for
distribution distance measurement. Instead of directly estimating the pixel
value distribution, we apply a feature extractor that can project the pixel
values onto an embedding. With the feature extractor, the computed
distribution can focus on style related features, including colors, edges,
textures, patterns, and shapes. We define

$\text{SFID Score}=\mathcal{F}(f(\mathcal{X}),f(\mathcal{Y})),$ (9)
$\mathcal{F}(\mathbf{X},\mathbf{Y})=\mid\mu_{X}-\mu_{Y}\mid+tr(\sigma_{X}+\sigma_{Y}-2\sqrt{\sigma_{X}*\sigma_{Y}}),$

where $\mathcal{X}$ is the generated set of objects, $\mathcal{Y}$ is a set of
well designed (ground-truth) objects, function $f$ is a feature extractor,
$\mu$ and $\sigma$ are the mean and variance. Please refer supplementary for
more details about SFID.

###  4.3 Compatibility Learning Results

Method | FITB $\uparrow$ | Condition
---|---|---
CSA-Net w. triplet loss | $60.0$ | /
CSA-Net w. outfit loss | $57.4$ | /
Visual Similarity Learning | $62.0$ | /
OutfitTransformer | $77.6$ | /
ICAR (Ours) | 83.9 | Predict category
ICAR (Ours) | 87.1 | Given category

Table 1: FITB Results on DeepRooms (Gadde, Feng, and Martinez 2021). Our
approach improved FITB accuracy by 9.5% over Visual Similarity Learning, CSA-
Net (Lin, Tran, and Davis 2020), and OutfitTransformer (Sarkar et al. 2022).

#### Quantitative Results

Here we present quantitative evaluation results on four different scales for
benchmark datasets. We first evaluate our algorithms on the largest DeepHomes
(Gadde, Feng, and Martinez 2021) dataset. As shown in Table 1 and 3, ICAR
improves performance over the state-of-the-art by 9.5% for FITB and 11.2
(about 23.3%) for SFID score. To further test the effectiveness of our
category embedding, we compare the FITB results when specify or not the
category during inference time. If category is not specified, the FITB score
drops from $87.1\%$ to $83.9\%$.

Method | STL-Fashion $\uparrow$ | STL-Home $\uparrow$ | S2S $\uparrow$
---|---|---|---
IBR | 58.5 | 57.0 | 56.5
Siamese Nets | 67.1 | 72.4 | 63.0
BPR-DAE | 61.1 | 64.2 | 59.3
Complete the Look | 70.0 | 75.0 | 63.1
OutfitTransformer | 65.0 | 77.0 | 67.2
ICAR (Ours) | 75.3 | 86.6 | 71.4

Table 2: FITB Results on STL and Street2Shop (S2S). Our approach can improve
FITB accuracy by 4.2%–9.6% over IBR (McAuley et al. 2015), Siamese Nets (Veit
et al. 2015), BPR-DAE (Song et al. 2017a), Complete the Look (Kang et al.
2019), & OutfitTransformer (Sarkar et al. 2022).

We then evaluate our method on more existing datasets: STL-Home (Kang et al.
2019), STL-Fashion (Kang et al. 2019) and Street2Shop (Hadi Kiapour et al.
2015). For these three datasets, the human annotators also label products that
have a similar style to the observed product and are compatible with the
scene. This study suggests that the scene images and the products may not have
one-to-one correspondence, thereby increasing the difficulty of style
matching. We show in Table 2 and Table 3, that ICAR outperforms SOTA by 5.3%
on STL-Fashion and 9.6% on STL-Home in FITB metric, and outperforms SOTA by
3.4 (22.3%) on STL-Fashion and 2.9 (31.8%) on STL-Home in terms of SFID
metric.

Method | DeepRooms $\downarrow$ | STL-F $\downarrow$ | STL-H $\downarrow$ | S2S $\downarrow$
---|---|---|---|---
GT VS Neg | 49.0 | 15.8 | 10.2 | 9.1
OutfitTransformer | 48.0 | 15.2 | 9.1 | 8.7
ICAR (Ours) | 36.8 | 11.8 | 6.2 | 7.1

Table 3: SFID Results on DeepRooms, STL(F: fashion, H: home) and
Street2Shop(S2S). Our approach can improve SFID accuracy by 11.2 (23.3%,
DeepRooms) and 2.9 (31.8%, STL-Home) on furniture and by 3.4 (22.3%, STL-F)
and 1.6 (18.4%, S2S) on fashion images, respectively, over OutfitTransformer
(Sarkar et al. 2022). Figure 7: Learned Scene Image Embedding Clustering
Results. To validate the style implicitly learned through our network, First
column is the t-SNE on the randomly sampled 2k STL-home and STL-fashion test
in split scene images (Columns 2-5).

#### Qualitative Results

Next we present qualitative results on scene-based cross-domain CIR results.
We sample scene images from the STL-home and STL-fashion dataset test-split.
Then we specify the item categories. As shown in Figure 6, from left to right
and from top to bottom, our model generates diverse style recommendations,
including but not only Contemporary (e.g. 1st, 2nd, 4th and 9th set), Classic
(e.g. 3rd, 7th and 8th set), Industrial (e.g. 5th set), Rustic (e.g. 6th set).
Refer to supplementary materials for more examples.

#### Sequence Masking Validation

Method | FITB $\uparrow$ | Condition
---|---|---
ICAR - fixed length masking | 82.2 | Predict category
ICAR - random length masking | 83.9 | Predict category
ICAR - fixed length masking | 84.8 | Given category
ICAR - random length masking | 87.1 | Given category

Table 4: Masking Method Comparison. Our random length masking outperforms
fixed length masking.

In our FBT model, we propose a random shuffle and random length masking
technique for unordered set generation. Here we ablate our proposed random
length masking technique. In this experiment, we compare the results when we
apply random length masking vs. fixed length masking to the input sequence. As
shown in Table 4, we show that our random length masking approach outperforms
fixed length masking for unordered set generation with or without target
category been given.

#### Visual Compatibility Embedding Validation

To validate our model’s ability in learning the visual style implicitly, we
perform t-SNE analysis on the learned embedding of the scene images sampled
from the STL-home and STL-fashion datasets. In Figure 7, we show the
clustering results on the left and the scene images from the clusters in
column 2-5. The cluster labels are computed using the k-means ($K=6$) method.
We find some typical interior design styles such as the industrial style,
rustic style and contemporary style in the clusters. More interestingly, we
also find that our model pays attention to color. For example, in the C home
scene-image cluster, our model learns to extract the Morandi type of color
scheme. Given that our model is trained in a self-supervised way, we observe
some mix in style within some of the clusters.

#### Human Perception Validation

Figure 8: Human Ratings on Different Datasets. Our SFID score correlates
better than SOTA with human judgement.

We conduct user studies by asking interior design experts to rate (Good: items
are compatible, Neutral: one or two items are incompatible) 100 generated sets
of furniture item images from the Random sample, OutfitTransformer(Sarkar et
al. 2022), our method (ICAR), and Ground Truth for different datasets. As
shown in Figure 8, ICAR outperforms all other methods, and those results are
consistent with our SFID score (Table 3). We further compute the Pearson
correlation coefficient to measure the association between SFID and human
rating score (normalized as Ground truth score/Method score). We obtain a 0.7
average Pearson correlation value ([-1, 1]). This demonstrates that there is a
strong positive association between our SFID score and human perception.

###  4.4 Similarity Learning Results

Here we validate the effectiveness of focusing on visual similarity learning
in the first stage. We compare our method against four different types of
learning target i.e. image classification, image reconstruction, image
generation and image representation. And for each type of learning target, we
choose the state-of-the-art method (shown in the first column in Table 5)
except the image representation. For image representation, we train the model
in a contrastive learning manner. All the models in this experiment are
trained using the DeepRooms (Gadde, Feng, and Martinez 2021) dataset. As shown
in Table 5, our model performs the best when the first stage focus on the
visual similarity learning.

Method | FITB $\uparrow$ | Learning Target Type
---|---|---
ICAR - VQGAN | $73.1$ | Reconstruction
ICAR - Swin | $80.5$ | Classification
ICAR - BEiT | $73.3$ | Generation
ICAR - Contrastive Learning | $75.0$ | Image Representation
ICAR - Visual Similarity | $87.1$ | Retrieval

Table 5: Similarity Learning: visual similarity learning is the most suitable
for scene-based CIR. VQGAN (Esser, Rombach, and Ommer 2021), Swin (Liu et al.
2022), BEiT (Bao, Dong, and Wei 2021)

##  5 Conclusion, Limitation, and Future Work

In this paper, we introduce a compatibility learning framework using a novel
category-aware “Flexible Bidirectional Transformer” (FBT), based on Visual
similarity and complementarity to effectively retrieve a set of stylistically
compatible items across domains, given a scene-image query. This learning
framework is also generalizable and can be extended to other types of
conditional cross-domain CIR tasks.

While our results show a promising direction, there is more to be explored.
First, there are other informations, like text descriptors and video contents,
to be used as compatibility signals. Secondly, the metric to measure styles
can be further broadened to encode regional preference and cultural influence,
interpretation of styles can be expanded to include more global and diverse
perspectives.

## References

  * Ak et al. (2018) Ak, K. E.; Kassim, A. A.; Lim, J. H.; and Tham, J. Y. 2018.  Learning attribute representations with localization for flexible fashion search.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 7708–7717.
  * Bao, Dong, and Wei (2021) Bao, H.; Dong, L.; and Wei, F. 2021.  Beit: Bert pre-training of image transformers.  _arXiv preprint arXiv:2106.08254_.
  * Chen et al. (2015) Chen, Q.; Huang, J.; Feris, R.; Brown, L. M.; Dong, J.; and Yan, S. 2015.  Deep domain adaptation for describing people based on fine-grained clothing attributes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 5315–5324.
  * Chen et al. (2019) Chen, W.; Huang, P.; Xu, J.; Guo, X.; Guo, C.; Sun, F.; Li, C.; Pfadler, A.; Zhao, H.; and Zhao, B. 2019.  POG: personalized outfit generation for fashion recommendation at Alibaba iFashion.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, 2662–2670.
  * Cheng et al. (2021) Cheng, W.-H.; Song, S.; Chen, C.-Y.; Hidayati, S. C.; and Liu, J. 2021.  Fashion meets computer vision: A survey.  _ACM Computing Surveys (CSUR)_ , 54(4): 1–41.
  * Cheng et al. (2017) Cheng, Z.-Q.; Wu, X.; Liu, Y.; and Hua, X.-S. 2017.  Video2shop: Exact matching clothes in videos to online shopping images.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 4048–4056.
  * Di et al. (2013) Di, W.; Wah, C.; Bhardwaj, A.; Piramuthu, R.; and Sundaresan, N. 2013.  Style finder: Fine-grained clothing style detection and retrieval.  In _Proceedings of the IEEE Conference on computer vision and pattern recognition workshops_ , 8–13.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * El-Nouby et al. (2021) El-Nouby, A.; Neverova, N.; Laptev, I.; and Jégou, H. 2021.  Training vision transformers for image retrieval.  _arXiv preprint arXiv:2102.05644_.
  * Esser, Rombach, and Ommer (2021) Esser, P.; Rombach, R.; and Ommer, B. 2021.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 12873–12883.
  * Fu et al. (2021) Fu, H.; Cai, B.; Gao, L.; Zhang, L.-X.; Wang, J.; Li, C.; Zeng, Q.; Sun, C.; Jia, R.; Zhao, B.; et al. 2021.  3d-front: 3d furnished rooms with layouts and semantics.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 10933–10942.
  * Gadde, Feng, and Martinez (2021) Gadde, R.; Feng, Q.; and Martinez, A. M. 2021.  Detail Me More: Improving GAN’s Photo-Realism of Complex Scenes.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 13950–13959.
  * Gallagher and Chen (2008) Gallagher, A. C.; and Chen, T. 2008.  Clothing cosegmentation for recognizing people.  In _2008 IEEE conference on computer vision and pattern recognition_ , 1–8. IEEE.
  * Girdhar et al. (2019) Girdhar, R.; Carreira, J.; Doersch, C.; and Zisserman, A. 2019.  Video action transformer network.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 244–253.
  * Guan et al. (2022a) Guan, W.; Jiao, F.; Song, X.; Wen, H.; Yeh, C.-H.; and Chang, X. 2022a.  Personalized Fashion Compatibility Modeling via Metapath-guided Heterogeneous Graph Learning.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 482–491.
  * Guan et al. (2022b) Guan, W.; Song, X.; Zhang, H.; Liu, M.; Yeh, C.-H.; and Chang, X. 2022b.  Bi-directional Heterogeneous Graph Hashing towards Efficient Outfit Recommendation.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , 268–276.
  * Guan et al. (2021) Guan, W.; Wen, H.; Song, X.; Yeh, C.-H.; Chang, X.; and Nie, L. 2021.  Multimodal compatibility modeling via exploring the consistent and complementary correlations.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , 2299–2307.
  * Hadi Kiapour et al. (2015) Hadi Kiapour, M.; Han, X.; Lazebnik, S.; Berg, A. C.; and Berg, T. L. 2015.  Where to buy it: Matching street clothing photos in online shops.  In _Proceedings of the IEEE international conference on computer vision_ , 3343–3351.
  * Han et al. (2017) Han, X.; Wu, Z.; Jiang, Y.-G.; and Davis, L. S. 2017.  Learning fashion compatibility with bidirectional lstms.  In _Proceedings of the 25th ACM international conference on Multimedia_ , 1078–1086.
  * Hermans, Beyer, and Leibe (2017) Hermans, A.; Beyer, L.; and Leibe, B. 2017.  In defense of the triplet loss for person re-identification.  _arXiv preprint arXiv:1703.07737_.
  * Heusel et al. (2017) Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017\.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30.
  * Hsiao and Grauman (2017) Hsiao, W.-L.; and Grauman, K. 2017.  Learning the latent” look”: Unsupervised discovery of a style-coherent embedding from fashion images.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 4203–4212.
  * Hsiao and Grauman (2018) Hsiao, W.-L.; and Grauman, K. 2018.  Creating capsule wardrobes from fashion images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 7161–7170.
  * Hu, Yi, and Davis (2015) Hu, Y.; Yi, X.; and Davis, L. S. 2015.  Collaborative fashion recommendation: A functional tensor factorization approach.  In _Proceedings of the 23rd ACM international conference on Multimedia_ , 129–138.
  * Huang et al. (2015) Huang, J.; Feris, R. S.; Chen, Q.; and Yan, S. 2015.  Cross-domain image retrieval with a dual attribute-aware ranking network.  In _Proceedings of the IEEE international conference on computer vision_ , 1062–1070.
  * Jun et al. (2019) Jun, H.; Ko, B.; Kim, Y.; Kim, I.; and Kim, J. 2019.  Combination of multiple global descriptors for image retrieval.  _arXiv preprint arXiv:1903.10663_.
  * Kalantidis, Kennedy, and Li (2013) Kalantidis, Y.; Kennedy, L.; and Li, L.-J. 2013.  Getting the look: clothing recognition and segmentation for automatic product suggestions in everyday photos.  In _Proceedings of the 3rd ACM conference on International conference on multimedia retrieval_ , 105–112.
  * Kang et al. (2019) Kang, W.-C.; Kim, E.; Leskovec, J.; Rosenberg, C.; and McAuley, J. 2019.  Complete the look: Scene-based complementary product recommendation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10532–10541.
  * Kiapour et al. (2014) Kiapour, M. H.; Yamaguchi, K.; Berg, A. C.; and Berg, T. L. 2014.  Hipster wars: Discovering elements of fashion styles.  In _European conference on computer vision_ , 472–488. Springer.
  * Li et al. (2020) Li, X.; Wang, X.; He, X.; Chen, L.; Xiao, J.; and Chua, T.-S. 2020.  Hierarchical fashion graph network for personalized outfit recommendation.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 159–168.
  * Li et al. (2017) Li, Y.; Cao, L.; Zhu, J.; and Luo, J. 2017.  Mining fashion outfit composition using an end-to-end deep learning approach on set data.  _IEEE Transactions on Multimedia_ , 19(8): 1946–1955.
  * Lin, Tran, and Davis (2020) Lin, Y.-L.; Tran, S.; and Davis, L. S. 2020.  Fashion outfit complementary item retrieval.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3311–3319.
  * Liu et al. (2012a) Liu, S.; Feng, J.; Song, Z.; Zhang, T.; Lu, H.; Xu, C.; and Yan, S. 2012a.  Hi, magic closet, tell me what to wear!  In _Proceedings of the 20th ACM international conference on Multimedia_ , 619–628.
  * Liu et al. (2012b) Liu, S.; Song, Z.; Wang, M.; Xu, C.; Lu, H.; and Yan, S. 2012b.  Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set.  In _Proceedings of the 20th ACM international conference on Multimedia_ , 1335–1336.
  * Liu et al. (2022) Liu, Z.; Hu, H.; Lin, Y.; Yao, Z.; Xie, Z.; Wei, Y.; Ning, J.; Cao, Y.; Zhang, Z.; Dong, L.; et al. 2022.  Swin transformer v2: Scaling up capacity and resolution.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12009–12019.
  * Liu et al. (2016) Liu, Z.; Luo, P.; Qiu, S.; Wang, X.; and Tang, X. 2016.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1096–1104.
  * Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * McAuley et al. (2015) McAuley, J.; Targett, C.; Shi, Q.; and Van Den Hengel, A. 2015.  Image-based recommendations on styles and substitutes.  In _Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval_ , 43–52.
  * Radenović, Tolias, and Chum (2018) Radenović, F.; Tolias, G.; and Chum, O. 2018.  Fine-tuning CNN image retrieval with no human annotation.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(7): 1655–1668.
  * Sarkar et al. (2022) Sarkar, R.; Bodla, N.; Vasileva, M.; Lin, Y.-L.; Beniwal, A.; Lu, A.; and Medioni, G. 2022.  OutfitTransformer: Outfit Representations for Fashion Recommendation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2263–2267.
  * Shih et al. (2018) Shih, Y.-S.; Chang, K.-Y.; Lin, H.-T.; and Sun, M. 2018.  Compatibility family learning for item recommendation and generation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Simo-Serra and Ishikawa (2016) Simo-Serra, E.; and Ishikawa, H. 2016.  Fashion style in 128 floats: Joint ranking and classification using weak data for feature extraction.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 298–307.
  * Song et al. (2017a) Song, X.; Feng, F.; Liu, J.; Li, Z.; Nie, L.; and Ma, J. 2017a.  Neurostylist: Neural compatibility modeling for clothing matching.  In _Proceedings of the 25th ACM international conference on Multimedia_ , 753–761.
  * Song et al. (2017b) Song, Y.; Li, Y.; Wu, B.; Chen, C.-Y.; Zhang, X.; and Adam, H. 2017b.  Learning unified embedding for apparel recognition.  In _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2243–2246.
  * Su et al. (2021) Su, T.; Song, X.; Zheng, N.; Guan, W.; Li, Y.; and Nie, L. 2021.  Complementary factorization towards outfit compatibility modeling.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , 4073–4081.
  * Tan et al. (2019) Tan, R.; Vasileva, M. I.; Saenko, K.; and Plummer, B. A. 2019.  Learning similarity conditions without explicit supervision.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 10373–10382.
  * Tangseng, Yamaguchi, and Okatani (2017) Tangseng, P.; Yamaguchi, K.; and Okatani, T. 2017.  Recommending outfits from personal closet.  In _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2275–2279.
  * Taraviya et al. (2021) Taraviya, M.; Beniwal, A.; Lin, Y.-L.; and Davis, L. 2021.  Personalized compatibility metric learning.  In _KDD Workshop_ , volume 1.
  * Teh, DeVries, and Taylor (2020) Teh, E. W.; DeVries, T.; and Taylor, G. W. 2020.  Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis.  In _European Conference on Computer Vision_ , 448–464. Springer.
  * Tran et al. (2019) Tran, S.; Du, M.; Chanda, S.; Manmatha, R.; and Taylor, C. 2019.  Searching for Apparel Products from Images in the Wild.  _arXiv preprint arXiv:1907.02244_.
  * Vasileva et al. (2018) Vasileva, M. I.; Plummer, B. A.; Dusad, K.; Rajpal, S.; Kumar, R.; and Forsyth, D. 2018.  Learning type-aware embeddings for fashion compatibility.  In _Proceedings of the European conference on computer vision (ECCV)_ , 390–405.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Veit et al. (2015) Veit, A.; Kovacs, B.; Bell, S.; McAuley, J.; Bala, K.; and Belongie, S. 2015.  Learning visual clothing style with heterogeneous dyadic co-occurrences.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 4642–4650.
  * Yamaguchi, Hadi Kiapour, and Berg (2013) Yamaguchi, K.; Hadi Kiapour, M.; and Berg, T. L. 2013.  Paper doll parsing: Retrieving similar styles to parse clothing items.  In _Proceedings of the IEEE international conference on computer vision_ , 3519–3526.
  * Yamaguchi et al. (2012) Yamaguchi, K.; Kiapour, M. H.; Ortiz, L. E.; and Berg, T. L. 2012.  Parsing clothing in fashion photographs.  In _2012 IEEE Conference on Computer vision and pattern recognition_ , 3570–3577. IEEE.
  * Yang and Yu (2011) Yang, M.; and Yu, K. 2011.  Real-time clothing recognition in surveillance videos.  In _2011 18th IEEE international conference on image processing_ , 2937–2940. IEEE.
  * Yang et al. (2019) Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R. R.; and Le, Q. V. 2019\.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ , 32.
  * Yu et al. (2012) Yu, L.-F.; Yeung, S. K.; Terzopoulos, D.; and Chan, T. F. 2012.  DressUp!: outfit synthesis through automatic optimization.  _ACM Trans. Graph._ , 31(6): 134–1.
  * Zhai et al. (2017) Zhai, A.; Kislyuk, D.; Jing, Y.; Feng, M.; Tzeng, E.; Donahue, J.; Du, Y. L.; and Darrell, T. 2017.  Visual discovery at pinterest.  In _Proceedings of the 26th International Conference on World Wide Web Companion_ , 515–524.
  * Zhai and Wu (2018) Zhai, A.; and Wu, H.-Y. 2018.  Classification is a strong baseline for deep metric learning.  _arXiv preprint arXiv:1811.12649_.
  * Zheng et al. (2021) Zheng, N.; Song, X.; Niu, Q.; Dong, X.; Zhan, Y.; and Nie, L. 2021.  Collocation and try-on network: Whether an outfit is compatible.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , 309–317.

ICAR: Image-based Complementary Auto Reasoning - Supplementary Material

##  6 Supplementary Material

###  6.1 Implementation Details

For our visual similarity learning, we apply a standard ResNet50 followed by a
combination of average pooling and generalized mean pooling global
descriptors. We average the two pooling global descriptors as the final
2048-dimensional image embedding. This model is trained with Normalized
Softmax Loss and Soft-margin Triplet Loss for retrieval task.

In our complementarity learning, we employ 6 self-attention layers with 8
heads. The input and output token dimension is 256. We set the number of
output query to 1 for training and 9 for testing. During the training phase,
we randomly shuffle all the items in the scene and randomly mask out
$[0,M],M\leq N$ (where N is the total number of items in the scene) items in
the sequence as input. Then the model predicts the next $(M+1)th$ item in the
sequence. We put the scene embedding at the beginning of input set as the
start token. We set a zero vector as the end token. During testing phase, we
use the auto-regressive manner to predict products and stop when the predicted
output number reach 9 or the predicted output is the end token. For
OutfitTransformer(Sarkar et al. 2022), we use our Visual Similarity model as
encoder and take it as the visual compatibility learning model. To keep the
fair competition, we replace the text input with the label input and keep all
the training setting the same as in other experiments in this paper.

Figure 9: SFID Composed Images Comparison. Here we show how we compose the set
images. In (a), we randomly place the item images with fixed size (same
height, aspect ratio reserved). In (b), we place the item images using the
bbox of the item in the original scene images. After the study, we apply (a)
in our SFID computation.  Figure 10: FBT: Flexible Bidirectional Transformer.
We randomly sample $M\in[0,N]$ items from the total item number $N$ of items
(in a scene) as input set, and the $(M+1)_{th}$ item not in the input set as
output target. We put the scene embedding at the beginning of input set, and
take the scene embedding as the start token $E_{X_{s}}$. We set a zero vector
as the end token $E_{X_{e}}$. The output will be the end token when the input
set contains all the items in the scene.

###  6.2 More Details of Our Approach

#### Visual Similarity Learning

For visual similarity learning, we apply a standard ResNet50 followed by a
combination of average pooling and generalized mean pooling global
descriptors. We average the two pooling global descriptors as the final
2048-dimensional image embedding. This model is trained with Normalized
Softmax Loss and Soft-margin Triplet Loss for retrieval task.

##### Normalized Softmax Loss (Zhai and Wu 2018)

The core idea is to use the standard cross-entropy loss for metric learning.
The main insight is that there is no need to do exhaustive all-pair
comparisons as would be ideally done with contrastive and triplet losses.
Instead, the comparison can be done only to “class proxies” which can be
thought of as class representatives or class centroids.

##### Soft-margin Triplet Loss (Hermans, Beyer, and Leibe 2017)

In Triplet loss, the hinge function can help avoid over correcting. But for
similarity learning, it can be beneficial to over correct. In this loss, the
hinge function is replaced with a softplus function $ln(1+exp(\cdot))$. The
softplus function has similar behavior to the hinge, but it decays
exponentially.

#### Complementarity Learning with Flexible Bidirectional Transformer

As shown in Figure 10, we randomly mask out $[0,M],M\leq N$ (where N is the
total number of items in the scene) items in the sequence as input. Then the
model predicts the next $(M+1)th$ item in the sequence. We put the scene
embedding at the beginning of input set as the start token. We set a zero
vector as the end token. The output will be the end token when the input set
contains all the products in the scene.

#### SFID

For better visual sets compatibility evaluation, we propose the SFID metric.
The goal is to propose a metric that correlates the compatibility of a
generated set with the ground-truth design set of objects. How to compose a
set of object images to form an image is a critical part of the SFID design.

We identify three major factors impacting the effectiveness of the score when
composing the set image, i.e. background, position, and size of the objects.
In the ideal scenario, the score computed from the composed images of the
generated sets and the ground truth sets should be as low as possible. To
compute the metric, we compute images from the generated and groundtruth
object sets. And for all the experiments, we set the maximum number of items
to be included in one composed image to 5 with the observation of number of
items exist in most of the scene images.

From the experiments, we find that the background images have larger impact
than the compatibility between two sets of items (shown first set in Table 6).
And varying the size produces negative effect on the score (shown in the
second set in Table 6). But, according to the third set in Table 6, location
has small impact on the score compared to other two factors. Thus, we conclude
to use white background, fix the size and randomly position the items when
composing the set image when computing the SFID score. Our perceptual study
4.3 shows strong positive correlation ($\text{p-value}=0.7$) between SFID and
human perception.

After studying the major factors, we conclude fixing the size and randomly
place the item images on a white background (shown in Figure 9a) is better
than other setting with bbox (from the scene image) (shown in Figure 9b).

Factor | Pair | SFID $\downarrow$
---|---|---
Background | bbox GT VS (bbox GT+scene image) | 98.1
bbox Neg VS (bbox GT+scene image) | 107.5
bbox GT VS bbox Neg | $75.2$
Size | bbox GT VS non-bbox GT | 104.9
bbox GT VS bbox Neg | $75.2$
non-bbox GT VS non-bbox Neg | $49.0$
Position | non-bbox GT VS random non-bbox GT | 35.1

Table 6: Composing Method Comparison. Here bbox means using the bbox from the
original scene image to place in the composed image and non-bbox means using a
fixed size. GT is the ground truth set and Neg is the randomly chosen set.
With the results, we find using white background and a fixed size for all the
items and random placement in the composed set of images produces the lowest
score.

###  6.3 More Qualitative Evaluation Results

In this qualitative evaluation, we give our ICAR model: 1) the sampled scene
images from the STL-home and STL-fashion dataset test-split; 2) a set of item
categories; 3) a pool of cross-domain item images. We test our model on
different input setup: different sets of item categories and seen or unseen
cross-domain item pool.

For home recommendation, we test living room and bedroom scene generation. For
living room scene (shown in Figure 12), we set sofa, chair, coffee table, end
table, tv-stand, and ceiling light as given categories. For bedroom scene
(shown in Figure 13), we use bed, nightstand, vanity, dresser, and chair as
given categories. For fashion outfit generation (shown in Figure 14), we use
top, coat, pant, shoes, handbag, and necklace as given categories. Both living
room and bedroom scene generation, we test our model’s ability in generalizing
to unseen domain (3D-FRONT (Fu et al. 2021) synthetically rendered).

As shown in Figure 12 and Figure 13, our model can generate stylistically
compatible object set from unseen domain. For fashion examples, we test our
model on the STL white-background object image pool. For home recommendation
(shown in Figure 15), we also show results of generating from the STL white-
background object image pool. As shown in Figure 14 and Figure 15, the
recommended sets in both scenarios are style compatible.

In summary, our model can generate high quality style compatible items given
different set of categories and can generalize to unseen domains.

Figure 11: Binary FITB Results. We show the FITB results on the STL-home
datasets when there are two candidates (second row). Our model chooses the
item in the green box.

In addition, we conduct the binary FITB (1 negative candidate) experiments
(Kang et al. 2019), as shown in Figure 11, our model chooses the more
stylistically compatible item in the green box.

Figure 12: More Visualization for living room. In this example, given an
inspirational home scene image (sampled from STL-home as shown in column 1, 5)
with a pool of products (from 3D-FRONT). Our model auto-regressively retrieves
a set of stylistically compatible items (column 2, 3, 4, 6,7,8 in the
picture).  Figure 13: More Visualization for bedroom. In this example, given
an inspirational home scene image (sampled from STL-home as shown in column 1,
5) with a pool of products (from 3D-FRONT). Our model auto-regressively
retrieves a set of stylistically compatible items (column 2, 3, 4, 6,7,8 in
the picture).  Figure 14: More Visualization for Fashion. In this example,
given an inspirational fashion scene image (sampled from STL-fashion as shown
in column 1, 5) with a pool of products (from STL-fashion). Our model auto-
regressively retrieves a set of stylistically compatible items (column 2, 3,
4, 6, 7, 8 in the picture).  Figure 15: More Visualization for STL-home. In
this example, given an inspirational home scene image (sampled from STL-home
as shown in column 1, 5) with a pool of products (from STL-home). Our model
auto-regressively retrieves a set of stylistically compatible items (column 2,
3, 4, 6,7,8 in the picture).