# Cross-Domain Product Representation Learning for Rich-Content E-Commerce

Xuehan Bai Yan Li 11footnotemark: 1 Yanhua Cheng Wenjie Yang Quan Chen
22footnotemark: 2 Han Li
Kuaishou Technology
{baixuehan03, liyan26, chengyanhua, chenquan06<EMAIL_ADDRESS><EMAIL_ADDRESS>
https://github.com/adxcreative/COPE Equal contribution.Corresponding authors.

###### Abstract

The proliferation of short video and live-streaming platforms has
revolutionized how consumers engage in online shopping. Instead of browsing
product pages, consumers are now turning to rich-content e-commerce, where
they can purchase products through dynamic and interactive media like short
videos and live streams. This emerging form of online shopping has introduced
technical challenges, as products may be presented differently across various
media domains. Therefore, a unified product representation is essential for
achieving cross-domain product recognition to ensure an optimal user search
experience and effective product recommendations. Despite the urgent
industrial need for a unified cross-domain product representation, previous
studies have predominantly focused only on product pages without taking into
account short videos and live streams. To fill the gap in the rich-content
e-commerce area, in this paper, we introduce a large-scale cRoss-dOmain
Product rEcognition dataset, called ROPE. ROPE covers a wide range of product
categories and contains over 180,000 products, corresponding to millions of
short videos and live streams. It is the first dataset to cover product pages,
short videos, and live streams simultaneously, providing the basis for
establishing a unified product representation across different media domains.
Furthermore, we propose a Cross-dOmain Product rEpresentation framework,
namely COPE, which unifies product representations in different domains
through multimodal learning including text and vision. Extensive experiments
on downstream tasks demonstrate the effectiveness of COPE in learning a joint
feature space for all product domains.

##  1 Introduction

Figure 1: Illustrate the importance of cross-domain product representation for
rich-content e-commerce. There are two solid demands for such new e-commerce:
1) The platform needs to provide accurate product results of product page,
short video, and live streaming corresponding to user’s query; 2) The platform
is able to recommend similar products of interest according to user’s behavior
history. Both of the two tasks highly depend on a high-performance cross-
domain product representation. Examples are from the popular rich-content
e-commerce platforms, including TikTok, Kwai and Taobao.

Recently, with the vicissitude in time spent by customers on entertainment
media, the way consumers shop online has transformed significantly, and rich-
content e-commerce is becoming increasingly popular. In the rich-content
e-commerce area, the products are sold not only with traditional product pages
but also with dynamic and interactive media formats, _i.e_., short videos and
live streams. As a result, consumers are increasingly relying on these formats
to make informed purchase decisions. This shift has facilitated a more
engaging shopping experience, bridging the gap between consumers and sellers
while presenting new opportunities for platforms to capitalize on.

Despite the advantages of rich-content e-commerce, it presents several
technical challenges. One of the most significant challenges is the
inconsistency in product presentation across different media domains. For
instance, a product may appear entirely different in a live stream than on a
traditional product page. Establishing a unified product representation across
different domains is curial and desperately needed in industrial scenarios to
address the inconsistency problem. As shown in Figure 1, when users search for
a particular product, the unified product representation ensures an enjoyable
search experience that the returned product pages, short videos, and live
streams precisely describe the same product. When the platform recommends
products for users, the unified product representations are beneficial to
exploiting users’ consuming behaviors in different media for comprehensive
product recommendations.

In spite of the urgent industrial need for a unified cross-domain product
representation, prior efforts have concentrated solely on the product page
domain. The most common way to learn the product representations is to train a
product classification model with product images and titles [14, 12, 26, 27].
However, such representations are far from acceptable in rich-content
e-commerce. Specifically, the pictures displayed on the product pages are
generally well-shot by professionals, while in short videos and live
streaming, the posture of the products and the positions they occupy in the
scene often change a lot. Moreover, in live streams and short videos, it is
not always guaranteed that products are visible at every moment. Short videos
may be mixed with the story plot, while live streams may contain chats between
the sellers and their audiences. These contents are generally irrelevant to
the products. To bridge this gap and push forward the related research, we
collect a large amount of real data from online shopping platforms and present
the first large-scale cRoss-dOmain Product rEcognition dataset, ROPE. Our
dataset contains 3,056,624 product pages, 5,867,526 short videos, and
3,495,097 live streams of 189,958 different products. It covers all product
categories of online shopping scenarios. To the best of our knowledge, ROPE is
the rich-content e-commerce dataset, including product pages, short videos,
and live streams. We hope that the publication of ROPE will attract more
researchers to the field of content commerce and drive the development of
related technologies.

In addition to the ROPE dataset, we propose a Cross-dOmain Product
rEpresentation baseline, COPE, that maps product pages, short videos, and live
streams into the same feature space to build a unified product representation.
Based on the ROPE dataset, we evaluate the COPE model on the cross-domain
retrieval and few-shot classification tasks. The experimental results show
significant improvement over the existing state-of-the-arts.

In summary, our contributions are as follows:

1) As far as we know, our work is the first exploration that tries to build a
unified product representation across the product pages, short videos, and
live streams to meet the urgent industrial need of the emerging rich-content
e-commerce.

2) We collect realistic data from online e-commerce platforms and build a
large-scale cRoss-dOmain Product rEcognition dataset, ROPE. It contains
3,056,624 product pages, 5,867,526 short videos, and 3,495,097 live streams
belonging to 189,958 different products. The included product categories cover
the full spectrum of online shopping scenarios.

3) A Cross-dOmain Product rEpresentation model, COPE, is proposed to learn the
cross-domain product representations. The experimental results prove the
superiority of the COPE model to the existing methods.

##  2 Related Work

###  2.1 E-Commerce Datasets

A large number of e-commerce datasets have been proposed to advance the
technical developments in the area [2, 6, 8, 11, 25, 33, 32]. The earlier
datasets traditionally have limited size. Corbiere _et al_. introduce the
Dress Retrieval [6] dataset in 2017, which has 20,000 pairs of the product
image and text pairs. Rostamzadeh _et al_. propose the FashionGen [25]
dataset, which includes 293,000 samples, covering only 48 product categories.
In recent years, large-scale product recognition datasets have been introduced
with the development of deep-learning-based methods. Product 1M [33] increases
the scale of the training samples to a million level, but all samples come
from 48 cosmetic brands. The coverage of the products is quite limited. MEP-3M
[2] dataset includes more than three million samples, and each sample consists
of the product image, product title, and hierarchical classification labels.
However, all these datasets focus solely on the product page domain. In the
experiment section, we will demonstrate that the representations learned on
the product page domain are insufficient to handle the cross-domain product
recognition task. The most related datasets to our ROPE dataset are M5Product
[8] and MovingFashion [11]. M5Product comprises six million samples, and for
each sample, it provides product images, product titles, category labels,
attribute tables, assigned advertising videos, and audios extracted from
videos. However, the provided videos in M5Product are quite different from the
live streams introduced in our ROPE dataset. The videos in M5Product all come
from the product page and are usually closely related to the advertised
products, with products displayed in the center and described throughout. By
contrast, there are many chat contents between the sellers and audiences in
live streams of ROPE, which are unrelated to the products. Furthermore, the
poses and locations of the products vary significantly in live streams, making
the ROPE dataset more challenging for product recognition.

Dataset | Samples | Categories | Products | Domains
---|---|---|---|---
FashionGen [25] | 293,008 | 48 | 78850 | product page
Dress Retrieval [6] | 20,200 | 50 | 20,200 | product page
Product1M [33] | 1,182,083 | 458 | 92,200 | product page
MEP-3M [2] | 3,012,959 | 599 | - | product page
M5Product [8] | 6,313,067 | 6,232 | - | product page
MovingFashion [11] | 15,000 | - | - | product page/short video
ROPE(ours) | 12,027,068 | 1,396 | 187,431 | product page/short video/live streaming
Table 1:  Comparisons with other product datasets. “-” means not mentioned.

MovingFashion [11] also focuses on aligning videos and product pages. It only
comprises 15,000 videos, covering 13 product categories. The scale of the
MovingFahsion is much smaller than our ROPE dataset, which covers more than
1,000 product categories and provides million-level samples of the product
page, short video, and live streaming domains.

###  2.2 Cross-Domain Retrieval Methods

Existing cross-domain retrieval methods typically learn unified
representations between the visual and text domains. Some of the most popular
models following the single-stream architecture, such as VL-bert [28],
Imagebert [23], Videobert [29], Visualbert [13], and Uniter [4]. These models
concatenate visual and text features and then use a binary classifier to
predict whether the image-text pairs match. Although these methods usually
perform better, they suffer from inferior inference efficiency. The ViLBERT
[18], LXMERT [30], CLIP [24], and CoOp [34] utilize the two-stream
architecture. In this approach, the visual and text features are extracted
using independent encoders, and the visual-text similarity is efficiently
calculated using the dot product operation. The proposed COPE model learns
representations of different domains using contrastive loss to ensure
efficient cross-domain retrieval.

##  3 ROPE Dataset

###  3.1 Data Collection and Cleaning

We collect data from the online e-commerce platform over 1,300 product
categories. Three steps are taken to construct the ROPE dataset. Firstly, we
collect a large amount of unsupervised multi-modal samples from the product
page domain, short video domain, and live streaming domain. For the product
page domain, we offer the product images and titles; for the short video and
live streaming domains, the extracted frames and ASR (automatic speed
recognition) texts are provided. The resulting dataset includes over 200
million samples and is defined as $\mathcal{D}_{\textit{raw}}$.

Secondly, a small portion of $\mathcal{D}_{\textit{raw}}$ (0.1%, 200K data
points) is sampled and defined as $\mathcal{D}_{\textit{sample}}$. For each
sample in $\mathcal{D}_{\textit{sample}}$, we ask the human annotators to find
other samples from $\mathcal{D}_{\textit{raw}}$ that shares the same product.
To reduce the annotation costs, the extracted features with the public Chinese
CLIP model [31]111For short videos and live streamings, the average of frame-
level image features are adopted as the visual representations. The visual and
text features are concatenated as the final multi-model representations for
retrieving relevant samples. are utilized to find relevant samples for further
human checkout. The annotated samples are used to train a baseline COPE model.

Thirdly, for remaining unannotated samples in $\mathcal{D}_{\textit{raw}}$,
the baseline COPE model is employed to filter out relevant samples, and only
the samples whose matching scores are higher than 0.7 are kept. Afterward, the
product pages, short videos, and live streams belonging to the same product
are aggregated. We only retain the completed paired samples, including data
from all three domains.

Figure 2: The distribution of training samples over product categories. It is
biased and long-tailed. Figure 3: The overall framework of the proposed COPE
model. The text encoder and visual encoder are utilized to extract features
from the single modality, and the fusion encoder is adopted to aggregate the
two features. To model the temporal information in videos and live streams, we
insert the cross-frame communication transformer into each block of the visual
encoder. The multi-frame integration transformer is placed at the top of the
visual encoder to summarize the whole video’s representation.

###  3.2 Datasets Statistics

The final ROPE dataset comprises 3,056,624 product pages, 5,867,526 short
videos, and 3,495,097 live streams associated with 189,958 products. Table 1
compares the ROPE and previous product datasets. We divided the ROPE dataset
into train and test sets. The train set has 187,431 products with 3,025,236
product pages, 5,837,716 short videos, and 3,464,116 live streams. On average,
each product has 16 product pages, 31 short videos, and 18 live streams. The
distribution of training samples across product categories is illustrated in
Figure 2, showing a long-tailed pattern that reflects online shopping
interests. The top five categories are Carpet, Calligraphy/Painting, Quilt
Cover, Emerald, and Sheet. The test set contains 2,527 products, with 31,388
product pages, 29,810 short videos, and 30,981 live streams. The average
duration of short videos and live streams is 31.78 seconds and 129.09 seconds,
respectively, and each product has an average of 12 product pages, 11 short
videos, and 12 live streamings. The product categories in the test set are
different from those in the train set to ensure an accurate evaluation, and
human annotators have thoroughly reviewed the test set.

###  3.3 Evaluation Tasks

We propose two evaluation tasks based on the ROPE dataset to verify the
unified cross-domain product representations. The first is the cross-domain
product retrieval task, which aims to find the matched samples from the
identical product from the two domains. There are six variations of the task:
$\mathrm{P}$$\rightarrow$$\mathrm{V}$, $\mathrm{V}$$\rightarrow$$\mathrm{P}$,
$\mathrm{P}$$\rightarrow$$\mathrm{L}$, $\mathrm{L}$$\rightarrow$$\mathrm{V}$,
$\mathrm{V}$$\rightarrow$$\mathrm{L}$, and
$\mathrm{L}$$\rightarrow$$\mathrm{V}$, where $\mathrm{P}$, $\mathrm{V}$, and
$\mathrm{L}$ indicate product page domain, short video domain, and live
streaming domain, respectively. The second one is the cross-domain few-shot
($k$=1) classification task. Similar to the retrieval task, it also has six
variations.

Taking the $\mathrm{P}$$\rightarrow$$\mathrm{V}$ variation as an example, we
elaborate on the detailed evaluation processes for the two tasks. For the
retrieval task, we collect all the short videos in the test set as the gallery
set $G_{\mathrm{V}}$ and regard all the product pages in the test set as the
query set $Q_{\mathrm{P}}$. For each query in $Q_{\mathrm{P}}$, the goal is to
find a matched short video from $G_{\mathrm{V}}$, whose product label is the
same as the query product page. For the few-shot ($k$=1) classification task,
we randomly sample one short video from each product in the test set. The
sampled shot videos are considered anchors. Then we try to classify all the
product pages in test set by finding the nearest short video anchor.

##  4 Method

The overall framework of the proposed COPE model is illustrated in Figure 3.
It comprises the visual encoder, the text encoder, the fusion encoder, and the
domain projection layers. The visual and text encoders are shared between the
three domains, and the parameters of the domain projection layers in each
domain are not shared.

###  4.1 Architectural Design

As stated in Section 3.2, we provide training samples with multiple modalities
for each domain. Specifically, we offer product titles and images for the
product domain, while for the short video and live streaming domain, we
provide extracted frames and ASR (automatic speech recognition) texts. The
COPE model is designed with a two-stream pipeline to handle both visual and
textual modalities. At the bottom of the model, we utilize a shared text
encoder and visual encoder to extract representations for raw texts and
images/frames for each domain. These extracted features are fed into three
domain-specific projection layers to obtain domain-specific representations.
Additionally, we employ a fusion encoder module, followed by a projection
layer, to aggregate visual and text features. The parameters of the fusion
encoder are shared across domains, while the projection layers are domain-
specific. It is important to note that we do not utilize the ASR texts, and we
remove text-modal related modules for the short video and live streaming
domains in our initial version of COPE. The excessive noise information in raw
ASR texts can negatively impact the final presentations for videos and live
streams. In our future work, we will explore possible approaches to utilize
the ASR texts by extracting product-related keywords from the raw texts.

The visual encoder follows the same architecture as in [21], which consists of
$N$ cross-frame communication transformer (CCT) modules and a multi-frame
integration transformer (MIT) module. The CCT module is a revised ViT [9]
block by inserting the temporal encoder to enable temporal information
exchangeability. The MIT module is placed at the top of $N$ stack CCT modules
to integrate the sequence of frame-level features into a unified video
representation.

Given an input video $\mathbf{V}\in\mathbb{R}^{T*H*W*3}$ (the product image
can be regarded as a video with only one frame), where $T$ denotes the number
of frames. $H$ and $W$ indicate the spatial resolution of the video; we split
the $t$-th frame into $M$ non-overlapping patches,
$\mathbf{X}_{\mathit{vis}}^{t}\in\mathbb{R}^{M*3}$. The learnable class token
is inserted at the beginning of the patch sequence, and the spatial position
encoding is added to the patch sequence. Formally,

$\displaystyle\mathbf{z}_{t}^{(0)}$
$\displaystyle=[e^{\mathit{cls}}_{\mathit{vis}};\;\mathbf{X}_{\mathit{vis}}]+e^{\mathit{spa}}$
(1)

Then we feed $\mathbf{z}_{t}^{(0)}$ into $N$ CCT modules to obtain the frame-
level representations:

$\displaystyle\mathbf{z}_{t}^{(n)}$
$\displaystyle=\mathrm{CCT}^{\mathit{(n)}}(\mathbf{z}_{t}^{\mathit{(n-1)}}),\;n=1,...,N$
(2)
$\displaystyle=[h_{\mathit{t,cls}}^{\mathit{(n),vis}},\;h_{\mathit{t,1}}^{\mathit{(n),vis}},\;h_{\mathit{t,2}}^{\mathit{(n),vis}},\;,...,\;,h_{\mathit{t,M}}^{\mathit{(n),vis}}]$

where $n$ denotes the CCT module index.

We take the final output of the class token at the $N$-th CCT module,
$h_{\mathit{t,cls}}^{\mathit{(N),vis}}$, to represent the $t$-th frame. Then
the global representation of the video is obtained by aggregating frame-level
features with the MIT module. Formally,

$Z_{\mathit{vis}}=\mathrm{AvgPool}(\mathrm{MIT}([h_{\mathit{1,cls}}^{\mathit{(N),vis}},...,h_{\mathit{T,cls}}^{\mathit{(N),vis}}]+e^{\mathit{temp}}))$
(3)

where $\mathrm{AvgPool}$ and $e^{\mathit{temp}}$ denote the average pooling
operator and temporal position encoding, respectively.
$Z_{\mathit{vis}}\in\mathbb{R}^{d}$ is utilized as the visual representation
for the input product image or videos.

The text encoder is a three-layer RoBERTa [15, 7] model. The input raw texts
are firstly tokenized and defined as
$\mathbf{X}_{\mathit{txt}}\in\mathbb{R}^{L}$ where $L$ indicates the length of
the token sequence. Then the class token is inserted at the beginning of the
sequence, and the position embeddings are added to retrain positional
information. The final obtained text sequence is fed into the text encoder to
extract text representations. Formally,

$\displaystyle\mathbf{H}_{\mathit{txt}}$
$\displaystyle=\mathrm{RoBERTa}([e^{\mathit{cls}};\;\mathbf{X}_{\mathit{txt}}]+e^{\mathit{pos}})$
(4)
$\displaystyle=[h_{\mathit{cls}}^{\mathit{txt}},\;h_{1}^{\mathit{txt}},\;h_{2}^{\mathit{txt}},\;...,\;,h_{L}^{\mathit{txt}}]$

where $e^{\mathit{cls}}$ and $e^{\mathit{pos}}$ denote the input class token
embedding and position embeddings, respectively.
$h_{\mathit{cls}}^{\mathit{txt}}\in\mathbb{R}^{d}$ indicates the extracted
feature of the class token. We utilize $h_{\mathit{cls}}^{\mathit{txt}}$ as
the text representation for input raw texts.

The visual representation $Z_{\mathit{vis}}$ and text representation
$h_{\mathit{cls}}^{\mathit{txt}}$ of the three domains are extracted with the
shared visual and text encoders, even though the samples in different domains
vary significantly. Such a scheme is expected to enhance the generalization
capability of the basic feature extractors. The characteristics of each domain
are retained and magnified by utilizing different projection layers that are
not shared across domains to transform the general representations into
domain-specific representations. For each domain, the projection layer is a
linear layer with weight $\mathbf{W}$ and bias $b$, and the domain-specific
representations are obtained as:

$\displaystyle E_{\mathit{vis}}^{\mathrm{P}}$
$\displaystyle=\mathbf{W}_{\mathit{vis}}^{\mathrm{P}}Z_{\mathit{vis}}^{\mathrm{P}}+b_{\mathit{vis}}^{\mathrm{P}}$
(5) $\displaystyle E_{\mathit{txt}}^{\mathrm{P}}$
$\displaystyle=\mathbf{W}_{\mathit{txt}}^{\mathrm{P}}h_{\mathit{txt}}^{\mathrm{P}}+b_{\mathit{txt}}^{\mathrm{P}}$
$\displaystyle E_{\mathit{vis}}^{\mathrm{V}}$
$\displaystyle=\mathbf{W}_{\mathit{vis}}^{\mathrm{V}}Z_{\mathit{vis}}^{\mathrm{V}}+b_{\mathit{vis}}^{\mathrm{V}}$
$\displaystyle E_{\mathit{vis}}^{\mathrm{L}}$
$\displaystyle=\mathbf{W}_{\mathit{vis}}^{\mathrm{L}}Z_{\mathit{vis}}^{\mathrm{L}}+b_{\mathit{vis}}^{\mathrm{L}}$

where $\mathrm{P}$, $\mathrm{V}$, and $\mathrm{L}$ denote the product page
domain, short video domain, and the live streaming domain. It should be noted
that in the short video domain and live streaming domain, we do not include
the text modality, and only visual representations, _i.e_.,
$E_{\mathit{vis}}^{\mathrm{V}}$ and $E_{\mathit{vis}}^{\mathrm{L}}$, are
utilized for the two domains.

Finally, the fusion encoder, followed by a projection layer, is proposed to
aggregate the visual and text representations. The fusion encoder is
implemented with a self-attention layer, and the projection layer is a linear
layer. Also, in our initial version of COPE, the fusion operation is only
applied to the product page domain. Formally,

$\displaystyle E_{\mathit{fus}}^{\mathrm{P}}$
$\displaystyle=\mathrm{SelfAtten}([E_{\mathit{vis}}^{\mathrm{P}};\;E_{\mathit{txt}}^{\mathrm{P}}])$
(6) $\displaystyle E_{\mathit{fus}}^{\mathrm{P}}$
$\displaystyle=\mathbf{W}_{\mathit{fus}}^{\mathrm{P}}E_{\mathit{fus}}^{\mathrm{P}}+b_{\mathit{fus}}^{\mathrm{P}}$

where $\mathrm{SelfAttn}$ denotes the self-attention layer.
$E_{\mathit{fus}}^{\mathrm{P}}$ is the obtained multi-modal representation for
the product page domain $\mathrm{P}$. For the other two domains $\mathrm{V}$
and $\mathrm{L}$, the visual representations $E_{\mathit{vis}}^{\mathrm{V}}$
and $E_{\mathit{vis}}^{\mathrm{L}}$ are the final obtained representations for
them.

###  4.2 Training Objective

To learn a unified product representation across the different domains, we
first leverage contrastive learning to train the proposed COPE model following
previous self-supervised learning methods [22, 3, 10]. The basic formulation
of the contrastive loss function [22] is defined as:

$\mathcal{L}_{\mathit{con}}=-\log\frac{\exp(s_{\mathit{qk_{+}}}/\tau)}{\sum_{i=0}^{i=K}\exp(s_{\mathit{qk_{i}}}/\tau)}$
(7)

where $s_{\mathit{qk_{i}}}$ denotes the cosine similarity between the sample
$q$ and the sample $k_{i}$. The positive sample $k_{+}$ indicates the sample
that has the same product label with $q$.

The similarity $s_{\mathit{qk}}$ can be calculated with a different form of
representations (vision, text, or fusion), and the samples $q$ and $k$ can
come from different domains (product page, short video, or live streaming). In
this paper, we choose seven different implementations of the similarity
$s_{\mathit{qk}}$, resulting in seven different contrastive loss functions.
The details of the implementations are summarized in Table 2. Based on the
seven contrastive loss functions, we define cross-domain loss as the sum of
them. Formally,

$\mathcal{L}_{\mathit{cd}}=\sum_{n=1}^{n=7}\alpha_{n}\mathcal{L}_{\mathit{con}}^{n}$
(8)

where $\alpha_{n}$ is the weight of $n$-th contrastive learning loss function.

In addition to the cross-domain loss, we also adopt the product classification
loss to train our COPE model. Specifically, we use an MLP (multi-layer
perceptron) with shared parameters to predict product classification scores
for each domain with the domain-specific representations. For the product page
domain, the multi-modal representation $E_{\mathit{fus}}^{\mathrm{P}}$ is
utilized. For the short video and live streaming domains, the visual
representations $E_{\mathit{vis}}^{\mathrm{V}}$ and
$E_{\mathit{vis}}^{\mathrm{L}}$ are adopted. Formally,

$\displaystyle s^{\mathrm{P}}$
$\displaystyle=\mathrm{MLP}(E_{\mathit{fus}}^{\mathrm{P}})$ (9) $\displaystyle
s^{\mathrm{V}}$ $\displaystyle=\mathrm{MLP}(E_{\mathit{vis}}^{\mathrm{V}})$
$\displaystyle s^{\mathrm{L}}$
$\displaystyle=\mathrm{MLP}(E_{\mathit{vis}}^{\mathrm{L}})$

where $s$ denotes the classification score for each domain. Then the standard
softmax loss is used to train the model. Formally,

$\displaystyle\mathcal{L}_{\mathit{cls}}$
$\displaystyle=-(\log\frac{e^{s^{\mathrm{P}}_{i}}}{\sum_{j}^{C}e^{s^{\mathrm{P}}_{j}}}+\log\frac{e^{s^{\mathrm{V}}_{i}}}{\sum_{j}^{C}e^{s^{\mathrm{V}}_{j}}}+\log\frac{e^{s^{\mathrm{L}}_{i}}}{\sum_{j}^{C}e^{s^{\mathrm{L}}_{j}}})$
(10)

similarity $s_{\mathit{qk}}$ | domain | modality
---|---|---
$<E_{\mathit{fus}}^{\mathrm{P}}(q),E_{\mathit{vis}}^{\mathrm{V}}(k)>$ |  $\mathit{product}$-$\mathit{video}$ |  $\mathit{fusion}$-$\mathit{vision}$
$<E_{\mathit{fus}}^{\mathrm{P}}(q),E_{\mathit{vis}}^{\mathrm{L}}(k)>$ |  $\mathit{product}$-$\mathit{live}$ |  $\mathit{fusion}$-$\mathit{vision}$
$<E_{\mathit{vis}}^{\mathrm{V}}(q),E_{\mathit{vis}}^{\mathrm{L}}(k)>$ |  $\mathit{video}$-$\mathit{live}$ |  $\mathit{vision}$-$\mathit{vision}$
$<E_{\mathit{vis}}^{\mathrm{P}}(q),E_{\mathit{vis}}^{\mathrm{V}}(k)>$ |  $\mathit{product}$-$\mathit{video}$ |  $\mathit{vision}$-$\mathit{vision}$
$<E_{\mathit{txt}}^{\mathrm{P}}(q),E_{\mathit{vis}}^{\mathrm{V}}(k)>$ |  $\mathit{product}$-$\mathit{video}$ |  $\mathit{text}$-$\mathit{vision}$
$<E_{\mathit{vis}}^{\mathrm{P}}(q),E_{\mathit{vis}}^{\mathrm{L}}(k)>$ |  $\mathit{product}$-$\mathit{live}$ |  $\mathit{vision}$-$\mathit{vision}$
$<E_{\mathit{txt}}^{\mathrm{P}}(q),E_{\mathit{vis}}^{\mathrm{L}}(k)>$ |  $\mathit{product}$-$\mathit{live}$ |  $\mathit{text}$-$\mathit{vision}$
Table 2: The implementations of different similarity functions
$s_{\mathit{qk}}$.

The total loss to train the COPE model is the combination of the cross-domain
loss and the classification loss. Formally,

$\displaystyle\mathcal{L}_{\mathit{f}}=\mathcal{L}_{\mathit{cd}}+\beta\mathcal{L}_{\mathit{cls}}$
(11)

where $\beta$ indicates the weight of classification loss.

###  4.3 Implementation Details.

We initialize the text encoder with the public Chinese RoBERTa model [15, 7]
and the visual encoder with the pre-trained model in [21]. Eight frames are
extracted to obtain features for short videos and live streams. The training
batch size is set to 84, and the training process continues for 80 epochs. We
optimize the model using AdamW [17], and the cosine schedule with a linear
warmup is used for adjusting the learning rate. The warmup approach continues
for two epochs, and the max learning rates are set to 5e-5, 5e-7, and 5e-3 for
the text encoder, visual encoder, and other layers.

|  | cross domain retrieval | few-shot classification
---|---|---|---
models | cross domain setting | R@1 | R@5 | R@10 | R@20 | R@50 | R@mean | Top1 Acc
CLIP4CLIP [19] | P2V | 59.06 | 79.31 | 86.02 | 91.01 | 95.03 | 82.08 | 27.94
V2P | 38.48 | 52.25 | 59.16 | 66.54 | 74.65 | 58.21 | 26.55
P2L | 23.68 | 38.14 | 45.32 | 54.27 | 66.79 | 45.64 | 9.97
L2P | 14.46 | 24.52 | 30.77 | 38.09 | 48.91 | 31.35 | 10.75
V2L | 18.10 | 29.83 | 35.65 | 42.22 | 52.01 | 35.56 | 9.47
L2V | 20.14 | 33.51 | 40.44 | 48.05 | 58.68 | 40.16 | 7.22
TS2-Net [16] | P2V | 57.42 | 77.88 | 85.29 | 90.44 | 94.92 | 81.19 | 26.11
V2P | 36.56 | 50.93 | 58.02 | 65.12 | 73.89 | 56.90 | 24.09
P2L | 22.85 | 38.49 | 45.91 | 54.11 | 65.89 | 45.45 | 9.83
L2P | 14.16 | 24.52 | 30.50 | 37.52 | 48.37 | 31.01 | 10.57
V2L | 17.69 | 29.63 | 34.84 | 41.27 | 50.95 | 34.87 | 9.68
L2V | 20.55 | 33.80 | 40.91 | 48.46 | 59.16 | 40.57 | 7.40
X-CLIP [20] | P2V | 56.61 | 77.46 | 84.84 | 90.11 | 94.51 | 80.70 | 26.97
V2P | 35.29 | 49.41 | 56.82 | 64.13 | 72.54 | 55.63 | 23.55
P2L | 22.66 | 37.47 | 44.33 | 52.11 | 63.38 | 43.98 | 9.72
L2P | 13.52 | 23.08 | 28.92 | 35.98 | 46.14 | 29.52 | 8.88
V2L | 17.64 | 28.71 | 34.03 | 40.17 | 49.67 | 34.04 | 9.05
L2V | 19.60 | 32.73 | 39.51 | 47.07 | 57.25 | 39.23 | 7.42
ChineseCLIP [31] | P2V | 56.93 | 79.80 | 87.43 | 92.48 | 96.51 | 82.65 | 31.44
V2P | 40.48 | 57.85 | 66.74 | 75.25 | 84.03 | 64.87 | 29.10
P2L | 34.37 | 50.83 | 58.66 | 67.05 | 78.57 | 57.89 | 19.23
L2P | 22.49 | 37.11 | 46.78 | 56.30 | 68.14 | 46.16 | 15.73
V2L | 25.51 | 38.28 | 45.02 | 52.27 | 62.53 | 44.72 | 13.24
L2V | 28.28 | 45.87 | 53.67 | 62.18 | 72.27 | 52.45 | 14.16
FashionClip [5] | P2V | 44.31 | 67.06 | 75.25 | 82.57 | 89.29 | 71.69 | 18.59
V2P | 25.51 | 40.75 | 48.71 | 56.63 | 65.94 | 47.50 | 15.88
P2L | 19.54 | 31.14 | 36.98 | 43.91 | 54.39 | 37.19 | 8.70
L2P | 11.22 | 24.23 | 31.90 | 40.05 | 50.96 | 31.67 | 7.57
V2L | 15.55 | 24.88 | 29.51 | 35.07 | 42.68 | 29.53 | 6.80
L2V | 21.20 | 35.72 | 42.55 | 49.60 | 58.77 | 41.56 | 10.40
COPE (Ours) | P2V | 82.58 | 94.88 | 97.54 | 98.89 | 99.65 | 94.70 | 59.84
V2P | 65.20 | 76.56 | 82.04 | 86.86 | 91.69 | 80.47 | 57.12
P2L | 54.06 | 71.07 | 77.14 | 82.86 | 89.70 | 74.96 | 34.95
L2P | 42.33 | 56.48 | 63.67 | 71.11 | 80.22 | 62.76 | 36.51
V2L | 45.95 | 63.63 | 70.64 | 77.50 | 85.47 | 68.63 | 30.43
L2V | 48.28 | 67.20 | 74.70 | 81.52 | 89.15 | 72.17 | 33.30
Table 3: Retrieval and classification results on COPE. P, V, and L means
product page, short video, and live stream domains.

###  4.4 Experimental Results

In this section, we evaluate our proposed COPE model and compare it with the
state-of-the-arts on the ROPE dataset. The cross-domain product retrieval task
and one-shot cross-domain classification task are considered. Since no
existing methods are precisely suitable to our cross-domain setting, we
compare the COPE model with the multi-modal vision-language models [5, 16, 19,
20, 31], which are not fine-tuned on our dataset. The product page
representation is obtained for these models by averaging the image and text
features. The short video and live streaming representations are extracted by
averaging the representations of all frames.

The vision-language models trained with general image-text pairs are compared
in the first compartment of Table 3. We can see that all of them obtain
inferior performance to our COPE model on each setting of the two evaluation
tasks. In both the retrieval and classification tasks, the performance in the
live related settings, _i.e_., $\mathrm{P}$$\rightarrow$$\mathrm{L}$,
$\mathrm{L}$$\rightarrow$$\mathrm{P}$, $\mathrm{V}$$\rightarrow$$\mathrm{L}$,
and $\mathrm{L}$$\rightarrow$$\mathrm{V}$, is obviously lower than others. For
example, the COPE model obtains 82.58% R@1 in the
$\mathrm{P}$$\rightarrow$$\mathrm{V}$ retrieval task and 59.84% Acc in the
$\mathrm{P}$$\rightarrow$$\mathrm{V}$ classification task. By contrast, the
performance of COPE in the $\mathrm{P}$$\rightarrow$$\mathrm{L}$ settings is
54.06% and 34.95%, respectively. The scales and views of products in live
streams differ from the product pages. Moreover, the products are not always
visible in the live stream durations. These situations significantly improve
the difficulty of recognizing products in live streams. In the second
compartment of Table 3, we compare the COPE model with the FashionClip model.
Although the FashionClip model is trained with the product images and titles
rather than the general data, there are still large margins between the
obtained results of FashionClip and COPE. As described in Section 2.1, the
representations learned on the product page domain only are insufficient to
deal with the cross-domain product recognition problem.

model | mAP@10 | mAP@50 | mAP@100 | mAR@10 | mAR@50 | mAR@100 | Prec@10 | Prec@50 | Prec@100
---|---|---|---|---|---|---|---|---|---
SOTA | 79.36 | 74.79 | 74.63 | 34.69 | 30.04 | 30.08 | 73.97 | 72.12 | 73.86
COPE (Ours) | 86.02 | 80.51 | 77.35 | 53.53 | 57.03 | 58.03 | 80.30 | 72.39 | 66.58
Table 4: Retrieval results of COPE on Product1M.

###  4.5 Performance on other datasets

In order to verify the generalization of our ROPE dataset. We directly utilize
the learned COPE model on ROPE to extract product representations and conduct
evaluations on other datasets, such as Product1M [33] and M5Product [8]. The
reulsts are shown in Table 4 and Table 5. We can see that without any fine-
tuning approach, the COPE model can achieve better performance to the origin
SOTAs.

model | mAP@1 | mAP@5 | Prec@1 | Prec@5
---|---|---|---|---
SOTA(I+T) | 62.20 | 66.97 | 62.20 | 49.85
SOTA(ALL) | 69.25 | 74.08 | 69.25 | 58.76
COPE(Ours) | 80.89 | 83.66 | 80.89 | 75.96
Table 5: Retrieval results of COPE on M5Product.

###  4.6 Effectiveness of Classification Loss

In this section, we examine the influence of classification loss on our model.
Due to a large number of categories in our dataset, we utilize Partial-FC [1]
to enhance training efficiency. As indicated in Table 6, including the
classification loss substantially improves the model’s performance across all
retrieval tasks. The model with $\mathcal{L}\mathit{cls}$ outperforms the
model without $\mathcal{L}\mathit{cls}$ by 30% and 19% in rank-1 accuracy on
the P2V and L2P tasks, respectively. It provides compelling evidence for the
efficacy of the classification loss.

tasks | loss | R@1 | R@5 | R@10 | R@20 | R@50
---|---|---|---|---|---|---
P2V |  w/o $\mathcal{L}_{\mathit{cls}}$ | 51.88 | 76.45 | 84.58 | 90.58 | 95.50
w $\mathcal{L}_{\mathit{cls}}$ | 82.58 | 94.88 | 97.54 | 98.89 | 99.65
V2P |  w/o $\mathcal{L}_{\mathit{cls}}$ | 44.17 | 60.01 | 68.24 | 75.86 | 84.29
w $\mathcal{L}_{\mathit{cls}}$ | 65.20 | 76.56 | 82.04 | 86.86 | 91.69
P2L |  w/o $\mathcal{L}_{\mathit{cls}}$ | 26.41 | 44.76 | 53.25 | 62.72 | 75.26
w $\mathcal{L}_{\mathit{cls}}$ | 54.06 | 71.07 | 77.14 | 82.86 | 89.70
L2P |  w/o $\mathcal{L}_{\mathit{cls}}$ | 23.11 | 38.28 | 47.97 | 57.88 | 71.04
w $\mathcal{L}_{\mathit{cls}}$ | 42.33 | 56.48 | 63.67 | 71.11 | 80.22
V2L |  w/o $\mathcal{L}_{\mathit{cls}}$ | 29.39 | 47.54 | 55.88 | 64.47 | 75.81
w $\mathcal{L}_{\mathit{cls}}$ | 45.95 | 63.63 | 70.64 | 77.50 | 85.47
L2V |  w/o $\mathcal{L}_{\mathit{cls}}$ | 29.50 | 52.07 | 62.60 | 72.30 | 83.12
w $\mathcal{L}_{\mathit{cls}}$ | 48.28 | 67.20 | 74.70 | 81.52 | 89.15
Table 6: The classification loss significantly improves the performances on
all the tasks.

###  4.7 Sampling Strategy

In Table 7, we present a comparison between random sampling and product-
balance sampling. In a mini-batch with $N$ samples, random sampling refers to
randomly selecting $N$ samples from the training set. By contrast, product-
balance sampling selects $P$ products and then samples $K$ instances from each
product, resulting in $N=P\times K$ samples. The experimental results indicate
that balanced sampling significantly enhances the model’s performance.

tasks | strategy | R@1 | R@5 | R@10 | R@20 | R@50
---|---|---|---|---|---|---
P2V | rs | 70.08 | 88.49 | 93.18 | 96.22 | 98.40
pb | 82.58 | 94.88 | 97.54 | 98.89 | 99.65
V2P | rs | 55.26 | 68.74 | 75.44 | 81.87 | 88.45
pb | 65.20 | 76.56 | 82.04 | 86.86 | 91.69
P2L | rs | 40.85 | 60.39 | 68.51 | 76.42 | 85.79
pb | 54.06 | 71.07 | 77.14 | 82.86 | 89.70
L2P | rs | 33.10 | 48.67 | 57.39 | 66.03 | 76.70
pb | 42.33 | 56.48 | 63.67 | 71.11 | 80.22
V2L | rs | 37.66 | 56.10 | 64.28 | 72.30 | 82.04
pb | 45.95 | 63.63 | 70.64 | 77.50 | 85.47
L2V | rs | 38.31 | 60.06 | 68.99 | 77.40 | 86.41
pb | 48.28 | 67.20 | 74.70 | 81.52 | 89.15
Table 7: Comparison of the two sampling strategies, _i.e_., random sampling
(rs) and product-balance sampling (pb).

###  4.8 Visualization

Figure 4: The t-SNE visualization of the COPE embeddings. Points of the same
product have the same color.

(a) P2V and V2P retrieval results

(b) P2L and L2P retrieval results

(c) V2L and L2V retrieval results

Figure 5: Visualization of the retrieval results, where the red box denotes
false positive.

In Figure 4, we present the t-SNE visualization of the embeddings of product
pages, short videos, and live streams. We randomly selected 30 products and
their corresponding product pages, short videos, and live streams to generate
this visualization. The visualization clearly illustrates that the embeddings
of the same product are positioned closely together, which indicates the
effectiveness of our COPE approach in distinguishing between different
products. Furthermore, Figure 5 displays some of our retrieval results.
Notably, most of the false positive results belong to the same category as the
query and possess similar visual characteristics.

##  5 Conclusion

To enable the creation of a unified cross-domain product representation, we
introduce a large-scale E-commerce cross-domain dataset that includes three
domains (product pages, short videos, and live streams) and two modalities
(vision and language). It is the first dataset that encompasses various
domains in the e-commerce scenario. We propose our COPE as the baseline and
evaluate it on cross-domain retrieval and few-shot classification tasks.
Finally, we provide an analysis and visualization of the results. This task
applies to most e-commerce platforms, and both the dataset and the proposed
framework will inspire research on cross-domain product representation.

## References

  * [1] Xiang An, Xuhan Zhu, Yuan Gao, Yang Xiao, Yongle Zhao, Ziyong Feng, Lan Wu, Bin Qin, Ming Zhang, Debing Zhang, et al.  Partial fc: Training 10 million identities on a single machine.  In ICCV, pages 1445–1449, 2021.
  * [2] Delong Chen, Fan Liu, Xiaoyu Du, Ruizhuo Gao, and Feng Xu.  Mep-3m: A large-scale multi-modal e-commerce products dataset.  In IJCAI, volume 21, 2021.
  * [3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [4] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In ECCV, pages 104–120. Springer, 2020.
  * [5] Patrick John Chia, Giuseppe Attanasio, Federico Bianchi, Silvia Terragni, Ana Rita Magalhães, Diogo Goncalves, Ciro Greco, and Jacopo Tagliabue.  Fashionclip: Connecting language and images for product representations.  arXiv preprint arXiv:2204.03972, 2022.
  * [6] Charles Corbiere, Hedi Ben-Younes, Alexandre Ramé, and Charles Ollion.  Leveraging weakly annotated data for fashion image retrieval and label prediction.  In ICCV, pages 2268–2274, 2017.
  * [7] Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang.  Pre-training with whole word masking for chinese bert.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3504–3514, 2021.
  * [8] Xiao Dong, Xunlin Zhan, Yangxin Wu, Yunchao Wei, Michael C Kampffmeyer, Xiaoyong Wei, Minlong Lu, Yaowei Wang, and Xiaodan Liang.  M5product: Self-harmonized contrastive learning for e-commercial multi-modal pretraining.  In CVPR, pages 21252–21262, 2022.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [10] Tianyu Gao, Xingcheng Yao, and Danqi Chen.  Simcse: Simple contrastive learning of sentence embeddings.  In EMNLP, pages 6894–6910, 2021.
  * [11] Marco Godi, Christian Joppi, Geri Skenderi, and Marco Cristani.  Movingfashion: a benchmark for the video-to-shop challenge.  In WACV, pages 1678–1686, 2022.
  * [12] Brendan Kolisnik, Isaac Hogan, and Farhana Zulkernine.  Condition-cnn: A hierarchical multi-label fashion image classification model.  Expert Systems with Applications, 182:115195, 2021.
  * [13] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [14] Huidong Liu, Shaoyuan Xu, Jinmiao Fu, Yang Liu, Ning Xie, Chien-Chih Wang, Bryan Wang, and Yi Sun.  Cma-clip: Cross-modality attention clip for image-text classification.  arXiv preprint arXiv:2112.03562, 2021.
  * [15] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [16] Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin.  Ts2-net: Token shift and selection transformer for text-video retrieval.  In ECCV, pages 319–335. Springer, 2022.
  * [17] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR.
  * [18] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In NeurIPS, pages 13–23, 2019.
  * [19] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.  Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.  Neurocomputing, 508:293–304, 2022.
  * [20] Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji.  X-clip: End-to-end multi-grained contrastive learning for video-text retrieval.  In ACMMM, pages 638–647, 2022.
  * [21] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling.  Expanding language-image pretrained models for general video recognition.  2022\.
  * [22] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [23] Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti.  Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data.  arXiv preprint arXiv:2001.07966, 2020.
  * [24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [25] Negar Rostamzadeh, Seyedarian Hosseini, Thomas Boquet, Wojciech Stokowiec, Ying Zhang, Christian Jauvin, and Chris Pal.  Fashion-gen: The generative fashion dataset and challenge.  2018\.
  * [26] Majuran Shajini and Amirthalingam Ramanan.  An improved landmark-driven and spatial–channel attentive convolutional neural network for fashion clothes classification.  The Visual Computer, 37(6):1517–1526, 2021.
  * [27] Majuran Shajini and Amirthalingam Ramanan.  A knowledge-sharing semi-supervised approach for fashion clothes classification and attribute prediction.  The Visual Computer, 38(11):3551–3561, 2022.
  * [28] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  In ICLR.
  * [29] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In ICCV, pages 7464–7473, 2019.
  * [30] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  In EMNLP-IJCNLP, pages 5100–5111, 2019.
  * [31] An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, and Chang Zhou.  Chinese clip: Contrastive vision-language pretraining in chinese.  arXiv preprint arXiv:2211.01335, 2022.
  * [32] Wenjie Yang, Yiyi Chen, Yan Li, Yanhua Cheng, Xudong Liu, Quan Chen, and Han Li.  Cross-view semantic alignment for livestreaming product recognition.  arXiv preprint arXiv:2308.04912, 2023.
  * [33] Xunlin Zhan, Yangxin Wu, Xiao Dong, Yunchao Wei, Minlong Lu, Yichi Zhang, Hang Xu, and Xiaodan Liang.  Product1m: Towards weakly supervised instance-level product retrieval via cross-modal pretraining.  In ICCV, pages 11782–11791, 2021.
  * [34] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In CVPR, pages 16816–16825, 2022.


# Multi-label Annotation for Visual Multi-Task Learning Models

Gaurang Sharma1, Alexandre Angleraud1 and Roel Pieters1 1Unit of Automation
Technology and Mechanical Engineering, Tampere University, Tampere, Finland;
<EMAIL_ADDRESS>
###### Abstract

Deep learning requires large amounts of data, and a well-defined pipeline for
labeling and augmentation. Current solutions support numerous computer vision
tasks with dedicated annotation types and formats, such as bounding boxes,
polygons, and key points. These annotations can be combined into a single data
format to benefit approaches such as multi-task models. However, to our
knowledge, no available labeling tool supports the export functionality for a
combined benchmark format, and no augmentation library supports
transformations for the combination of all. In this work, these
functionalities are presented, with visual data annotation and augmentation to
train a multi-task model (object detection, segmentation, and key point
extraction). The tools are demonstrated in two robot perception use cases.

##  I Introduction

Advancements in deep learning have helped to address numerous problems in
different domains (e.g., robotics[1], medicine[2] and agriculture[3]).
Perception in particular has been of major focus, where single purpose models
have been optimized and fine-tuned for one specific task at hand [4]. More
recently also multi-task models have been developed [5, 6], where a shared
model concurrently learns multiple tasks. A fundamental aspect of these models
is to increase data efficiency, reduce overfitting, and enable quick learning
due to the use of auxiliary information. They use a unified single backbone
structure having multiple descriptive heads to address technical computer
vision challenges like boundary detection, semantic segmentation, object
detection, etc. To effectively utilize multi-task models, multi-label
annotations should be supported by the framework. This implies that images and
the objects inside them can be annotated with multiple different labels (e.g.,
bounding boxes, polygons and key points), to be used for learning different
tasks and thus generate multiple outputs. Compared to single-task models this
is less computationally expensive and requires less annotated records. In
addition, multiple outputs per image or object provides additional information
compared to single-task models.

Figure 1: Motivation for multi-task detection models. Single-task models can
detect an object (bounding box), but cannot verify whether their orientation
is correct. Multi-task models can provide additional information from
different model outputs. In this case key points (three blue points) indicate
a correct (green box) and incorrect (red box) object placement.

Fig. 1 depicts this visually in an object detection problem. Two objects are
annotated and correctly detected, however, in this context, the lower object
is incorrectly placed. From detection or segmentation alone, this cannot be
determined, yet with the estimated key points as additional information (i.e.,
three blue points), incorrect placement can be detected. Existing labeling and
annotation solutions[7] do not provide the combined annotation export
functionality, which results in solutions that combine several export formats
to accomplish a common task. This served as motivation for the development of
a novel data pipeline to provide domain-specific annotation and augmentation
strategies. Best-of-breed libraries such as Albumentation [8] and Label Studio
[9] are integrated to create a generic data generation solution for training
deep learning models. Our developments are evaluated by applications in object
detection, object configuration estimation and depth estimation. In this work,
our contributions are:

  1. 1.

A novel pipeline that enables labeling of image datasets with different
annotations and formats

  2. 2.

The augmentation of this dataset without conversion to different formats

  3. 3.

The validation of results in two industrial use cases with different computer
vision tasks

##  II RELATED WORK

TABLE I: Well-known annotation tools and their main exported formats and supported annotation types. None of the existing tools support a combined annotation strategy in a benchmark format.  | Export format | Annotation type | Comment
---|---|---|---
Annotation tool |  | Bounding box | Polygon | Masks | keypoints |
Label Studio[9] | COCO [10] | ✓ | ✓ |  |  |
YOLO [11] | ✓ |  |  |  |
PASCAL VOC XML [12] | ✓ |  |  |  |
JSON | ✓ | ✓ | ✓ | ✓ | JSON supports any annotation type
NumPy arrays |  |  | ✓ |  |
V7 [13] | COCO [10] [10] | ✓ | ✓ |  |  |
YOLO [11] | ✓ |  |  |  |
PASCAL VOC XML [12] | ✓ |  |  |  |
Darwin JSON | ✓ | ✓ | ✓ | ✓ | JSON supports any annotation type
PNG |  |  | ✓ |  |
CVAT [14] | COCO object detection [10] | ✓ | ✓ |  |  |
COCO keypoint [10] |  |  |  | ✓ |
YOLO [11] | ✓ |  |  |  |
PASCAL VOC XML [12] | ✓ |  | ✓ | ✓ |
MOTS [15] |  | ✓ |  |  |
LabelBox [16] | JSON | ✓ | ✓ | ✓ | ✓ | Separate JSON for each annotation
LabelMe[17] | JSON | ✓ | ✓ | ✓ | ✓ | JSON supports any annotation type
Our work | COCO [10] | ✓ | ✓ | ✓ | ✓ |

###  II-A Multi-task Learning

Multi-task learning [5] in the field of computer vision [6] has shown recent
success with methods such as Cross-Stitch [18] and UberNet [19]. As
comparison, single-task models are more expensive to train and require a
larger amount of annotated data than multi-task models. Moreover, knowing
which tasks should be trained together could aid in improving prediction
quality and acting as a preventative measure against the occurrence of
negative transfer [20]. Performance improvement of a specific task can thus
lead to performance degradation of other tasks [21]. Currently, libraries such
as YOLOR[22] and YOLOv7[11] provide configuration features to facilitate
multi-task learning.

###  II-B Image Annotation and Storing Formats

Labeling data is a critical component for deep learning as it provides the
ground truth for estimating errors. Incorrectly labeled datasets lead to poor
training and higher validation losses [23]. Nowadays, there are several
machine learning based tools available that speed up the automatic labeling
process and reduce human labor and time [24]. MS COCO[10], PASCAL VOC[12], and
YOLO are some commonly used benchmarks with different storage format styles.
MS COCO is a rich (.json) format that offers a large variety of annotation
storing structures. PASCAL VOC, on the other hand, supports a (.xml) structure
that is useful for detection and segmentation tasks but has limitations in
supporting key points. YOLO models accept (.txt) file format for detection
tasks, however, YOLOv7[11] accepts polygon segmentation and key point
representations as well. While numerous labeling tools are available, each
provides support for different export formats and annotation types (see Table
I). For example, only a few tools offer direct labeling options for polygon
segmentation, bounding boxes and key points simultaneously. And when the
opportunity is provided, only a combined JSON export format is available,
thereby not directly supporting state-of-the-art deep learning libraries. As a
result, custom data loaders need to be developed to enable library-specific
data formats.

###  II-C Image Augmentation

Data augmentation is a form of data expansion that can improve model
performance by assisting in its generalization, robustness and convergence.
While augmentation is a well-known approach, libraries such as TensorFlow [25]
and PyTorch [26] provide very few transformation strategies. Albumentations
[8], on the other hand provides around six different libraries to apply a
broad range of transformations within a single tool. However, if custom and
complex transformations are required, then libraries like OpenCV [27] can be
utilized for basic image manipulation algorithms. Smart augmentation
techniques also exist [28], in which suitable augmentations are learned during
the process of training a deep neural network.

###  II-D Object detection and Pose estimation

Traditionally, keypoint estimation, segmentation, and object detection were
considered separate problems. The solution for object detection evolved in the
last 20 years from hand-crafted features to deformable transformers [29].
Research on segmentation started with thresholding and later led to methods
such as semantic, instance, and panoptic segmentation. On the other hand,
local features, called keypoints, became popular through corner and edge
detectors [30], and today there are several ways to predict them, including
Faster R-CNN [31], which is based on local keypoints. In the field of robotics
and computer vision, segmentation masks are used to predict keypoints, as
presented in [32]. Recently, combinations of both to predict human and object
poses is addressed by [33]. In this, COCO is used as the primary format,
emphasizing its relevance.

##  III Methodology

In this section, we present our methodology which enables the annotation of
images with multiple labels in multiple formats and the export of these in a
generic format, for training a multi-task detection model.

Figure 2: The multi-task learning architecture from Detectron2 [34] supports
the combined data annotations (bounding box, polygon and key points) for
multi-task learning. The final output is the combination all three detection
formats.

###  III-A Architecture

The multi-task architecture of ResNet-50 FPN Keypoint R-CNN [34] supports the
combined data annotations and is utilized for training the detection model
(see Fig. 2). The output of the model contains all three detection formats,
for object recognition, segmentation and feature extraction at the same time.

###  III-B Data Annotation

Our pipeline starts with the image annotation of polygons, key points and
bounding boxes in Label Studio. As Label Studio does not provide the
functionality for simultaneously annotating features for polygons and key
points, our approach updates polygon configurations by adding a key points
configuration script to the user interface, maintaining a single image via the
’toName’ attribute. Bounding boxes are extracted from the polygon annotations
by selecting the outer bounds of the polygon coordinates. Following, all
annotations are exported into a JSON-min format and converted into COCO
format, containing all three data annotation types in a single output format.

###  III-C Data Augmentation

Tools such as Albumentations [8] offer a wide range of image transformations.
However, it does not offer augmentation support for polygon and run-length
encoding COCO formats, limiting the support for an integrated augmentation
strategy. To enable the augmentation of multi-label annotated data, such as
bounding boxes, key points and COCO segmentation formats, the following steps
are performed. First, the combined data (in COCO format) is loaded and
extracted into separate arrays, such as bounding boxes, key points, area, etc.
Second, polygons are converted to key points and appended to a key point
array. Third, the conversion of (x,y) key points to (x,y,v) is done to
maintain COCO standards, followed by appending invisible key points to keep
the number of labeled key points the same. Dataset augmentation is then
performed using Albumentation’s key point transformation strategy, after which
the key points are converted back to polygons. As a result, this procedure
enables the transformation of COCO polygon format by the Albumentation
library.

##  IV Results and Discussion

This section presents the results of our multi-label annotation pipeline, its
detection results and a discussion on its limitations.

###  IV-A Practical Use Cases and System Integration

The use cases aim to detect different objects for automation purposes. Two
assembly sets are selected including a diesel engine and a planetary gearbox,
and their internal parts (e.g., rocker arms, pushrods, bolts, gears, housing,
etc.). The diesel engine [35] and planetary gearbox [36] datasets contain 195
and 150 RGB images, respectively, manually annotated by drawing polygons and
labeling key points (see Fig. 3(a) and 3(c)). Augmentation is then performed
on the annotated data by different transformation strategies, completing
datasets of 280,000 and 170,000 images. Training of the multi-task models
followed typical training and evaluation steps [34] with hyper-parameters such
as momentum, weight decay, and regularization to achieve loss convergence in
about 2000 epochs. The images were captured with an Intel Realsense D435
camera. All tools are open-source available from the dataset documentations
[35, 36].

###  IV-B Object and Key point Detection

Fig. 3(b) and 3(d) present the output of the trained multi-task models on the
two use cases with results for object detection and segmentation, and key
point estimations. This demonstrates that a single multi-task model can
utilize multiple data annotation formats as part of the dataset, train a model
and achieve successful detection outputs. On average, the detection confidence
is over 90 percent. Furthermore, as main beneficial outcome Table II presents
that the achieved frame rate of the multi-task model is higher than running
multiple models in parallel. For high-resolution camera input
(1280$\times$720), the difference is almost double, i.e., 8.7 FPS versus 4.8
FPS, respectively. Likewise, the memory requirements for a multi-task model is
less than running multiple single-task models in parallel (i.e., 500 MB vs 700
MB).

(a) (b) (c) (d)

Figure 3: Results for the Diesel engine (top row) and planetary gear (bottom row) datasets. (a) and (c) depict examples of image annotations with polygons and key points. (b) and (d) depict the results of the trained multi-task detection models (bounding box and segmentation mask with class number and confidence score, and red key points).  TABLE II: Frame rate and memory usage comparison between single- and multi-task models. Model | Output |  | Inference
---
(FPS)
| Memory
---
(MB)
Faster R-CNN | BB, class | 11.1 | $\sim 350$
Mask R-CNN | Mask, BB, class | 10.5 | $\sim 350$
Keypoint R-CNN | Kps, BB, class | 9.4 | $\sim 350$
Multi-task model | Mask, kps, BB, class | 8.7 | $\sim 500$
Mask + keypoint (in parallel) | Mask, kps, BB, class | 4.8 | $\sim 700$

###  IV-C Object Configuration and Depth Estimation

Two further functionalities are developed to demonstrate the use of multiple
detection format outputs from the multi-task model. These are object
configuration and depth estimation, based on the object segmentation masks and
key points, respectively. Segmentation masks generated by the multi-task model
can be used to determine rocker arm orientations with respect to the image,
with the 2nd order moment of the mask. The three estimated key points on the
rocker arm then verify correct configuration of the rocker arm object with
respect to the engine. The process for finding the outer side for the correct
configuration is to minimize the axis of the smallest 2nd order moment, then
apply a cross-product between the axis coordinates and predicted key points.
The one key point with an opposite sign to the others indicates the outer
side. Fig. 1 illustrates the correct orientation of the rocker arm on the
engine and verifies (in)correct placement.

To estimate the depth of the rocker arm target, with respect to the camera,
two methods were considered: by segmentation mask or by key points (see Fig.
3(b)). In both cases the depth value from corresponding RGB pixel coordinates
are retrieved from a depth camera (Intel Realsense D435). For a segmentation
mask, the entire mask area as depth range is taken and an average depth value
is returned, and for a key point, a single depth value is returned. The key
point approach provides 10 times less difference in depth range (i.e., $468\pm
5mm$) as compared to the segmentation mask, and is therefore found to be more
reliable.

###  IV-D Discussion

The proposed approach enables images to be annotated by multiple different
labels, including bounding boxes, polygons and key points, during a single
annotation step, and store the result in a single convenient format (COCO).
The annotations are then all included in the same data augmentation process to
collect a dataset utilized as input for training a multi-task model. In
effect, this provides a single model with multiple detection outputs in
different detection formats (bounding box, segmentation masks and key points).
Compared to single-task models that only consider single annotation format
input and detection output, the following benefits are identified. 1. Time-
saving for image annotation and augmentation. All annotations can be done in a
single image, without changing any formats for different annotation types. For
example, in case of a single task-model, one image would need to be annotated
three times separately for three different annotation types. 2. Memory space
requirements saving in model training. As only a single multi-task model needs
to be trained instead of multiple single-task models, size of the model is
reduced. 3. Combining detection outputs. Detection modalities from the multi-
task model can be combined to solve additional detection tasks, not directly
provided by the model. For example, depth and object configuration from key
points as shown by our use cases, or other post-prediction applications such
as boundary or grasp pose detection, or the estimation of object properties.

A current limitation of the annotation pipeline is that it relies on existing
annotation (Label Studio [9]) and augmentation (Albumentations [8]) tools.
While these tools can be used under an open source license, access and support
might change over time. Another limitation is that, still, annotation is a
time-consuming step in the process of generating a learning-based detection
model. Selection of which data to annotate and in which format (e.g., depth
from pixel masks or key points) requires manual decisions and an iterative
process to select a best outcome for the tasks. Automated and interactive
annotation (e.g., based on machine learning [37, 23]) and semi-automated,
interactive image annotation for visual detection exist [38], but are still in
early stages of research. This is a natural next-step in developing assistive
tools for model training and planned as future work.

##  V CONCLUSION

Learning-based multi-task models require suitable tools for the collection and
annotation of data, especially when data needs multiple labels. In this work
we proposed a single framework that collects and annotates all required labels
(bounding boxes, polygons and key points) from a single image and stores it as
a single benchmark format (COCO). Augmentation can then utilize this single
format to generate a full dataset. The approach is validated with two use
cases, where three different annotation types are needed per image. Results
show that multi-task models provide benefits with respect to inference time
and memory usage as compared to multiple single-task models. In addition,
results show that multi-task models allow to combine detection outputs and
solve additional detection tasks from a single model, such as the estimation
of depth and object configuration.

## ACKNOWLEDGMENT

Project funding was received from European Union’s Horizon 2020 research and
innovation programme, grant no. 871449 (OpenDR) and 871252 (METRICS).

## References

  * [1] A. I. Károly et al., “Deep learning in robotics: Survey on model structures and training strategies,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 51, no. 1, pp. 266–279, 2021.
  * [2] F. Piccialli et al., “A survey on deep learning in medicine: Why, how and when?,” Information Fusion, vol. 66, pp. 111–137, 2021.
  * [3] A. Kamilaris and F. X. Prenafeta-Boldú, “Deep learning in agriculture: A survey,” Computers and electronics in agriculture, vol. 147, pp. 70–90, 2018.
  * [4] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Communications of the ACM, vol. 60, no. 6, pp. 84–90, 2017.
  * [5] R. Caruana, “Multitask learning,” Machine Learning, vol. 28, pp. 41–75, 1997.
  * [6] S. Vandenhende et al., “Multi-task learning for dense prediction tasks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 7, pp. 3614–3633, 2022.
  * [7] B. Pande et al., “A review of image annotation tools for object detection,” in International Conference on Applied Artificial Intelligence and Computing, pp. 976–982, 2022.
  * [8] Albumentations. https://albumentations.ai/, 2023.
  * [9] Label Studio. https://labelstud.io/, 2023.
  * [10] T.-Y. Lin et al., “Microsoft COCO: Common objects in context,” in European Conf. on Computer Vision (ECCV), pp. 740–755, 2014.
  * [11] C.-Y. Wang et al., “Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,” arXiv preprint arXiv:2207.02696, 2022.
  * [12] M. Everingham, L. V. Gool, C. K. Williams, J. Winn, and A. Zisserman, “The Pascal Visual Object Classes (VOC) challenge,” International Journal of Computer Vision, vol. 88, pp. 303–338, 6 2010.
  * [13] V7Labs. https://www.v7labs.com/, 2023.
  * [14] CVAT. https://www.cvat.ai/, 2023.
  * [15] P. Voigtlaender et al., “MOTS: Multi-object tracking and segmentation,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [16] Labelbox. https://labelbox.com, 2023.
  * [17] LabelMe. http://labelme.csail.mit.edu, 2023.
  * [18] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch networks for multi-task learning,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3994–4003, 2016.
  * [19] I. Kokkinos, “UberNet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6129–6138, 2017.
  * [20] C. Fifty, E. Amid, Z. Zhao, T. Yu, R. Anil, and C. Finn, “Efficiently identifying task groupings for multi-task learning,” Advances in Neural Information Processing Systems, vol. 34, pp. 27503–27516, 2021.
  * [21] Z. Meng, X. Yao, and L. Sun, “Multi-task distillation: Towards mitigating the negative transfer in multi-task learning,” in IEEE International Conference on Image Processing (ICIP), pp. 389–393, 2021.
  * [22] C.-Y. Wang, I.-H. Yeh, and H.-Y. M. Liao, “You Only Learn One Representation: Unified network for multiple tasks,” arXiv preprint arXiv:2105.04206, 2021.
  * [23] P. Bhagat and P. Choudhary, “Image annotation: Then and now,” Image and Vision Computing, vol. 80, pp. 1–23, 2018.
  * [24] C. Sager, C. Janiesch, and P. Zschech, “A survey of image labelling for computer vision applications,” Journal of Business Analytics, vol. 4, no. 2, pp. 91–110, 2021.
  * [25] M. Abadi et al., “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015.  Software available from tensorflow.org.
  * [26] A. Paszke et al., “PyTorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019.
  * [27] G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software Tools, 2000.
  * [28] J. Lemley, S. Bazrafkan, and P. Corcoran, “Smart augmentation learning an optimal data augmentation strategy,” IEEE Access, vol. 5, pp. 5858–5869, 2017.
  * [29] Z. Zou et al., “Object detection in 20 years: A survey,” Proceedings of the IEEE, vol. 111, no. 3, pp. 257–276, 2023.
  * [30] C. Harris, M. Stephens, et al., “A combined corner and edge detector,” in Alvey vision conference, vol. 15, pp. 10–5244, 1988.
  * [31] X. Ding et al., “Local keypoint-based Faster R-CNN,” Applied Intelligence, vol. 50, pp. 3007–3022, 10 2020.
  * [32] C.-C. Wong et al., “Manipulation planning for object re-orientation based on semantic segmentation keypoint detection,” Sensors, vol. 21, no. 7, p. 2280, 2021.
  * [33] L. Manuelli, W. Gao, P. Florence, and R. Tedrake, “kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation,” in The International Symposium of Robotics Research, pp. 132–157, 2019.
  * [34] Detectron2. https://github.com/facebookresearch/detectron2, 2023.
  * [35] G. Sharma, R. Pieters, and A. Angleraud, “Engine assembly dataset.” http://dx.doi.org/10.5281/zenodo.7669593, Feb. 2023.
  * [36] G. Sharma and R. Pieters, “Planetary gear dataset.” http://dx.doi.org/10.5281/zenodo.8223473, Aug. 2023.
  * [37] Q. Cheng et al., “A survey and analysis on automatic image annotation,” Pattern Recognition, vol. 79, pp. 242–259, 2018.
  * [38] J. Bragantini, A. X. Falcão, and L. Najman, “Rethinking interactive image segmentation: Feature space annotation,” Pattern Recognition, vol. 131, p. 108882, 2022.