# Collaborative Image Understanding

Koby Bibas<EMAIL_ADDRESS>MetaTel AvivIsrael ,  Oren Sar Shalom
<EMAIL_ADDRESS>AmazonTel AvivIsrael and  Dietmar Jannach
<EMAIL_ADDRESS>University of KlagenfurtKlagenfurtAustria

(2022)

###### Abstract.

Automatically understanding the contents of an image is a highly relevant
problem in practice. In e-commerce and social media settings, for example, a
common problem is to automatically categorize user-provided pictures.
Nowadays, a standard approach is to fine-tune pre-trained image models with
application-specific data. Besides images, organizations however often also
collect _collaborative_ signals in the context of their application, in
particular how users interacted with the provided online content, e.g., in
forms of viewing, rating, or tagging. Such signals are commonly used for item
recommendation, typically by deriving _latent_ user and item representations
from the data. In this work, we show that such collaborative information can
be leveraged to improve the classification process of new images.
Specifically, we propose a multitask learning framework, where the auxiliary
task is to reconstruct collaborative latent item representations. A series of
experiments on datasets from e-commerce and social media demonstrates that
considering collaborative signals helps to significantly improve the
performance of the main task of image classification by up to 9.1%.

collaborative filtering, computer vision, multitask learning

Information Extraction, Image Categorization, Collaborative Filtering,
Multitask Learning

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
31st ACM International Conference on Information and Knowledge Management;
October 17–21, 2022; Atlanta, GA, USA††booktitle: Proceedings of the 31st ACM
International Conference on Information and Knowledge Management (CIKM ’22),
October 17–21, 2022, Atlanta, GA, USA††price: 15.00††doi:
10.1145/3511808.3557260††isbn: 978-1-4503-9236-5/22/10††ccs: Information
systems Information extraction††ccs: Computing methodologies Multi-task
learning††ccs: Computing methodologies Computer vision

##  1\. Introduction

Image understanding can be described as the process of automatically analyzing
unstructured image data in order to extract knowledge for specific tasks.
Automatically assigning images to pre-defined categories (i.e., image
categorization) is one important form of image understanding and an area in
which we observed substantial progress in recent years, largely due to
innovations in deep learning (Deng et al., 2009; Krizhevsky et al., 2012;
Bibas et al., 2021b). Nowadays, one standard way of developing an image
categorization solution for a specific task or application is to rely on pre-
trained image models and to fine-tune them with application-specific data for
the particular problem at hand.

In a specific application setting, additionally to the raw image information
(pixels), other types of information may be available as well. In particular,
providers of e-commerce or social media sites nowadays often have access to
_collaborative_ information that is collected in the context of an
application. Such sites usually often record how users interact with the
provided online content, e.g., in the form of viewing, rating, commenting,
tagging, or resharing.

Such collaborative signals can be used for different purposes and in
particular for personalization. Most importantly, these signals have been
successfully used for many years by online retailers and service providers to
build _collaborative filtering_ (CF) recommender systems (Resnick et al.,
1994). Today’s most effective CF systems are based on deriving _latent_
representations (_embeddings_) of users and items from a given user-item
interaction matrix through matrix factorization (Hu et al., 2008) or deep
learning techniques (Liang et al., 2018; Hidasi et al., 2017).

The latent representations that are derived from collaborative information
cannot be directly interpreted. However, it seems intuitive to assume that the
_item representations_ may encode some information about the relevant
categories in a domain.

In this work, we explore an approach which we call _collaborative image
understanding._ Specifically, the idea is to leverage the collaborative
information as additional supervision for the _training_ phase of image
understanding models. By incorporating this collaborative information only
during training, the model can support also new images for which collaborative
information is not available. Such problems for example arise when online
platforms have to automatically process images as soon as they are uploaded,
before they gain any collaborative information. Any application that combines
images and interaction data suits to this setting, where applications include
e-commerce, social media and online content.

The main contributions of our work are as follows.

  1. (1)

We introduce a new signal (i.e., collaborative information) to support the
training procedure of automated image understanding processes.

  2. (2)

We propose a general multitask-learning (MTL) (Caruana, 1997) framework to
incorporate collaborative information in the form of latent item
representations into existing single-task content understanding models.

  3. (3)

We explore several alternative approaches to combining collaborative
information in the training procedure of image classification models.

  4. (4)

We show that our approach is also particularly effective in typical real-world
situations when labels are missing for a fraction of the training images.

Our MTL approach, where the auxiliary task is to reconstruct collaborative
latent item representations, is highly promising. An in-depth empirical
evaluation on the Pinterest (Zhong et al., 2015), MovieLens (Harper and
Konstan, 2015), Amazon Clothing and Amazon Toys (McAuley et al., 2015)
datasets revealed that the classification performance can be improved by 5.2%,
2.6%, 9.1%, and 3.0% respectively.

The rest of this paper is organized as follows. In Section 2 we provide the
necessary background and review earlier work. Section 3 details the proposed
approach; thorough experiments are described in Section 4. Section 5 concludes
the paper.

##  2\. Background and Related Work

In this section, we first briefly provide some relevant background in three
research areas: image classification, collaborative filtering and multi-task
Learning (Section 2.1). Then, we discuss earlier works, which use side
information or collaborative information in combination with image data
(Section 2.2).

###  2.1. Background

####  2.1.1. Image Classification with Neural Networks

Classifying images to predefined categories based on their content is an
important task in the areas of computer vision and information extraction. The
training input for this learning problem is a set $\\{(x_{i},y_{i})\\}$, where
$x_{i}$ is an image of an entity $i$ and $y_{i}$ is the corresponding
category. The ImageNet database (Deng et al., 2009) for example, contains
millions of hand-labeled images, which are assigned to thousands of
categories. Usually models trained on one domain do not perform well on other
domains and labels from the target domain are required (Zhou et al., 2021). As
manual annotation is expensive and time consuming, label paucity is a
ubiquitous problem.

In the last ten years, substantial progress was made in the field due to
advances in deep learning technology and in particular through the use of
convolutional neural networks (CNN) (Krizhevsky et al., 2012; Simonyan and
Zisserman, 2015; Szegedy et al., 2015). Continuously new architectures were
proposed over the years, including the ResNet (He et al., 2016), MobileNet
(Howard et al., 2017) and the recent RegNet (Radosavovic et al., 2020)
approaches that we use in the experiments in this paper.

Given a particular application problem, a common approach to mitigate the
paucity of labeled data is to rely on existing models that are pre-trained on
a large corpus like ImageNet, and to then fine-tune them with the data of the
particular application. Such an approach is also followed in our work.
However, in addition to the application-specific images and labels, we also
consider collaborative information to further improve classification
performance.

####  2.1.2. Collaborative Filtering and Latent Factor Models

Collaborative filtering recommender systems are commonly based on a matrix
that contains recorded interactions between users and items (Resnick et al.,
1994), with the typical goal of predicting the missing entries of the matrix
and/or ranking the items for each user in terms of their estimated relevance.
Matrix factorization (MF) techniques prove to be highly effective for this
task and are based on representing users and items in a lower-dimensional
latent space. While early methods were based on Singular Value Decomposition
(Billsus and Pazzani, 1998), later approaches used various types of machine
learning models to compute the latent factors, e.g., (Hu et al., 2008). Note
that while various deep learning approaches were proposed for the task in
recent years, latent factor models remain highly competitive and relevant
today (Rendle et al., 2020).

Technically, MF methods project users and items into a shared latent space,
where each user and each item is represented through a real-valued vector. A
user vector indicates to what extent the corresponding user prefers each of
the latent traits; likewise, an item vector represents to what extent the
corresponding item possesses them (Koren et al., 2009). The traits are called
latent because their _meaning_ is unknown. Yet, we know that latent factor
models are highly effective for recommendations and furthermore, that user
vectors hold information on the demographics of the users (Resheff et al.,
2018). Therefore, we hypothesize that item vectors encode certain types of
relevant information about the items—like category or brand—which we try to
leverage in our work for improved image understanding.

####  2.1.3. Multi-Task Learning

MTL is a machine learning approach which optimizes multiple learning tasks _in
parallel_ based on a shared representation for all tasks. The promise of MTL
is that if the tasks share significant commonalities, then this approach may
lead to improved performance on each individual task, compared to training
task-specific models separately (Caruana, 1997). The rationale behind MTL and
its success lies in the potential of improved generalization by using the
information contained in the signals of related tasks. Thus, each individual
task can “help” other tasks, often in order to also prevent overfitting for
one particular task. Concretely, this is achieved by having some shared
parameters for all tasks. That way, during training time each task directly
affects the other ones.

Various successful applications of MTL are reported in the literature, both
for computer vision problems and other areas, see Zhang and Yang (2017) for a
survey. However, using MTL to combine collaborative signals and image
information for higher categorization accuracy has not been explored in the
literature before. As our experimental evaluations will show, these two types
of information indeed have commonalities that contribute to the learning
process in an MTL setting. In addition, the proposed MTL approach is favorable
over other combination approaches such as training the models independently.

###  2.2. Related Work

There is a plethora of classification models that combine raw image
information together with additional metadata, which generally fall under the
umbrella of multi-modal classification. Notable examples are image and text
classification, see Baltrušaitis et al. (2018) and Ramachandram and Taylor
(2017) for related surveys on multi-modal learning. Our work is fundamentally
different than multi-modal learning since in this those works the other
modalities are part of the features of the model. That is, they serve as input
to the model both at training and inference time. In our work we assume that
the other modality, namely collaborative information, is not available at
inference time, yet we aim to leverage its existence at training time by using
it to guide the learning of the model. Thus, the model may benefit from this
signal without relying on it for inference.

Several works have tried to harness the power of CF for various image
understanding problems. The work of Dabov et al. (2007), for example, applies
a CF technique to improve image denoising. Choi et al. (2015), on the other
hand, showed how to leverage collaborative information for better pose
estimation results. However, the neighborhood of an image in these works is
determined not by the interactions with it, but according to visual
similarity.

There are also existing several works that aim to improve the image content
understanding process using collaborative information as we do in our work (Ma
et al., 2010; Xu et al., 2019; Liu et al., 2019; Sigurbjörnsson and Van Zwol,
2008). However, in these works the user interaction signal is used as an input
to their image understanding models. Therefore, the collaborative information
cannot be used to model _new_ images, which is the focus of our work. Our
proposed method in contrast does not treat user interactions as a feature for
the model, but only as means of guidance.

Another type of related work can be found in Li and Tang (2016) and Li et al.
(2018), which both aim to improve the training of content understanding models
using user data. However, they treat the user interaction data as a type of
content. That is, they omit the identities of the interacting users and do not
apply collaborative filtering as we do. As a result of such an approach, if
the user interaction is, for example, image _tagging_ , then their work is
limited to enhancing image tagging processes. An earlier work described in
Sang et al. (2012) retains the users’ identities and its objective is to
improve image tagging using interaction data. This is done by applying tensor
factorization to model users, images and tags in the same joint latent space.
However, this approach also binds the type of the interactions and the
prediction task. In our approach, the prediction task is not tied to a
specific type of user interactions.

Common to all discussed approaches based on user data is their need to process
the raw image data for each user interaction. As the number of pixels $P$ in a
typical image reaches to million ($10^{6}$) and the number $N$ of interactions
in real-life datasets can be hundreds of millions ($10^{8}$), their running
time is in $O(N\cdot P)=10^{14}$, which is computationally expensive or even
infeasible for large real-life datasets. Differently from that, in our
approach we first apply a CF method on the interaction data to create a
compact representation for each item of dimensionality $f$, and we then train
a content understanding model using those fixed-size vectors. Therefore, the
running time is $O(f\cdot(P+N))\ll O(N\cdot P)$. This is achieved since
instead of taking a naive end-to-end solution, we adopt a two-phase solution,
which first transforms the interaction data to a label set.

Finally, it is worth mentioning that the other direction, i.e., content
understanding to enhance recommender systems, has shown striking success
(Barkan et al., 2019; He and McAuley, 2016; Sanchez et al., 2012; Shalom et
al., 2019; Eden et al., 2022). The relationship of these works to our current
approach is however limited, as our goal is to improve the image understanding
process and not item recommendations.

##  3\. Proposed Technical Approach

###  3.1. Overview

Figure 1. Overview of the proposed method.

Borrowing terminology from the recommender systems community, the term _item_
refers to an entity that users can interact with, and furthermore may have an
image as side-information. In this paper, as the main focus is image
understanding, we use the terms image and item mostly interchangeably. In some
cases the image can be the item itself, as in image sharing services (e.g.,
Pinterest) for example, where users interact with images.

Our proposed approach consists of two main phases, as illustrated in Figure 1.
The input to the learning task is a collection of images, where (i) each image
has category labels assigned and (ii) for each image we are given a set of
users who have interacted with it, e.g., through rating, tagging, or sharing.
In the _first phase_ of our approach, a user-item interaction matrix is
derived from the input data and a CF technique is applied to create fixed-size
latent representations (vectors) of users and items. The purpose of this phase
is solely to generate augmented labels for the classification problem. In the
_second phase_ , multitask learning is applied, where the main task is to
predict the categories of the images, and the auxiliary task is to reconstruct
the latent item representations created in the first phase111The latent user
vectors are not used in this process.. As mentioned, we assume that the latent
item vectors—dubbed _CF vectors_ from here on—encode category information that
can help the training of the main task. Technically, instead of learning the
image categorization from scratch, we rely on a pre-trained image model, which
we then fine-tune with the given image data of the problem at hand.

Once the model is trained, we use it to predict categories of _new images_ ,
i.e., for images for which we do not have collaborative information. Typical
important use cases could be pictures that users upload on a social media
platform or pictures of newly added shop items that should be automatically
categorized.

It is important to note that at this level, we consider our technical
contribution as a _framework_ because we propose a general form of
incorporating collaborative information into the image understanding process.
In a given application scenario, the framework can then be instantiated in
appropriate ways, using a suitable combination of matrix factorization
technique, training set for pre-training, and deep learning architecture for
image categorization.

###  3.2. Technical Approach

In the following, we describe our approach more formally. In this
formalization, we use $i\in\\{1,\ldots,I\\}$ to index items. Table 1
summarizes the notations in this paper.

Symbol | Explanation
---|---
$x_{i}$ | Content of item $i$
$y_{i}$ | Category label of item $i$ (a binary vector)
$\hat{y}_{i}$ | A predicted label of item $i$ (a real-valued vector)
$U_{i}$ | Set of users who interacted with item $i$
$q_{i}$ | CF vector of item $i$
$\hat{q}_{i}$ | A predicted CF vector of item $i$
$\omega_{i}$ | Weight of CF vector of item $i$
Table 1. Main symbols used in the paper.

At training time, we assume each item $i$ is associated with an _image_
$x_{i}$ and with some _label_ $y_{i}$, a category in our use case.
Furthermore, the item may be associated with a set $U_{i}$ of users who
interacted with it. At inference time, given an image to be modeled we assume
it is _not_ associated with historical interaction data.

In many practical scenarios, the content understanding model must be invoked
upon introducing the new item to the system, and therefore it has not gathered
usage data (also known as _cold item_ , a ubiquitous problem in recommender
systems (Sar Shalom et al., 2015)). We emphasize that integrating
collaborative filtering information at inference time has been proposed
earlier, see Section 2.1. These approaches assume the existence of such
information for the images to be modeled. In contrast, this work focuses on
situations where such information is not available. Integrating existing
collaborative information into our framework is deferred to future work.

Therefore, the input for the model is a set of triplets
$\\{(x_{i},y_{i},U_{i})\\}$, where $x_{i}$ serves as the feature set and
$y_{i}$ and $U_{i}$ provide guidance for training the model. We note that at
training time some items $i$ might miss wither the label $y_{i}$ or the
collaborative data $U_{i}$, which frames the problem as a semi-supervised one
(Chapelle et al., 2009). As we show in Section 4, when only a portion of the
training data is labeled, the relative contribution of our approach is even
higher.

###  3.3. Deriving CF vectors

In the first phase, the collaborative data $U_{i}$ is transformed into a
binary user-item interaction matrix $M$. A collaborative filtering technique
is then used to construct a user embedding matrix $P$ and an item embedding
matrix $Q$ of a given size $f$ (number of factors). We highlight that any CF
method can be applied, as long as it represents items by a fixed-size vector.
After this process, we are given, for each item $i$, a corresponding latent CF
vector $q_{i}$. Although these vectors are uninterpretable, we hypothesize
that they hold valuable information for the classification task, as indeed is
empirically shown in Section 4.

###  3.4. CF-guided Image Categorization

In the second phase, the image understanding (categorization) process is
guided by the CF vectors obtained in the previous phase.

In our multitask-learning approach, predicting the labels is the main task and
the auxiliary task is based on the given CF vectors. After performing the
first task, the input to the model is a set of triplets
$\\{(x_{i},y_{i},q_{i})\\}$, where $y_{i}$ and $q_{i}$ are the labels. The
model has two outputs, corresponding to the main and auxiliary tasks.
$\hat{y}_{i}$ and $\hat{q}_{i}$ are the predicted categories and CF vector,
respectively.

##### Main task: Classification loss

Generally, for classification problems with $N$ classes, the label $y$ is an
$N$-dimensional binary vector (which can support multi-label classification),
where $y^{n}$ denotes the $n$th entry in the vector. The single-task
categorization model processes the input and then outputs a prediction vector
$\hat{y}$ that aims to minimize the binary cross-entropy (Ruby and Yendapalli,
2020). Since labels might be imbalanced, that is, some categories may appear
much more often than others, we also apply class balancing as follows. Let
$\eta_{n}$ denote the number of appearances of class $n$. Then the raw weight
of this class is $\frac{1}{\eta_{n}}$. We normalize the weights such that the
average weight is 1, that is: $\frac{1}{N}\sum_{n=1}^{N}\frac{1}{\eta_{n}}=1$.
Then the class-balanced binary cross-entropy is defined as:

(1)
$\ell_{MAIN}(\hat{y},y)=\sum_{n=1}^{N}\frac{1}{\eta_{n}}y^{n}\cdot\log\hat{y}^{n}+(1-y^{n})\cdot\log(1-\hat{y}^{n}).$

##### Auxiliary task: CF-based loss

Inspired by Gou et al. (2021), we propose a method that considers all of the
information embedded in the CF vectors to reconstruct the CF vector using the
content only. We therefore name this method MTL-reconstruct. Let $L_{1}$ and
$L_{2}$ denote the Manhattan and Euclidean distances between the CF vector $q$
and its reconstruction $\hat{q}$, respectively. The reconstruction loss
(denoted as $l_{CF\\_R}$) asks to minimize the difference between the vectors
as follows222We experimented with several other metrics, like convex
combinations and single distance loss. Yet, this one yielded the best
results.:

$\ell_{CF\\_R}(q_{i},\hat{q}_{i})=exp(L_{1}(q_{i},\hat{q}_{i})+L_{2}(q_{i},\hat{q}_{i}))$

##### Considering CF vector relevance weights.

Since CF methods do not converge to a global optimum, some items might be
poorly represented and therefore the CF vectors induce weak supervision
(Hoffmann et al., 2011). To cope with such “noisy labels” in our problem
setting, where some CF vectors might be more informative than others, we
introduce a per-instance confidence weighting method which affects the impact
of the auxiliary task. This method may depend on the CF vector $q$ or even on
the raw collaborative information $U$, annotated as $h(\cdot)=\omega$. To
avoid shattering gradients, $\omega$ is upper bounded by some hyperparameter
$\Omega$. The auxiliary loss function is then:

(2)
$\ell_{AUX}(\hat{q},q,\omega)=min(\omega,\Omega)\cdot\ell_{CF\\_R}(q,\hat{q})$

We explored three ways of weighting the CF vectors when training the model
with the MTL-reconstruct approach.

  * •

_Uniform confidence_ : This approach neglects the possible variability in the
fitness and information encoded in the CF vectors. Hence, $h$ is a nullary
function which assigns the same weight to each CF vector.

  * •

_Interaction confidence_ : The intuition behind this approach is as more users
interacted with an item, then there is more collaborative information about
it, and as a consequence we trust its produced CF vector more. Therefore,
according to these considerations, there should be a positive correlation
between the assigned weight of a CF vector and the number of interacting
users. Let $|U_{i}|$ denote the number of interacting users with item $i$.
Then333Another plausible alternative is linear correlation, however it was
inferior to the proposed method in preliminary experiments.,
$h(U_{i})=\sqrt{|U_{i}|}$.

  * •

_Loss-based confidence_ : The utility of a CF vector for the category
prediction task might not always be proportional to its amount of interacting
users. We therefore suggest a novel method that explicitly captures the
utility level of each individual CF vector according to its intrinsic traits.
To this end, we train a category prediction model, called _CF2Label_ , that
takes as input only the CF vector (without the image itself). See Section 4.2
for more details on this model. We then approximate the fitness of a CF vector
by its ability to predict the labels, measured by the mean reciprocal loss
value on the positive labels. Formally, let $y$ be the actual label of a given
item and $\hat{y}$ be the predicted label: $CF2Label(q)=\hat{y}$. The mean
loss on the positive labels is:

$\ell_{WEIGHT}(\hat{y},y)=\frac{1}{\sum_{n}y^{n}}\sum_{n=1}^{N}y^{n}\cdot\log\hat{y}^{n}.$

Hence, the weight of this CF vector is:

$h(q_{i},y_{i})=\frac{1}{\ell_{WEIGHT}(CF2Label(q_{i}),y_{i})}.$

Our experiments will show that this novel weighting method considerably
contributes to the performance of the model. We also experimented with the
mean across all labels, including the negative ones. However, this
deteriorated results, as the performance on positive labels is more
informative.

##### Weighting the tasks.

Various ways of learning how to balance the weights of the tasks in multitask-
learning were proposed in the literature (Chen et al., 2018; Guo et al., 2018;
Kendall et al., 2018). However, since our problem involves only two tasks, we
resort to a simple hyperparameter tuning strategy to find suitable weights.
All in all, the final loss function is:

(3)
$\ell_{MTL}(\hat{y},y,\hat{q},q,\omega)=\ell_{MAIN}(\hat{y},y)+\alpha\cdot\ell_{AUX}(\hat{q},q,\omega),$

where $\alpha$ is a hyperparameter that controls the relative importance of
the two tasks. In all experiments, the optimal weight of the auxiliary task
was non-negligible ($1<\alpha<2$), and our experiments clearly demonstrate the
importance of the auxiliary task.

##  4\. Experimental Evaluation

Next, we report the results of an in-depth experimental evaluation of our MTL-
based image understanding approach. Specifically, we seek to answer the
following research questions (RQs).

  * •

(RQ1) To what extent does our method improve image understanding processes
_(a)_ compared to models that do not use collaborative information and _(b)_
compared to alternative ways of incorporating collaborative information?

  * •

(RQ2) Given that missing labels are common in real-world scenarios, we ask:
What is the relative gain of the proposed method in case of data paucity,
i.e., when many labels are missing in the training data?

###  4.1. Experiment Setup

####  4.1.1. Datasets

To show the robustness of our approach we made experiments with four real-
world datasets. These datasets differ in fundamental properties like domain
and types of labels. Each of them contains item images, category labels, as
well as user interaction data. Furthermore, they were selected such that they
cover very popular domains: e-commerce, online content and social media.

Two of the datasets are from the e-commerce domain and were collected from
Amazon.com (McAuley et al., 2015). The items in these two public datasets are
from the area of “Clothing” and “Toys”, where products are associated with
their catalog images. The labels in these datasets correspond to the item
categories and user interactions refer to purchases. The third dataset is a
version of the widely-used public MovieLens 10M dataset (Harper and Konstan,
2015), which includes movie posters. In this dataset, the movie genres are
used as category labels and user ratings as interactions. Finally, as a fourth
dataset we include a dataset from the social media domain. It is an anonymized
version of the Pinterest image dataset used in (Zhong et al.,
2015)444https://nms.kcl.ac.uk/netsys/datasets/social-curation/dataset.html.
Here, the items are the images themselves; the labels correspond to user-
provided tags and interactions are repin
activity555https://sproutsocial.com/glossary/repin/. The labels here are
single-class, unlike the rest of the datasets, which have multiple labels per
item.

For all datasets, the recorded user interactions, e.g., ratings, serve as
collaborative signals. The main statistics of the datasets are provided in
Table 2. Observe that the datasets not only vary in terms of size and density,
but also with respect to the average number of categories each item is
assigned to (“Avg. pos. labels”).

####  4.1.2. Pre-Trained Image Models

We conducted preliminary experiments with different pre-trained models versus
randomly initialized image models. These experiments confirmed that in all
cases fine-tuning a pre-trained image model leads to substantially better
classification results. In the rest of the paper, we therefore only report
results that were obtained with pre-trained models.

Statistic | Pinterest | MovieLens | Clothing | Toys
---|---|---|---|---
#Images | 27,080 | 10,677 | 23,033 | 11,924
#Interactions | 265,252 | 10,000,054 | 278,677 | 167,597
#Users | 170,978 | 68,532 | 39,387 | 19,412
Sparsity | 5.73e-5 | 1.37e-2 | 3.07e-4 | 7.24e-4
#Labels | 32 | 19 | 199 | 54
Avg. pos. labels | 1.0 | 2.03 | 3.22 | 1.28
Gini index | 0.48 | 0.51 | 0.74 | 0.58
Table 2. Dataset statistics.

Over the last years, a variety of deep learning architectures were proposed
that can be used for image categorization, as discussed in Section 2. One goal
of our experiments is to demonstrate the robustness of our MTL-based approach
with respect to the underlying image model, i.e., that the effectiveness of
our approach is not tied to a particular image model and may contribute to new
models to come in the future. Therefore, we considered the following three
state-of-the-art architectures in our experiments: ResNet18 (He et al., 2016),
MobileNet (Howard et al., 2017), and RegNet (Radosavovic et al., 2020). To
avoid any bias, we randomly assigned each of them to one of the datasets,
ending up with ResNet18 for the Amazon datasets, RegNet for Pinterest, and
MobileNet for MovieLens. All models were pre-trained on the ImageNet set
(Krizhevsky et al., 2012).

####  4.1.3. CF Models

Generally, we hypothesize that the CF vectors carry meaningful information
about the items, and that this information can be advantageous for our
problem, which we validate in Section 4.2. To incorporate a well fitted CF
model in our approach, we seek a method that performs well for the given
datasets. A large variety of CF models that use item embeddings were proposed
over the last fifteen years, and we experimented with several landmark CF
methods. Among other methods, we considered Bayesian Personalized Ranking
(BPR) as a more traditional model (Rendle et al., 2009) and Variational
Autoencoders (VAE) (Liang et al., 2018), which is a state-of-the-art
technique666We relied on the implementations provided in the Cornac framework
(https://github.com/PreferredAI/cornac). Motivated by (Press and Wolf, 2016),
the last layer of the decoder in the autoencoder architecture serves as the
item embedding matrix $Q$. Moreover, we supplemented the original VAE
implementation with bias terms as this led to even better results. Note that
we also tried an autoencoder-based recommender based on the VAE model proposed
in (Ouyang et al., 2014). This however did not lead to competitive results.

In Table 3, we report the results of BPR and VAE in terms of the AUC metric
for the four datasets after careful fine-tuning. For comparison purposes, we
also include the results obtained by a non-personalized recommender that
always suggests the most popular items to users. Not surprisingly, the results
show that VAE consistently works better than BPR across all datasets.
Therefore, we use VAE as the CF model for all experiments reported in this
paper. Remember that we do not rule out that other CF models may exist that
lead to even better recommendation results for individual datasets, and we
iterate that finding the best existing CF model is not the main focus of our
present research. Nonetheless, we acknowledge that future advances in CF
models may be beneficial for our MTL-based image categorization as well.

Method | Pinterest | MovieLens | Clothing | Toys
---|---|---|---|---
Most popular | 0.47 | 0.94 | 0.63 | 0.65
BPR | 0.56 | 0.97 | 0.66 | 0.69
VAE | 0.96 | 0.97 | 0.75 | 0.82
Table 3. Recommendation performance of CF models.

####  4.1.4. Evaluation Metric

To measure classification performance apply the widely-used Mean Average
Precision (mAP) metric (Hossin and Sulaiman, 2015), arguably one of the most
adopted evaluation metrics for multi-label classification (e.g., (Xu and Li,
2007; Revaud et al., 2019; Zha et al., 2015; Kaufman et al., 2019)). mAP is
computed as:

$\frac{1}{N}\sum_{c=1}^{N}AP(c)$

where $N$ is the number of classes and $AP(c)$ denotes the average precision
in class $c$. To compute statistical significance we use bootstrapping (Park,
2011) and paired sample $t$-tests (Weisstein, 2001).

####  4.1.5. Implementation & Hyperparameter Tuning

All of the implementation for our experiments was done in Python, using the
PyTorch framework. We share all material online for
reproducibility777https://github.com/anonymous1e6/cactus. Model selection and
hyperparameter tuning (e.g., learning rate, regularization and number of
hidden layers and sizes) and test were generally done in a systematic way on a
held-out set, evenly divided into evaluation and test sets. The held-out set
for the classification models was created by randomly selecting 30% of the
items. We emphasize that the collaborative information of these items was
omitted as well, to compute the performance on images with no prior user
interactions.

The held-out dataset for the recommender systems was created by randomly
selecting 20% of the interactions and AUC (Lü et al., 2012) was used as the
evaluation metric. This metric was chosen instead of top-$k$ evaluation
metrics like precision@$k$ or nDCG@$k$ because all test items directly
contribute to the final measure and not just those (possibly few) items that
made it to a top-$k$ list. This is an important trait because we care about
the fitness of all items, as each of them contributes to the auxiliary task in
the MTL setting.

###  4.2. Pre-Study: CF-Based Category Prediction

Before we discuss the results of the evaluation of our approach to
collaborative image understanding in Section 4.4, we report the outcomes of a
pre-study in which we aimed to validate the value of collaborative information
for category prediction.

To this end, we conducted the following experiment in which the image category
prediction task is solely based on the available interaction data, i.e., the
matrix describing user-item interactions. As a first step in the experiment,
we trained a collaborative filtering model on the interaction matrix to obtain
the needed latent item vectors. Specifically, we carefully tuned BPR and VAE
CF models on the four evaluated datasets. The resulting fixed-size888A latent
vector size of 64 worked best for all datasets; smaller or larger values led
to degraded recommendation performance. latent item vectors $q_{i}$ then serve
as the input to a neural network, which predicts the corresponding image
category label $y_{i}$. We name these models _CF2Label_. The model used in our
experiments consists of a two-layered MLP with ReLU as the activation function
and Binary Cross-Entropy per each class as the loss function999Other model
architectures are possible as well. Remember that the main here goal is to
demonstrate the general value of interaction data, but not necessarily to find
the best possible architecture..

The results of this experiment are shown in Table 4, where we report the image
categorization performance in terms of mAP. As a reference point to gauge the
effectiveness of the trained models, we provide the mAP results that we obtain
when making predictions solely based on the popularity of the categories in
the different datasets, i.e., when we always predict the most popular
category.

Approach | Pinterest | MovieLens | Clothing | Toys
---|---|---|---|---
Popularity based | 0.031 | 0.107 | 0.016 | 0.024
CF2Label BPR | 0.075 | 0.465 | 0.021 | 0.074
CF2Label VAE | 0.266 | 0.528 | 0.124 | 0.241
Table 4. Performance results for _CF2Label_ (mAP).

The results clearly show that even though the CF item vectors were trained
oblivious to the categories, they indeed contain valuable information for the
category prediction task. The mAP values obtained with the _CF2Label_ variants
are up to ten times higher than those of the popularity-based approach.
Aligned with our assumption stated in Section 4.1.3, we furthermore obtained
evidence that the VAE CF model, which outperformed BPR on the recommendation
task, also outperformed it on this task and therefore should be used in our
experiments. On an _absolute_ scale, the highest mAP values are obtained for
the MovieLens dataset. This is not surprising, given that it has the lowest
number of categories to predict and at the same time the highest user-item
interaction density across all datasets. In terms of the _relative_
improvement over the baseline, the gains obtained by _CF2Label_ are however
the lowest across the datasets, i.e., a five-time improvement.

Let us emphasize here that the mAP results reported in Table 4 must not be
compared on an absolute scale with the mAP values reported next in Section
4.4. In the pre-study reported here, the CF-based image category predictions
were based on knowledge about the observed user interactions for a given image
to categorize. Our MTL-based approach described in Section 3, however, is
designed for situations in which no collaborative information is yet
available, i.e., for new images, and where the collaborative information is
merely used to guide the learning process of the main task. Remember here that
we use the _CF2Label_ method to inform the weighting of the CF vectors in the
loss-based approach described in Section 3.

###  4.3. Baselines

To answer RQ1 on the effectiveness of MTL-reconstruct and its variations, we,
first of all, benchmark its performance against the performance that is
achieved with the fine-tuned image model alone, i.e., with a model that does
not use collaborative information. We refer to this baseline as Image-only in
the rest of the paper.

Regarding other possible baselines, we are not aware of any other work which
used collaborative information in combination with image models to categorize
_new_ images, that is, images for which no collaborative information is
available yet. Therefore, we designed two alternative ways of how one could
intuitively approach the problem: one of them based on multitask learning, and
one based on sequential fine-tuning of the auxiliary task and the main task.

####  4.3.1. Contrastive-Loss Approach

A possible advantage of the proposed MTL-reconstruct approach is its per-
dimension learning guidance by utilizing the entire information in the CF
vectors. However, reconstructing the entire CF vector means simultaneously
predicting tens or even hundreds of latent features. This might be an
overwhelming task, and consequently might introduce noise to the training
process. Hence, we propose an alternative approach and thereby an additional
baseline for our experiments, that resorts to supervised contrastive loss
(Khosla et al., 2020). Namely, at each epoch it generates a list of triplets,
where each item participates once as an _anchor_. For any anchor item $a$
(associated with CF vector $q_{a}$), it first chooses a _positive_ item $p$
such that its associated CF vector $q_{p}$ is the most similar to $q_{a}$ in
terms of the $L_{2}$ distance. Then it chooses at random a _negative_ item
$n$. The auxiliary task is then to distinguish between the positive and the
negative items solely based on the predicted CF vectors. More formally, the
task is to minimize the triplet loss (Schroff et al., 2015):

$\ell_{cont}(q_{a},q_{p},q_{n})=max\\{L_{2}(q_{a},q_{p})-L_{2}(q_{a},q_{n})+\tau,0\\}$

where $\tau$ is a hyperparameter, which serves as a positive margin. We refer
to this baseline in our experiments as Contrastive-loss.

Compared to the MTL-reconstruct method, this approach induces an easier
learning task, namely a binary classification task rather than a
multidimensional regression task. However, a substantial amount of information
is lost, as it reduces the entire CF vector into a single scalar that
represents the distance to the anchor.

####  4.3.2. Sequential Approach

A possible complication with multitask learning in general is that the
auxiliary task might overshadow the main task. In such situations, the model
tends to excel at the auxiliary task at the expense of the primary one.
Therefore, we explore fine-tuning (Too et al., 2019; Campos et al., 2017) as
an alternative way to leverage the available collaborative information for the
main task. In this approach, the model in the first step is optimized only for
the auxiliary task. Then, a fine-tuning phase follows where parameters are
optimized for the primary task only. We refer to this method as Sequential in
the rest of the paper. We note that this baseline cannot benefit from a joint
learning process as is the case for MTL.

###  4.4. Results

####  4.4.1. RQ1: Effectiveness of MTL-reconstruct

Table 5 shows the main results of our experimental evaluation with respect to
RQ1. Considering the last row of Table 5, we observe that our method MTL-
reconstruct with the loss-based weighting scheme leads to consistent and solid
performance improvements from 2.2% up to 9.5% over the Image-only baseline
across the different datasets. The greatest improvements are observed for the
Clothing dataset, which also has the highest number of categories.

The smallest improvements were achieved for the MovieLens dataset. This is
consistent with the findings from Section 4.2, where we also found the
smallest relative gain from the latent item vectors with the _CF2Label_ method
for the MovieLens data. Using other weighting schemes for MTL-reconstruct was
similarly beneficial except for one case for the MovieLens dataset, where the
interaction-based loss did not lead to any improvements.

While MTL-reconstruct is effective for all datasets, the extent of the gain
varies across the datasets. Various aspects may contribute to these
differences, including dataset characteristics (e.g., class distributions,
density of user interactions), aspects related to the item pictures (e.g.,
movie posters are artistic whereas e-commerce items are not), or the
underlying idiosyncrasies of each domain.

Figure 2. Samples count for each class.

With respect to dataset characteristics, Figure 2 illustrates the number of
items per category for all four datasets. Let us consider Figure 2(c) and
Figure 2(d), which show the distribution for the Clothing and Toys datasets.
Note that even though both datasets are from the e-commerce domain and even
from the same platform, we can observe a clear difference in terms of the
shape of the long tail distribution. In particular, the concentration on the
“short head” is much more pronounced for the Clothing dataset, i.e., a large
fraction of the items are assigned to a small set of broad (or popular)
categories. The Gini index for these datasets is given in Table 2 and it
supports these visual observations101010Higher Gini values indicate a strong
concentration of the samples.. Our experiments therefore indicate that the use
of collaborative information may be particularly helpful to improve the image
categorization process when there is a long tail of rather rare categories. A
deeper analysis of factors that influence the effectiveness of MTL-reconstruct
is part of our future work.

We found no consistent performance gains for the baseline Contrastive-loss.
While it worked very well for the Clothing dataset, for the Toys dataset, it
almost had no effect; and it led to a performance degradation compared to the
Image-only model for the MovieLens and Pinterest data. Still, we conclude that
the Contrastive-loss method might be a candidate to explore in certain
practical settings, given that there are indications that it may work well in
some domains.

Looking at the Sequential approach, we find that it did not help to improve
the results compared to the Image-only baselines. In contrast, we actually
consistently observed a small degradation when combining collaborative
information with the image-based models in this way. This highlights the
importance and value of combining collaborative information with image-based
models using a shared representation.

##### Weighting the CF vectors

Comparing the naive weighting method MTL-reconstruct
${}_{\text{\\!\emph{interaction}}}$ to the method that does not weigh the CF
vectors MTL-reconstruct ${}_{\text{\\!\emph{uniform}}}$, we do not find a
clear winner. However, our novel weighting method MTL-reconstruct
${}_{\text{\\!\emph{loss}}}$ consistently outperforms the rest of the methods.
This proves the need to deeply inspect the CF vectors to assess their weights.

Approach | Pinterest | MovieLens | Clothing | Toys
---|---|---|---|---
Image-only | 0.345 | 0.329 | 0.387 | 0.412
Sequential | 0.344 (-0.3%) | 0.308 (-6.5%*) | 0.352 (-9.0%*) | 0.41 (-0.4%)
Contrastive-loss | 0.34 (-1.4%*) | 0.325 (-1.3%*) | 0.416 (+7.6%*) | 0.411 (-0.2%)
MTL-reconstruct ${}_{\text{\\!\emph{uniform}}}$ | 0.36 (+4.3%*) | 0.336 (+2.0%*) | 0.391 (+1.1%*) | 0.424 (+3.0%*)
MTL-reconstruct ${}_{\text{\\!\emph{interaction}}}$ | 0.361 (+4.6%*) | 0.329 (0.0%) | 0.392 (+1.4%*) | 0.417 (+1.3%)
MTL-reconstruct ${}_{\text{\\!\emph{loss}}}$ | 0.363 (+5.2%*) | 0.338 (+2.6%*) | 0.422 (+9.1%*) | 0.424 (+3.0%*)
Table 5. Image categorization performance results for four datasets (mAP). The
best results for each dataset are printed in bold font. Increases over the
Image-only baseline that were found to be statistically significant are marked
with * ($p<0.01$).

####  4.4.2. RQ2: Gains in situations of data paucity.

A certain lack of labels in a given target domain is a common problem in many
practical applications, given that this task is often manual and thus time-
consuming, see also Section 2. Therefore, we analyzed the effectiveness of our
proposed method in such situations of data paucity through a series of
additional experiments. Specifically, we measured the performance of our
method compared to the baseline model Image-only after randomly removing a
certain fraction of the labels from the training data.

Figure 3 shows the results of this measurement for the compared methods in
terms of the mAP metric. This figure is supplemented with confidence
intervals, implemented using bootstrapping (Park, 2011). Note that in the
Pinterest dataset the area of the confidence interval is almost imperceptible,
due to low variance in the results. Figure 4 shows the relative improvement of
MTL-reconstruct over the Image-only baseline approach for the four datasets.
On the horizontal axis, we show the “Label ratio”, which denotes the
percentage of labels that were retained in the training data. A value of 0.1
for the label ratio thus means that 90% of the labels were discarded for this
measurement.

Note that we used the MTL-reconstruct ${}_{\text{\\!\emph{uniform}}}$ variant
for these experiments (and not the best-performing MTL-reconstruct
${}_{\text{\\!\emph{loss}}}$) because only the uniform weighting scheme
described in Section 3 can be applied due to the missing labels in the
training data.

Looking at the results shown in Figure 3 and Figure 4, we find that MTL-
reconstruct is consistently better or at least performing equally well as the
Image-only method for all datasets and label ratio levels. For all datasets
except MovieLens, we furthermore observe the pattern that the gains of using
CF information in our approach are stronger when there are more missing
labels. In these cases, considering the collaborative information seems
particularly helpful and consistent with the intuition that the value of CF
information increases when other information is lacking. For the dense
MovieLens dataset, such a phenomenon is not observed, but the accuracy gains
are again rather small on average (around 2%) for this dataset in general.

Note that for the figure showing relative improvements (Figure 4), the gains
appear very small or non-existent in some cases due to the scale of the
y-axis. In fact, the improvements indicated by the smaller bars are in the
range of a few percent as shown in Table 5.

Figure 3. mAP values for Image-only and MTL-reconstruct
${}_{\text{\\!\emph{uniform}}}$ for different levels of missing labels. Figure
4. Relative mAP improvement of MTL-reconstruct ${}_{\text{\\!\emph{uniform}}}$
over Image-only for different levels of missing labels.

##  5\. Conclusions and Outlook

We showed how collaborative information, which is often available in practical
applications, can support image understanding processes. Our technical
proposal is built upon principles of multitask learning and leads to
consistent performance improvements over different baselines on four datasets.
Considering that nowadays improvements over state-of-the-art models are rather
marginal, our improvements are even more striking. Differently from earlier
approaches, our work focuses on the categorization of new images for which
also no collaborative information is yet available. Overall, our results
however show that relevant signals for image categorization may be contained
in the collaborative information that already exists for other images.

In terms of ongoing and future work, one important direction is to further
explore the relative importance of the CF model used in the MTL framework and
if alternative CF models help to further improve the overall performance. In
addition, we would like to apply our MTL-reconstruct
${}_{\text{\\!\emph{loss}}}$ variant to the missing labels case instead of
MTL-reconstruct ${}_{\text{\\!\emph{uniform}}}$ we used in this work. This can
be done by estimating the prediction uncertainty as done in recent works
(e.g., (Bibas et al., 2021a; Sastry and Oore, 2020; Gal and Ghahramani, 2016;
Bibas et al., 2019; Bibas and Feder, 2021)).

There are two interesting directions to expand the general contribution of our
proposed framework. First, while classification problems are arguably the most
prominent problems in content understanding, it would be very useful to assess
the usefulness of our proposed framework also for regression problems. Second,
it is important to investigate if the proposed approach to collaborative
content understanding is viable and effective also for other domains, in
particular for text documents or multimedia content.

Going beyond such additional technical analyses and explorations, we
anticipate future extensions to our work. Two natural extensions of key
importance are as follows. One intuitive extension is to enable improved
inference for items with existing collaborative information. Another worthy
research approach is to achieve mutual reinforcement between collaborative
filtering and content understanding by leveraging end-to-end learning. A
plausible approach is to simultaneously train the content understanding model
and the CF model, which allows to transfer valuable information between them.

## References

  * (1)
  * Baltrušaitis et al. (2018) Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018.  Multimodal machine learning: A survey and taxonomy.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 41, 2 (2018), 423–443.
  * Barkan et al. (2019) Oren Barkan, Noam Koenigstein, Eylon Yogev, and Ori Katz. 2019.  CB2CF: a neural multiview content-to-collaborative filtering model for completely cold item recommendations. In _Proceedings of the 13th ACM Conference on Recommender Systems_. 228–236.
  * Bibas and Feder (2021) Koby Bibas and Meir Feder. 2021.  Distribution Free Uncertainty for the Minimum Norm Solution of Over-parameterized Linear Regression.  _arXiv preprint arXiv:2102.07181_ (2021).
  * Bibas et al. (2021a) Koby Bibas, Meir Feder, and Tal Hassner. 2021a.  Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection.  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Bibas et al. (2019) Koby Bibas, Yaniv Fogel, and Meir Feder. 2019.  Deep pNML: Predictive normalized maximum likelihood for deep neural networks.  _arXiv preprint arXiv:1904.12286_ (2019).
  * Bibas et al. (2021b) Koby Bibas, Gili Weiss-Dicker, Dana Cohen, Noa Cahan, and Hayit Greenspan. 2021b.  Learning rotation invariant features for cryogenic electron microscopy image reconstruction. In _2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)_. 563–566.
  * Billsus and Pazzani (1998) Daniel Billsus and Michael J. Pazzani. 1998.  Learning Collaborative Information Filters. In _Proceedings of the Fifteenth International Conference on Machine Learning_. 46–54.
  * Campos et al. (2017) Víctor Campos, Brendan Jou, and Xavier Giró i Nieto. 2017\.  From pixels to sentiment: Fine-tuning CNNs for visual sentiment prediction.  _ArXiv_ abs/1604.03489 (2017).
  * Caruana (1997) Rich Caruana. 1997\.  Multitask Learning.  _Machine Learning_ 28, 1 (1997), 41–75.
  * Chapelle et al. (2009) Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. 2009\.  Semi-supervised learning.  _IEEE Transactions on Neural Networks_ 20, 3 (2009), 542–542.
  * Chen et al. (2018) Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018.  Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In _International Conference on Machine Learning_. PMLR, 794–803.
  * Choi et al. (2015) Chiho Choi, Ayan Sinha, Joon Hee Choi, Sujin Jang, and Karthik Ramani. 2015.  A collaborative filtering approach to real-time hand pose estimation. In _Proceedings of the IEEE International Conference on Computer Vision_. 2336–2344.
  * Dabov et al. (2007) Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2007.  Image denoising by sparse 3-D transform-domain collaborative filtering.  _IEEE Transactions on Image Processing_ 16, 8 (2007), 2080–2095.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  ImageNet: A Large-Scale Hierarchical Image Database. In _2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Eden et al. (2022) Sagi Eden, Amit Livne, Oren Sar Shalom, Bracha Shapira, and Dietmar Jannach. 2022.  Investigating the Value of Subtitles for Improved Movie Recommendations. In _Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization_. 99–109.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In _international Conference on Machine Learning_. 1050–1059.
  * Gou et al. (2021) Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021.  Knowledge distillation: A survey.  _International Journal of Computer Vision_ 129, 6 (2021), 1789–1819.
  * Guo et al. (2018) Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. 2018.  Dynamic task prioritization for multitask learning. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 270–287.
  * Harper and Konstan (2015) F Maxwell Harper and Joseph A Konstan. 2015.  The MovieLens datasets: History and context.  _ACM Transactions on Interactive Intelligent Systems (TiiS)_ 5, 4 (2015), 1–19.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 770–778.
  * He and McAuley (2016) Ruining He and Julian McAuley. 2016.  VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 30.
  * Hidasi et al. (2017) Balázs Hidasi, Alexandros Karatzoglou, Oren Sar Shalom, Sander Dieleman, Bracha Shapira, and Domonkos Tikk. 2017.  DLRS 2017: Second Workshop on Deep Learning for Recommender Systems. In _Proceedings of the Eleventh ACM Conference on Recommender Systems_. 370–371.
  * Hoffmann et al. (2011) Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011.  Knowledge-based weak supervision for information extraction of overlapping relations. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. 541–550.
  * Hossin and Sulaiman (2015) Mohammad Hossin and Md Nasir Sulaiman. 2015.  A review on evaluation metrics for data classification evaluations.  _International Journal of Data Mining & Knowledge Management Process_ 5, 2 (2015), 1.
  * Howard et al. (2017) Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017\.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ (2017).
  * Hu et al. (2008) Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008.  Collaborative filtering for implicit feedback datasets. In _2008 Eighth IEEE International Conference on Data Mining_. 263–272.
  * Kaufman et al. (2019) Dotan Kaufman, Koby Bibas, Eran Borenstein, Michael Chertok, and Tal Hassner. 2019.  Balancing specialization, generalization, and compression for detection and tracking.  _arXiv preprint arXiv:1909.11348_ (2019).
  * Kendall et al. (2018) Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 7482–7491.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  _arXiv preprint arXiv:2004.11362_ (2020).
  * Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009\.  Matrix factorization techniques for recommender systems.  _Computer_ 42, 8 (2009), 30–37.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  ImageNet classification with deep convolutional neural networks.  _Advances in Neural Information Processing Systems_ 25 (2012), 1097–1105.
  * Li and Tang (2016) Zechao Li and Jinhui Tang. 2016.  Weakly supervised deep matrix factorization for social image understanding.  _IEEE Transactions on Image Processing_ 26, 1 (2016), 276–288.
  * Li et al. (2018) Zechao Li, Jinhui Tang, and Tao Mei. 2018.  Deep collaborative embedding for social image understanding.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 41, 9 (2018), 2070–2083.
  * Liang et al. (2018) Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018.  Variational autoencoders for collaborative filtering. In _Proceedings of the 2018 World Wide Web Conference_. 689–698.
  * Liu et al. (2019) Jianran Liu, Shiping Wang, and Wenyuan Yang. 2019\.  Sparse autoencoder for social image understanding.  _Neurocomputing_ 369 (2019), 122–133.
  * Lü et al. (2012) Linyuan Lü, Matúš Medo, Chi Ho Yeung, Yi-Cheng Zhang, Zi-Ke Zhang, and Tao Zhou. 2012.  Recommender systems.  _Physics Reports_ 519, 1 (2012), 1–49.
  * Ma et al. (2010) Hao Ma, Jianke Zhu, Michael Rung-Tsong Lyu, and Irwin King. 2010.  Bridging the semantic gap between image contents and tags.  _IEEE Transactions on Multimedia_ 12, 5 (2010), 462–473.
  * McAuley et al. (2015) Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015.  Image-based recommendations on styles and substitutes. In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 43–52.
  * Ouyang et al. (2014) Yuanxin Ouyang, Wenqi Liu, Wenge Rong, and Zhang Xiong. 2014\.  Autoencoder-based collaborative filtering. In _International Conference on Neural Information Processing_. Springer, 284–291.
  * Park (2011) Laurence Park. 2011\.  Bootstrap confidence intervals for mean average precision. In _Proceedings of the 4th Applied Statistics Education and Research Collaboration (ASEARC)_.
  * Press and Wolf (2016) Ofir Press and Lior Wolf. 2016.  Using the output embedding to improve language models.  _arXiv preprint arXiv:1608.05859_ (2016).
  * Radosavovic et al. (2020) Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. 2020\.  Designing network design spaces. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10428–10436.
  * Ramachandram and Taylor (2017) Dhanesh Ramachandram and Graham W Taylor. 2017.  Deep multimodal learning: A survey on recent advances and trends.  _IEEE Signal Processing Magazine_ 34, 6 (2017), 96–108.
  * Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009.  BPR: Bayesian personalized ranking from implicit feedback. In _Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence_. 452–461.
  * Rendle et al. (2020) Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. 2020.  Neural Collaborative Filtering vs. Matrix Factorization Revisited. In _Proceedings of the Fourteenth ACM Conference on Recommender Systems_. 240–248.
  * Resheff et al. (2018) Yehezkel S Resheff, Yanai Elazar, Moni Shahar, and Oren Sar Shalom. 2018.  Privacy and fairness in recommender systems via adversarial training of user representations.  _arXiv preprint arXiv:1807.03521_ (2018).
  * Resnick et al. (1994) Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom, and John Riedl. 1994.  GroupLens: An Open Architecture for Collaborative Filtering of Netnews. In _Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work_. 175–186.
  * Revaud et al. (2019) Jerome Revaud, Jon Almazán, Rafael S Rezende, and Cesar Roberto de Souza. 2019.  Learning with average precision: Training image retrieval with a listwise loss. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5107–5116.
  * Ruby and Yendapalli (2020) U Ruby and V Yendapalli. 2020.  Binary cross entropy with deep learning technique for image classification.  _International Journal of Advanced Trends in Computer Science and Engineering_ 9, 10 (2020).
  * Sanchez et al. (2012) Faustino Sanchez, Marta Barrilero, Silvia Uribe, Federico Alvarez, Agustin Tena, and Jose Manuel Menendez. 2012.  Social and content hybrid image recommender system for mobile social networks.  _Mobile Networks and Applications_ 17, 6 (2012), 782–795.
  * Sang et al. (2012) Jitao Sang, Changsheng Xu, and Jing Liu. 2012.  User-aware image tag refinement via ternary semantic analysis.  _IEEE Transactions on Multimedia_ 14, 3 (2012), 883–895.
  * Sar Shalom et al. (2015) Oren Sar Shalom, Shlomo Berkovsky, Royi Ronen, Elad Ziklik, and Amir Amihood. 2015.  Data quality matters in recommender systems. In _Proceedings of the 9th ACM Conference on Recommender Systems_. 257–260.
  * Sastry and Oore (2020) Chandramouli Shama Sastry and Sageev Oore. 2020.  Detecting out-of-distribution examples with gram matrices. In _International Conference on Machine Learning_. PMLR, 8491–8501.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015\.  Facenet: A unified embedding for face recognition and clustering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 815–823.
  * Shalom et al. (2019) Oren Sar Shalom, Guy Uziel, and Amir Kantor. 2019\.  A generative model for review-based recommendations. In _Proceedings of the 13th ACM Conference on Recommender Systems_. 353–357.
  * Sigurbjörnsson and Van Zwol (2008) Börkur Sigurbjörnsson and Roelof Van Zwol. 2008.  Flickr tag recommendation based on collective knowledge. In _Proceedings of the 17th International Conference on World Wide Web_. 327–336.
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very Deep Convolutional Networks for Large-Scale Image Recognition. In _3rd International Conference on Learning Representations, ICLR 2015_.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.  Going deeper with convolutions. In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 1–9.
  * Too et al. (2019) Edna Chebet Too, Li Yujian, Sam Njuki, and Li Yingchun. 2019\.  A comparative study of fine-tuning deep learning models for plant disease identification.  _Comput. Electron. Agric._ 161 (2019), 272–279.
  * Weisstein (2001) Eric W Weisstein. 2001\.  Student’s t-Distribution.  _https://mathworld. wolfram. com/_ (2001).
  * Xu et al. (2019) Chaoyang Xu, Yuanfei Dai, Renjie Lin, and Shiping Wang. 2019\.  Stacked autoencoder based weak supervision for social image understanding.  _IEEE Access_ 7 (2019), 21777–21786.
  * Xu and Li (2007) Jun Xu and Hang Li. 2007\.  AdaRank: a boosting algorithm for information retrieval. In _Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_. 391–398.
  * Zha et al. (2015) Shengxin Zha, Florian Luisier, Walter Andrews, Nitish Srivastava, and Ruslan Salakhutdinov. 2015\.  Exploiting Image-trained CNN Architectures for Unconstrained Video Classification. In _26th British Machine Vision Conference (BMVC’15)_.
  * Zhang and Yang (2017) Yu Zhang and Qiang Yang. 2017.  An overview of multi-task learning.  _National Science Review_ 5, 1 (2017), 30–43.
  * Zhong et al. (2015) Changtao Zhong, Dmytro Karamshuk, and Nishanth Sastry. 2015\.  Predicting pinterest: Automating a distributed human computation. In _Proceedings of the 24th International Conference on World Wide Web_. 1417–1426.
  * Zhou et al. (2021) Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. 2021.  Domain generalization: A survey.  _arXiv preprint arXiv:2103.02503_ (2021).