# Do LLMs Understand User Preferences? Evaluating LLMs On User Rating
Prediction

Wang-Cheng Kang<EMAIL_ADDRESS>Google Research, Brain TeamUSA ,  Jianmo Ni
<EMAIL_ADDRESS>Google Research, Brain TeamUSA ,  Nikhil Mehta
<EMAIL_ADDRESS>Google Research, Brain TeamUSA ,  Maheswaran
Sathiamoorthy<EMAIL_ADDRESS>Google Research, Brain TeamUSA ,  Lichan Hong
<EMAIL_ADDRESS>Google Research, Brain TeamUSA ,  Ed Chi<EMAIL_ADDRESS>Google Research, Brain TeamUSA and  Derek Zhiyuan Cheng<EMAIL_ADDRESS>Google Research, Brain TeamUSA

###### Abstract.

Large Language Models (LLMs) have demonstrated exceptional capabilities in
generalizing to new tasks in a zero-shot or few-shot manner. However, the
extent to which LLMs can comprehend user preferences based on their previous
behavior remains an emerging and still unclear research question.
Traditionally, Collaborative Filtering (CF) has been the most effective method
for these tasks, predominantly relying on the extensive volume of rating data.
In contrast, LLMs typically demand considerably less data while maintaining an
exhaustive world knowledge about each item, such as movies or products. In
this paper, we conduct a thorough examination of both CF and LLMs within the
classic task of user rating prediction, which involves predicting a user’s
rating for a candidate item based on their past ratings. We investigate
various LLMs in different sizes, ranging from 250M to 540B parameters and
evaluate their performance in zero-shot, few-shot, and fine-tuning scenarios.
We conduct comprehensive analysis to compare between LLMs and strong CF
methods, and find that zero-shot LLMs lag behind traditional recommender
models that have the access to user interaction data, indicating the
importance of user interaction data. However, through fine-tuning, LLMs
achieve comparable or even better performance with only a small fraction of
the training data, demonstrating their potential through data efficiency.

*The two authors contributed equally to this work.

##  1\. Introduction

Large language models (LLMs) have shown an uncanny ability to handle a wide
variety of tasks such as text generation (Chowdhery et al., 2022; Devlin et
al., 2018; Raffel et al., 2020a; Brown et al., 2020), translation (Wang et
al., 2019; Yang et al., 2020), and summarization (Liu and Lapata, 2019). The
recent fine-tuning of LLMs on conversations and the use of techniques like
instruction fine-tuning (Chung et al., 2022) and reinforcement learning from
human feedback (RLHF) (Christiano et al., 2017) led to tremendous success to
bring highly human-like chatbots (e.g. ChatGPT (OpenAI, 2022b), and Bard
(Google, 2023)) to the average household. There are three key factors that
directly contribute to LLMs’ versatility and effectiveness:

  1. (1)

Knowledge from internet-scale real world information: LLMs are trained on
enormous datasets of text, providing access to a wealth of real-world
information. This information is converted to knowledge that can be used to
answer questions, creatives writing (e.g., poems, and articles), and translate
between languages.

  2. (2)

Incredible generalization ability through effective few-shot learning: within
certain context, LLMs are able to learn new tasks from an extremely small
number of examples (a.k.a., few-shot learning). Strong few-shot learning
capability gears up LLMs to be highly adaptable to new tasks.

  3. (3)

Strong reasoning capability: LLMs are able to reason through a chain-of-
thought process (Wei et al., 2022c, b), significantly improving their
performance across many tasks (Wang et al., 2023).

Recently, there has been some early exploratory work to make use of LLMs for
Search (Microsoft, 2023), Learning to Rank (Zou et al., 2021; Han et al.,
2020), and Recommendation Systems (Geng et al., 2022; Cui et al., 2022; Liu et
al., 2023). Specifically for recommendation systems, P5 (Geng et al., 2022)
fine-tunes T5-small (60M) and T5-base(220M) (Raffel et al., 2020b), unifying
both ranking, retrieval and other tasks like summary explanation into one
model. M6-Rec (Cui et al., 2022) tackles the CTR prediction task by finetuning
a LLM called M6 (300M) (Lin et al., 2021). Liu et al. (Liu et al., 2023)
looked into whether conversational agents like ChatGPT could serve as an off-
the-shelf recommender model with prompts as the interface and reported zero-
shot performance on rating prediction against baselines like MF and MLP.
However, there is a noticeable absence of a comprehensive study that
meticulously evaluates LLMs of varying sizes and contrasts them against
carefully optimized, strong baselines.

In this paper, we explore the use of off-the-shelf large language models
(LLMs) for recommendation systems. We study a variety of LLMs of various sizes
ranging from 250M to 540B parameters. We focus on the specific task of user
rating prediction, and evaluate the performance of these LLMs under three
different regimes: 1. zero-shot 2. few-shot, and 3. fine-tuning. We then
carefully compare them with the state-of-the-art recommendation models on two
widely adopted recommendation benchmark datasets.

Figure 1. Zero-shot performance for rating prediction of LLMs with different
model sizes, including Flan-T5 (base to XXL), GPT3 (Curie, Davinci) and Flan-
U-PaLM. We see a performance gain when increasing the model size. Among which,
models greater than 100B (Flan-U-PaLM 540B and text-davinci-003 175B)
outperform or on-par with the global average rating baseline on both RMSE and
AUC-ROC.

Our contributions are three-fold:

  * •

We empirically study the zero-shot and few-shot performance of off-the-shelf
LLMs with a wide spectrum of model sizes. We found that larger models (over
100B parameters) can provide reasonable recommendations under the cold-start
scenario, achieving comparable performance to decent heuristic-based
baselines.

  * •

We show that zero-shot LLMs still fall behind traditional recommender models
that utilize human interaction data. Zero-shot LLMs only achieve comparable
performance than two surprisingly trivial baselines that always predicts the
average item or user rating. Furthermore, they significantly underperform
traditional supervised recommendation models, indicating the importance of
user interaction data.

  * •

Through numerous experiments that fine-tune LLMs on human interaction data, we
demonstrate that fine-tuned LLMs can achieve comparable or even better
performance than traditional models with only a small fraction of the training
data, showing its promise in data efficiency.

##  2\. Related Work

###  2.1. Use of Natural Language in Recommender System

One of the earliest works that explored formulating the recommendation problem
as a natural language task is (Zhang et al., 2021). They used BERT (Devlin et
al., 2018) and GPT-2 (Radford et al., 2019) on the Movielens dataset (Harper
and Konstan, 2015) to show that such language models perform surprisingly
well, though not as good as well tuned baselines like GRU4Rec (Hidasi et al.,
2015).

P5 (Geng et al., 2022) fine-tunes a popular open-sourced T5 (Raffel et al.,
2020b) model, unifying both ranking, retrieval and other tasks like summary
explanation into one model. M6-Rec (Cui et al., 2022) is another related work,
but they tackle the CTR prediction task by finetuning a LLM called M6 (Lin et
al., 2021).

Two recent works explore the use of LLMs for zero-shot prediction. ChatRec
(Gao et al., 2023) handles zero-shot prediction as well as being interactive
and providing explanations. (Wang and Lim, 2023) takes a three-stage prompting
approach to generate next item recommendation in the Movielens dataset and
achieves competitive metrics, although not being able to beat strong
sequential recommender baselines such as SASRec (Kang and McAuley, 2018).

###  2.2. Large Language Models

Once people realized that scaling up sizes of data and model helps language
models, there has been a series of large language models proposed and built:
e.g. PaLM (Chowdhery et al., 2022), GPT-3 (Brown et al., 2020) and recent ones
such as OPT (Zhang et al., 2022) and LLaMA (Touvron et al., 2023). One of the
unique abilities of LLMs has been in their ability to reason about things,
which is further improved by techniques such as chain-of-thought prompting
(Wei et al., 2022c), self-consistency (Wang et al., 2022) and self-reflection
(Shinn et al., 2023).

Another major strong capability of LLMs is instruction following that models
can generalize to unseen tasks by following the given natural language
instructions. Researchers have found that techniques like instruction fine-
tuning (Chung et al., 2022) and RLHF (Christiano et al., 2017) can
significantly improve LLMs’ capability to perform tasks given natural language
descriptions that align with human’s preferences. As one of the tasks that can
be described in natural language, ‘recommendation’ has become a promising new
capability for LLMs. In this work, we focus on the models that have been fine-
tuned to improve their instruction following capability such as ChatGPT
(OpenAI, 2022b), GPT-3 (text-davinci-003 (OpenAI, 2022a)), Flan-U-PaLM and
Flan-T5 (Chung et al., 2022).

##  3\. Method

###  3.1. Problem formulation

We study the task of user rating prediction, formulated as: Given a user
$u\in\mathcal{U}$, a sequence of user $u$’s historical interactions
$E^{u}=\\{e^{u}_{1},e^{u}_{2},...,e^{u}_{n}\\}$ and an item $i\in\mathcal{I}$,
predict the rating that the user $u$ will give to the item $i$, where the user
historical interaction sequence $E^{u}$ is ordered by time ($e^{u}_{n}$ is the
most recent item that the user consumed), and each interaction $e^{u}_{k}$ is
represented by information about the item (e.g., ID, title, metadata, etc.)
that the user has consumed as well as the rating the user gave to the item.

###  3.2. Zero-shot and Few-shot LLMs for Rating Prediction

Figure 2. Zero and few shot LLM prompt for rating prediction.

We demonstrate the zero-shot and few-shot prompts used for the rating
prediction task on the MovieLens dataset in figure 2. As shown in the figure,
the input prompts depict several important features represented as text,
including user’s past rating history and candidate item features (title and
genre). Finally, to elicit a numeric rating from the model with the rating
scale, the input prompt specifies a numerical rating scale. The model response
is parsed to extract the rating output from the model. However, we discovered
that LLMs can be highly sensitive to the input prompts and do not always
follow the provided instruction. For instance, we found that certain LLMs may
offer additional reasoning or not provide a numerical rating at all. To
resolve this, we performed additional prompt engineering by adding additional
instructions such as "Give a single number as rating without explanation" and
"Do not give reasoning" to the input prompt.

###  3.3. Fine-tuning LLMs for Rating Prediction

In traditional recommender system research, it has been widely shown that
training models with human interaction data is effective and critical to
improve recommender’s capability of understanding user preference.

Here, we explore training the LLMs with human interaction and study how it
could improve the model performance. We focus on fine-tuning a family of LLMs,
namely Flan-T5, since they are publicly available and have competitive
performance on a wide range of benchmarks. The rating prediction task could be
formulated into one of two tasks: (1) multi-class classification; or (2)
regression, as shown in Figure 3(b).

##### Multi-class Classification

LLMs (either Decoder-only or Encoder-Decoder architecture) are essentially
pre-trained with a $K$-way classification task that predicts the token from a
fixed vocabulary with size $K$. As shown in Figure 3(b), there is a projection
layer that projects the outputs from the last layer to the vocabulary size
then the pre-training optimizes the cross-entropy loss for the token
classification. The output logits is computed as
$\mathrm{logits}_{\mathrm{dec}}=W_{\mathrm{proj}}h_{\mathrm{dec}}$, where
$W_{\mathrm{proj}}$ is the projection matrix of size $(d,|V|)$,
$h_{\mathrm{dec}}$ is the output from the decoder’s last transformer layer,
$d$ is the hidden dimension size of the decoder and $|V|$ is the vocabulary.

Following (Raffel et al., 2020a; Chung et al., 2022), we formulate the rating
regression task as a 5-way classification task, where we take the rating 1 to
5 as 5 classes. During training, we use the cross-entropy loss as other
classification tasks, as shown below:

(1)
$L_{\mathrm{cross\\_entropy}}=-\sum_{i=1}^{N}r^{i}\mathrm{log}(\mathrm{logits}_{\mathrm{dec}}^{i}),$

where $r^{i}$ is the ground-truth rating for the $i$-th item and $N$ is the
number of total training examples.

During inference, we compute the log-likelihood for the model output each
class and choose the class with the largest probability as the final
prediction.

##### Regression

. To enable LLMs for regression tasks, we set the shape of the projection
matrix $W_{\mathrm{proj}}$ to be $(d,1)$, so that it will only output a
1-digit logits. As shown in Equation 2, during training we apply a mean-
squared-error (MSE) loss based on the output logits and the ground-truth
rating.

(2)
$L_{\mathrm{regression}}=\frac{1}{|N|}\sum_{i=1}^{N}(\mathrm{logits}_{\mathrm{dec}}^{i}-r^{i})^{2}.$

(a) Decoder-Only model.

(b) Encoder-Decoder model.

Figure 3. Two types of LLMs for the rating prediction task.

##  4\. Experiments

We conduct extensive experiments to answer the following research questions:
RQ1: Do off-the-shelf LLMs perform well for zero-shot and few-shot
recommendations?
RQ2: How do LLMs compare with traditional recommenders in a fair setting
RQ3: How much does model size matter for LLMs when used for recommenders?
RQ4: Do LLMs converge faster than traditional recommender models?

###  4.1. Datasets and Evaluation Setup

####  4.1.1. Datasets

To evaluate the user rating prediction task, we use two widely adopted
benchmark datasets for evaluating model performance on recommendations. Both
datasets consist of user review ratings that range from 1 to 5.

  * •

MovieLens (Harper and Konstan, 2016): We use the version MovieLens-1M that
includes 1 million user ratings for movies.

  * •

Amazon-Books (Ni et al., 2019): We use the “Books” category of the Amazon
Review Dataset with users’ ratings on items. We use the 5-core version that
filters out users and items with less than 5 interactions.

####  4.1.2. Training / Test Split

To create the training and test sets, we follow the single-time-point split
(Sun, 2022). We first filter out the ratings associated with items that don’t
have metadata, then sort all user ratings in chronological order. Finally, we
take the first 90% ratings as the training set and the remaining as the test
set. Each training example is a tuple of
$<user\\_id,item\\_id,item\\_metadata,rating>$, where the label is a 5 Likert-
scale rating. The input features are $user\\_id$, $item\\_id$, and a list of
item_metadata features. The statistics of the datasets are shown in the Table
1. Due to the high computation cost of zero-shot and few-shot experiments
based on LLMs, we randomly sample from the test set of each dataset into 2,000
tuples as a smaller test set. For all our experiments, we report results on
the sampled test set. And we truncate the user sequence to the most recent 10
interactions during training and evaluation.

Table 1. Statistics of the dataset Datasets | #Users | #Items | # of Training Examples | # of Test Examples | Features
---|---|---|---|---|---
Movielens-1M | 6,040 | 3,689 | 882,002 | 2,000 (75,880) | Title, Genre
Amazon-Books | 1,850,187 | 483,579 | 17,566,711 | 2,000 (2,324,503) | Title, Brand

####  4.1.3. Evaluation Metrcis

We use the widely adopted metrics RMSE (Root Mean Squared Error) and MAE (Mean
Average Error) to measure model performance on rating prediction. Moreover, we
use ROC-AUC to evaluate the model’s performance on ranking, where ratings
greater than or equal to 4 are considered as positives and the rest as
negatives. In this case, AUC measures whether the model ranks the positives
higher than negatives.

###  4.2. Baselines and LLMs

####  4.2.1. Baselines

  * •

Traditional Recommeder: We consider several traditional recommendation models
as strong baselines, including 1. Matrix Factorization (MF) (Rendle et al.,
2012), and 2. Multi-layer Perceptrons (MLP) (He et al., 2017). For MF and MLP,
only user ID and item ID are used as input features.

  * •

Attribute and Rating-aware Sequential Rating Predictor: In our experiments, we
supply the LLM with historical item metadata, such as titles and categories,
along with historical ratings. However, to the best of our knowledge, there is
no existing method designed for this setting111The most related works are
SASRec (Kang and McAuley, 2018) and CARCA (Rashed et al., 2022), however they
are designed for next item prediction instead of rating prediction, and thus
not directly applicable to our case.. To ensure a fair comparison, we
construct a Transformer-MLP model to efficiently process the same input
information provided to the LLM.

There are three key design choices: (i) feature processing: We treat all
features as sparse features, and learn their embeddings end-to-end. For
example, we use one-hot encoding for genres, and create an embedding table,
where the i-th row is genre i’s embedding. Similarly, we obtain bag-of-words
encodings via applying a
tokenizer222https://www.tensorflow.org/text/api_docs/python/text/WhitespaceTokenizer
on titles, and then look up the corresponding embedding. (ii) user modeling:
for each user behavior, we use Add or Concat to aggregate all embeddings (e.g.
item ID, title, genres/category, rating) into one, and then adopt bi-
directional self-attention (Vaswani et al., 2017) layers with learned position
embeddings to model users’ past behaviors. Similar to SASRec, we use the most
recent behavior’s output embedding as the user summary; (iii) Fuse user and
candidate for rating prediction: we apply a MLP on top of the user embedding
along with other candidate item features to generate the final rating
prediction, and optimize for minimizing MSE.

To properly tune the baseline models, we defined a hyper-parameter search
space (e.g. for embedding dimension, learning rate, network size, Add or
Concat aggregation, etc.), and perform more than 100 search trials using
Vizier (Golovin et al., 2017), a black-box hyper-parameter optimization tool.

  * •

Heuristics: We also include three heuristic-based baselines: (1) global
average rating: (2) candidate item average rating, and (3) user past average
rating, meaning the model’s prediction is depending on (1) the average rating
among all user-item ratings, (2) the average rating from the candidate item or
(3) the user’s average rating in the past.

####  4.2.2. LLMs for Zero-shot and Few-shot Learning

: We used the LLMs listed below for zero-shot and few-shot learning. We use a
temperature of 0.1 for all LLMs, as the LLM’s output in our case is simply a
rating prediction. We use GPT-3 models from OpenAI (OpenAI, 2023): (i) text-
davinci-003 (175B): The most capable GPT-3 model with Reinforcement Learning
from Human Feedback (RLHF) (Stiennon et al., 2020); (ii) ChatGPT: the default
model is gpt-3.5-turbo, fine-tuned on both human-written demonstrations and
RLHF, and further optimized for conversation. Flan-U-PaLM (540B) is the
largest and strongest model in (Chung et al., 2022), it applies both FLAN
instruction tuning (Wei et al., 2022a) and UL2 training objective (Chung et
al., 2022) on PaLM (Chowdhery et al., 2022).

Table 2. User rating prediction results. The best performing method is boldfaced in each column, and underlined in each group. Model | MovieLens | Amazon-Books
---|---|---
|  RMSE$\downarrow$ |  MAE$\downarrow$ |  AUC$\uparrow$ |  RMSE$\downarrow$ |  MAE$\downarrow$ |  AUC$\uparrow$
_Zero-Shot LLMs_ |  |  |  |  |  |
Flan-U-PALM | 1.0677 | 0.7740 | 0.7084 | 0.9565 | 0.5569 | 0.7676
ChatGPT | 1.0081 | 0.8193 | 0.6794 | 1.0081 | 0.8093 | 0.6778
text-davinci-003 | 1.0460 | 0.7850 | 0.6951 | 0.8890 | 0.5442 | 0.7416
_Few-Shot LLMs_ |  |  |  |  |  |
Flan-U-PALM | 1.0721 | 0.7605 | 0.7094 | 1.0712 | 0.5855 | 0.7439
ChatGPT | 1.0862 | 0.8203 | 0.6930 | 1.0618 | 0.7760 | 0.7470
text-davinci-003 | 1.0867 | 0.8119 | 0.6963 | 1.0716 | 0.7753 | 0.7739
_Simple Dataset Statistics_ |  |  |  |  |  |
Global Avg. Rating | 1.1564 | 0.9758 | 0.5 | 0.9482 | 0.7609 | 0.5
Candidate Item Avg. Ratings | 0.9749 | 0.7778 | 0.7395 | 0.9342 | 0.7078 | 0.6041
User Past Avg. Ratings | 1.0196 | 0.7959 | 0.7266 | 0.8527 | 0.5502 | 0.8047
_Supervised Recommendation Methods_ |  |  |  |  |  |
MF | 0.9552 | 0.7436 | 0.7734 | 1.7960 | 1.1070 | 0.7638
MLP | 0.9689 | 0.7452 | 0.7393 | 0.8607 | 0.6384 | 0.6932
Transformer+MLP | 0.8848 | 0.7036 | 0.7979 | 0.8143 | 0.5541 | 0.8042
_Fine-tuned LLMs_ |  |  |  |  |  |
Flan-T5-Base (classification) | 1.0110 | 0.6805 | 0.7590 | 0.9856 | 0.4685 | 0.6292
Flan-T5-Base (regression) | 0.9187 | 0.7092 | 0.7949 | 0.8413 | 0.5317 | 0.8182
Flan-T5-XXL (regression) | 0.8979 | 0.6986 | 0.8042 | 0.8301 | 0.5122 | 0.8312

(a) MovieLens

(b) Amazon Books

Figure 4. Data efficiency: convergence curve.

####  4.2.3. LLMs for Fine-tuning

For fine-tuning methods, we use Flan-T5-Base (250M) and Flan-T5-XXL (11B)
models in the experiments. We set the learning rate to 5e-5, batch size to 64,
drop out rate to 0.1 and train 50k steps on all datasets.

###  4.3. Zero-Shot and Few-shot LLMs (RQ1)

As shown in Table 2, we conduct experiments on several off-the-shelf LLMs in
the zero-shot setting. We observed that LLMs seem to understand the task from
the prompt description, and predict reasonable ratings. LLMs outperform global
average rating in most cases, and perform comparable with item or user average
ratings. For example, text-davinci-003 performs slightly worse than candidate
item average ratings on Movielens but outperforms on Amazon-Books. For few-
shot experiments, we provide 3 examples in the prompt (3-shot). Compared
against zero-shot, we found that the AUC for few-shot LLMs are improved, while
there is no clear pattern in RMSE and MAE.

Furthermore, we found that both zero-shot and few-shot LLMs under-perform
traditional recommendation models trained with interaction data. As shown in
Table 2, GPT-3 and Flan-U-PaLM models achieve significantly lower performance
compared to supervised models. The inferior performance could be due to the
lack of user-item interaction data in LLMs’ pre-training, and hence they don’t
have knowledge about human preference for different recommendation tasks.
Moreover, recommendation tasks are highly dataset-dependent: (e.g.) the same
movie can have different average ratings on different platform. Hence, without
knowing the dataset-specific statistics, it’s impossible for LLM to provide a
universal prediction that is suitable for every datasets.

###  4.4. LLMs vs. Traditional Recommender Models (RQ2)

Fine-tuning LLMs is an effective way to feed dataset statistics into LLMs, and
we found the performance of fine-tune LLMs are much better than zero/few-shot
LLMs. Also, when fine-tuning the Flan-T5-base model with the classification
loss, the performance is much worse than fine-tuning with the regression loss
on all three metrics. This indicates the importance of choosing the right
optimizing objective for fine-tuning LLMs.

Comparing against the strongest baseline Transformer-MLP, we found fine-tuned
Flan-T5-XXL has better MAE and AUC, implying fine-tune LLMs may be more
suitable for ranking tasks.

###  4.5. Effect of Model Size (RQ3)

For all the LLMs we studied of different model sizes vary from 250M and 500B
parameters, we were able to use zero-shot or few-shot prompts to let them
output a rating prediction between 1 to 5. This shows the effectiveness of
instruction tuning that enables these LLMs (Flan-T5, Flan-U-PaLM, GPT-3) to
follow the prompt. We further found that only LLMs with size greater than 100B
perform reasonably well on rating prediction in the zero-shot setting, as
shown in Figure 1. For fine-tuning experiments, we also found that Flan-T5-XXL
outperforms Flan-T5-Base on both datasets, as shown in the last two rows of
Table 2.

###  4.6. Data Efficiency of LLMs (RQ4)

As LLMs have learned vast amounts of world knowledge during pre-training,
while traditional recommender models are trained from scratch, we compare
their convergence curves in Figure 4 to examine whether LLMs have better data
efficiency. We can see that for RMSE, both methods could converge to
reasonable performance with a small fraction of data. This is probably because
that even average rating of all items has a relatively low RMSE, and thus as
long as a model learns to predict a rating near the average rating, it could
achieve reasonable performance. For AUC the trend is more clear, as simply
predicting average rating results in an AUC of 0.5. We found that a small
fraction of data is required for LLM to achieve good performance, while
Transformer+MLP needs much more training data (at least 1 epoch) for
convergence.

##  5\. Conclusion

In this paper, we evaluate the effectiveness of large language models as a
recommendation system for user rating prediction in three settings: 1. zero-
shot; 2. few-shot; and 3. fine-tuning. Compared to traditional recommender
methods, our results revealed that LLMs in zero-shot and few-shot LLMs fall
behind fully supervised methods, implying the importance of incorporating the
target dataset distribution into LLMs. On the other hand, fine-tuned LLMs can
largely close the gap with carefully designed baselines in key metrics. LLM-
based recommenders have several benefits: (i) better data efficiency; (ii)
simplicity for feature processing and modeling: we only need to convert
information into a prompt without manually designing feature processing
strategies, embedding methods, and network architectures to handle various
kind of information; (iii) potential for unlock conversational recommendation
capabilities. Our work sheds light on the current status of LLM-based
recommender systems, and in the future we will further look into improving the
performance via methods like prompt tuning, and explore novel recommendation
applications enabled by LLMs.

## References

  * (1)
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022\.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ (2022).
  * Christiano et al. (2017) Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep Reinforcement Learning from Human Preferences.  _ArXiv_ abs/1706.03741 (2017).
  * Chung et al. (2022) Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022\.  Scaling Instruction-Finetuned Language Models.  _ArXiv_ abs/2210.11416 (2022).
  * Cui et al. (2022) Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022.  M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems.  _arXiv preprint arXiv:2205.08084_ (2022).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Gao et al. (2023) Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. 2023\.  Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System.  _arXiv preprint arXiv:2303.14524_ (2023).
  * Geng et al. (2022) Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2022.  Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5).  _arXiv preprint arXiv:2203.13366_ (2022).
  * Golovin et al. (2017) Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. 2017.  Google Vizier: A Service for Black-Box Optimization. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017_. ACM, 1487–1495.  https://doi.org/10.1145/3097983.3098043
  * Google (2023) Google. 2023.  Bard: A Large Language Model from Google AI.  https://bard.google.com/
  * Han et al. (2020) Shuguang Han, Xuanhui Wang, Mike Bendersky, and Marc Najork. 2020.  Learning-to-Rank with BERT in TF-Ranking.  arXiv:2004.08476 [cs.IR]
  * Harper and Konstan (2015) F Maxwell Harper and Joseph A Konstan. 2015.  The movielens datasets: History and context.  _Acm transactions on interactive intelligent systems (tiis)_ 5, 4 (2015), 1–19.
  * Harper and Konstan (2016) F. Maxwell Harper and Joseph A. Konstan. 2016.  The MovieLens Datasets: History and Context.  _ACM Trans. Interact. Intell. Syst._ 5 (2016), 19:1–19:19.
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017\.  Neural Collaborative Filtering. In _Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017_ , Rick Barrett, Rick Cummings, Eugene Agichtein, and Evgeniy Gabrilovich (Eds.). ACM, 173–182.  https://doi.org/10.1145/3038912.3052569
  * Hidasi et al. (2015) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2015.  Session-based recommendations with recurrent neural networks.  _arXiv preprint arXiv:1511.06939_ (2015).
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _2018 IEEE international conference on data mining (ICDM)_. IEEE, 197–206.
  * Lin et al. (2021) Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, et al. 2021\.  M6: A chinese multimodal pretrainer.  _arXiv preprint arXiv:2103.00823_ (2021).
  * Liu et al. (2023) Junling Liu, Chao Liu, Renjie Lv, Kang Zhou, and Yan Zhang. 2023.  Is ChatGPT a Good Recommender? A Preliminary Study.  arXiv:2304.10149 [cs.IR]
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text Summarization with Pretrained Encoders.  arXiv:1908.08345 [cs.CL]
  * Microsoft (2023) Microsoft. 2023\.  Reinventing search with a new AI-powered Bing and Edge, your copilot for the web.  https://news.microsoft.com/the-new-Bing/
  * Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019.  Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects. In _Conference on Empirical Methods in Natural Language Processing_.
  * OpenAI (2022a) OpenAI. 2022a.  Aligning language models to follow instructions.  https://openai.com/research/instruction-following.
  * OpenAI (2022b) OpenAI. 2022b.  Introducing ChatGPT.  https://openai.com/blog/chatgpt.
  * OpenAI (2023) OpenAI. 2023.  GPT Models Documentation.  https://platform.openai.com/docs/models/overview
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Raffel et al. (2020a) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020a.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _J. Mach. Learn. Res._ 21, 1, Article 140 (jan 2020), 67 pages.
  * Raffel et al. (2020b) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020b.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ 21, 1 (2020), 5485–5551.
  * Rashed et al. (2022) Ahmed Rashed, Shereen Elsayed, and Lars Schmidt-Thieme. 2022\.  CARCA: Context and Attribute-Aware Next-Item Recommendation via Cross-Attention.  _arXiv preprint arXiv:2204.06519_ (2022).
  * Rendle et al. (2012) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012.  BPR: Bayesian personalized ranking from implicit feedback.  _arXiv preprint arXiv:1205.2618_ (2012).
  * Shinn et al. (2023) Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023.  Reflexion: an autonomous agent with dynamic memory and self-reflection.  _arXiv preprint arXiv:2303.11366_ (2023).
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020.  Learning to summarize from human feedback.  _CoRR_ abs/2009.01325 (2020).  arXiv:2009.01325 https://arxiv.org/abs/2009.01325
  * Sun (2022) Aixin Sun. 2022\.  Take a Fresh Look at Recommender Systems from an Evaluation Standpoint.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023\.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang and Lim (2023) Lei Wang and Ee-Peng Lim. 2023.  Zero-Shot Next-Item Recommendation using Large Pretrained Language Models.  _arXiv preprint arXiv:2304.03153_ (2023).
  * Wang et al. (2019) Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019.  Learning Deep Transformer Models for Machine Translation.  arXiv:1906.01787 [cs.CL]
  * Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023\.  Self-Consistency Improves Chain of Thought Reasoning in Language Models.  arXiv:2203.11171 [cs.CL]
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022\.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_ (2022).
  * Wei et al. (2022a) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a.  Finetuned Language Models are Zero-Shot Learners. In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.  https://openreview.net/forum?id=gEZrGCozdqR
  * Wei et al. (2022b) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022b.  Emergent Abilities of Large Language Models.  arXiv:2206.07682 [cs.CL]
  * Wei et al. (2022c) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022c.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ (2022).
  * Yang et al. (2020) Jiacheng Yang, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Weinan Zhang, Yong Yu, and Lei Li. 2020.  Towards making the most of bert in neural machine translation. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 34. 9378–9385.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022\.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ (2022).
  * Zhang et al. (2021) Yuhui Zhang, Hao Ding, Zeren Shui, Yifei Ma, James Zou, Anoop Deoras, and Hao Wang. 2021.  Language models as recommender systems: Evaluations and limitations.  (2021).
  * Zou et al. (2021) Lixin Zou, Shengqiang Zhang, Hengyi Cai, Dehong Ma, Suqi Cheng, Shuaiqiang Wang, Daiting Shi, Zhicong Cheng, and Dawei Yin. 2021.  Pre-Trained Language Model Based Ranking in Baidu Search. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_ (Virtual Event, Singapore) _(KDD ’21)_. Association for Computing Machinery, New York, NY, USA, 4014–4022.  https://doi.org/10.1145/3447548.3467147


# Semi-Supervised Object Detection with Sparsely Annotated Dataset

Jihun Yoon
hutom
Republic of Korea
<EMAIL_ADDRESS>Seungbum Hong
hutom
Republic of Korea
<EMAIL_ADDRESS>Sanha Jeong
hutom
Republic of Korea
<EMAIL_ADDRESS>Min-Kook Choi
hutom
Republic of Korea
<EMAIL_ADDRESS>
###### Abstract

In training object detector based on convolutional neural networks, selection
of effective positive examples for training is an important factor. However,
when training an anchor-based detectors with sparse annotations on an image,
effort to find effective positive examples can hinder training performance.
When using the anchor-based training for the ground truth bounding box to
collect positive examples under given IoU, it is often possible to include
objects from other classes in the current training class, or objects that are
needed to be trained can only be sampled as negative examples. We used two
approaches to solve this problem: 1) the use of an anchorless object detector
and 2) a semi-supervised learning-based object detection using a single object
tracker. The proposed technique performs single object tracking by using the
sparsely annotated bounding box as an anchor in the temporal domain for
successive frames. From the tracking results, dense annotations for training
images were generated in an automated manner and used for training the object
detector. We applied the proposed single object tracking-based semi-supervised
learning to the Epic-Kitchens dataset. As a result, we were able to achieve
runner-up performance in the Unseen section while achieving the first place in
the Seen section of the Epic-Kitchens 2020 object detection challenge under
IoU $>0.5$ evaluation.

##  1 Introduction

Thanks to the rapid development of CNN (Convolutional Neural Networks), the
performance of object recognition networks using CNN has also been improved
dramatically [16]. As the performance of the object detection network has been
improved, the dataset for evaluating it was also started from a dataset with
low complexity such as PASCAL VOC [8] and developed to have a high complexity
such as MS-COCO [15]. Among the object detection datasets, the relatively
recently released Epic-Kitchens dataset has the following characteristics
different from other object detection datasets [7].

  * •

Images for training detector are collected from the original video, and
corresponding frame sequences are provided.

  * •

In a training image, only some of the trainable objects are sparsely
annotated.

  * •

The difference in the amount of annotations between the few and many shot
classes is large, depending on the distribution of the appearance of the
objects in the training dataset.

Figure 1: An example of anchor-based detector training on a sparsely annotated
dataset. The solid green line represents the label information in each
training image, and the red dotted line is an example of positive examples.
Light blue dashed lines indicate objects that are included in the label in
other training images (top), but are not labeled in the given image to train
(bottom). As such, in the Epic-Kitchens object detection dataset, it is an
object to learn when training an anchor-based detector, but training
performance is impaired because label information is missing.

As described above, the annotation of Epic-Kitchens for object detection is
provided in a different way from the existing dataset, and has a
characteristic that it is difficult to apply the method of training the
existing object detection model as it is. Typically, in the case of detectors
that train positive examples based on anchors [21, 17] or detectors that train
the objectness of a candidate object with the structure of an RPN [23, 6],
batch sampling is performed considering the Intersection on Union (IoU) with
the ground truth bounding box for effective training. However, if an anchor-
based hard example mining is performed on a sparsly annotated training image,
the efficiency of training is hindered by the distribution of objects near the
ground truth bounding box. Figure 1 shows the negative effects of trying to
select a good positive example when training anchor-based object detectors
with sparse annotation images.

We have trained the object detector through two approaches to solve this
problem. First, a detector using different parameterization for estimating the
bounding box was used instead of the anchor-based detection model. We were
able to minimize the effect of sparsely annotated training images affected by
anchor-based sampling by utilizing the Fully Convolutional One-Stage Object
Detection (FCOS) network [24]. The second is to utilize the features of the
Epic-Kitchens dataset, and the object tracking technique is used to semi-
supervise all detectable objects in each training image. To this end, the
bounding box label existing in a specific frame is set as an initial bounding
box on the time domain and used as an input of a single object tracker.
Subsequently, the predictive output exceeding the threshold from the tracker
was assumed to be a pesudo annotation, and labels for all learnable objects
were provided in the image for training. Figure 2 schematically shows the
proposed goal of semi-supervised learning.

With the proposed approach, we were able to train an object detection network
effectively with the Epic-Kitchens object detection dataset. Subsequently,
joint NMS-based ensemble [11] was performed for FCOS models with trained
inhomogeneous backbones. As a result, we were able to achieve first place in
the Seen evaluation set and runnerup in the Unseen evaluation set under the
IoU $>0.5$ of the Epic-Ktichens 2020 object detection challenge.

Figure 2: The final goal of the proposed supervised learning. $f$ denotes an
action clip composed of $N$ frames, and $o$ denotes a total of $M$ objects
present in the action clip. We performed semi-supervised learning through
bidirectional tracking to obtain dense labels for all learnable objects
present in the action clip.

##  2 Related Work

Object detection. CNN-based object detection models are largely divided into
one-stage or two-stage models. In the one-stage model, the process of
predicting the class and position of an object is performed in one structure,
and examples include YOLO [21, 22], SSD [17, 9], and RetinaNet [14].
Additional structures such as Feature Pyramid Networks (FPN) [21] are often
used to efficiently process the output features obtained from the backbone in
the head structure. In general, it is known that the regression accuracy is
lower than the two-stage model because classification and regression are
performed in one structure. In the case of the two-stage model, the prior
knowledge of the location of the object is estimated from the RPN (Region
Proposal Network), which is a subnetwork in the detector [23]. RPN determines
objectness by class-agnostic subnet, and performs class-aware detection
through the subsequent head structure. Faster R-CNN [23], R-FCN [6], Cascade
R-CNN [2], Cascade RPN [25], etc. are representative of various head
structures, and are known to have relatively high regression accuracy. Models
such as RefineDet [29] that combine the philosophy of one-stage and two-stage
models have also been proposed, and detectors that utilize other
parameterizations for bounding box regression rather than structural
advantages, such as FCOS [24], have also been proposed.

Semi-supervised learning for object detection. Object detection using semi-
supervised learning is used in situations where it is difficult to manually
acquire a sufficient number of annotations to learn, or when pseudo labels are
to be obtained from a relatively large number of unlabeled data. [18] proposed
an iterative framework for evaluating and retraining pseudo labels using pre-
trained object detectors and robust trackers to obtain good pseudo labels in
successive frames. In [19], it was possible to achieve improved detection
performance in Open Image Dataset V4 by utilizing part-aware sampling and RoI
proposals to obtain good pseudo labels for sparsely annotated large-scale
datasets. In [5], in order to efficiently use unlabeled data from the MS-COCO
dataset, co-occurrent matrix analysis was used to generate good pseudo labels
by using prior information of the labeled dataset. The proposed single object
tracker-based semi-supervised learning is similar to [18] in that it uses a
tracker, but has a difference in obtaining dense annotation information for a
specific image by using the existing lean annotation information. At the same
time, since the object detector is not used for the initial input for
tracking, the training is not applied as an iterative training scenario.

Single object visual tracking. We used a single object tracking network to
generate pseudo labels for sparsely annotated datasets. In single object
tracking [1, 13, 27, 3, 12], the Siamese network-based visual tracker shows
balanced accuracy and speed across various datasets. The Siamese network-based
tracker basically trains with the similarity of the CNN feature for the target
image and the input image for tracking. We used SiamMask [27] as a single
object tracker, which uses box and mask information together with similarity
of features to the tracking target.

##  3 Fully Convolutional One-Stage Object Detection (FCOS)

We used the FCOS model [24] to exclude the computational process for selecting
a good positive example with an anchor from detector training. FCOS defined
the parameters for regression of the bounding box differently, and presented
an anchor-free detector. The loss function of FCOS is defined as follows.

$\displaystyle\begin{split}L(a_{p},m_{p})=\frac{1}{N}\sum_{\\{u,v\\}\in
p}L_{cls}(a_{p},c_{p})+\\\ \frac{1}{N}\sum_{\\{u,v\\}\in
p}[c_{p}>0]L_{reg}(m_{p},\hat{m}_{p}),\end{split}$ (1)

where $p$ denotes a position $(u,v)$ on the feature map, and $a_{p}$ denotes a
prediction vector for class estimation. $c_{p}$ denotes the class label for
the input example, and $m_{p}$ denotes the specific spatial location of the
feature map and the distance from the ground truth bounding box
$m_{p}=(l,t,r,b)$. Given the label $B=(x_{0},y_{0},x_{1},y_{1},c)$ for the
bounding box, FCOS parameterizes to find the bounding box by $l=u-x_{0}$,
$t=v-y_{0}$, $r=u-x_{1}$, $b=v-y_{1}$. We used pretrained ResNet [10], ResNeXt
[28], HRNet [26] as the backbone network for FCOS to perform inhomogeneous
ensemble.

##  4 Semi-Supervised Learning with Single Object Tracker

The Epic-Kitchens dataset simultaneously provides a bounding box label for a
particular object and a sequence frame for the action clip in which the object
appears. The bounding box for the object is not densely given every frame, but
rather sparsely in the action sequence. We used a single object tracker to
achieve the goal in Figure 2 with an automated procedure. Among various single
object trackers, SiamMask [27] was used, which shows a balanced performance
for tracking accuracy and speed. We performed a bidirectional tracking using
the SiamMask model trained from the DAVIS dataset, using each bounding box as
the initial value for a single object. The details of forward tracking with
SiamMask for one action clip input are described in Algorithm 1. Algorithm 1
is used in the same way for backward tracking to complete bidirectional
tracking. Figure 3 shows an example of the start and end of tracking according
to Algorithm 1 on a single object, and Figure 4 shows training images with
pseudo labels generated after tracking an object.

Input: Action clip ($A$), pretrained tracking model ($T$), a set of bbox for
initial input ($BB$), threshold of tracking score ($\rho_{1}$), threshold of
IoU between two pair of tracked bbox ($\rho_{2}$)

Output: $BB$ in $Q$ from $T$

Initialize an empty queue $Q$

while  _bbox $b_{c,i}$ with class $c$ at $i$-th frame available from $BB$_ do

Get a list of frames $FF$ in forward from $i$-th frame in $A$;

Initialize $T$ with $b_{i}$ from $A$;

Initialize a variable $pre_{s}$ with a size of $b_{c,i}$ to store a size of
object from $T$ at previous frame;

while  _each frame in FF_ do

Get a bbox $b_{c,k}$ from $T$ at $k$-th frame;

$crnt_{s}$ $:=$ a size of $b_{c,k}$;

if  _$IoU(prev_{s},crnt_{s})\geq\rho_{1}$ _ then

$prev_{s}:=crnt_{s}$;

Add $b_{c,k}$ to $Q$;

else

break;

end if

end while

end while

Algorithm 1 Forward tracking Figure 3: Example of the result of Algorithm 1.
Frames marked with blue boxes are frames that have been tracked with the same
object since tracking started, and a frame marked with a red box is a frame
whose tracking is terminated due to the termination condition of algorithm 1.
Figure 4: Changes in training images after tracking. An example of the final
annotations to be used for semi-supervised learning is shown on the training
images indicated by the red dotted line.

##  5 Epic-Kitchens Object Detection Results

Training details. We used Faster R-CNN [23] and Cascade R-CNN [2] as anchor-
based detectors and FCOS [24] as an anchorless detector to compare performance
in the Epic-Kitchens object detection dataset [7]. As the backbone CNN for
training the detector, ResNet-50, ResNet-101, ResNeXt-101, and HRNet-V2p-W32
pretrained with ImageNet were used, and training details for each combination
of backbone and head structure are shown in Table 1. All experiments were
conducted using the MMDetection library [4].

Table 1: Training detail of each detector. ‘lr’ represents the learning rate.
Learning schedule is indicated as (scheduler, drop rate) [drop epoch1:drop
epoch2:max epoch].

Detector | Backbone | Optimizer (lr) | Learning schedule | Warmup(iter, ratio)
---|---|---|---|---
Faster R-CNN | ResNet-101 | SGD (0.02) | (step, 0.1)[8:11:24] | linear(500, $1/3$)
Cascade R-CNN | HRNet-V2P | SGD (0.02) | (step, 0.1)[16:19:30] | linear(500, $1/3$)
FCOS | HRNet-V2P | SGD (0.01) | (step, 0.1)[8:11:20] | constant(500, $1/3$)
FCOS | ResNet-50 | SGD (0.01) | (step, 0.1)[8:11:16] | constant(500, $1/3$)
FCOS | ResNet-101 | SGD (0.01) | (step, 0.1)[16:20:22] | constant(500, $1/3$)
FCOS | ResNeXt-101 | SGD (0.01) | (step, 0.1)[16:22:39] | constant(500, $1/3$)

Anchor-based vs. anchorless detector Table 2 shows the training performance in
a single model of an anchor-based detector and an anchorless detector.
According to Table 2, it can be seen that in the basic performance of
training, the performance of the detector without an anchor is excellent and
shows stable learning results. Figure 5 shows the loss change during training
of an anchor-based detector and an anchorless detector. The detector without
anchor shows a relatively stable loss curve. At the same time, Table 2 shows
the performance change of the FCOS model according to different backbones. For
a single model, it was confirmed that the FCOS model utilizing the ResNet-101
backbone achieved the best generalization performance in the Seen set, and the
HRNet backbone model performed the best in the Unseen set.

Table 2: Performance comparison of anchor-based and anchorless detectors. If
the model name has a ‘$+$’, it is the result of evaluation using tracker-based
semi-supervised learning. The highest performance in a single model and the
highest performance in an entire model are shown in bold.

Detector | backbone | Seen | Unseen
---|---|---|---
- | - |  $>$ 0.05 |  $>$ 0.5 |  $>$ 0.75 |  $>$ 0.05 |  $>$ 0.5 |  $>$0.75
Faster R-CNN | ResNet-101 | 37.54 | 28.64 | 6.92 | 32.83 | 23.16 | 5.55
Cascade R-CNN | HRNet-V2P | 30.44 | 24.17 | 8.73 | 23.87 | 18.05 | 6.81
FCOS | HRNet-V2P | 48.44 | 34.87 | 11.02 | 43.88 | 30.68 | 9.27
FCOS | ResNet-50 | 46.96 | 34.51 | 10.09 | 42.46 | 29.49 | 7.48
FCOS | ResNet-101 | 49.77 | 35.8 | 10.15 | 43.39 | 28.98 | 7.86
FCOS | ResNeXt-101 | 48.17 | 33.95 | 9.86 | 41.79 | 27.27 | 7.19
FCOS+ | ResNet-101 | 50.27 | 35.89 | 10.57 | 43.14 | 29.82 | 7.76
FCOS Ensemble+ | - | 58.27 | 44.48 | 15.36 | 55.72 | 41.12 | 12.5

Figure 5: Training losses for anchor-based and anchorless detectors. It shows
that the loss curve of the anchorless detectors is relatively stable.

Semi-supervised learning. We used the pretrained SiamMask model from the DAVIS
dataset [20] to generate dense labels to train the FCOS models with dense
annotations. Table 2 shows that the generalization performance of the FCOS
models under $IoU>0.5$ is consistently improved when using semi-supervised
learning based on single object tracking.

Inhomogeneous backbone ensemble. We performed model ensemble for each trained
model in Table 2 to achieve the best detection performance from trained
detectors. We performed the ensemble using the joint NMS technique [11], where
we can achieve an ensemble by applying NMS to a bounding box with up to 300
high prediction scores obtained from every detectors. Table 2 shows the
performance change of the Seen and Unseen sets according to the ensemble
combination, and Figure 6 shows the performance published on the Epic-Kitchen
object detection challenge page. We were finally able to achieve the first
rank in Seen set and runnerup performance in Unseen set through an
inhomogeneous backbone ensemble under IoU $>0.5$ evaluation.

Visualizations. We visualized the inference results of each model to confirm
the effect of the inhomogeneous backbone ensemble. Figure 7 shows the
visualization of the inference bounding box of the FCOS model with different
backbones. As shown in Figure 7, the detectors have the same structure, but
only different backbones can be used to obtain very different types of
inference results. Through this, it was confirmed that the ensemble model can
achieve a very large performance improvement compared to a single model.

Figure 6: Epic-Kitchens 2020 object detection challenge evaluation page. The
entry marked with a red box is the final performance evaluated by the our
proposed approach. Each entry is ranked under IoU $>0.5$ evaluation. Figure 7:
Visualization for each network prediction. It shows that the prediction scores
can be variously distributed for each network when they have the same
structure but have a heterogeneous backbone. The predicted score threshold for
visualization was set at 0.5.

##  6 Conclusion

We performed single object tracker-based semi-supervised object detection to
effectively train datasets with sparse annotations on sequence images. The
Epic-Kitchens object detection dataset was used to verify the utility of the
proposed technique, and the proposed semi-supervised learning showed good
performance in the ensemble as well as in the single model. However, it needs
to be analyzed more closely with semi-supervised learning about the advantages
and disadvantages of the anchor-based model, and there is a limitation that a
simple rule-based engine is used to obtain a pseudo label. For future
improvement, it is necessary to perform quantitative analysis on the effect of
anchor and RPN on sparse annotation data training, and at the same time, it is
possible to consider how to improve the tracking rules or utilize the results
obtained in the tracking process during training.

## References

  * [1] Luca Bertinetto, Jack Valmadre, João F. Henriques, Andrea Vedaldi, and Philip H. S. Torr.  Fully-convolutional siamese networks for object tracking.  In In Proc. of ECCV, 2016.
  * [2] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In In Proc. of CVPR, 2018.
  * [3] Bao Xin Chen and John K. Tsotsos.  Fast visual object tracking with rotated bounding boxes.  In In Proc. of ICCVW, 2019.
  * [4] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin.  Mmdetection: Open mmlab detection toolbox and benchmark.  arXiv:1906.07155, 2019.
  * [5] Min-Kook Choi, Jaehyeong Park, Jihun Jung, Heechul Jung, Jin-Hee Lee, Woong Jae Won, Woo Young Jung, Jincheol Kim, and Soon Kwon.  Co-occurrence matrix analysis-based semi-supervised training for object detection.  In In Proc. of ICIP, 2018.
  * [6] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun.  R-fcn: Object detection via region-based fully convolutional networks.  In In Proc. of NIPS, 2016.
  * [7] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Scaling egocentric vision: The epic-kitchens dataset.  In In Proc. of ECCV, 2018.
  * [8] Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  International Journal of Computer Vision, 111:98–136, 2015.
  * [9] Cheng-Yang Fu, Wei Liu, Ananth Ranga, Ambrish Tyagi, and Alexander C. Berg.  Dssd : Deconvolutional single shot detector.  arXiv:1701.06659, 2017.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In In Proc. of CVPR, 2016.
  * [11] Heechul Jung, Min-Kook Choi, Jihun Jung, Jin-Hee Lee, Soon Kwon, and Woo Young Jung.  Resnet-based vehicle classification and localization in traffic surveillance systems.  In In Proc. of CVPR, 2017.
  * [12] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan.  Siamrpn++: Evolution of siamese visual tracking with very deep networks.  In In Proc. of CVPR, 2019.
  * [13] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu.  High performance visual tracking with siamese region proposal network.  In In Proc. of CVPR, 2018.
  * [14] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In In Proc. of ICCV, 2017.
  * [15] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro PeronaDeva, Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco: Common objects in context.  In In Proc. of ECCV, 2014.
  * [16] Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen.  Deep learning for generic object detection: A survey.  arXiv:1809.02165, 2018.
  * [17] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg.  Ssd: Single shot multibox detector.  In In Proc. of ECCV, 2016.
  * [18] Ishan Misra, Abhinav Shrivastava, and Martial Hebert.  Watch and learn: Semi-supervised learning for object detectors from video.  In In Proc. of CVPR, 2015.
  * [19] Yusuke Niitani, Takuya Akiba, Tommi Kerola, Toru Ogawa, Shotaro Sano, and Shuji Suzuki.  Sampling techniques for large-scale object detection from sparsely annotated objects.  In In Proc. of CVPR, 2019.
  * [20] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbelaez, Alexander Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  arXiv:1704.00675, 2017.
  * [21] Joseph Redmon and Ali Farhadi.  Yolo9000: Better, faster, stronger.  In In Proc. of CVPR, 2017.
  * [22] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv:1804.02767, 2018.
  * [23] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In In Proc. of NIPS, 2015.
  * [24] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In In Proc. of ICCV, 2019.
  * [25] Thang Vu, Hyunjun Jang, Trung X. Pham, and Chang Yoo.  Cascade rpn: Delving into high-quality region proposal network with adaptive convolution.  In In Proc. of NIPS, 2019.
  * [26] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao.  Deep high-resolution representation learning for visual recognition.  arXiv:1908.07919, 2019.
  * [27] Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip H.S. Torr.  Fast online object tracking and segmentation: A unifying approach.  In In Proc. of CVPR, 2019.
  * [28] Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In In Proc. of CVPR, 2017.
  * [29] Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei, and Stan Z. Li.  Single-shot refinement neural network for object detection.  In In Proc. of CVPR, 2018.