# Discovering the Effectiveness of Pre-Training
in a Large-scale Car-sharing Platform

Kyung Ho Park1 and Hyunhee Chung1 1K. H. Park is with SOCAR AI Research. kp at
socar.kr1H. Chung is with SOCAR AI Research. esther at socar.kr

###### Abstract

Recent progress of deep learning has empowered various intelligent
transportation applications, especially in car-sharing platforms. While the
traditional operations of the car-sharing service highly relied on human
engagements in fleet management, modern car-sharing platforms let users upload
car images before and after their use to inspect the cars without a physical
visit. To automate the aforementioned inspection task, prior approaches
utilized deep neural networks. They commonly employed pre-training, a de-facto
technique to establish an effective model under the limited number of labeled
datasets. As candidate practitioners who deal with car images would presumably
get suffered from the lack of a labeled dataset, we analyzed a sophisticated
analogy into the effectiveness of pre-training is important. However, prior
studies primarily shed a little spotlight on the effectiveness of pre-
training. Motivated by the aforementioned lack of analysis, our study proposes
a series of analyses to unveil the effectiveness of various pre-training
methods in image recognition tasks at the car-sharing platform. We set two
real-world image recognition tasks in the car-sharing platform in a live
service, established them under the many-shot and few-shot problem settings,
and scrutinized which pre-training method accomplishes the most effective
performance in which setting. Furthermore, we analyzed how does the pre-
training and fine-tuning convey different knowledge to the neural networks for
a precise understanding.

###### Index Terms:

Pretraining, Representation Learning, Self Supervised Learning, Intelligent
Transportation

##  I Introduction

The recent development of deep learning has empowered convenience and
efficiency toward various areas of the transportation industry, such as
traffic surveillance [6, 9], car insurance, and especially car-sharing
services [26, 27]. In the past decades, the traditional car-sharing company’s
operational procedures mostly necessitated human engagement. Human operators
should have resided in the rental station to make a rental contract with the
customers, and they manually checked every car’s condition for quality
assurance; thus, the car-sharing business was known to necessitate a large
amount of human labor. On the other hand, modern car-sharing services such as
Zipcar in the United States have reduced the aforementioned burdensome human
engagements in their operations, leveraging mobile technologies. As most
people have smartphones, car-sharing companies started to let customers
conveniently reserve the car and open the car’s door through the smartphone
application [25, 24]. Moreover, as shown in Figure. 1, the company requires
the customers to take pictures of the car before and after its use to ensure
that they did not damage it. The human operators inspect retrieved images to
monitor the cars’ condition without visiting the station. Throughout the
aforementioned efforts, modern car-sharing companies aim to provide a
convenient experience to the customers and accomplish efficiency in fleet
management.

---
Figure 1: Illustration of how car images are retrieved from the customers.

To automate the image-based fleet management procedures, several studies
started to automate fleet management procedures with computer vision
algorithms. Along with the development of deep neural networks, recent studies
primarily employ convolutional neural networks (CNN) under the fully
supervised, high data availability regime [26]. However, the prior supervised
learning-based approaches bear several limits. First, the practitioner should
manually annotate every image, and it creates a burdensome resource
consumption. Second, especially in the car-sharing service, there frequently
exists the case where the practitioner cannot acquire a particular amount of
samples.

Among several mitigating methods against these limits, pre-training has become
a de-facto technique to establish a practical model in the real world. Given a
target task where the amount of training samples is limited or the
practitioner cannot annotate many samples, the pre-training method trains the
feature extractor of the neural networks on the other task (source task) and
fine-tunes the model on the target task [23]. One paradigm of pre-training is
transfer learning, which trains the network on the external source dataset
such as ImageNet [7] under the supervised manner. The other paradigm is self-
supervised learning, which trains the network on the source task consisting of
the target dataset in an unsupervised manner [15]. These pre-training
approaches are known to convey powerful representation power to the neural
network [23]; thus, it enables the model to hedge the risk of overfitting
under the limited number of samples at the target training set. While the
image recognition studies related to the operations of car-sharing services
conventionally utilized pre-training, we analyzed there has been less
spotlight on the following questions: 1) Which pre-training method is most
effective at various problem settings to learn the representation power of car
images?, 2) If an effective pre-training method exists, why does it contribute
to better performance?

To provide meaningful takeaways for the candidate practitioners who deal with
car images, our study proposes a series of analyses to unveil the
effectiveness of two pre-training paradigms in various image recognition tasks
at the car-sharing service. Under the cooperation with the SOCAR, which is the
largest car-sharing platform in Korea, we present two real-world image
recognition problems: car model recognition and car defect recognition. To
examine the most effective pre-training method in various problem settings, we
established two learning paradigms: many-shot learning and few-shot learning.
Throughout solving the aforementioned image recognition tasks in various
problem settings, we discovered meaningful takeaways related to the
effectiveness of pre-training. Furthermore, we scrutinized how pre-training
methods convey knowledge to the target task, and which layers are affected by
the pre-training and the fine-tuning. We highly expect the candidate
practitioners can utilize these takeaways to efficiently solve real-world
problems related to the image recognition tasks with car images.

##  II Related Works

###  II-A Understanding Car-related Images

Recently-proposed car-related studies aim to understand various attributes of
the car with deep neural networks [1, 2]. As early works, car part recognition
methods have been proposed. Prior works resolved this task under the
supervised regime, which trains the model with a finely-labeled large dataset.
[26] acquired car images, annotated corresponding car parts, and trained CNNs
under various architectures. [27] additionally merge both classifier and
object detector to precisely understand car images. [25] also employed an
unsupervised learning paradigm to cluster car images including similar parts.
Not only understand car parts, but several studies also proposed various
studies understanding various car statuses. They suggested numerous approaches
to understanding damaged patterns of the car to automate post-accident claims
of the car-insurance entities [24]. Throughout these works, we confirm that
cars have become an important object in modern computer vision studies, and
both academia and industry have a keen interest in discovering effective
representations of car-related images. However, we scrutinize most prior works
of application studies to solve particular tasks (i.e., defect recognition),
not focusing on why a particular learning strategy effectively understands car
images. Furthermore, most previous studies utilize web-crawled or surveillance
camera-taken images, which do not reflect the realistic characteristics of the
car in the real world. Under the aforementioned limits of prior works, our
study aims to focus on effective learning strategies regarding real-world car
images and how a particular strategy contributes to the improved inductive
biases of the model.

###  II-B Representation Learning

Transfer learning aims to utilize the knowledge gained from solving the source
task to escalate the target task performance [23]. A conventional method of
transfer learning utilizes the feature extractor weight generated from the
source task of the ILSVRC task (i.e., ImageNet classification task) [7]. We
employed two transfer learning methods: a pre-training with the ImageNet
dataset and a pre-training with Stanford-Cars dataset [20]. We employed
ImageNet pre-trained weight as it has been conventionally utilized in numerous
computer vision tasks due to its generality. The Stanford-Cars dataset is a
dataset of cars; thus, we expected utilization of the Stanford-Cars dataset as
a source dataset would share similar knowledge to our image recognition tasks.

Self-supervised learning utilizes Training set during the pre-training and
trains any human-annotated tasks [15, 14]. There have been proposed a wide
range of pretext tasks for the source task, such as jigsaw puzzles [22], image
colorization [4], or rotation prediction [11]. One recent paradigm of self-
supervised learning is contrastive learning [14], which trains the model to
discriminate whether a pair of the original image and the augmented one come
from the same source (i.e., SimCLR [5], BYOL [12]). While contrastive learning
approaches accomplished precise performance, it bears a limit in that the
model necessitates a massive amount of GPU computing environments [5]. As our
study aims to provide meaningful takeaways that general practitioners can
utilize, we dropped instance discrimination-based self-supervised learning
approaches. Instead, our study employed a rotation prediction-based self-
supervised learning approach [11], which achieved a promising performance in
the public benchmark dataset without a huge computing environment.

##  III Preliminaries

###  III-A Pre-Training Methods

Throughout the study, we employed four pre-training methods in various problem
settings as below:

  * •

Random: It randomly initializes target feature extractor $F_{T}$’s weight. It
implies the target neural network $N_{T}$ solves the target task from the
randomly initialized feature extractors and classification layers without any
pre-training.

  * •

TL-ImageNet: Under the regime of transfer learning, it initializes target
feature extractor $F_{T}$’s weight from the trained feature extractor $F_{S}$
trained on the ImageNet dataset.

  * •

TL-Cars: Under the regime of transfer learning, it initializes target feature
extractor $F_{T}$’s weight from the trained feature extractor $F_{S}$ trained
on the Stanford-Cars dataset. As the Stanford-Cars dataset and our image
recognition tasks simultaneously deal with car images, we expect this option
to provide contextual knowledge to the target neural network $N_{T}$.

  * •

SSL-Rotation: Under the regime of self-supervised learning, it initializes
target feature extractor $F_{T}$’s weight from the trained feature extractor
$F_{S}$ trained from the rotation prediction as a pretext task.

|
---|---
(a) Car Model Dataset | (b) Car Defect Dataset
Figure 2: Sample images at car model dataset and car defect dataset

###  III-B Target Tasks

Our study employed two image recognition tasks that are widely utilized in
real-world transportation industries: car model recognition and car defect
recognition. The datasets for both tasks are accumulated from May 2021 to
August 2021 from SOCAR, which is the largest car-sharing platform in Korea. As
every image in both datasets is taken by real-world customers, sizes and
resolutions vary at each sample.

####  III-B1 Car model recognition

Car model recognition is a $N$-class classification task given $N$ car models.
The car model recognition model is an effective tool for internal human
operators or data analysts at the car-sharing platform. When operators have to
categorize a large number of unlabeled car images, they can utilize the
classified results from the car model recognition model instead of manual
inspections in every sample. The car model recognition model is also utilized
to identify whether a customer correctly uploads the image of the car he or
she borrowed, not the other car. In our study, we acquired car model dataset
consisting of 10 car models (2 light cars, 4 mid-sized cars, 4 Sport Utility
Vehicles) for the car model recognition task, and samples from particular car
models are visualized in Figure 2-(a).

####  III-B2 Car Defect Recognition

Car defect recognition is a binary classification task between normal car
images and damaged car images. The car defect recognition model is primarily
utilized for fleet management to ensure whether a car has a damaged part or
not. If the model figures out a damaged car, human operators in the car-
sharing platform visit the rental station to repair it. Our study retrieved
car defect dataset, which consists of both normal car images and damaged car
images as shown in Figure 2-(b). While the normal car images have a common
characteristic of a clean surface, the damaged car images bear various damage
types such as scratch, dent, separation of car parts, and severe crashes.

|  |
---|---|---
(a) 2-Class Car Model Recognition | (b) 10-class Car Model Recognition | (c) Car Defect Recognition
Figure 3: Results on the effectiveness of various pre-training methods at car
model recogntion and car defect recognition at the many-shot problem setting.

##  IV Effectiveness of Pre-Training under Many-shot setting

###  IV-A Problem Definition

First and foremost, we examined how the pre-training methods escalate the
target performance under the many-shot learning paradigm. Many-shot learning
is a supervised learning paradigm where a particular amount of samples are
available in the training set. To examine the effectiveness of various pre-
training methods at two target tasks above, we established several research
questions to provide essential takeaways toward candidate practitioners. The
research questions are illustrated as follows.

  * •

RQ1: Does the pre-training contribute to the escalation of target task
performance?

  * •

RQ2: Does the effectiveness of pre-training differs at different sizes of the
target training set? If it differs, which pre-training method is optimal in
which size of training set?

###  IV-B Many-shot Car Model Recognition

####  IV-B1 Setup

To validate the effectiveness of pre-training in car model recognition, we
established two classification tasks with a car model dataset consisting of 10
car models (classes): 2-class classification and 10-class classification. We
established two classification tasks to validate whether the pre-training
method escalates the target performance well regardless of the number of
classes. In response to the RQ1, we initialized the target feature extractor
$F_{T}$ with four pre-training options proposed in the section III-A and
trained the $N_{T}$ with the target dataset $D_{T}$. We aim to figure out
three pre-training options (TL-ImageNet, TL-Cars, SSL-Rotation) to elevate the
target task performance compared to the random initialization (Random).
Moreover, we configured three types of the target training set with a
different number of samples at each class to make a response to the RQ2:
Model-10%, Model-50%, Model-100%. The Model-10% training set option implies a
circumstance where the practitioners could not acquire many annotated samples
at each class, and we expected this circumstance to frequently happen in the
real world. The Model-100% training set illustrates the case where the
practitioners acquired a comparatively large amount of labeled samples. Note
that detailed training set options are illustrated in Table I.

As an implementation detail, we employed the deep neural networks architecture
of ResNet-50 [13] both in pre-training and the target task as it is a widely-
utilized network architecture in various computer vision tasks [16, 13]. We
employed a cross-entropy loss as a learning objective, Adam [17] as an
optimizer, and applied weight decay [21] to accomplish the convergence during
the target training stage. Note that we established pre-trained feature
extractor weights $F_{S}$ on the ResNet-50 architecture before training the
target model $N_{T}$, and we also checked the convergence during the pre-
training. Lastly, we employed an evaluation metric of the target task
performance as classification accuracy. Based on the aforementioned setups, we
checked the target task performance along with different pre-training methods
and target training set options. The experiment results at 2-class
classification and 10-class classification are illustrated in Figure 3-(a),
Figure 3-(b), respectively.

TABLE I: Configuration of the training and test for the car model recognition.

# (Number) of | Training Set Options | Test
---|---|---
images | Model-10% | Model-50% | Model-100% | Set
per class | 100 | 500 | 1,000 | 500
Total # (2-class) | 200 | 1,000 | 2,000 | 1,000
Total # (10-class) | 1,000 | 5,000 | 10,000 | 5,000

####  IV-B2 Experiment Results

Following the experiment results shown in Figure 3-(a), Figure 3-(b), we
discovered several takeaways regarding the effectiveness of pre-training.
First, as a response to the RQ1, every pre-training method (TL-ImageNet, TL-
Cars, SSL-Rotation) accomplished superior accuracy than random initialization
(Random) regardless of training set options. We analyzed this escalation of
target task accuracy concretely derives from the acquired representation power
during the pre-training, and examined the pre-training methods are valid in
the car model recognition task. Furthermore, as a response to the RQ2, we
discovered the magnitude of target task accuracy escalation differs along with
the size of the target training set. Given the small size of the target
training set (Model-10%), SSL-Rotation was the most effective one among pre-
training methods. Along with the increased size of the training set (from
Model-10% option to the Model-50%, Model-100%), SSL-Rotation consistently
accomplishes the best target task accuracy, but the other pre-training methods
also achieve similar performance to it.

###  IV-C Many-shot Car Defect Recognition

####  IV-C1 Setup

In this section, we examined whether the takeaways discovered in section IV-B
are also valid in the car defect recognition task, which is a binary
classification between the normal and damaged images. In response to the RQ1,
we employed four pre-training options presented in the sectionIII-A, but
additionally utilized one extra pre-training option denoted as TL-Cars-OURS,
which is the fine-tuned network on 10-class car model recognition. We analyzed
the trained neural networks in the car model recognition task would yield a
particular amount of knowledge to the car defect recognition; thus, the
trained feature extractor $F_{T}$ at car model recognition can become a
trained source feature extractor $F_{S}$ at the car defect recognition. Note
that we regard the car model dataset as a source dataset $D_{S}$ and car
defect recognition as a target dataset $D_{T}$ in this experiment. For the TL-
Cars-OURS pre-training method, we utilized the feature extractor of the car
model recognition model trained under 10-class classification setting with
Model-100% training set option and TL-ImageNet pre-trained weight. In response
to the RQ2, as shown in Table II, we also configured three training set
options (Defect-10%, Defect-50%, Defect-100%) consisting of different numbers
of images at each class.

TABLE II: Configuration of the car defect recognition.

# (Number) of | Training Set Options | Defect
---|---|---
images at | Defect-10% | Defect-50% | Defect-100% | -Test
Normal Class | 296 | 1,474 | 2,946 | 685
Damaged Class | 122 | 611 | 1,222 | 700
Total | 418 | 2,085 | 4,168 | 1,385

As an implementation detail, we utilized the same configurations presented in
section IV-B except for the neural networks architecture. In car defect
recognition, we employed Progressive Multi-Granularity nerual networks (PMG-
Net) [8], which achieved a precise performance in fine-grained classification
task [31, 30]. Compared to the car model recognition task, we analyzed that
the car defect recognition task requires the model to capture more finer-
grained discriminative characteristics between normal and damaged car images.
Therefore, we employed PMG-Net as a neural networks architecture in this car
defect recognition as it is originally designed to capture fine-grained
discriminative characteristics between classes similar to each other. Note
that we tried to solve the car defect recognition task with ResNet-50
architecture, but it failed to converge during the training stage under
various hyperparameters. Following the experiment setups, the results are
shown in Figure 3-(c).

####  IV-C2 Experiment Result

Referring to the experiment result shown in Figure 3-(c), we figured out the
proposed takeaways at car model recognition are also valid in car defect
recognition, which requires the model to understand fine-grained
discriminative characteristics among each class. As a response to the RQ1,
every pre-training method accomplished superior target accuracy rather than
random initialization. Furthermore, as a response to the RQ2, the self-
supervised learning (SSL-Rotation) method accomplished precise target accuracy
regardless of the training set options.

While every pre-training method accomplished similar target task accuracy in
car model recognition task, only SSL-Rotation and TL-ImageNet achieved
similar, superior performance under the large size of the target training set
(Defect-100%). We analyzed these results derived from the biasedness of the
knowledge learned during the pre-training. Both TL-Cars and TL-Cars-OURS
methods convey the knowledge biased to the classes in the source dataset
$D_{S}$. TF-Cars and TF-Cars-OURS pre-train the representation power to
extract meaningful features biased to car models biased to 196 car models and
10 car models, respectively. As discriminative characteristics among car
classes are less fine-grained than car defect recognition, we analyzed the
representation power biased to the coarse-grained characteristics (learned at
TL-Cars and TL-Cars-OURS) could not provide a significant knowledge for the
car defect recognition, which is a finer-grained recognition task. On the
other hand, we interpret TL-ImageNet and SSL-Rotation methods yield
comparatively less biased knowledge to the target task. We analyzed TL-
ImageNet is less biased to the classes in the source dataset as there exist a
thousand classes, and SSL-Rotation is also less biased due to its inherent
characteristics [15, 14]. Note that self-supervised learning is inherently
designed to evade the convey of biased representation power. We analyzed this
less biased representation power conveys better knowledge to the target
feature extractor on fine-grained image recognition tasks; thus, TL-ImageNet
and SSL-Rotation consistently could increase the target task accuracy rather
than other pre-training methods.

##  V Effectiveness of Pre-Training under Few-shot setting

###  V-A Problem Definition

In this section, we validated how the pre-training methods elevate the target
task performance under the few-shot learning paradigm. The few-shot learning
paradigm aims to recognize classes with a few labeled samples at the target
dataset. With the conventional case, training the target network on this
dataset presumably causes overfitting as the few samples cannot provide
sufficient knowledge to the model; thus, the model cannot achieve satisfactory
performance. Instead, the few-shot learning employs a meta-learning stage with
an external auxiliary task to let the model understand how to extract features
from the image. During the training phase, the few-shot learning paradigm
first trains the neural networks with a labeled, easily acquirable auxiliary
dataset consisting of many samples. Then, the trained neural networks adapt to
the novel classes (the target classes) during the test phase, where a few
samples exist at each novel class [3, 18]. As the few-shot learning paradigm
does not necessitate many labeled samples in the target dataset, practitioners
utilize this paradigm when they cannot acquire the labeled target dataset.
Under the scope of our study at the few-shot learning paradigm, we presume the
auxiliary dataset as car model dataset and the target dataset as car defect
dataset. While car model dataset can be easily acquired from the operations or
web search, we presumed collecting damaged car images reflecting real-world
damages (i.e., various types of scratch, dent, separated car parts) is more
challenging to the practitioners. To validate the effectiveness of pre-
training in the few-shot learning paradigm, we also established a research
question as below.

  * •

RQ3: Does the pre-training contribute to the escalation of target task
performance?

TABLE III: Configuration of datasets in the few-shot setting

Descriptions of | Dataset for Few-Shot Classification
---|---
the dataset | Auxiliary Set | Support Set | Query Set
Source | Model-100% | Defect-100% | Defect-Test
Label Space | 6-class | 2-class | 2-class
Total # of images | 6,000 | 4,168 | 1,385

###  V-B Few-Shot Car Defect Recognition

####  V-B1 Setup

The few-shot learning paradigm bears two stages: 1) training stage with the
auxiliary set (which is car model dataset), 2) test stage with the support set
and the test set (where both support set and test set are derived from the
target task, which is car defect dataset). The auxiliary set is a set of data
utilized to let the neural networks understand how to extract the knowledge at
given images. The support set and query set consists of the unseen classes
during the training stage; thus, their label space differs from the auxiliary
set. The support set includes a few labeled samples from the target task. It
is utilized as a guideline to the trained model to infer discriminative
characteristics of the target task, which consists of novel classes in the
model’s perspective. Lastly, the query set bears a particular amount of
samples from the target task, which is utilized for the model validation
during the test stage. If the support set includes $K$ samples for each of $N$
classes, we conventionally denote this few-shot recognition as $N$-way
$K$-shot setting. In this study, we established the few-shot recognition as a
2-way 5-shot setting (2-way derives from two classes exist in the car defect
dataset). Considering the aforementioned descriptions regarding the few-shot
learning paradigm, we configured auxiliary dataset, support set, and a query
set as shown in Table III. Note that we utilized 6 randomly selected classes
from car model dataset to reduce the burden of computation.

Among numerous few-shot classification approaches[28, 10], we employed
Relation Network proposed in [29] due to its precise performance in the public
benchmark dataset. Given a query image, Relation Network identifies the most
similar class in the support set by selecting the smallest distance between
support samples and a query sample. To examine the effectiveness of pre-
training at a few-shot classification setting, we first initialized the
feature extractor of the target network ($N_{T}$) with five options: Random,
TL-ImageNet, TL-Cars, TL-Cars-OURS, and SSL-Rotation. Note that source labels
in the TL-Cars-OURS option do not duplicate the labels that exist in the
auxiliary set. We then let the target network be trained on the auxiliary set
and examined the few-shot classification performance with the support and
query sets. As the target task performance can vary along with the selected
samples in the support set, we applied the episode learning method [28], which
averages the test accuracy at $E$ iterations of the test stage. In a single
iteration, episode learning randomly selects five support samples at each
class of support set and calculates the classification accuracy; thus, the
averaged accuracy can measure an overall accuracy regardless of the selected
support samples. In this study, we recorded the target task accuracy scores
within 90% of the confidence interval from 100 iterations. The experiment
result is shown in Table IV.

TABLE IV: 2-way 5-shot car damage recognition performance

Pre-Training | Accuracy(%)
---|---
Methods | Interval | Lower Bound | Upper Bound
Random | 53.38$\pm$5.2 | 48.63 | 59.03
TL-ImageNet | 75.59$\pm$ 2.2 | 73.39 | 77.79
TL-Cars | 75.16$\pm$2.3 | 72.86 | 77.46
SSL-Rotation | 75.91$\pm$2.4 | 73.51 | 78.31

####  V-B2 Experiment Result

Following the experiment result elaborated in Table IV, we figured out several
takeaways similar to the lessons learned in experiments under the many-shot
setting. In response to the RQ3, we discovered that the pre-training method
contributed to the escalated target accuracy rather than the random
initialization. We interpret pre-training methods conveyed a particular amount
of representation power to the target neural network to extract meaningful
features in a given image. Moreover, while the self-supervised learning method
achieved the best target task performance, there exists a minimal difference
among various pre-training methods. Therefore, we expect candidate
practitioners can utilize any pre-training method in the real-world problem to
escalate the few-shot classification performance.

|
---|---
(a) Random | (b) TL-ImageNet
|
(c) TL-Cars | (d) SSL-Rotation
Figure 4: Layer-wise similarity matrices in a single neural network.

##  VI How does the Pre-Training escalate target task performance?

Throughout experimental analyses in section IV and V-B, we discovered pre-
training methods concretely contribute to the escalated target task
performance in every image recognition task and problem settings. Based on the
aforementioned discoveries, we aim to scrutinize an underlying reason behind
the effectiveness of pre-training by analyzing learned knowledge at the neural
networks. To analyze the representations in neural networks consisting of
different weights, we employed Central Kernel Alignment (CKA) [19] as an index
of measuring similarity between two representations from different neural
networks. Suppose we provide image samples into two neural networks ($N_{1}$,
$N_{2}$) trained from different pre-trained weights. Then, we can extract a
pair of representation vectors from any layers of $N_{1}$ and $N_{2}$ denoted
as ($R_{1}$, $R_{2}$). Given these representation vectors $R_{1}$ and $R_{2}$,
CKA effectively measures the similarity between layers in the same neural
networks with different weights and across entirely different architectures.
The CKA yields a similarity metric lying between 0 to 1, where 0 implies less
similarity and 1 illustrates high similarity. Due to its convenience and
effectiveness in measuring similarity between two representations, we utilized
it. Please refer to the original publication [19] for a detailed elaboration
on the CKA.

To scrutinize the reason behind the superior contribution of various pre-
training methods, we established the following research questions.

  * •

RQ 4: What does the pre-trained weight convey to the target task? Does the
pre-trained weight help the neural networks understand data a priori to the
target task training?

  * •

RQ 5: Which layers of the network are primarily affected by the pre-training
and the fine-tuning?

TABLE V: Representation similarity in lower-level blocks

Pre-Training | Representation Similarity
---|---
Methods | (PT, PT) | (PT, FT) | Difference
TL-ImageNet | 0.9192 | 0.9212 | 0.0020
TL-Cars | 0.9447 | 0.9336 | 0.0111
SSL-Rotation | 0.9224 | 0.9240 | 0.0016

TABLE VI: Representation similarity in higer-level blocks

Pre-Training | Representation Similarity
---|---
Methods | (PT, PT) | (PT, FT) | Difference
TL-ImageNet | 0.9819 | 0.8526 | 0.1293
TL-Cars | 0.9579 | 0.8548 | 0.1031
SSL-Rotation | 0.9757 | 0.7942 | 0.1815

###  VI-A Analyis on the Overall Knowledge

In response to the RQ 4, we visualized layer-wise similarity among every layer
of the neural networks. We would like to highlight that this section employs
four pre-trained weights proposed in section V, and target network (ResNet-50)
trained on the 2-class car model dataset (Model-100%). Given a set of samples
at Model-Test, we provide these samples into the neural network and acquire
the representations at $i$th block and $j$th block, denoted as $R_{i}$ and
$R_{j}$, respectively. We measured the representation similarity between
$R_{i}$ and $R_{j}$, and created a similarity matrix consists of $(i,j)$. Just
as [19] utilized this similarity matrix as a proxy to overall knowledge
learned by the neural network, we also employed it in the analysis. To
scrutinize the impact of pre-training on the target task, we compared the
similarity matrix between the pre-trained feature extractor and the fine-tuned
feature extractor. Suppose the similarity matrix of the pre-trained feature
extractor looks similar to the fine-tuned feature extractor. In that case, we
can infer that the pre-training conveys a particular amount of knowledge to
the target neural network before the target training stage. Following these
setups, we visualized a pair of similarity matrices with four pre-training
options in Figure 4.

Referring to Figure 4, we figured out several interesting takeaways regarding
the effectiveness of pre-training. First, similarity matrices of TL-ImageNet,
TL-Cars, SSL-Rotation share similar patterns while each pre-trained weight was
established from a different source dataset. (Refer the similarity matrices
denoted as PT-PT). Unlike the similarity matrix at the randomly initialized
network, three pre-trained weights (PT-PT) shared a similar shape of a square-
like structure. Accordingly, we analyzed the existence of common knowledge
among each pre-trained weight, and this knowledge provides a precedent
knowledge to the target neural network. Second, we figured out the fine-tuned
network also maintains similar knowledge to the pre-trained weights. Comparing
each pair of (PT-PT, PT-FT) in Figure 4, a pair consisting of pre-trained
weights seems to be similar while the similarity matrix at random
initialization (PT-PT) changed in a particular manner at PT-FT. We analyzed
the target task training provides additional knowledge to the neural networks
based on the common knowledge from the pre-training. As the amount of change
from PT-PT to PT-FT is not magnificent, we also expect target task training to
yield a smaller amount of knowledge to the neural networks than pre-training.
Throughout the analysis on the overall knowledge in the neural networks, we
inferred that various pre-training methods establish common knowledge
regarding image understanding, and it becomes a solid basis for the target
neural network to accomplish precise target task performance.

###  VI-B Analysis on the Layer-level Knowledge=

While we figured out the pre-training method concretely yields a common
knowledge to the target neural network, we scrutinized which block (layers at
conventional CNN architectures) of the neural network primarily illustrates a
common knowledge conveyed from the pre-trained weight. Not only the conveyed
common knowledge, but we also scrutinized which block of the neural network
includes additional knowledge acquired during the fine-tuning into the target
task. As a response to the RQ5, we compared the similarity between the pre-
trained weight (PT) and the fine-tuned weight (PT-FT) at two blocks: low-level
block and high-level block. A low-level block is the first block at ResNet-50
architecture, which first receives the image as an input and extracts low-
level characteristics of the image. A high-level block is the last block at
ResNet-50, where yields a condensed representation to the classification
layers. We scrutinized how does the knowledge changes during the fine-tuning
stage by comparing the representation similarity. The representation
similarity between PT and PT-FT at low-level block and high-level block are
illustrated in Table V, Table VI, respectively.

We resulted in the common knowledge conveyed by the pre-trained weight
primarily exist in the low-level block, while the knowledge gained from the
fine-tuning stage lies in the high-level block. As shown in Table V,
representations after the fine-tuning do not change in a particular manner.
Therefore, we expect a common knowledge generated by the pre-training
presumably exists in the neural network’s low-level blocks, which implies the
pre-training conveys the target network how to extract low-level features.
Referring to Table VI, we discovered representation similarity at the high-
level block decreases in a particular manner after the fine-tuning; thus, we
infer the fine-tuning provides knowledge regarding the semantics of the target
dataset.

##  VII Conclusion

Our study performed in-depth analyses of which learning paradigm fits with
real-world car images and how the representation landscape looks in the
trained neural networks. First, we highly recommend that practitioners utilize
the pre-training method as they accomplish improved target task performances
rather than random initialization in every image recognition task. Second, the
practitioners shall be cautious about maintaining unbiasedness in the pre-
trained weight under the fine-grained target task. Lastly, we emphasize that
the conveyed representation power from the pre-training primarily exists in
the low-level layers of the neural networks, while the fine-tuning provides
knowledge in the high-level layers. Still, we acknowledge there exist several
improvement avenues for our study. Further studies shall compare various self-
supervised learning methods or compare the effectiveness of transfer learning
from non-natural images.

## References

  * [1] J. Azimjonov and A. Özmen, “A real-time vehicle detection and a novel vehicle tracking systems for estimating and monitoring traffic flow on highways,” _Advanced Engineering Informatics_ , vol. 50, p. 101393, 2021\.
  * [2] ——, “Vision-based vehicle tracking on highway traffic using bounding-box features to extract statistical information,” _Computers & Electrical Engineering_, vol. 97, p. 107560, 2022.
  * [3] P. Bateni, R. Goyal, V. Masrani, F. Wood, and L. Sigal, “Improved few-shot visual classification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 14 493–14 502.
  * [4] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” _arXiv preprint arXiv:2006.09882_ , 2020.
  * [5] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [6] S. R. E. Datondji, Y. Dupuis, P. Subirats, and P. Vasseur, “A survey of vision-based traffic monitoring of road intersections,” _IEEE transactions on intelligent transportation systems_ , vol. 17, no. 10, pp. 2681–2698, 2016.
  * [7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [8] R. Du, D. Chang, A. K. Bhunia, J. Xie, Z. Ma, Y.-Z. Song, and J. Guo, “Fine-grained visual classification via progressive multi-granularity training of jigsaw patches,” in _European Conference on Computer Vision_. Springer, 2020, pp. 153–168.
  * [9] J. Fang, Y. Zhou, Y. Yu, and S. Du, “Fine-grained vehicle model recognition using a coarse-to-fine convolutional neural network architecture,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 18, no. 7, pp. 1782–1792, 2016.
  * [10] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1126–1135.
  * [11] S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” _arXiv preprint arXiv:1803.07728_ , 2018\.
  * [12] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar _et al._ , “Bootstrap your own latent-a new approach to self-supervised learning,” _Advances in neural information processing systems_ , vol. 33, pp. 21 271–21 284, 2020.
  * [13] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [14] A. Jaiswal, A. R. Babu, M. Z. Zadeh, D. Banerjee, and F. Makedon, “A survey on contrastive self-supervised learning,” _Technologies_ , vol. 9, no. 1, p. 2, 2021.
  * [15] L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [16] A. Khan, A. Sohail, U. Zahoora, and A. S. Qureshi, “A survey of the recent architectures of deep convolutional neural networks,” _Artificial Intelligence Review_ , vol. 53, no. 8, pp. 5455–5516, 2020.
  * [17] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [18] G. Koch, R. Zemel, R. Salakhutdinov _et al._ , “Siamese neural networks for one-shot image recognition,” in _ICML deep learning workshop_ , vol. 2. Lille, 2015.
  * [19] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 3519–3529.
  * [20] J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3d object representations for fine-grained categorization,” in _4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13)_ , Sydney, Australia, 2013.
  * [21] A. Krogh and J. A. Hertz, “A simple weight decay can improve generalization,” in _Advances in neural information processing systems_ , 1992, pp. 950–957.
  * [22] M. Noroozi and P. Favaro, “Unsupervised learning of visual representations by solving jigsaw puzzles,” in _European conference on computer vision_. Springer, 2016, pp. 69–84.
  * [23] S. J. Pan and Q. Yang, “A survey on transfer learning,” _IEEE Transactions on knowledge and data engineering_ , vol. 22, no. 10, pp. 1345–1359, 2009.
  * [24] K. H. Park and H. Chung, “Don’t wait until the accident happens: Few-shot classification framework for car accident inspection in a real world,” in _International Conference on Image Analysis and Processing_. Springer, 2022, pp. 560–571.
  * [25] K. H. Park, Y. Kwon, Y. Song, and S. Byeon, “Visual representation learning for automating car part recognition in a large-scale car sharing platform,” in _2021 IEEE 17th International Conference on Automation Science and Engineering (CASE)_. IEEE, 2021, pp. 1104–1110.
  * [26] K. Patil, M. Kulkarni, A. Sriraman, and S. Karande, “Deep learning based car damage classification,” in _2017 16th IEEE international conference on machine learning and applications (ICMLA)_. IEEE, 2017, pp. 50–54.
  * [27] R. Singh, M. P. Ayyar, T. V. S. Pavan, S. Gosain, and R. R. Shah, “Automating car insurance claims using deep learning techniques,” in _2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM)_. IEEE, 2019, pp. 199–207.
  * [28] J. Snell, K. Swersky, and R. S. Zemel, “Prototypical networks for few-shot learning,” _arXiv preprint arXiv:1703.05175_ , 2017.
  * [29] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales, “Learning to compare: Relation network for few-shot learning,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 1199–1208.
  * [30] X.-S. Wei, J. Wu, and Q. Cui, “Deep learning for fine-grained image analysis: A survey,” _arXiv preprint arXiv:1907.03069_ , 2019.
  * [31] B. Zhao, J. Feng, X. Wu, and S. Yan, “A survey on deep learning-based fine-grained object classification and semantic segmentation,” _International Journal of Automation and Computing_ , vol. 14, no. 2, pp. 119–135, 2017.