# Deep Learning-Based Connector Detection for Robotized Assembly of Automotive
Wire Harnesses*

Hao Wang1 and Björn Johansson1 *This work was supported by the Swedish
innovation agency, Vinnova, and the strategic innovation program,
Produktion2030, under grant no. 2022-01279. The work was carried out within
Chalmers’ Area of Advance Production. The connectors used in this study were
provided by Wiretronic AB. The support is gratefully acknowledged. 1Hao Wang
and Björn Johansson are with the Division of Production Systems, Department of
Industrial and Materials Science, Chalmers University of Technology,
Hörsalsvägen 7A, SE-412 96 Gothenburg, Sweden<EMAIL_ADDRESS>
###### Abstract

The shift towards electrification and autonomous driving in the automotive
industry makes automotive wire harnesses increasingly more critical for
various functions of automobiles, such as maneuvering, driving assistance, and
safety system. It leads to more and more wire harnesses installed in modern
automobiles, which stresses the great significance of guaranteeing the quality
of automotive wire harness assembly. The mating of connectors is essential in
the final assembly of automotive wire harnesses due to the importance of
connectors on wire harness connection and signal transmission. However, the
current manual operation of mating connectors leads to severe problems
regarding assembly quality and ergonomics, where the robotized assembly has
been considered, and different vision-based solutions have been proposed to
facilitate the robot control system’s better recognition of connectors.
Nonetheless, there has been a lack of deep learning-based solutions for
detecting wire harness connectors in previous studies. This paper presents a
deep learning-based connector detection for robotized automotive wire harness
assembly. A dataset of twenty types of automotive wire harness connectors was
created to train and evaluate a two-stage object detection model and a one-
stage object detection model, respectively. The experiment results indicate
the effectiveness of deep learning-based connector detection for automotive
wire harness assembly but are limited by the design of the exteriors of
connectors.

##  I INTRODUCTION

Electrification and autonomous driving have driven a paradigm shift in the
current automotive industry, making the electronic system increasingly
critical in modern automobiles. Numerous automotive wire harnesses have been
installed in current vehicles as an essential infrastructure for supporting
signal transmission within the electronic system. Meanwhile, more and more
wire harnesses are expected to be installed, considering the increase of
automotive wire harnesses in vehicles in the past decades and the paradigm
shift in the industry. Thus, it is crucial to guarantee the quality of the
assembly of automotive wire harnesses.

However, the current final assembly of automotive wire harnesses into vehicles
remains mostly manual and skill-demanding, which makes it challenging to
control and improve the quality and productivity of the assembly. Some manual
operations also involve heavy lifting (for example, approximately 40 kg for
some low-voltage automotive wire harnesses) and high-pressure manual
manipulations on different components of automotive wire harnesses, which
poses severe ergonomic problems to human operators. In particular, the mating
of connectors is one of the sub-process relating to ergonomic issues due to
the repetitive high-pressure manual pressing in the assembly line. Fig. 1
demonstrates an example of an automotive wire harness, where red rectangles
highlight connectors on the automotive wire harness.

Figure 1: An example of an automotive wire harness, with connectors
highlighted by red rectangles.

Connectors are essential components on automotive wire harnesses, among the
others, such as clamps and cables. Automotive wire harnesses are connected to
the target unit or other automotive wire harnesses via connectors so the
signal can be transmitted continuously within the electronic systems
responsible for various functions of automobiles, which are safety-critical in
particular. Thus, ensuring the quality of mating connectors in the final
assembly of wire harnesses into vehicles is critical. However, the current
manual process of mating connectors constrains the productivity and quality of
assembly and generates ergonomic problems for human operators. To relieve the
problems regarding productivity, assembly quality, and ergonomics, robotized
wire harness assembly is of great interest to the automotive industry,
considering its better replicability, transparency, and comprehensibility, and
has been discussed in different studies previously [1, 2, 3, 4]. Nevertheless,
the robotized mating of connectors is non-trivial as the robotic operation
needs to address not only high manipulation accuracy but also intricate
structures and non-rigid materials of connectors [5]. It is also fundamental
to retrieve the geometrical information of connectors beforehand so that a
robot arm can flexibly reach, grasp, and manipulate the perceived connector.

Computer vision has demonstrated a significant potential on the robotized
assembly to the manufacturing industry in solving ergonomic issues while
increasing quality and productivity [6]. Previously, there have also been
studies on computer vision techniques for robotized manipulation of wire
harness connectors [5, 7, 8, 9, 10, 11, 12]. However, a few studies discussed
the task of connector detection [9, 11, 12], where methods based on basic
image processing techniques were mainly explored [9, 11]. Considering the
various designs of connectors on automotive wire harnesses, such as colors,
shapes, and sizes, it is intricate to manage the manual feature engineering on
connectors for flexible robotized manipulation. The recent advancement in
implementing convolutional neural networks (CNN) and deep learning in computer
vision research has demonstrated the extraordinary effectiveness of learning-
based solutions for object detection compared to traditional image processing-
based solutions [13]. Zhou et al. [12] have previously explored deep learning-
based connector detection for the robotized wire harness connection, but the
proposal mainly focused on one-connector detection. The learning-based
detection on multiple connectors remained unsolved but is required for the
robotized assembly of automotive wire harnesses in actual production.

This paper presents a study on the deep learning-based connector detection for
the robotized mating of connectors on automotive wire harnesses and discusses
the feasibility and potential problems of implementing deep learning-based
object detection on the task of mating connectors in robotized automotive wire
harness assembly under laboratory conditions. As there is no publicly
available dataset on automotive wire harness connectors, a dataset comprising
twenty different types of connectors was collected initially. Then, two
different detection models, a two-stage object detection model, Faster R-CNN
[14], and a one-stage object detection model, YOLOv5 [15], were adopted for
the training and inference. The experiment results demonstrate the
effectiveness of deep learning-based connector detection as both detection
methods achieved remarkable detection outcomes with various combinations of
connectors presenting in the scene. Yet, detection performance can be improved
further, and a more extensive dataset comprising more connectors and more
images per connector is needed. Some detection errors on classes and positions
of connectors in inference results further reflect the effect of the design of
the exterior of connectors, which motivates the future connector detection
based on multi-view images of connectors and with new exterior design of
connectors so that more visually distinguishable features of the connector can
be extracted.

This paper is organized in the following structure: Section II introduces the
related research in connector detection and deep learning-based object
detection. Section III introduces the data collection and annotation strategy
and the statistics of the collected dataset of connectors. Section IV
introduces the experiment setups of two-stage and one-stage connector
detection, whose results are presented and further discussed in Section V. The
study is concluded in Section VI with an outlook on the future work of this
study.

##  II RELATED WORK

###  II-A Connector Detection for Robotized Mating of Connectors

Connector detection is needed to acquire the position and categories of
connectors so that the robot can flexibly reach, grasp, and manipulate
connectors. Although some vision-based solutions have been proposed for
facilitating different sub-tasks in robotized mating of connectors [5, 7, 8,
9, 10, 11, 12], connector detection has yet gathered few attention in previous
studies [9, 11, 12], where the basic image processing-based methods are
dominant [9, 11].

Tamada et al. [9] proposed to recognize the types and poses of connectors
using a high-speed vision system. An image processing method was adopted in
Tamada et al. [9] to detect the positions of connectors via detecting the
corners of connectors, which was further processed to calculate the
orientations of connectors. Yumbla et al. [11] later proposed a basic image
processing-based method to detect multiple connectors, including converting
color space and applying color thresholding. However, the task in Yumbla et
al. [11] was a one-class detection, where all connectors were considered the
same class. Deep learning-based connector detection has also been discussed in
a recent study [12], which proposed to roughly locate the position of a
connector and then zoom in to the detected connector to acquire the finer pose
of the connector. Nevertheless, the proposal in Zhou et al. [12] mainly
focused on manipulating one pair of connectors instead of multi-connector
manipulation, which is more common in actual production.

###  II-B Deep Learning-Based Object Detection

The rebirth of convolutional neural networks (CNNs) in 2012 [16] initiated the
research on introducing deep learning [13] to object detection [17], which
further promoted the remarkable development of two major groups of detectors
for object detection based on deep learning in previous years: two-stage
detection and one-stage detection [17].

Similar to the attentional mechanism of the human brain, the two-stage
detection model first scans the whole scenario coarsely and then focuses on
regions of interest (ROIs) to distinguish the object [18]. The region-based
convolutional neural network (R-CNN) proposed by Girshick et al. [19, 20]
symbolized the inauguration of two-stage object detection. In R-CNN [19, 20],
a set of object proposals were extracted and fed into a CNN model to extract
features for classification. However, the redundant feature computations due
to many overlapped proposals made the detection speed extremely slow, which
was improved later by Spatial Pyramid Pooling Networks (SPPNet) [21]. A
Spatial Pyramid Pooling (SPP) layer was introduced in SPPNET [21] to enable a
CNN to generate a fixed-length representation to avoid re-scaling.
Nevertheless, SPPNET [21] remained multi-stage training and only fine-tuning
fully-connected layers. To improve R-CNN [19] and SPPNet [21], Fast R-CNN [22]
was proposed later, where the detector and the bounding box regressor could be
trained under the same network configurations simultaneously. Furthermore,
Faster R-CNN [14] was proposed to accelerate the detection further by
introducing a Region Proposal Network (RPN), but the problem of computation
redundancy remained at the subsequent detection stage. Besides the R-CNN
family, Lin et al. [23] proposed Feature Pyramid Networks (FPNs), which can be
integrated into other detectors to enable high-level semantics building at all
scales besides the feature maps of the networks’ top layer.

Though able to attain high-precision detection, two-stage detection methods
are constrained by their ponderous detection speed and computation,
stimulating the research on one-stage detection. You Only Look Once (YOLO)
[24] was the first deep learning-based one-stage detection that simultaneously
predicted bounding boxes and probabilities for each sub-region of an image.
Although the detection speed was improved significantly, the localization
accuracy dropped remarkably compared to two-stage detection, especially for
some small objects, which was enhanced in YOLO’s subsequent versions [25, 26,
27, 28]. There were also other one-stage detection methods besides the YOLO
family proposed to improve the detection accuracy while maintaining the
advantage of high detection speed, including Single-Shot Multibox Detector
(SSD) [29], RetinaNet [30], and CornerNet [31].

Recent years have also witnessed the profound influence of Transformer models
[32] in deep learning and computer vision [33], which has spawned DEtection
TRansformer (DETR) [34] and Deformable DETR [35] and promoted deep learning-
based object detection to higher performance.

##  III THE DATASET OF CONNECTORS

The dataset is essential for learning-based object detection [36, 37, 38] and
scalable deep learning-based solutions in industry [39]. However, to the best
of the authors’ knowledge, there is no publicly available benchmark dataset
dedicated to the detection of automotive wire harness connectors. Thus, to
facilitate the study of deep learning-based connector detection for the
robotized assembly of automotive wire harnesses, a dataset was collected and
annotated first, consisting of $20$ types of connectors commonly occurring on
automotive wire harnesses installed in passenger vehicles. Fig. 2 demonstrates
one example image for each of the $20$ connectors. The following subsections
will introduce the strategy for image collection and annotation and summarize
the statistics of the dataset used in the experiments.

Figure 2: The twenty types of connectors collected for dataset creation. The
class of each connector is simplified and labeled below images.

###  III-A Image Collection Procedure

Connectors are placed on a white workbench for image acquisition using the
main camera of an iPhone 11. The original image format is RGB, and each image
has a size of $4032\times 3024$ pixels. The distance between the camera and
connectors was not fixed, considering the various locations of connectors in
the three-dimensional (3D) space in actual assembly situations.

There are $360$ images captured in total. Initially, $60$ images of various
combinations of connectors with random poses were collected to simulate the
random distribution of connectors in the actual assembly scenario. Fig. 3
demonstrates some examples of these $60$ images. For clarification, the
distribution of connectors in each of these $60$ images does not represent the
actual distribution of connectors on practical automotive wire harnesses or in
the final assembly of automotive wire harnesses.

Figure 3: Examples of images with different combinations of connectors. Figure
4: The six images captured from the front, back, top, down, left, and right of
A0. These images are cropped from the raw data for demonstration. Figure 5:
The other nine images of A0 were captured from random perspectives. These
images are cropped from the raw data for demonstration.

In addition, images of each of the $20$ connectors were also collected to
train the detector with more features of respective classes. For each
connector, $15$ images were captured from different views, including six
images captured from the front, back, top, down, left, and right of the
connector, as an example of class A0 shown in Fig. 4, and nine images captured
from random perspectives, as an example of class A0 shown in Fig. 5.

###  III-B Image Annotation Procedure

The image annotation procedure of the dataset of connectors followed the
methodology implemented in the PASCAL visual object classes (VOC) challenge
2007 [36].

The image annotation includes the class and the bounding box for every
connector in the target set of classes. As shown in Fig. 2, this study
simplified the $20$ classes of connectors into A0, A1, B0, B1, C, D, E, F, G,
H, I, J, K, L, M, N, O, P, Q, and R, which can be easily mapped to the actual
types of connectors in practical applications. An axis-aligned rectangular
bounding box surrounding the connector was drawn for each connector visible in
each image in the dataset. Though relatively quick to annotate, choosing an
axis-aligned rectangular bounding box for the annotation is a compromise. Some
connectors in images fit well because of their rectangular or approximately
rectangular profiles, for example, class A0 shown in Fig. 4. However, for
other connectors presented in images, an axis-aligned bounding box can be a
poor fit because either they are not axis-aligned, for example, been captured
from random perspectives (Fig. 5) or placed randomly (Fig. 3), or the
connector is not in the shape of a box, for example, class I shown in Fig. 2.

The actual image annotation was conducted using an annotation platform,
Labelme [40]. It was trivial to annotate images with a single connector due to
the structured storage of images. For images with multiple connectors, a list
of visible connectors in each image was documented first during the image
collection procedure. Then, each connector visible in the images was compared
with the original physical counterpart and annotated exhaustively. The
annotation results were compared to the documented list to guarantee the
consistency and accuracy of the image annotation.

###  III-C Dataset Statistics

The total number of annotated images is $360$. The data is primarily divided
into three main subsets: training data (Train), validation data (Validation),
and test data (Test), with a ratio of $90\%/5\%/5\%$. The images in the
validation set and test set were selected randomly. For each subset of the
connector dataset and class of connectors, the number of object instances is
shown in TABLE I. In the collected dataset, the most frequent class is “L”,
with $46$ object instances, and the least frequent class is “M”, with $31$
object instances. Fig. 6 illustrates a histogram of the number of object
instances presented in different subsets of the collected connector dataset
for each class of connectors.

TABLE I: The Numbers of Annotated Object Instances in the Collected Connector Dataset. | A0 | A1 | B0 | B1 | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Train | $39$ | $38$ | $29$ | $36$ | $33$ | $35$ | $33$ | $34$ | $36$ | $32$ | $32$ | $33$ | $38$ | $40$ | $26$ | $37$ | $29$ | $36$ | $36$ | $28$
Validation | $1$ | $1$ | $3$ | $1$ | $2$ | $1$ | $1$ | $3$ | $3$ | $3$ | $3$ | $2$ | $1$ | $5$ | $2$ | $2$ | $2$ | $2$ | $2$ | $3$
Test | $4$ | $4$ | $2$ | $3$ | $4$ | $2$ | $2$ | $2$ | $3$ | $2$ | $1$ | $2$ | $2$ | $1$ | $3$ | $3$ | $4$ | $3$ | $1$ | $5$
Total | $44$ | $43$ | $34$ | $40$ | $39$ | $38$ | $36$ | $39$ | $42$ | $37$ | $36$ | $37$ | $41$ | $46$ | $31$ | $42$ | $35$ | $41$ | $39$ | $36$
Figure 6: Histogram of the numbers of object instances shown in the collected
connector dataset. The classes and the corresponding counts are shown on the
x-axis and the y-axis, respectively. Figure 7: Detection results of Faster
R-CNN [14] with different threshold values and YOLOv5 [15] with bounding boxes
around detected connectors and inferred classes on the upper left corners of
bounding boxes. The first row demonstrates the corresponding ground truth. The
second row demonstrates the Faster R-CNN [14] detection results with a
threshold of $0.1$. The third row demonstrates the Faster R-CNN [14] detection
results with a threshold of $0.5$. The last row demonstrates the YOLOv5 [15]
detection results.

##  IV EXPERIMENT SETTINGS

This study investigated a two-stage detector and a one-stage detector for
automotive wire harness connector detection. The experiment on two-stage
detection was conducted based on Faster R-CNN [14], and the one-stage
detection was achieved based on YOLO [24]. Both models were trained using the
union of the train and validation set of the collected connector dataset and
evaluated on the test set using an NVIDIA GeForce RTX 4090. The following
subsections introduce the detailed implementation of the two-stage detection
and the one-stage detection, respectively.

###  IV-A Two-Stage Detection

This study investigated the two-stage detection based on Faster R-CNN [14] and
implemented Faster R-CNN [14] with ResNet [41] plus Feature Pyramid Network
(FPN) [23] as the backbone. The overall baselines and hyper-parameters
followed Faster R-CNN [14] provided in the publicly available code of
Detectron2 [42]. Specifically, the model was trained with a learning rate of
$0.00025$ using Stochastic Gradient Descent (SGD) as the optimizer. The batch
size was $8$. The weights of the model were initiated with the pre-trained
checkpoint, faster_R-CNN_R_101_FPN_3x, provided by Detectron2 [42].

###  IV-B One-Stage Detection

YOLO [24] was selected as the backbone of the one-stage detection in the
experiment. The overall baselines and hyper-parameters of the one-stage
detection in this study followed the publicly available code of YOLOv5 [15].
Specifically, the model was trained with an initial learning rate of $0.01$
using SGD as the optimizer. The weight decay was $0.0005$, and the momentum
was $0.937$. The batch size was $16$. The weights of the model were initiated
with the pre-trained checkpoint, yolov5x, provided by YOLOv5 [15]. An early-
stop module was adopted to control the end of the training process, which
terminated the training if there was no improvement after $300$ consecutive
epochs.

##  V RESULTS AND DISCUSSION

The initialization, training, and evaluation of the two-stage detection model
based on Faster R-CNN [14] and the one-stage detection model based on YOLOv5
[15] were conducted following the experiment protocol explained in section IV.
Fig. 7 demonstrates some inference results of Faster R-CNN [14] with two
threshold values and YOLOv5 [15] as well as the corresponding ground-truth
images with original bounding boxes and labels.

There are several sub-stages of processing involved in the Faster R-CNN [14]
algorithm. In one of these sub-stages, which classifies regions of an image as
either object or background, a threshold value is required to be set by the
user to determine the confidence score needed for a region to be considered as
an object, i.e., a region is considered as background and discarded if its
confidence score is below the threshold value, otherwise, an object and
retained. In this study, two threshold values, $0.1$ and $0.5$, were set to
evaluate the Faster R-CNN-based model, whose inference results are shown in
the second and the third row of Fig. 7.

As shown in the second row of Fig. 7, all connectors are located with the
threshold value of $0.1$, but there are many detection errors in the classes
of connectors and uncertain detection of the positions of connectors in the
detection results. With the elevation of the threshold values to $0.5$, the
inference results present a more precise detection on different connectors, as
shown in the third row of Fig. 7, but some bounding boxes are excluded from
the final detection results, which left some connectors in images not
detected, leading to a deteriorated recall rate. In general, the detection
results with both threshold values demonstrate the deep learning-based two-
stage detection model’s effectiveness on the task of detection on automotive
wire harness connectors. Nevertheless, more data on connectors is desired to
train a better detection model with higher precision and recall rates. Further
study on selecting the appropriate threshold value is also critical to make
the detection more accurate and robust for practical applications.

The last row in Fig. 7 demonstrates some detection results from the one-stage
detection model based on YOLOv5 [15]. The detection results indicate the
effectiveness of the deep learning-based one-stage detection model on
connector detection. However, there are also some detection errors on the
positions and classes of connectors presented in the detection results, where
the augmentation of the dataset [43] can be helpful for better training and
inference.

TABLE II: The Precision ($\%$) of Faster R-CNN [14] with Threshold Values of $0.1$ and $0.5$ and YOLOv5 [15] Among Classes. | A0 | A1 | B0 | B1 | C
---|---|---|---|---|---
Faster R-CNN [14] ($0.1$) | 85.1 | $69.9$ | $20.2$ | $68.0$ | $28.4$
Faster R-CNN [14] ($0.5$) | $67.8$ | $43.1$ | $0.0$ | $59.7$ | $11.6$
YOLOv5 [15] | $76.4$ | 79.2 | 100.0 | 73.8 | 69.7
| D | E | F | G | H
Faster R-CNN [14] ($0.1$) | 12.6 | $16.4$ | $75.1$ | 56.4 | $50.5$
Faster R-CNN [14] ($0.5$) | 12.6 | $0.0$ | $75.1$ | 56.4 | $50.5$
YOLOv5 [15] | $0.0$ | 47.2 | 100.0 | $30.9$ | 90.7
| I | J | K | L | M
Faster R-CNN [14] ($0.1$) | $35.0$ | 60.1 | 80.1 | $45.0$ | $73.2$
Faster R-CNN [14] ($0.5$) | $0.0$ | $35.3$ | 80.1 | $45.0$ | $63.1$
YOLOv5 [15] | 88.5 | $53.0$ | $63.2$ | 85.5 | 100.0
| N | O | P | Q | R
Faster R-CNN [14] ($0.1$) | $91.1$ | $53.1$ | $76.6$ | $80.0$ | $82.9$
Faster R-CNN [14] ($0.5$) | $91.1$ | $35.6$ | $56.4$ | $80.0$ | $69.7$
YOLOv5 [15] | 91.6 | 96.4 | 94.6 | 93.0 | 96.8
TABLE III: The Mean Average Precision ($\%$) of Faster R-CNN [14] with Threshold Values of $0.1$ and $0.5$ and YOLOv5 [15]. | mAP50 | mAP50-95
---|---|---
Faster R-CNN [14] ($0.1$) | $69.4$ | $58.0$
Faster R-CNN [14] ($0.5$) | $54.4$ | $46.7$
YOLOv5 [15] | 88.5 | 82.1
Figure 8: Class A1, B1, C, D, and E with highly similar profiles. Figure 9:
Inference result by YOLOv5 (left) on class G and J, whose exteriors are highly
similar but the colors of seal rings inside are different (highlighted by red
rectangles).

Quantitatively, TABLE II and TABLE III present the rate of precision and mean
Average Precision (mAP) of the Faster R-CNN-based model with threshold values
of $0.1$ and $0.5$ and the YOLOv5-based model. In general, the YOLOv5-based
model outperforms the Faster R-CNN-based model regarding mAP on the collected
connector dataset under the experiment settings in this study. However, there
are several rates of precision in TABLE II lower than $50\%$, including the
ones of both detection model on class D and E, the ones of the Faster R-CNN-
based model on class C, and the one of the YOLOv5-based model on the class G.

By observing the exteriors of the connectors in the collected dataset, we find
that similar designs of some connectors may affect detection performance. For
example, the widths of classes A1, B1, C, D, and E are different, but their
left and right profiles are highly similar, as shown in Fig. 8, and classes G
and J have identical exteriors but different seal rings inside the connectors,
which are occluded when the images are captured from specific perspectives, as
shown in Fig. 9. These observations indicate that if some connectors share
similar exterior designs and are placed with specific poses, their
distinguishable features can be occluded, making it hard to recognize them.
Nonetheless, similar exteriors motivate two feasible strategies to relieve
this detection problem: 1) conducting further connector detection based on
multi-view images or videos; 2) re-design the exteriors of connectors with
more distinguishable features. Specifically, for the former solution, if the
inference of the class of a connector is uncertain, multi-view images of the
connector or a video capturing different views of the connector can be
acquired for further classification. And for the latter solution, changing the
design of the exteriors of connectors, for example, changing the color of the
whole connector or part of the connector, may substantially facilitate the
detection, which calls for collaboration with the manufacturers of connectors.

##  VI CONCLUSIONS AND FUTURE WORK

This study collects a dataset with twenty types of connectors commonly used on
automotive wire harnesses and trains a two-stage Faster R-CNN-based detection
model and a one-stage YOLOv5-based detection model to validate the feasibility
of deep learning-based connector detection for robotized automotive wire
harnesses assembly. The experiment results indicate the effectiveness of both
types of object detection methods and demonstrate the better performance
achieved by the one-stage YOLOv5-based model on detecting automotive wire
harness connectors but also reflect problematic detection outcomes that
require further study with other detection algorithms and more data, which
will be investigated in the future research. In addition, observations on
collected connectors motivate the problematic detection potentially affected
by the similar designs of some connectors, especially the exteriors, which
leads to future studies on multi-view image-based and video-based connector
detection as well as on new exterior designs of connectors.

## References

  * [1] K.-m. Koo, X. Jiang, K. Kikuchi, A. Konno, and M. Uchiyama, “Development of a robot car wiring system,” in _2008 IEEE/ASME International Conference on Advanced Intelligent Mechatronics_. IEEE, 2008, pp. 862–867.
  * [2] K. Koo, X. Jiang, A. Konno, and M. Uchiyama, “Development of a wire harness assembly motion planner for redundant multiple manipulators,” _Journal of Robotics and Mechatronics_ , vol. 23, no. 6, p. 907, 2011.
  * [3] X. Jiang, K.-m. Koo, K. Kikuchi, A. Konno, and M. Uchiyama, “Robotized assembly of a wire harness in a car production line,” _Advanced Robotics_ , vol. 25, no. 3-4, pp. 473–489, 2011.
  * [4] X. Jiang, Y. Nagaoka, K. Ishii, S. Abiko, T. Tsujita, and M. Uchiyama, “Robotized recognition of a wire harness utilizing tracing operation,” _Robotics and Computer-Integrated Manufacturing_ , vol. 34, pp. 52–61, 2015\.
  * [5] B. Sun, F. Chen, H. Sasaki, and T. Fukuda, “Robotic wiring harness assembly system for fault-tolerant electric connectors mating,” in _2010 International Symposium on Micro-NanoMechatronics and Human Science_. IEEE, 2010, pp. 202–205.
  * [6] L. Zhou, L. Zhang, and N. Konz, “Computer vision techniques in manufacturing,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , 2022.
  * [7] P. Di, J. Huang, F. Chen, H. Sasaki, and T. Fukuda, “Hybrid vision-force guided fault tolerant robotic assembly for electric connectors,” in _2009 International Symposium on Micro-NanoMechatronics and Human Science_. IEEE, 2009, pp. 86–91.
  * [8] P. Di, F. Chen, H. Sasaki, J. Huang, T. Fukuda, and T. Matsuno, “Vision-force guided monitoring for mating connectors in wiring harness assembly systems,” _Journal of robotics and mechatronics_ , vol. 24, no. 4, pp. 666–676, 2012\.
  * [9] T. Tamada, Y. Yamakawa, T. Senoo, and M. Ishikawa, “High-speed manipulation of cable connector using a high-speed robot hand,” in _2013 IEEE International Conference on Robotics and Biomimetics (ROBIO)_. IEEE, 2013, pp. 1598–1604.
  * [10] H.-C. Song, Y.-L. Kim, D.-H. Lee, and J.-B. Song, “Electric connector assembly based on vision and impedance control using cable connector-feeding system,” _Journal of Mechanical Science and Technology_ , vol. 31, pp. 5997–6003, 2017\.
  * [11] F. Yumbla, M. Abeyabas, T. Luong, J.-S. Yi, and H. Moon, “Preliminary connector recognition system based on image processing for wire harness assembly tasks,” in _2020 20th International Conference on Control, Automation and Systems (ICCAS)_. IEEE, 2020, pp. 1146–1150.
  * [12] H. Zhou, S. Li, Q. Lu, and J. Qian, “A practical solution to deformable linear object manipulation: A case study on cable harness connection,” in _2020 5th International Conference on Advanced Robotics and Mechatronics (ICARM)_. IEEE, 2020, pp. 329–333.
  * [13] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [14] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [15] “Yolov5,” https://github.com/ultralytics/yolov5, accessed: 2023-02-07.
  * [16] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 2, 2012.
  * [17] Z. Zou, K. Chen, Z. Shi, Y. Guo, and J. Ye, “Object detection in 20 years: A survey,” _Proceedings of the IEEE_ , 2023.
  * [18] Z.-Q. Zhao, P. Zheng, S.-t. Xu, and X. Wu, “Object detection with deep learning: A review,” _IEEE transactions on neural networks and learning systems_ , vol. 30, no. 11, pp. 3212–3232, 2019.
  * [19] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 580–587.
  * [20] ——, “Region-based convolutional networks for accurate object detection and segmentation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 38, no. 1, pp. 142–158, 2015.
  * [21] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep convolutional networks for visual recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 37, no. 9, pp. 1904–1916, 2015\.
  * [22] R. Girshick, “Fast r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1440–1448.
  * [23] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [24] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788.
  * [25] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [26] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” _arXiv preprint arXiv:2004.10934_ , 2020.
  * [27] J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7263–7271.
  * [28] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, “Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,” _arXiv preprint arXiv:2207.02696_ , 2022.
  * [29] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14_. Springer, 2016, pp. 21–37.
  * [30] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [31] H. Law and J. Deng, “Cornernet: Detecting objects as paired keypoints,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 734–750.
  * [32] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [33] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” _ACM computing surveys (CSUR)_ , vol. 54, no. 10s, pp. 1–41, 2022.
  * [34] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16_. Springer, 2020, pp. 213–229.
  * [35] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable {detr}: Deformable transformers for end-to-end object detection,” in _International Conference on Learning Representations_ , 2021. [Online]. Available: https://openreview.net/forum?id=gZ9hCDWe6ke
  * [36] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International journal of computer vision_ , vol. 88, pp. 303–308, 2009.
  * [37] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 2014, pp. 740–755.
  * [38] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, pp. 211–252, 2015.
  * [39] H. G. Nguyen, R. Habiboglu, and J. Franke, “Enabling deep learning using synthetic data: A case study for the automotive wiring harness manufacturing,” _Procedia CIRP_ , vol. 107, pp. 1263–1268, 2022.
  * [40] “Labelme,” https://github.com/wkentaro/labelme, accessed: 2023-02-16.
  * [41] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [42] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2,” https://github.com/facebookresearch/detectron2, 2019.
  * [43] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” _International journal of computer vision_ , vol. 111, pp. 98–136, 2015.


Self-Supervised Visual Representation Learning Using Lightweight Architectures

Prathamesh Sonawane1* Sparsh Drolia1* Saqib Shamsi2 Bhargav Jain2 1 Pune
Institute of Computer Technology, Maharashtra, India 2 Whirlpool Corporation
(pratt3000, sparshdrolia<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In self-supervised learning, a model is trained to solve a pretext task, using
a data set whose annotations are created by a machine. The objective is to
transfer the trained weights to perform a downstream task in the target
domain. We critically examine the most notable pretext tasks to extract
features from image data and further go on to conduct experiments on resource
constrained networks, which aid faster experimentation and deployment. We
study the performance of various self-supervised techniques keeping all other
parameters uniform. We study the patterns that emerge by varying model type,
size and amount of pre-training done for the backbone as well as establish a
standard to compare against for future research. We also conduct comprehensive
studies to understand the quality of representations learned by different
architectures.

##  1 INTRODUCTION

Self-supervised learning is a class of methods that leverage pretext tasks to
create labels automatically from the data. These labels are used to learn
representations. The representations learned from the pretext task are then
used to train the same model on some task of interest called the downstream
task. Self-supervised learning is prominently used in various tasks including
NLP(Natural Language Processing), computer vision and reinforcement learning
to learn spatial and temporal features depending on the task, data set and
model architecture.

Although we have observed significant progress in Self-Supervised learning
research, it still remains to be outperform the traditional way of training
models which is by using human annotated data sets. Although in recent
research Self-Supervised learning techniques have come close to performing on
par with Supervised training, they appear somewhat unapproachable due to the
amount of compute as well as time demanded for experimentation and research.
Despite its said drawback, it does provide a way of eliminating label
acquisition cost and reducing time required to create a data set. Moreover the
networks trained aren’t hinged on to a specific task like classification or
object detection, the models weights remain universal for a variety of
downstream tasks.

It is to these benefits that we are observing an increased amount of research
in similar fields like transfer learning, semi-supervised learning, weakly
supervised learning and unsupervised learning. In this paper we focus on self-
supervised learning. We study the most notable pretext tasks for visual
representation learning on lightweight architectures in a resource constrained
environment.

Precisely, we aim to answer the rather intriguing question, ”How well do
techniques like Rotnet Gidaris . (2018), BYOL Grill . (2020), SimCLR T. Chen .
(2020), etc., which are traditionally trained on computationally heavier
architectures like AlexNetKrizhevsky . (2012), ResNet He . (2016) learn
features when they are trained on lightweight architectures like an
EfficientNet-lite0 Tan Le (2020)?” We conduct a comparative analysis across
various techniques with a controlled set of hyper parameters, and see how they
perform in a low resource setting. We hope to establish a standard against
which future research can be conducted and evaluated prematurely. Our aim is
to establish a standard for comparison before dedicating huge chunks of time
and capital in order to perform pretext tasks on bigger architectures and
consequently bigger data sets. Additionally in our study we also found that
lightweight architectures could achieve comparable accuracies with
significantly less carbon footprints(Tab. 1).

To measure the computation complexity we have not only used the number of
Floating Point Operations (FLOPs) performed by each model, as it is an
indirect metric, but also have measured amount of training time, which is a
more direct measure Ma . (20181). You can find all of the code for our
experiments here.[Once the the paper gets accepted we will release the link to
our code base here]

##  2 RELATED WORK

With the success of deep neural networks, a lot of tasks can be solved very
well by collecting a labelled data set and using supervised learning. In order
to perform well, the models usually need a large corpus of labeled examples.
However, getting labels for data turns out to be expensive and scaling it up
is a major challenge. With the vast amount of unlabeled data being generated
in the form of text, images, videos, everyday, the goal of self-supervised
learning is to get supervision from the data itself to learn useful
representations instead of using explicit labels. Once a good representation
is learned on a pretext task, the model can be fine tuned on a downstream task
with comparatively fewer data than what would have been required were the
model trained from scratch on the downstream task.

The general nature of self-supervised learning allows it to be used across
different modalities including images, text, video, audio and even in
robotics. The use of context to predict words Mikolov . (2013); Pennington .
(2014); Howard Ruder (2018); Devlin . (2019) is a popular technique in the
domain of natural language processing. In the similar vein, the temporal
information in video can also be leveraged to learn representations Misra .
(20161, 20162). One could also use different modalities in videos for the same
Sayed . (2018); Owens Efros (2018).

There has been a lot of work over the years in the field of visual
representation learning. Doersch et al. Doersch . (2015) proposed a patch
based method to predict the placement of patches relative to each other within
an image. They motivated a line of patch based methods such as the ”jigsaw
puzzle” based task Noroozi Favaro (2016), where they used nine patches from
the full image instead of just two patches to make the pretext task more
challenging. There have been more works following the two Noroozi . (2018,
2017).

In contrast to patch based methods, there have been other methods that used
the image information to create a classification pretext task. Notable
examples of these include RotNet Gidaris . (2018), where the authors rotate
the images in multiples of 90 degrees and the model performs a 4-way
classification predicting the angle the image was rotated by. Another class of
methods have focused on generative modeling to learn representations from
images. Researchers have relied on tasks like predicting a subset of channels
of the image from another subset of the same image such as in a colorization
pretext task Zhang . (2016) and its improvement Split-Brain Auto Encoders
Zhang . (2017). Pathak et al. used image in painting as a pretext task for
self-supervised learning Pathak . (2016).

Recently there has been a rise in the use of contrastive methods which have
performed extremely well in this domain. Contrastive methods aim to learn a
model which transforms an input into an embedding vector such that examples
from the same class have similar embeddings relative to embeddings of samples
from different classes. SimCLR T. Chen . (2020) is a framework which learns
representations by maximizing the agreement between different augmented views
of the same input in a latent space. Barlow Twins Zbontar . (2021) method
learns to make the cross-correlation matrix between the two distorted versions
of the same image close to the identity matrix. MOCO He . (2020) and MOCO-v2
X. Chen . (2020) rely on a framework of representation learning from images as
a dynamic dictionary look-up. BYOL Grill . (2020) aims to learn a
representation using two neural networks, which are referred to as online and
target networks, without the use of negative samples. The two networks have
the same architecture, with the target network having polyak averaged weights.

There has been a lot of focus and effort from the research community on coming
up with new pretext tasks. We aim to take a complementary approach in the
self-supervised research landscape by evaluating various pretext tasks on low
resource architectures like MobileNetv2 Sandler . (2018), ShuffleNetv2 Ma .
(20182), SqueezeNet Iandola . (2016) and EfficientNetLite0 Tan Le (2020) and
compare and contrast the performance on a computationally expensive
architecture like a ResNet He . (2016). Our work is similar to the study by
Kolesnikov et al. Kolesnikov . (2019) where they investigate how architectural
choices affect the performance of various pretext tasks for visual
representation learning. However, while they use different variants of
Residual Networks He . (2016), which are computationally expensive
architectures, the aim of our work is to examine how self-supervised visual
representation learning fares on computationally lighter architectures.

Table 1: Max validation accuracy achieved for downstream classification task on different pretext tasks and backbone architectures. C.T. refers to contrastive techniques. The column mentions if the respective technique is contrastive or not. CO2 Gen. is CO2 generated, as in the carbon emitted in kg CO2 equivalent for training on average for the respective architecture. Val. Acc. | Efficientnetlite0 | Mobilenetv2 | ResNet-18 | Shufflenetv2 | Squeezenet | . | C.T.
---|---|---|---|---|---|---|---
Rotnet | 0.28 | 0.31 | 0.33 | 0.37 | 0.30 |  | No
MOCOv2 | 0.47 | 0.51 | 0.78 | 0.65 | 0.10 |  | Yes
Split Brain | 0.58 | 0.57 | 0.59 | 0.53 | 0.45 |  | No
SimCLR | 0.69 | 0.65 | 0.70 | 0.59 | 0.60 |  | Yes
BYOL | 0.51 | 0.71 | 0.78 | 0.71 | 0.68 |  | Yes
. |  |  |  |  |  |  |
Avg. FLOPs | 3.90E+08 | 3.27E+08 | 1.80E+09 | 4.01E+07 | 3.20E+08 |  |
CO2 Gen. | 1.9 | 1.8 | 3.22 | 1.84 | 1.99 |  |

##  3 EXPERIMENTAL SETUP

###  3.1 Dataset

We have used STL-10 Coates . (2011) for all of our experiments. STL-10 is made
up of a subset of images from the ImageNet Deng . (2009) . It is composed of
labeled and unlabeled subsets. The first set consists of $10$ classes with
$500$ training and $800$ validation RGB images per class. The images are of
size $96\times 96$. These are used for downstream training and evaluation. The
second set consists of $100,000$ unlabelled RGB images of size $96\times 96$,
which are used for pretext task training.

(a) Split-Brain Auto Encoders

(b) SimCLR

(c) Rotnet

(d) MOCOv2

(e) BYOL

Figure 1: The graphs represent max validation accuracies achieved on
downstream classification task with increased amount of pretraining done(in
intervals of 10 epochs) on different architectures for every technique. Each
graph represents one among 5 techniques evaluated. (x-axis : number of epochs
for which the model was pre-trained. y-axis : Max validation accuracy achieved
on downstream classification task.)

###  3.2 Architectures

####  3.2.1 ResNet-18

To compare the results of the different techniques against a computationally
expensive architecture we train a ResNet-18 He . (2016) on all of the
techniques separately. The network consists of an initial $7\times 7$
convolutional layer followed by a max pooling operation. The network then
consists of $16$ $3\times 3$ convolutions and ReLU non-linearity. Down
sampling is performed by a strided convolutions instead of pooling layers. The
network has $1.8$ GFLOPs (multiply-adds).

####  3.2.2 Mobilenet-v2

We chose Mobilenet-v2 Sandler . (2018) as one of the lightweight architectures
for this study. Mobilenet-v2 is built using inverted residual blocks with
shortcut connections between thin bottleneck layers. Its design includes an
initial fully convolutional layer with 32 filters, followed by 19 residual
bottleneck layers. The size of all the kernels in all of the convolution
layers is $3\times 3$

####  3.2.3 Efficientnet-lite0

We compared the performance of Efficientnet-lite0 Tan Le (2020) as it is one
of the modern state-of-the-art architectures. It is a smaller version of
EfficientNet Tan Le (2020) and uses compound scaling to uniformly scale depth
$\alpha$, width $\beta$, image size $\gamma$. Neural architecture search is
used to get the baseline model and bigger architectures are obtained by
scaling this up. It consist of convolutions followed by inverted residual
blocks found in MobileNet-v2.

####  3.2.4 Squeezenet

We also trained Squeezenet Iandola . (2016) in addition to the architectures
mentioned above on all the techniques separately. It is composed of fire
modules that further consists of a squeezed convolution layer (has only
$1\times 1$ filters), that feeds into an expand layer (that has a mix of
$1\times 1$ and 3×3 convolution filters). The architectures is constructed
using a convolution layer, followed by $9$ fire modules, followed by a
convolution and a Softmax unit at the end. The down sampling in SqueezeNet is
performed by pooling layers.

####  3.2.5 Shufflenet-v2

Shufflenet-v2 Ma . (20182) was our final lightweight architecture for this
study. Shufflenet-v2 unit is a residual block. In its residual branch, for the
$3\times$3 layer, a computational economical $3\times$3 depth wise convolution
on the bottleneck feature map is applied. Then the first $1\times$1 layer is
replaced with point wise group convolution followed by a channel shuffle
operation, to form a Shufflenet unit. For example, given the input size c x h
x w and the bottleneck channels m, Shufflenet unit requires only h*w(2c*m/g +
9m) FLOPs, where g means the number of groups for convolutions.

###  3.3 Techniques

####  3.3.1 Rotnet

The authors proposed rotating an image pseudo-randomly by a discrete set of
angles Gidaris . (2018). The angle is treated as the label and the rotated
image as the input. In the paper they tested rotating images in 45, 90 and 180
degree intervals separately. The best results were observed for 90 degree
interval set and the worst results were for 180 degree interval set. We used
90 degree intervals as our labels for training since they achieved the highest
accuracy in the paper.

####  3.3.2 MOCOv2

The authors introduced a contrastive learning approach – momentum contrast
(MOCO) X. Chen . (2020). The key ideas of this method are: 1) implementation
of a queue as the dictionary which stores plethora of keys; 2) updating the
key encoder using the momentum update from the query encoders(without the need
of passing the batch through key encoder). Compared with previous contrastive
learning methods based on memory bank and end-to-end learning, MOCOv2 not only
supports a large negative sample size but also maintains a consistent key
encoding.

####  3.3.3 SimCLR

The authors introduced a framework T. Chen . (2020) for contrastive learning
of visual representations. SimCLR learns representation by maximising
similarity of two differently augmented data set of the same image. It uses a
stochastic data augmentation module that randomly modifies a given data
instance, producing two correlated images of the same example, which is
regarded a positive pair. SimCLR uses three basic augmentations in sequence:
random cropping followed by resizing to the original size, random colour
distortions, and random Gaussian blur. The authors believe that random
cropping and colour distortion are critical to achieving high performance.

(a) Efficientlite0

(b) Mobilenetv2

(c) ResNet-18

(d) Shufflenet

(e) Squeezenet

Figure 2: The graph represents max validation accuracies achieved on
downstream classification task with the increase in pretraining done on
different techniques for each architecture. Each graph represents one among 5
architectures evaluated.(x-axis : number of epochs for which the model was
pre-trained. y-axis : Max validation accuracy achieved on downstream
classification task.)

####  3.3.4 BYOL

BYOL Grill . (2020) is a contrastive learning approach which uses two encoder
networks with the same architecture referred to as online and target networks
to learn representations and reduces the contrastive loss between the
representations learned by the two networks. Unlike other contrastive learning
BYOL does not use any negative samples. The output of the network is iterated
so as to serve as a target for enhanced representation. BYOL trains its online
network using an augmented view of an image and predicts the target network’s
representation of another augmented view of the same image. They use ResNet-50
as an encoder network. For the projection MLP, a 2048 dimensional feature
vector is projected onto 4096-dimensional vector space first with a sub-
network composed of batch norm followed by ReLU non-linear activation. The
resulting vector is then reduced to a 256-dimensional feature vector. The same
architecture is used for the predictor network.

####  3.3.5 Split-Brain Auto encoders

Split-Brain Auto encoders are a spin on the traditional auto encoder
architecture Zhang . (2017). The method adds a split to the network thereby
creating two sub-networks. The image is divided into two subsets of channels.
Each sub-network predicts one subset of channels from the other subset. For
this study, we use the Lab space and divide the image into perceptual
lightness L and color ab channels. Both sub-networks are trained for
classification using a cross-entropy objective. When predicting L from ab, the
output space is quantized into 50 bins of size 2. When predicting ab from L,
the quantized output space is binned into 313 bins of size 100.

##  4 EXPERIMENTS

We evaluated the performance of five of the most cited techniques: Rotnet,
MOCOv2, Split-Brain Auto encoders, SimCLR and BYOL across five different
architectures: Efficientnetlite0, Mobilenetv2, ResNet-18, Shufflenetv2 and
Squeezenet. All of them are relatively smaller architectures, except for
ResNet-18 which acts as a point of comparison of the performance of the four
lightweight architectures. All of the techniques and architectures have been
described in Section 3 above.

Every architecture was trained on the pretext task, using the unlabelled
subset of images of STL-10 until convergence. We saved the weights of the
network every 5 epochs during pretraining. These checkpoints were later used
to evaluate performance on the downstream classification task to evaluate the
performance as a function of the amount of pre-training the networks were
exposed to for the pretext task.

We used categorical crossentropy loss for the downstream training and Adam
Kingma Ba (2015) as the optimizer. For pretext tasks all of the configuration
was kept as stated in their respective papers, only the model architectures
were replaced.

To inspect the quality of the learned representations and understand what part
of the images most affect the outcome of predictions made by the models, we
used Saliency maps Simonyan . (2014). Saliency maps show us the degree of
importance of each pixel an image, in the visual field. Our approach to this
was to find the best true positives and false negatives for each experiment
and then examining those for any observable trends.

Another technique we used to understand the models’ performance is K-Nearest
Neighbours search (kNN) Peterson (2009). We use it to find the closest image
representations in space of representations learned by the models. We use
Euclidean distance as our distance measure.

(a) Bird

(b) Horse

(c) Aeroplane

(d) Ship

(e) Car

(f) Saliency heat map scale. Blue showing the least focused regions and red
showing the most focused regions

Figure 3: Saliency maps for BYOL using Mobilenetv2’s backbone. Although only a
few examples have been shown, These Saliency maps represent a trend which we
observed in most of our experiments. In Fig 3(a) we can observe that the model
focuses on the surrounding rather than the bird and hence throughout our
experiments bird images were the most misclassified. In Fig 3(b) we can
observe that the model focuses on the horse with a decent regard for its
surrounding as well, this must have been the reason that horse images were
classified correctly in most instances. In Fig 3(c) and Fig 3(d) we can
observe that the model focuses primarily on the blue pixels, and thus ends up
confusing between planes and ships in practice. In Fig 3(e) we can see that
the model is focusing on the red pixels and hence we get an explanation for
why red coloured cars and trucks are misclassified interchangeably by models

##  5 Results

###  5.1 General

As is evident from figures 1 Resnet-18 is the best performer in Fig 1(b), Fig
1(d) and Fig 1(e), with the different in performance increasing with the
amount of pretraining. This was expected as it is significantly heavier than
all architectures, . The highest validation accuracy in all our experiments
was achieved using ResNet-18, twice (Table 1). For Split-Brain Auto encoders
and Rotnet, ResNet-18 has the second best performance. Split-Brain appears to
be unstable when trained on ResNet-18 (as we can see from Fig 1(a)). For
Rotnet, after 100 epochs of pretraining all architectures have about the same
validation accuracy (Fig 1(c)), so it is unclear as to which is the best
performing architecture. Despite all this it is safe to say that ResNet-18
outperforms lighter models overall. On the other hand we can also observe in
Fig 1 that Squeezenet appears consistently in the worst 2 performing models.

Another key take away from Fig 1(d) and Fig 1(e) is that while Shufflenet has
trouble learning with less pretraining, it eventually overtakes the other
architectures as the amount of pretraining increases. Hence, while
Shufflenetv2 has a slower learning pace, with respect to the pretraining done,
it learns better feature representations over time, unlike the other 4
architectures.

We can observe from Fig 2 that BYOL is the best performing technique for 4 out
5 model architectures. From fig 1 we can observe that it also is the most
stable among other techniques.

Split-Brain Auto Encoders and Rotnet require relatively less pretraining to
reach their max validation accuracies, regardless of the architecture ( Fig
1(a) and Fig 1(c)). Rotnet is stable while doing so but the same cant be said
about Split-Brain Autoencoders.

Another observation is that excessive pre-training didn’t have any detrimental
or supplemental effects on performance during the downstream task.

###  5.2 Saliency Maps and kNN

We performed K nearest neighbors search for each architecture and technique.
One common trend we observed was that, most of the techniques misclassified
Aeroplanes as Ships (Fig 4(a), Fig 4(b)). This could be attributed to the
prevalence of the blue colored pixels in both of images. Moreover this
assumption is strengthened by visualization of such images through Saliency
maps(Fig 3(c)), which showed that most models classified those images
primarily based off of the blue pixels rather than the object.

Another observation is that images with substantial amount of red pixels are
directly classified as Trucks by most models (Fig 4(e), Fig 4(f)). This may be
due to skewed data in case of the Truck class and a general lack of other red
colored objects. Subsequently, in most techniques the truck label was wrongly
predicted. The Saliency map of a truck in Fig 3(e) further supports this idea.
We observe that the model classifies the image as a truck primarily based off
of the red colored section.

Models were most consistently accurate while predicting horse images and least
consistently accurate while predicting bird images(Fig 4(g), Fig 4(h), Fig
4(c), Fig 4(d)). For instance, we observed from Fig 3(b) that the model
focused on the horse with some regard for its surrounding. On the contrary, in
Fig 3(a) we observe that there is very little focus on the bird and its
classifying the image more on the basis of its surrounding. These observations
were a good representation of the many other observations we made during our
experiments with Saliency Maps and KNNs.

##  6 Assumptions and Limitations

###  6.1 Architectures without skip connections

After conducting most of our experiments we realized that all the
architectures used for evaluation had intrinsically incorporated skip
connections. Conducting experiments on architectures without skip connections
would have resulted in a more comprehensive study.

###  6.2 Heavier Architectures

Our experiments have 4 light weight architectures and 1 moderately heavy
architecture, but evaluation on even bigger architectures (Ex. ResNet-101)
could have provided a more concrete comparison benchmark. We were unable to
conduct these experiments due to the high compute requirement as well as the
huge amount of time required to conduct a single experiment across all the
tasks.

###  6.3 Dataset

We used a single data set for all of our experiments. While this removed the
chance of inherent variation in results due to varying data it also limited
our observation scope. Conducting all the experiments with a different data
set could have helped in more concretely establishing all the model
performances.

##  7 CO2 Emission Related to Experiments

Experiments were conducted using Google Cloud Platform in region asia-
southeast1, which has a carbon efficiency of 0.42 kgCO2eq/kWh. A cumulative of
250 hours of computation was performed on hardware of type Tesla P100 (TDP of
250W).

Total emissions are estimated to be 35 kgCO2eq of which 100 percents were
directly offset by the cloud provider.

Estimations were conducted using the Machine Learning Impact calculator
presented in Lacoste . (2019).

(a) Plane

(b) nearest neighbour results for Shufflenetv2 trained with BYOL as the
pretext task.

(c) Bird

(d) Nearest neighbour results SqueezeNet trained with BYOL as the pretext
task.

(e) Truck

(f) Nearest neighbour results for Mobilenetv2 trained with BYOL as the pretext
task.

(g) Horse

(h) Nearest neighbour results for Efficientnetlite0 trained with BYOL as the
pretext task.

Figure 4: The figure shows query image (left) with mapped images (right) whose
feature vectors have the least Euclidean distances from the query image. The
Euclidean distance increases from left to right. The KNN search performed in
above images is for the best performing pretext technique, BYOL, but similar
trends were observed in other techniques too. In Fig 4(a), Fig 4(b) we can
observe that the model groups together images with blue pixels and thus tends
to misclassify boats and aeroplanes. Evidence for this can be seen from Fig
Fig 3(c) and Fig 3(c) where the model is classifying those images on the basis
of blue regions. In Fig 4(c) and Fig 4(d) we can observe that the mapped
images appear somewhat random. This may be due to the fact that in Fig 3(a)
the model isn’t focusing on the bird but vaguely on the background. In Fig
4(e) and Fig 4(f) we can see that that the model group together images of cars
as well as trucks probably due to the red pixels. From Fig 3(e) we can observe
that the model focuses in the red colored pixels only. In Fig 4(g) and 4(h) we
can observe that the model has group together all horse images accurately,
this may be a result of the model focusing on the horse as we can see in Fig
3(b).

##  8 Conclusion

In conclusion we found that while having a heavier architecture does help,
having a good pretext technique has a bigger impact on the performance of
models. In our experiments BYOL was the best performing technique and Rotnet
was consistently among the worst performing techniques. Contrastive techniques
performed better than non-contrastive techniques most of the times even in the
realm of lightweight architectures.

Aeroplanes ans Ships are mis-labelled as each other because the models focus
on the blue pixels in the image rather than the object. Similarly Cars and
Trucks are mis-labelled as each other because the model focuses on the red
pixels. This could be tackled by using a more diverse data set. Bird images
are the most misclassified and horse images are the most successfully
classified this was attributed to weight of focus on the object and
environment.

##  9 Future Work

Some things that could be explored in the future are using more diverse and
balanced datasets. Experimenting on other lightweight architectures such as
condensenet Huang . (2018), ThiNet Luo . (2017) and the others can also be
explored. Computationally heavier architectures than ResNet-18, such as
ResNet-50, ResNet-101, DenseNet-121 and the others can also be used to gauge
the performance of various pre-text tasks. Also, incorporating more self-
supervised techniques would be one of the important goals. Finally, a more
diverse set of downstream tasks like object detection using the pretext task
trained models as backbones would enable a fairer study of the effectiveness
of architectural and method choices.

## References

  * T. Chen . (2020) SimCLRChen, T., Kornblith, S., Norouzi, M. Hinton, G.  2020\.  A Simple Framework for Contrastive Learning of Visual Representations A simple framework for contrastive learning of visual representations.
  * X. Chen . (2020) mocov2Chen, X., Fan, H., Girshick, RB. He, K.  2020\.  Improved Baselines with Momentum Contrastive Learning Improved baselines with momentum contrastive learning.  ArXivabs/2003.04297.
  * Coates . (2011) STL10Coates, A., Ng, A. Lee, H.  2011\.  An Analysis of Single-Layer Networks in Unsupervised Feature Learning An analysis of single-layer networks in unsupervised feature learning.  AISTATS. Aistats.
  * Deng . (2009) imagenetDeng, J., Dong, W., Socher, R., Li, LJ., Li, K. Fei-Fei, L.  2009\.  ImageNet: A large-scale hierarchical image database Imagenet: A large-scale hierarchical image database.  2009 IEEE Conference on Computer Vision and Pattern Recognition248-255.
  * Devlin . (2019) bertDevlin, J., Chang, MW., Lee, K. Toutanova, K.  2019\.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Bert: Pre-training of deep bidirectional transformers for language understanding.  NAACL. Naacl.
  * Doersch . (2015) patches_sslDoersch, C., Gupta, A. Efros, AA.  2015\.  Unsupervised Visual Representation Learning by Context Prediction Unsupervised visual representation learning by context prediction.  2015 IEEE International Conference on Computer Vision (ICCV)1422-1430.
  * Gidaris . (2018) RotnetGidaris, S., Singh, P. Komodakis, N.  2018\.  Unsupervised Representation Learning by Predicting Image Rotations Unsupervised representation learning by predicting image rotations.  ICLR. Iclr.
  * Grill . (2020) BYOLGrill, JB., Strub, F., Altch’e, F., Tallec, C., Richemond, PH., Buchatskaya, E.Valko, M.  2020\.  Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning Bootstrap your own latent: A new approach to self-supervised learning.  ArXivabs/2006.07733.
  * He . (2020) mocoHe, K., Fan, H., Wu, Y., Xie, S. Girshick, RB.  2020\.  Momentum Contrast for Unsupervised Visual Representation Learning Momentum contrast for unsupervised visual representation learning.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)9726-9735.
  * He . (2016) resnetHe, K., Zhang, X., Ren, S. Sun, J.  2016\.  Deep Residual Learning for Image Recognition Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)770-778.
  * Howard Ruder (2018) ulmfitHoward, J. Ruder, S.  2018\.  Universal Language Model Fine-tuning for Text Classification Universal language model fine-tuning for text classification.  ACL. Acl.
  * Huang . (2018) condensenetHuang, G., Liu, S., Van der Maaten, L. Weinberger, KQ.  2018\.  Condensenet: An efficient densenet using learned group convolutions Condensenet: An efficient densenet using learned group convolutions.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Proceedings of the ieee conference on computer vision and pattern recognition ( 2752–2761).
  * Iandola . (2016) squeezenetIandola, FN., Moskewicz, MW., Ashraf, K., Han, S., Dally, WJ. Keutzer, K.  2016\.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and ¡1MB model size Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ¡1mb model size.  ArXivabs/1602.07360.
  * Kingma Ba (2015) adamKingma, DP. Ba, J.  2015\.  Adam: A Method for Stochastic Optimization Adam: A method for stochastic optimization.  CoRRabs/1412.6980.
  * Kolesnikov . (2019) revisiting_sslKolesnikov, A., Zhai, X. Beyer, L.  2019\.  Revisiting Self-Supervised Visual Representation Learning Revisiting self-supervised visual representation learning.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)1920-1929.
  * Krizhevsky . (2012) alexnetKrizhevsky, A., Sutskever, I. Hinton, GE.  2012\.  ImageNet classification with deep convolutional neural networks Imagenet classification with deep convolutional neural networks.  Communications of the ACM6084 - 90.
  * Lacoste . (2019) lacoste2019quantifyingLacoste, A., Luccioni, A., Schmidt, V. Dandres, T.  2019\.  Quantifying the Carbon Emissions of Machine Learning Quantifying the carbon emissions of machine learning.  arXiv preprint arXiv:1910.09700.
  * Luo . (2017) thinetLuo, JH., Wu, J. Lin, W.  2017\.  ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression Thinet: A filter level pruning method for deep neural network compression.  2017 IEEE International Conference on Computer Vision (ICCV)5068-5076.
  * Ma . (20181) FLOPSvsTIMEMa, N., Zhang, X., Zheng, H. Sun, J.  20181\.  ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design Shufflenet v2: Practical guidelines for efficient cnn architecture design.  ArXivabs/1807.11164.
  * Ma . (20182) shufflenetv2Ma, N., Zhang, X., Zheng, H. Sun, J.  20182\.  ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design Shufflenet v2: Practical guidelines for efficient cnn architecture design.  ArXivabs/1807.11164.
  * Mikolov . (2013) word2vecMikolov, T., Chen, K., Corrado, GS. Dean, J.  2013\.  Efficient Estimation of Word Representations in Vector Space Efficient estimation of word representations in vector space.  ICLR. Iclr.
  * Misra . (20161) shuffle_and_learnMisra, I., Zitnick, CL. Hebert, M.  20161\.  Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification Shuffle and learn: Unsupervised learning using temporal order verification.  ECCV. Eccv.
  * Misra . (20162) arrow_of_timeMisra, I., Zitnick, CL. Hebert, M.  20162\.  Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification Shuffle and learn: Unsupervised learning using temporal order verification.  ECCV. Eccv.
  * Noroozi Favaro (2016) jigsawNoroozi, M. Favaro, P.  2016\.  Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles Unsupervised learning of visual representations by solving jigsaw puzzles.  ECCV. Eccv.
  * Noroozi . (2017) learning_to_countNoroozi, M., Pirsiavash, H. Favaro, P.  2017\.  Representation Learning by Learning to Count Representation learning by learning to count.  2017 IEEE International Conference on Computer Vision (ICCV)5899-5907.
  * Noroozi . (2018) jigsaw++Noroozi, M., Vinjimoor, A., Favaro, P. Pirsiavash, H.  2018\.  Boosting Self-Supervised Learning via Knowledge Transfer Boosting self-supervised learning via knowledge transfer.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition9359-9367.
  * Owens Efros (2018) multimodalOwens, A. Efros, AA.  2018\.  Audio-Visual Scene Analysis with Self-Supervised Multisensory Features Audio-visual scene analysis with self-supervised multisensory features.  ECCV. Eccv.
  * Pathak . (2016) inpaintingPathak, D., Krähenbühl, P., Donahue, J., Darrell, T. Efros, AA.  2016\.  Context Encoders: Feature Learning by Inpainting Context encoders: Feature learning by inpainting.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)2536-2544.
  * Pennington . (2014) glovePennington, J., Socher, R. Manning, CD.  2014\.  GloVe: Global Vectors for Word Representation Glove: Global vectors for word representation.  EMNLP. Emnlp.
  * Peterson (2009) KNNPeterson, LE.  2009\.  K-nearest neighbor K-nearest neighbor.  Scholarpedia41883.
  * Sandler . (2018) mobilenetv2Sandler, M., Howard, AG., Zhu, M., Zhmoginov, A. Chen, LC.  2018\.  MobileNetV2: Inverted Residuals and Linear Bottlenecks Mobilenetv2: Inverted residuals and linear bottlenecks.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition4510-4520.
  * Sayed . (2018) cross_and_learnSayed, N., Brattoli, B. Ommer, B.  2018\.  Cross and Learn: Cross-Modal Self-Supervision Cross and learn: Cross-modal self-supervision.  ArXivabs/1811.03879.
  * Simonyan . (2014) SaliencySimonyan, K., Vedaldi, A. Zisserman, A.  2014\.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps Deep inside convolutional networks: Visualising image classification models and saliency maps.  CoRRabs/1312.6034.
  * Tan Le (2020) effnetliteTan, M. Le, QV.  2020\.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Efficientnet: Rethinking model scaling for convolutional neural networks.
  * Zbontar . (2021) barlow_twinsZbontar, J., Jing, L., Misra, I., LeCun, YA. Deny, S.  2021\.  Barlow Twins: Self-Supervised Learning via Redundancy Reduction Barlow twins: Self-supervised learning via redundancy reduction.  ICML. Icml.
  * Zhang . (2016) colorful_colorizationZhang, R., Isola, P. Efros, AA.  2016\.  Colorful Image Colorization Colorful image colorization.  ECCV. Eccv.
  * Zhang . (2017) SplitBrainAEZhang2017Zhang, R., Isola, P. Efros, AA.  2017\.  Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction Split-brain autoencoders: Unsupervised learning by cross-channel prediction.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)645-654.