# An evaluation of pre-trained models for feature extraction in image
classification

Erick da Silva Puls
Institute of Informatics
UFRGS
Porto Alegre, Brazil
<EMAIL_ADDRESS>
& Matheus V. Todescato
Institute of Informatics
UFRGS
Porto Alegre, Brazil
<EMAIL_ADDRESS>
& Joel L. Carbonera
Institute of Informatics
UFRGS
Porto Alegre, Brazil
<EMAIL_ADDRESS>

###### Abstract

In recent years, we have witnessed a considerable increase in performance in
image classification tasks. This performance improvement is mainly due to the
adoption of _deep learning techniques_. Generally, deep learning techniques
demand a large set of annotated data, making it a challenge when applying it
to small datasets. In this scenario, _transfer learning_ strategies have
become a promising alternative to overcome these issues. This work aims to
compare the performance of different pre-trained neural networks for _feature
extraction_ in image classification tasks. We evaluated 16 different pre-
trained models in four image datasets. Our results demonstrate that the best
general performance along the datasets was achieved by CLIP-ViT-B and
ViT-H-14, where the CLIP-ResNet50 model had similar performance but with less
variability. Therefore, our study provides evidence supporting the choice of
models for _feature extraction_ in image classification tasks.

_Keywords_ Machine learning $\cdot$ Neural networks $\cdot$ Image
classification $\cdot$ Transfer learning $\cdot$ Feature extraction

##  1 Introduction

The rapid advancements in technology in the last decades have pushed
organizations to produce and accumulate all kinds of data. In the past,
critical organizational information was primarily represented by structured
data stored in databases. However, nowadays a significant part of this
information is represented in an unstructured way, for instance as images [1].

There is a need to develop approaches capable of recovering and evaluating
images in applications of several fields [1]. In that sense, one of the
challenges concerning image recovery is that the semantic content of images is
not apparent, so this information is not easily acquired through direct
queries. An alternative for recovering images is annotating them first [2] in
a way that allows us to retrieve them by querying for the annotations.
However, it is necessary to bear in mind that manual annotation of large
databases of images is time-consuming and impractical. In this context,
machine learning can be used to automatically classify these large databases
of images, thus enabling retrieval through direct queries.

Image classification (IC) tasks aim to classify the image as a whole by
assigning a specific label to it. Usually, labels in an IC task refer to
objects that appear in the image, kinds of images (photographs, drawings,
etc.), feelings (sadness, happiness, etc.), etc [3].

Most of the recent approaches for IC are based on deep neural network (DNN)
architectures. These architectures usually demand a large set of annotated
data, making it challenging to apply _deep learning_ when small amounts of
data are available. In this scenario, _transfer learning_ strategies have
become a promising alternative to overcome these issues. One of the main
alternatives of _transfer learning_ is through _feature extraction_ , where
models that were trained on large datasets can be used for producing
informative features that can be used by another classifier. By using
_transfer learning_ , we can leverage knowledge previously learned by neural
network models on a large dataset and use this knowledge in a context where
just small datasets are available.

There are currently several large datasets available, such as Imagenet [4],
and a range of models that were pre-trained on these datasets 111Some pre-
trained models can be found in https://pytorch.org/vision/stable/models.html.
The literature suggests that particular tasks on distinct datasets can benefit
from different pre-trained models [5, 6].

It is important to notice that there are different approaches for transfer
learning for image classification, such as _fine-tuning_ and _feature
extraction_. When adopting _fine-tuning_ , a neural network that was pre-
trained in a big dataset is retrained in a novel task, for which usually only
a small dataset is available. The goal, in this approach, is to use the
knowledge (represented by the weights of the model) acquired in the first
training process as a starting point for the training in the second task, and
the weights of the pre-trained model are updated during the training in the
target task. In the case of _feature extraction_ , the pre-trained model is
used just for extracting features that represent the images and that can be
used as input for a classifier. Notice that in this approach the pre-trained
model is kept frozen, that is, their weights are not updated during the
training of the classifier used in the target task. Some studies [7, 8]
comparing fine-tuning and feature extraction demonstrate that fine-tuning
achieves higher performance, but the results also suggest that feature
extraction achieves a comparable performance while requiring fewer
computational resources for training. In this context, the main goal of this
work is to compare and evaluate the performance of _feature extraction_ (FE)
of various pre-trained models in the task image classification.

In this study, Geological Images [9, 10, 11], Stanford Cars [12], CIFAR-10
[13], and STL10 [14] are the datasets adopted for analyzing the performance of
FE of the following pre-trained models: AlexNet, ConvNeXt Large, DenseNet-161,
GoogLeNet, Inception V3, MNASNet 1.3, MobileNet V3 Large, RegNetY-3.2GF,
ResNeXt101-64x4D, ShuffleNet V2 X2.0, SqueezeNet 1.1, VGG19 BN,
VisionTransformer-H/14, Wide ResNet-101-2, and both CLIP-ResNet50, and CLIP-
ViT-B. We evaluate the performance of the considered pre-trained models using
these metrics: accuracy, macro F1-measure, and weighted F1-measure.

Our results indicate that the pre-trained models CLIP-ResNet50, CLIP-ViT-B,
and VisionTransformer-H/14 had significantly better performance than the other
considered pre-trained models for all datasets. It is important to notice that
these are the only three models among those considered in our experiments that
include transformers in their architecture, while the others are based solely
on CNN architectures. Our analysis also indicates that there are differences
regarding the pattern of performances of these three transformer-based
architectures in comparison with the performances of the CNN-based
architectures across the datasets, in all the considered metrics. These
differences become evident when we analyze the Pearson correlation in Section
3.4. Moreover, our analysis suggests that the Stanford Cars dataset is the
most challenging of all datasets analyzed. We hypothesize that it is due to
its large number of classes, few samples per class, and the inclusion of
images with different sizes and features at different scales.

The remainder of this paper is structured as the following. In Section 2 we
discuss the related work. In Section 3 we present our experiments and discuss
our results. Finally, Section 4 presents the conclusions.

##  2 Related Works

The TL approach based on FE has been adopted for IC in several domains, such
as Biomedicine [15] and Geology [16, 17, 18]. In this work, we reviewed the
literature covering the last five years and focused on comparing the
performance of FE for different pre-trained models. In total, we selected
eighteen papers for our literature overview, of which five are specific to IC
in Geology. In our literature review, twenty-three different models were
found, among which the most recurrent pre-trained models were the VGG16 and
the Inception V3, both for studies in general and those specific to Geology,
and the third most used model was the AlexNet. Of the eighteen articles, only
two did not use the ImageNet dataset for pre-training the models. Therefore,
we can observe that ImageNet is one of the most used datasets for pre-training
models for IC.

There is a range of pre-trained models that can be applied for transfer
learning. The main expected result of FE from the pre-trained models is to
improve classification quality. The size and similarity of the target dataset
and the source task can be used to choose the pre-trained model [19].

The literature suggests that each dataset may need a different pre-trained
model. For instance, for plankton classification [20], when adopted as a
feature extractor the Inception V3, AlexNet, VGG16, VGG19, ResNet50,
ResNet101, DenseNet-161, and GoogLeNet pre-trained models with three different
datasets, the DenseNet-161 model presented the best results. On the other
hand, when classifying pathological brain images, Kaur & Gandhi (2020) found
that among eight pre-trained models, the AlexNet showed the best results.

Finally, using the CIFAR-10 dataset, and experimenting with the Inception V3,
GoogLeNet, SqueezeNet 1.1, and DarkNet53, ShuffleNet models, 21 found that,
overall, the Inception V3 model achieved the highest accuracy, as well as
higher values in other evaluation metrics including precision, sensitivity,
specificity, and F1-score [21].

In summary, for specific application needs, finding the right pre-trained
model can be challenging. Different models can present better results for
different datasets and different parameters of performance [22]. Therefore, it
is essential to systematically investigate the usability of several pre-
trained models to find the best match for specific datasets.

Although ML has been successfully applied in Geosciences in the last years, TL
for IC in this domain is still not as much explored as in other areas [23].

Recent papers are showing the capacity of DL and TL to facilitate the analysis
of uninterpreted images that have been neglected due to a limited number of
experts, such as fossil images, slabbed cores, or petrographic thin sections
[23], or even for environmental images [24]. The ability to create distinctive
models for specific sets of data allows a versatile application of those
techniques.

When comparing pre-trained models, de Lima and colleagues (2019) found that
both MobileNet V2 and Inception V3 showed promising results on geologic data
interpretation, with MobileNet V2 having slightly better results. Also, when
Sun and colleagues (2021) compared the performance of AlexNet, VGG16,
ResNet50, GLNet (AlexNet), GLNet (VGG16), and GLNet (ResNet) pre-trained
models on remote sensing scene classification using FE, the authors found that
their proposed new model shows better results compared to other traditional
DNN architectures. The proposed model GLNet (VGG16), which uses VGG16 as its
base, got over 95% accuracy in analyzing a clear environment and over 94% in a
cloudy environment. In contrast, the traditional VGG16 got over 93% and over
78%, for clear and cloud environments, respectively [24].

On seismic imaging classification, which is essential for oil and gas
exploration, 25, applied the TL of FE to train a DNN and found improvements in
accuracy [25, 26]. Finally, 27 also trained a CNN specially developed for the
fault identification problem using a dataset with synthetic patches whose
seismic signal frequency content does not match that of real data. They used
the first layers of this pre-trained CNN as a feature extractor to classify
another dataset [27].

Table 1 presents a summary of the literature review, representing the pre-
trained models used by each study identified in our review process. Table 2
presents the target datasets used in each study for comparing the performance
of the feature extractors.

Table 1 provides a summary of the reviewed literature, emphasizing which pre-
trained models were used for FE in each study considered in our review.

Table 1: Summary of literature review - pre-trained models

Table 2 summarizes the datasets in which different works evaluated different
pre-trained models for FE in the literature reviewed in this work.

Table 2: Summary of literature review - datasets

##  3 Experiments

In this section, we discuss the experiments to evaluate different pre-trained
models in different datasets. Firstly, we present the datasets and the models
used in our experiments. After, we describe the methodology adopted in these
experiments. Finally, we present and discuss the results of the experiments.

###  3.1 Datasets

Image datasets are collections of digital images that can be both annotated or
not. In general, image datasets can be used for various computer vision tasks,
such as image recognition, object detection, segmentation, object tracking,
etc. These datasets may contain images of different sizes, resolutions, and
formats, and they are often annotated with additional information such as
object classes, bounding boxes, segmentation masks, etc. They are essential
for training and evaluating computer vision models, and their availability has
enabled significant advances in the field of computer vision in recent years.
These datasets are often used to benchmark new algorithms and to improve the
performance of existing ones.

There are several widely used image datasets in computer vision research. In
this work, the following ones were considered for image classification, since
they are widely used in the literature, are colorful, and have different
characteristics: Geological Images dataset [9], Stanford Cars [12], CIFAR-10
[13], and STL10 [14].

The main focus of this work is to evaluate the performance of the pre-trained
models over the Geological Images dataset. However, as a secondary objective,
Stanford Cars, CIFAR-10, and STL10 datasets were also included in our
evaluation, in order to carry out a more comprehensive evaluation of the
models. Table 3 shows the main information of all datasets used in this work.

Table 3: Datasets Information

Dataset | Instances | Classes | Average Instances per Class $\pm$ Std
---|---|---|---
Geological Images [9] | 25725 | 45 | 571,67 $\pm$ 1290,90
Stanford Cars [12] | 16185 | 196 | 84 $\pm$ 6,28
CIFAR-10 [13] | 60000 | 10 | 6000 $\pm$ 0
STL10 [14] | 100000 | 10 | 10000 $\pm$ 0

Regarding the selected datasets, CIFAR-10 and STL10 are balanced and include
sets of images of homogeneous size. The Geological images and the Stanford
Cars datasets are unbalanced (Stanford Cars is slightly unbalanced) and have
images of heterogeneous sizes. In the following, we will provide more details
regarding each dataset adopted in this study.

####  3.1.1 Geological Images dataset

This is a domain-specific dataset [10] that includes a set of annotated images
that are relevant for applications in Geosciences. The adopted image dataset
includes images extracted from public and private sources. The images were
obtained from the Petrobras Geoscience Bulletins, Theses, and Dissertations
from the Geosciences Institute of UFRGS, as well as from several sources on
the internet. Cleaning procedures were performed to refine the set of images,
aiming to remove invalid and/or non-relevant images. In total, the dataset
includes 25,725 annotated images of 45 different classes. This dataset
presents an imbalance in the distribution of images across classes, where
there are classes with only 36 images and classes with 8450 images. The
average of images per class is 571,67 with a standard deviation of 1290,90.
The distribution of instances per class of this dataset is represented in
Figure 1. Figure 2 shows a set of images that are included in the Geological
Images dataset. Besides that, the images included in this dataset are
heterogeneous regarding height, width, and aspect ratio, as is represented in
Figure 3.

Figure 1: Illustration of the imbalance in the number of images for each class
on the Geological Images dataset [9]. Figure 2: Examples of images included in
the Geological Images dataset [9]. Figure 3: Statistics of the distribution of
height, width, and aspect ratio of images in the Geological Images dataset
[9].

####  3.1.2 Stanford Cars

It consists of 16,185 images of 196 different car classes, where each car
class has been carefully labeled with its make, model, and year. The images
were collected from various online sources, including auction websites, car
dealerships, and online classifieds. Each image in the dataset 222
https://ai.stanford.edu/~jkrause/cars/car_dataset.html is associated with a
text file that contains metadata about the car, including the car’s make,
model, and year, as well as its class ID, which ranges from 1 to 196. The
dataset also includes a set of bounding boxes for each image, which identify
the location of the car within the image. Figure 5 shows a set of images that
are included in the Stanford Cars dataset. Also, the images included in
Stanford Cars dataset compared to Geological Images dataset are partially
heterogeneous regarding height, width, and aspect ratio, as is represented in
Figure 4.

Figure 4: Statistics of the distribution of height, width, and aspect ratio of
images in the Stanford Cars dataset [9]. Figure 5: Images included in the
Stanford Cars dataset [12].

####  3.1.3 CIFAR-10

The dataset 333 https://www.cs.toronto.edu/~kriz/cifar.html contains 60,000
32x32 color images in 10 different classes. Each class contains 6,000 images,
and the dataset is split into 50,000 training images and 10,000 test images.
The classes in the dataset are airplane, automobile, bird, cat, deer, dog,
frog, horse, ship, and truck. Figure 6 shows a set of images that are included
in the CIFAR-10 dataset.

Figure 6: Images included in the CIFAR-10 dataset [13].

####  3.1.4 STL10

This is an image recognition dataset444
https://cs.stanford.edu/~acoates/stl10/ that contains a set of 10 classes,
each represented by 1,000 labeled images. The images in the dataset are 96x96
pixels and are derived from a larger set of unlabeled images collected from
the Internet. STL10 was developed to be a more challenging dataset for image
recognition tasks, as the images are larger than those in other popular
datasets like CIFAR-10 and ImageNet. The 10 classes are airplane, bird, car,
cat, deer, dog, horse, monkey, ship, and truck. It is also divided into a
training set of 5,000 images and a test set of 8,000 images, with each class
evenly represented in both sets. In addition to the labeled images, the STL10
dataset also includes an unlabeled set of 100,000 images, which can be used
for unsupervised learning tasks like FE and clustering. In this work, we
adopted only the labeled images. Figure 7 shows a set of images that are
included in the STL10 dataset.

Figure 7: Images included in the STL10 dataset [14].

###  3.2 Pre-trained Models

Due to the increasing in adoption of transfer learning, nowadays there are
several pre-trained models available in the literature. In our work, we use a
wide range of pre-trained models available in repositories555Can be accessed
through https://pytorch.org/vision/stable/models.html and
https://github.com/openai/CLIP. The majority of these models considered in
this work were pre-trained using ImageNet-1K666Can be accessed through
https://image-net.org/index.php[4] dataset except for the CLIP[28] based
models that were pre-trained in a dataset with 400 million images called
WebImageText (WIT). Table 4 presents the following properties of the selected
models: number of output features, number of parameters, training dataset, and
architecture. Notice that, clip-rn50 and clip-vit-b adopt two encoders, one
for images and the other for text. Thus, in Table 4, the notation CNN + Tr
means that the encoder for images is based on CNN and the encoder for texts is
based on transformers.

Table 4: Pre-trained Models Information. CNN indicates a convolutional neural networks architecture and Tr indicates a transformer-based architecture. Models |  | Output
---
Features
Parameters | Architecture
alexnet[29] | 256 | 61,100,840 | CNN
clip_rn50[30, 28] | 1024 | 63,000,000 | CNN + Tr
clip_vit_b[28] | 512 | 63,000,000 | Tr + Tr
convnext_large[31] | 1536 | 197,767,336 | CNN
densenet161[32] | 2208 | 28,681,000 | CNN
googlenet[33] | 1000 | 6,624,904 | CNN
inception_v3[34] | 1000 | 27,161,264 | CNN
mnasnet1_3[35] | 1000 | 6,282,256 | CNN
mobilenet_v3_large[36] | 960 | 5,483,032 | CNN
regnet_y_3_2gf[37] | 1000 | 19,436,338 | CNN
resnext101_64x4d[38] | 1000 | 83,455,272 | CNN
shufflenet_v2_x2_0[39] | 1000 | 7,393,996 | CNN
squeezenet1_1[40] | 512 | 1,235,496 | CNN
vgg19_bn[41] | 512 | 143,678,248 | CNN
vit_h_14[16] | 1000 | 632,045,800 | Tr
wide_resnet101_2[42] | 1000 | 126,886,696 | CNN

###  3.3 Methodology

Our goal is to evaluate the performance of different available pre-trained
models as feature extractors in the image classification task in different
datasets. We used the datasets and models previously detailed for conducting
our experiments. Notice also that, since there are different versions
available for each family of models, we have selected a single model for each
family that presented the best overall performances according to the
literature.

Since we are considering 16 models and four datasets, a total of 64
experiments considering pairs of models and datasets were performed.

In each experiment, each pre-trained model was used as a feature extractor.
Therefore, in this context, all initial layers (except the last one) of the
model, responsible for extracting relevant features from the input images,
were maintained, while the last layer was replaced by a new output layer, with
$N$ units (where $N$ is proportional to the number of classes in the used
dataset) and a linear activation function and a softmax layer. During the
model training, the weights of the initial layers (responsible for extracting
features) are not adjusted (they are kept "frozen"), while the weights of the
last layer are adjusted. That is, the pre-trained model is used just for
extracting features.

For each experiment, the datasets went through a homogeneous pre-processing.
The pre-processing consisted of applying resizing, center cropping, and
normalization. The resize is always done by decreasing or increasing the size
of the smallest dimension of the image to the size of the pre-trained model’s
input. Then the center crop is performed, where the central area of the image
is cut as a square that matches the size of the input, and this information is
used for representing the whole image. Finally, we ensure that all images are
converted to RGB.

Each model was evaluated considering a 5-fold cross-validation process. We
adopted the following metrics to evaluate our approach: accuracy, macro and
weighted f1-score. Thus, the reported metrics are averages obtained
considering the performance in each test fold of this cross-validation
process.

Regarding the training hyperparameters, the _learning rate_ used in this study
was $0.001$ with a _momentum_ of $0.9$. We adopted the _Adam optimizer_ with
default parameters, with the _Cross-Entropy_ loss function. All executions
were done using $100$ epochs and _early stopping_ with a minimal improvement
of $0.001$ and patience of $5$.

###  3.4 Results

The following tables represent the model’s performance according to the
selected metrics for each dataset. Table 5 represents the model’s evaluation
on the Geological Images dataset. Table 6 represents the model’s evaluation
according to Stanford Cars dataset. Table 7 represents the model’s evaluation
considering the CIFAR-10 dataset. Table 8 represents the model’s evaluation
for the STL10 dataset. In each table, we highlight the model with the best
performance in green and with the less performance in red.

Table 5: Geological Images Dataset

Geological Images Dataset
---
|  | Macro | Weighted
Model \Metrics | Accuracy | Precision | Recall | F-Score | Precision | Recall | F-Score
alexnet | 0,85 | 0,74 | 0,67 | 0,69 | 0,84 | 0,85 | 0,84
clip_rn50 | 0,93 | 0,86 | 0,83 | 0,84 | 0,92 | 0,93 | 0,92
clip_vit_b | 0,93 | 0,86 | 0,83 |  0,84 | 0,93 | 0,93 | 0,93
convnext_large | 0,91 | 0,84 | 0,80 | 0,82 | 0,91 | 0,91 | 0,91
densenet161 | 0,90 | 0,83 | 0,78 | 0,80 | 0,90 | 0,90 | 0,90
googlenet | 0,87 | 0,75 | 0,72 | 0,73 | 0,86 | 0,87 | 0,86
inception_v3 | 0,83 | 0,70 | 0,65 | 0,67 | 0,82 | 0,83 | 0,83
mnasnet1_3 | 0,88 | 0,77 | 0,73 | 0,75 | 0,87 | 0,88 | 0,87
mobilenet_v3_large | 0,90 | 0,82 | 0,77 | 0,79 | 0,90 | 0,90 | 0,90
regnet_y_3_2gf | 0,89 | 0,79 | 0,76 | 0,77 | 0,89 | 0,89 | 0,89
resnext101_64x4d | 0,88 | 0,79 | 0,74 | 0,76 | 0,88 | 0,88 | 0,88
shufflenet_v2_x2_0 | 0,89 | 0,80 | 0,76 | 0,78 | 0,89 | 0,89 | 0,89
squeezenet1_1 | 0,87 | 0,77 | 0,72 | 0,74 | 0,87 | 0,87 | 0,87
vgg19_bn | 0,88 | 0,79 | 0,74 | 0,76 | 0,88 | 0,88 | 0,88
vit_h_14 | 0,91 | 0,82 | 0,79 | 0,80 | 0,90 | 0,91 | 0,90
wide_resnet101_2 | 0,89 | 0,79 | 0,75 | 0,77 | 0,89 | 0,89 | 0,89
|  |  |  |  |  |  |
Average | 0,89 | 0,79 | 0,75 | 0,77 | 0,88 | 0,89 | 0,88
Standard Deviation | 0,02 | 0,04 | 0,05 | 0,05 | 0,03 | 0,02 | 0,03

Table 6: Stanford Cars Dataset

Stanford Cars Dataset
---
|  | Macro | Weighted
Model \Metrics | Accuracy | Precision | Recall | F-Score | Precision | Recall | F-Score
alexnet | 0,28 | 0,26 | 0,28 | 0,26 | 0,26 | 0,28 | 0,26
clip_rn50 | 0,82 | 0,82 | 0,82 | 0,82 | 0,82 | 0,82 | 0,82
clip_vit_b | 0,83 | 0,83 | 0,83 | 0,83 | 0,83 | 0,83 | 0,83
convnext_large | 0,65 | 0,65 | 0,64 | 0,64 | 0,65 | 0,65 | 0,64
densenet161 | 0,64 | 0,64 | 0,64 | 0,64 | 0,64 | 0,64 | 0,64
googlenet | 0,41 | 0,41 | 0,41 | 0,41 | 0,40 | 0,41 | 0,40
inception_v3 | 0,34 | 0,33 | 0,34 | 0,33 | 0,33 | 0,34 | 0,33
mnasnet1_3 | 0,42 | 0,42 | 0,42 | 0,42 | 0,41 | 0,42 | 0,41
mobilenet_v3_large | 0,56 | 0,56 | 0,56 | 0,55 | 0,56 | 0,56 | 0,55
regnet_y_3_2gf | 0,49 | 0,49 | 0,49 | 0,49 | 0,49 | 0,49 | 0,49
resnext101_64x4d | 0,35 | 0,35 | 0,35 | 0,34 | 0,34 | 0,35 | 0,34
shufflenet_v2_x2_0 | 0,50 | 0,50 | 0,50 | 0,50 | 0,50 | 0,50 | 0,50
squeezenet1_1 | 0,42 | 0,42 | 0,42 | 0,41 | 0,41 | 0,42 | 0,41
vgg19_bn | 0,51 | 0,50 | 0,51 | 0,50 | 0,50 | 0,51 | 0,50
vit_h_14 | 0,86 | 0,86 | 0,85 | 0,85 | 0,86 | 0,86 |  0,86
wide_resnet101_2 | 0,44 | 0,44 | 0,44 | 0,44 | 0,44 | 0,44 | 0,44
|  |  |  |  |  |  |
Average | 0,53 | 0,53 | 0,53 | 0,53 | 0,53 | 0,53 | 0,53
Standard Deviation | 0,18 | 0,18 | 0,18 | 0,18 | 0,18 | 0,18 | 0,18

Table 7: CIFAR-10 Dataset

CIFAR-10 Dataset
---
|  | Macro | Weighted
Model \Metrics | Accuracy | Precision | Recall | F-Score | Precision | Recall | F-Score
alexnet | 0,79 | 0,79 | 0,79 | 0,79 | 0,79 | 0,79 | 0,79
clip_rn50 | 0,88 | 0,88 | 0,88 | 0,88 | 0,88 | 0,88 | 0,88
clip_vit_b | 0,95 | 0,95 | 0,95 | 0,95 | 0,95 | 0,95 | 0,95
convnext_large | 0,96 | 0,96 | 0,96 | 0,96 | 0,96 | 0,96 | 0,96
densenet161 | 0,93 | 0,93 | 0,93 | 0,93 | 0,93 | 0,93 | 0,93
googlenet | 0,87 | 0,87 | 0,87 | 0,87 | 0,87 | 0,87 | 0,87
inception_v3 | 0,86 | 0,86 | 0,86 | 0,86 | 0,86 | 0,86 | 0,86
mnasnet1_3 | 0,90 | 0,90 | 0,90 | 0,90 | 0,90 | 0,90 | 0,90
mobilenet_v3_large | 0,91 | 0,91 | 0,91 | 0,91 | 0,91 | 0,91 | 0,91
regnet_y_3_2gf | 0,93 | 0,93 | 0,93 | 0,93 | 0,93 | 0,93 | 0,93
resnext101_64x4d | 0,95 | 0,95 | 0,95 | 0,95 | 0,95 | 0,95 | 0,95
shufflenet_v2_x2_0 | 0,92 | 0,92 | 0,92 | 0,92 | 0,92 | 0,92 | 0,92
squeezenet1_1 | 0,85 | 0,85 | 0,85 | 0,85 | 0,85 | 0,85 | 0,85
vgg19_bn | 0,88 | 0,88 | 0,88 | 0,88 | 0,88 | 0,88 | 0,88
vit_h_14 | 0,98 | 0,98 | 0,98 | 0,98 |  0,98 | 0,98 | 0,98
wide_resnet101_2 | 0,95 | 0,95 | 0,95 | 0,95 | 0,95 | 0,95 | 0,95
|  |  |  |  |  |  |
Average | 0,91 | 0,91 | 0,91 | 0,91 | 0,91 | 0,91 | 0,91
Standard Deviation | 0,05 | 0,05 | 0,05 | 0,05 | 0,05 | 0,05 | 0,05

Table 8: STL10 Dataset

STL10 Dataset
---
|  | Macro | Weighted
Model \Metrics | Accuracy | Precision | Recall | F-Score | Precision | Recall | F-Score
alexnet | 0,88 | 0,88 | 0,88 | 0,88 | 0,88 | 0,88 | 0,88
clip_rn50 | 0,97 | 0,97 | 0,97 | 0,97 | 0,97 | 0,97 | 0,97
clip_vit_b | 0,99 | 0,99 | 0,99 | 0,99 | 0,99 | 0,99 | 0,99
convnext_large | 0,99 | 0,99 | 0,99 | 0,99 | 0,99 | 0,99 | 0,99
densenet161 | 0,98 | 0,98 | 0,98 | 0,98 | 0,98 | 0,98 | 0,98
googlenet | 0,96 | 0,96 | 0,96 | 0,96 | 0,96 | 0,96 | 0,96
inception_v3 | 0,96 | 0,96 | 0,96 | 0,96 | 0,96 | 0,96 | 0,96
mnasnet1_3 | 0,97 | 0,97 | 0,97 | 0,97 | 0,97 | 0,97 | 0,97
mobilenet_v3_large | 0,96 | 0,96 | 0,96 | 0,96 | 0,96 | 0,96 | 0,96
regnet_y_3_2gf | 0,98 | 0,98 | 0,98 | 0,98 | 0,98 | 0,98 | 0,98
resnext101_64x4d | 0,99 | 0,99 | 0,99 | 0,99 | 0,99 | 0,99 | 0,99
shufflenet_v2_x2_0 | 0,97 | 0,97 | 0,97 | 0,97 | 0,97 | 0,97 | 0,97
squeezenet1_1 | 0,91 | 0,91 | 0,91 | 0,91 | 0,91 | 0,91 | 0,91
vgg19_bn | 0,96 | 0,96 | 0,96 | 0,96 | 0,96 | 0,96 | 0,96
vit_h_14 | 1,00 | 1,00 |  1,00 | 1,00 | 1,00 | 1,00 | 1,00
wide_resnet101_2 | 0,99 | 0,99 | 0,99 | 0,99 | 0,99 | 0,99 | 0,99
|  |  |  |  |  |  |
Average | 0,97 | 0,97 | 0,97 | 0,97 | 0,97 | 0,97 | 0,97
Standard Deviation | 0,03 | 0,03 | 0,03 | 0,03 | 0,03 | 0,03 | 0,03

In order to facilitate the data analysis, we have represented the data
generated in our experiments in the following line charts, which demonstrate
the performance (according to different metrics) of each pre-trained model for
classifying the images in the four selected datasets. Figure 8 represents the
accuracy. Figure 9 shows the macro precision. Figure 10 indicates the macro
recall. Figure 11 demonstrates the macro f1-score. Figure 12 presents the
weighted precision. Figure 13 shows the weighted recall. Figure 14 indicates
the weighted f1-score of each model on each dataset.

Figure 8: Line chart representing the accuracy of each model on each dataset.
Figure 9: Line chart representing the macro precision of each model on each
dataset. Figure 10: Line chart representing the macro recall of each model on
each dataset. Figure 11: Line chart representing the macro f1-score of each
model on each dataset. Figure 12: Line chart representing the weighted
precision of each model on each dataset. Figure 13: Line chart representing
the weighted recall of each model on each dataset. Figure 14: Line chart
representing the weighted f1-score of each model on each dataset.

The line charts in Figures 9-14 present a very similar pattern of variation of
the model’s performance across all datasets. We can notice also that, in
general, the model’s performance pattern increases and the differences among
patterns decrease (resulting in a smoother pattern) in the Geological Images
dataset when we consider the accuracy and the weighted averages of precision,
recall, and f-score in comparison with the macro averages of these measures.
This behavior is expected since the imbalance of this dataset is more
pronounced. We can notice that, in general, the CLIP-ViT-B and
VisionTransformer-H/14 models tend to show the best performances, considering
all metrics in most datasets. In the Geological Images dataset, the CLIP-ViT-B
presents the best performance, in all metrics. The CLIP-ResNet50 also tends to
perform well in the other datasets, although in the case of the CIFAR-10 and
Geological Images datasets, this model’s performance is reasonably lower than
the performance of CLIP-ViT-B and ViT-H/14, in all metrics. In the CIFAR-10
dataset, it is also worth highlighting the good performance of ConvNeXt Large
and ResNeXt101-64x4D. The ConvNeXt Large model also performs better than CLIP-
ViT-B and CLIP-ResNet50 in STL10. In our evaluation, AlexNet presents the
worst performance on most datasets, considering all metrics. Inception V3 also
performed poorly on the Stanford Cars, CIFAR-10, and Geological Images
datasets, where it performed the worst. Another model that had reasonably low
performance compared to the others was Squezenet1-1. The poor performance of
this model is more pronounced on the CIFAR-10 and STL10 datasets.

The following boxplots reveal the minimum value (excluding outliers), the
maximum value (excluding outliers), the median, the first quartile, the third
quartile, and outliers of the performance of the different models considering
all the datasets, according to different metrics. Figure 15 represents the
accuracy. Figure 16 shows the macro precision. Figure 17 indicates the macro
recall. Figure 18 demonstrates the macro f1-score. Figure 19 presents the
weighted precision. Figure 20 represents the weighted recall. Figure 21 shows
the weighted f1-score of each model on each dataset. Notice that each boxplot
is built from only four measures (one for each dataset), however, despite this
small amount of data, this visualization can provide further insights.

Figure 15: Boxplot of accuracy for each model. Figure 16: Boxplot of macro
precision for each model. Figure 17: Boxplot of macro recall for each model.
Figure 18: Boxplot of f1-score for each model. Figure 19: Boxplot of weighted
precision for each model. Figure 20: Boxplot of weighted recall for each
model. Figure 21: Boxplot of weighted f1-score for each model.

In the boxplots represented in Figures 15-21, it is possible to notice a
pattern in the different metrics, although some differences can be noticed in
the Geological Images dataset when comparing the macro averages of precision,
recall, and f-score with the accuracy and weighted averages of those metrics.
The CLIP-ResNet50, CLIP-ViT-B, and VisionTransformer-H/14 models present the
lowest variability in performance, in general. In the macro averages of
precision, recall, and f-score, CLIP-ResNet50 presents the lowest variability.
On the other hand, when considering the accuracy and weighted averages of
precision, recall, and f-score, CLIP-ViT-B presents less variability in its
performance. It is important to note that CLIP-ResNet50, CLIP-ViT-B, and
VisionTransformer-H/14 are the only models adopted in our experiments that
include transformers in their architecture. It is also worth noting that,
among the CNN-based architectures, ConvNeXt Large presented a higher median,
compared to the other CNN-based architectures, which is similar to the medians
of performances obtained by CLIP-ViT-B and VisionTransformer-H/14, although
with greater variability in performance. The boxplots also suggest that, in
the considered datasets, AlexNet tends to present the worst performances, in
general, presenting low medians and high variability. High variability is also
present in the performances of ResNeXt101-64x4D, Wide ResNet 101-2, and
Inception V3, which also presents low medians when compared to the other
models.

The previous analysis (Figures 8-14) suggests that some models present a very
similar performance behavior across the datasets while other models exhibit
behaviors that do not follow the general pattern. In order to emphasize how
similar are the model’s behaviors, we analyzed the Pearson correlation [43] of
the performances of each pair of models across the datasets according to all
the selected metrics. The following heat maps were created to visually
represent this information. In the following sequence of charts, the darker a
cell gets corresponds to the lower the correlation of a given pair of models,
according to a given performance metric. Figure 22 represents the correlation
regarding the accuracy. Figure 23 shows the correlation regarding the macro
precision. Figure 24 indicates the correlation regarding the macro recall.
Figure 25 demonstrates the correlation regarding the macro f1-score. Figure 26
presents the correlation regarding the weighted precision. Figure 27
represents the correlation regarding the weighted recall. Figure 28 shows the
correlation regarding the weighted f1-score between each pair of models.

Figure 22: Heat map representing the correlation between each pair of models
regarding accuracy. Figure 23: Heat map representing the correlation between
each pair of models regarding the macro precision. Figure 24: Heat map
representing the correlation between each pair of models regarding the macro
recall. Figure 25: Heat map representing the correlation between each pair of
models regarding the macro f1-score. Figure 26: Heat map representing the
correlation between each pair of models regarding the weighted precision.
Figure 27: Heat map representing the correlation between each pair of models
regarding the weighted recall. Figure 28: Heat map representing the
correlation between each pair of models regarding the weighted f1-score.

Figures 22-28 suggest that the correlation of the performances of each pair of
models presents a similar pattern in all metrics. We can notice also that, in
all performance metrics, the correlation between models based solely on CNN
architectures is high (in general, above 0.97). However, the performances of
CLIP-ResNet50, CLIP-ViT-B, and VisionTransformer-H/14 models present a lower
correlation with the performances of other models solely based on CNN. In the
case of CLIP-ViT-B, the correlation with the other models is subtly lower
considering accuracy and the weighted averages of precision, recall, and
f-score. However, when we consider the macro averages of precision, recall,
and f-score, this model’s correlation is significantly lower. It is important
to note that the CLIP-ResNet50, CLIP-ViT-B, and VisionTransformer-H/14 models
include transformers in their architectures. This performance correlation
analysis suggests that this difference in the basic principles of the
architecture of these models is correlated with this difference in the
performance pattern of these models when compared to architectures based
solely on CNN. Further analysis should be done in the future in order to
investigate this hypothesis. The heat maps also allow us to note that the
correlations among the performances of CLIP-ResNet50, CLIP-ViT-B, and
VisionTransformer-H/14 models are not high when compared with the correlations
among the performances of models based solely on CNN.

In the previous analyses, we focused on the performance of the models
considered in our experiments. In the following charts, we focused on
analyzing the datasets considered in our experiments. Each chart represents
how the performances of all models vary across the different datasets,
considering different metrics. This information is already present in Figures
8 \- 14. However, these charts provide an alternative visualization of this
information that can suggest, for example, how hard is to classify each
dataset. Figure 29 focuses on accuracy. Figure 9 represents macro precision.
Figure 31 shows macro recall. Figure 32 indicates macro f1-score. Figure 33
demonstrates weighted precision. Figure 34 presents weighted recall. Figure 35
focuses on the weighted f1-score of all models across the datasets.

Figure 29: Chart representing the accuracy of all models on different
datasets. Figure 30: Chart representing the macro precision of all models on
different datasets. Figure 31: Chart representing the macro recall of all
models on different datasets. Figure 32: Chart representing the macro f1-score
of all models on different datasets. Figure 33: Chart representing the
weighted precision of all models on different datasets. Figure 34: Chart
representing the weighted recall of all models on different datasets. Figure
35: Chart representing the weighted f1-score of all models on different
datasets.

The line charts represented in Figures 29-35 present a similar pattern that
can be seen across the different metrics. There are subtle differences when
comparing the macro averages of precision, recall, and f-score with accuracy
and weighted averages of those metrics. Note that, in general, the models tend
to perform better in the STL10 dataset, in second place CIFAR-10 has the best
overall results, in third place the dataset of Geological Images and, finally,
the dataset with the worst performances, in general, is the Stanford Cars.
This is an expected result since this dataset has a large number of classes,
few samples per class, and the inclusion of images with different sizes and
features at different scales. The Geological Images dataset has similar
properties but has fewer classes and more samples per class than Stanford
Cars, although it presents a greater imbalance.

The following boxplots represent the performances considering all the models
in each dataset, according to different metrics. Figure 36 represents the
variation of accuracy. Figure 37 shows the variation of macro precision.
Figure 38 indicates the variation of macro recall. Figure 39 demonstrates the
variation of macro f1-score. Figure 40 presents the variation of weighted
precision. Figure 41 represents the variation of weighted recall. Figure 42
shows the variation of weighted f1-score in each dataset.

Figure 36: Boxplot of accuracy for each dataset. Figure 37: Boxplot of macro
precision for each dataset. Figure 38: Boxplot of macro recall for each
dataset. Figure 39: Boxplot of macro f1-score for each dataset. Figure 40:
Boxplot of weighted precision for each dataset. Figure 41: Boxplot of weighted
recall for each dataset. Figure 42: Boxplot of weighted f1-score for each
dataset.

The boxplots represented in Figures 36-42 present a similar pattern across the
different metrics. There are subtle differences when comparing the macro
averages of precision, recall, and f-score with accuracy and the weighted
averages of those metrics. These boxplots emphasize some properties found in
the previous graphs (Figures 29 \- 35). Firstly, these charts provide evidence
that the Stanford Cars dataset is the most challenging among those analyzed,
with the worst performances and the greatest variability of performances in
all metrics. Besides that, we can notice also that the STL10 dataset and
Geological Images have a smaller variability in the performance of the
different models when compared with the other two datasets.

##  4 Conclusions

In this work, our goal was to compare the performance of sixteen pre-trained
neural networks for FE in four different datasets, with an emphasis on
geological image classification.

By analyzing the accuracy and macro and weighted averages of f-score, recall,
and precision, our experiments have shown that CLIP-ViT-B achieved the best
results for the Geological Images dataset, which was the main focus of this
work. Our experiments showed also that CLIP-ResNet50 and
VisionTransformer-H/14 also achieved similar performances.

When we focus on the general performances, considering all the datasets, our
experiments suggest that CLIP-ViT-B and VisionTransformer-H/14 achieved better
performance results for all datasets and low variability in their
performances. Besides that, CLIP-ResNet50 achieved performance similar to the
performance achieved by CLIP-ViT-B, and VisionTransformer-H/14 and even lower
variability. It is important to notice that CLIP-ViT-B,
VisionTransformer-H/14, and CLIP-ResNet50 include transformers in their
architectures. Thus, our results suggest that the principles underlying the
transformers can be the reason corroborating these remarkable results, but
further studies should be carried out to investigate this hypothesis.

Among the CNN-based architectures, ConvNeXt Large presents the best
performance, in general, and lower variability when compared to other CNN-
based architectures. AlexNet showed the worst performance and high
variability. Besides that, ResNeXt101-64x4D, Wide ResNet 101-2, and Inception
V3 also showed high variability.

Our analysis also showed that the performances of models based solely on CNN
architectures present a high Pearson correlation in all performance metrics.
However, the performances of CLIP-ResNet50, CLIP-ViT-B, and
VisionTransformer-H/14 models show a lower correlation with other models based
solely on CNN. This difference in performance is hypothesized to be due to
differences regarding the basic principles of the architecture of these
models. However, the correlations among the performances of CLIP-ResNet50,
CLIP-ViT-B, and VisionTransformer-H/14 models are not high, if compared to
correlations among the performances of CNN-based models. Further studies are
needed to better investigate this finding.

Our analysis also has shown that the selected models performed better on the
STL10 dataset, followed by CIFAR-10, then the Geological Images dataset, and
finally the Stanford Cars dataset. Thus, we can conclude that the Stanford
Cars dataset is the most challenging dataset evaluated in this work. The
Stanford Cars present a particularly large image size (when compared with
CIFAR-10 and STL10, for example) and a high amount of classes with just a few
samples per class. These characteristics may explain this result. The
Geological Images dataset shares some of the properties of the Stanford Cars
dataset, but it presents a higher imbalance, has a lower amount of classes,
and has more images per class, in general.

The investigation presented in this work can provide evidence that supports
the choice of models for transfer learning in image classification tasks
involving the Geological Images dataset. Since our evaluation also covered
other image datasets, it can also suggest reasonable choices for transfer
learning in other domains.

In future works, to make the analysis more comprehensive, it is important to
expand it by including other image datasets. Besides that, the investigation
can also be expanded to include more pre-trained models that eventually were
not considered in the scope of this work. Furthermore, future works could also
investigate the relationship between the underlying principles of each
architecture, the properties of the datasets used in the pre-training of these
models, and the properties of the target datasets, in which the pre-trained
models are applied to extract features. This investigation can reveal insights
into what makes the pre-trained model best suited for each task.

## References

  * [1] J.W. Pferd.  The challenges of integrating structured and unstructured data.  In 14th Petroleum Network Education Conference. s.n, S.l., 2010.
  * [2] Laura Hollink, Guus Schreiber, Jan Wielemaker, Bob Wielinga, et al.  Semantic annotation of image collections.  In Knowledge capture, volume 2, 2003.
  * [3] Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi.  General multi-label image classification with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16478–16488, June 2021.
  * [4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [5] Arafat Abu Mallouh, Zakariya Qawaqneh, and Buket D Barkana.  Utilizing cnns and transfer learning of pre-trained models for age range classification from unconstrained face images.  Image and Vision Computing, 88:41–51, 2019.
  * [6] Yusuf Arslan, Kevin Allix, Lisa Veiber, Cedric Lothritz, Tegawendé F Bissyandé, Jacques Klein, and Anne Goujon.  A comparison of pre-trained language models for multi-class text classification in the financial domain.  In Companion Proceedings of the Web Conference 2021, pages 260–268, 2021.
  * [7] Brady Kieffer, Morteza Babaie, Shivam Kalra, and Hamid R Tizhoosh.  Convolutional neural networks for histopathology image classification: Training vs. using pre-trained networks.  In 2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA), pages 1–6. IEEE, 2017.
  * [8] Romain Mormont, Pierre Geurts, and Raphaël Marée.  Comparison of deep transfer learning strategies for digital pathology.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 2262–2271, 2018.
  * [9] Matheus V. Todescato, Luan Fonseca Garcia, Dennis Giovani Balreira, and Joel Luís Carbonera.  Multiscale context features for geological image classification.  In Joaquim Filipe, Michal Smialek, Alexander Brodsky, and Slimane Hammoudi, editors, Proceedings of the 25th International Conference on Enterprise Information Systems, ICEIS 2023, Volume 1, Prague, Czech Republic, April 24-26, 2023, pages 407–418. SCITEPRESS, 2023.
  * [10] Mara Abel, Eduardo Simões Lopes Gastal, Cassiana Roberta Lizzoni Michelin, Luiza Gonçalves Maggi, Bruno Eduardo Firnkes, Felix Eduardo Huaroto Pachas, and Renata dos Santos Alvarenga.  A knowledge organization system for image classification and retrieval in petroleum exploration domain.  In ONTOBRAS, 2019.
  * [11] Matheus V Todescato, Luan F Garcia, Dennis G Balreira, and Joel L Carbonera.  Multiscale patch-based feature graphs for image classification.  Expert Systems with Applications, 235:121116, 2024.
  * [12] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
  * [13] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [14] Adam Coates, Andrew Ng, and Honglak Lee.  An analysis of single-layer networks in unsupervised feature learning.  In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215–223. JMLR Workshop and Conference Proceedings, 2011.
  * [15] Laith Alzubaidi et al.  Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions.  Journal of big Data, 8(1):1–74, 2021.
  * [16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [17] Hiren Maniar et al.  Machine-learning methods in geoscience.  In 2018 SEG International Exposition and Annual Meeting. OnePetro, 2018.
  * [18] Anuj Karpatne et al.  Machine learning for the geosciences: Challenges and opportunities.  IEEE Transactions on Knowledge and Data Engineering, 31(8):1544–1554, 2018.
  * [19] Hassan Ismail Fawaz et al.  Transfer learning for time series classification.  In 2018 IEEE international conference on big data (Big Data), pages 1367–1376. IEEE p, 2018.
  * [20] Alessandra Lumini and Loris Nanni.  Deep learning and transfer learning features for plankton classification.  Ecological informatics, 51:33–43, 2019.
  * [21] Jayapalan Senthil Kumar, Syahid Anuar, and Noor Hafizah Hassan.  Transfer learning based performance comparison of the pre-trained deep neural networks.  International Journal of Advanced Computer Science and Applications, 13:1, 2022.
  * [22] Nermeen Abou Baker, Nico Zengeler, and Handmann.  Uwe. a transfer learning evaluation of deep neural networks for image classification.  Machine Learning and Knowledge Extraction, 4(1):22–41, 2022.
  * [23] Rafael Pires De Lima et al.  Deep convolutional neural networks as a geological image classification tool.  The Sedimentary Record, 17(2):4–9, 2019.
  * [24] Huiming Sun et al.  Convolutional neural networks based remote sensing scene classification under clear and cloudy environments.  In Proceedings of the IEEE/CVF International Conference on Computer Vision p, pages 713–720, 2021.
  * [25] Daniel Chevitarese et al.  Transfer learning applied to seismic images classification.  AAPG Annual and Exhibition, 2018.
  * [26] Daniel Salles Chevitarese, Daniela Szwarcman, Emilio Vital Brazil, and Bianca Zadrozny.  Efficient classification of seismic textures.  In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2018.
  * [27] Augusto Cunha et al.  Seismic fault detection in real data using transfer learning from a convolutional neural network pre-trained with synthetic seismic data.  Computers & Geosciences, 135(10434):4, 2020.
  * [28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [29] Alex Krizhevsky.  One weird trick for parallelizing convolutional neural networks.  arXiv preprint arXiv:1404.5997, 2014.
  * [30] Kaiming He et al.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition p, pages 770–778, 2016.
  * [31] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976–11986, 2022.
  * [32] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [33] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [34] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * [35] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le.  Mnasnet: Platform-aware neural architecture search for mobile.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2820–2828, 2019.
  * [36] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019.
  * [37] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár.  Designing network design spaces.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10428–10436, 2020.
  * [38] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
  * [39] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018.
  * [40] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size.  arXiv preprint arXiv:1602.07360, 2016.
  * [41] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [42] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  arXiv preprint arXiv:1605.07146, 2016.
  * [43] Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen.  Pearson correlation coefficient.  Noise reduction in speech processing, pages 1–4, 2009.


#  SIR THOMAS WALTER BANNERMAN KIBBLE CBE

23rd December 1932 - 2nd June 2016

FRS 1980

M. J. Duff<EMAIL_ADDRESS>K. S. Stelle<EMAIL_ADDRESS>The
Blackett Laboratory, Imperial College London, Prince Consort Road,
London SW7 2BW, U.K.

###### Abstract

Professor Tom Kibble was an internationally-renowned theoretical physicist
whose contributions to theoretical physics range from the theory of elementary
particles to modern early-universe cosmology. The unifying theme behind all
his work is the theory of non-abelian gauge theories, the Yang-Mills extension
of electromagnetism. One of Kibble’s most important pieces of work in this
area was his study of the symmetry-breaking mechanism whereby the force-
carrying vector particles in the theory can acquire a mass accompanied by the
appearance of a massive scalar boson. This idea, put forward independently by
Brout and Englert, by Higgs and by Guralnik, Hagen and Kibble in 1964, and
generalised by Kibble in 1967, lies at the heart of the Standard Model and all
modern unified theories of fundamental particles. It was vindicated in 2012 by
the discovery of the Higgs boson at CERN. According to Nobel Laureate Steven
Weinberg, “Tom Kibble showed us why light is massless”; this is the
fundamental basis of electromagnetism.

††journal: Biographical Memoirs of the Royal Society Figure 1: Tom Kibble.
Portrait by Godfrey Argent 1980 (© Godfrey Argent Studio)

##  1 EARLY YEARS

###  1.1 India

Sir Thomas Walter Bannerman Kibble (FRS 1980), whose portrait is shown in Fig,
1, was born on the 23rd of December 1932 in Madras (now Chennai), Madras
Presidency, India. Both his parents were from missionary backgrounds. His
father Fred was a Professor of Mathematics at Madras Christian College. His
mother, Janet Bannerman, descended from a long line of ministers in the Church
of Scotland. Her father, William Burney Bannerman, was a doctor in the Indian
Medical Service. Tom spent his first ten years in India. He was educated in
the Doveton Corrie School, rated as one of the best schools in Madras. In the
summers, his family generally went to Kodaikanal in the Palni Hills to escape
the heat. He particularly enjoyed rowing on the lake there. See Fig. 2.

Figure 2: Tom rowing 1939 (courtesy of the Kibble family)

The principal mathematical influence on young Tom was his father, who was
particularly interested in the symmetry patterns of Moghul architecture in
Agra and elsewhere in India. From this exposure, he gained a love of geometry.
His father taught him about the regular and semi-regular solids, and he
enjoyed making paper models of them.

It had always been intended that at about the age of ten, Tom should return to
Britain for later schooling. However, this was during the Second World War,
and travel was difficult. In early 1943, passage became available at just two
days’ notice - he was not even able to say goodbye to his friends. He went on
this voyage alone. Since the Suez Canal was closed, the voyage was around
Africa, then across the Atlantic and back in convoy, taking eleven weeks. In
Capetown, Tom went ashore alone at one point but upon returning to the ship
found that it was gone. Happily, it had simply moved to another dock. Such
experiences gave young Tom a self-reliance which remained with him for life.

Figure 3: Tom Head Boy 1951 (courtesy of the Kibble family)

###  1.2 Edinburgh

From 1944 to 1951, Tom attended Melville College, Edinburgh, becoming Head Boy
in 1950-51. See Fig. 3. In his final years at school, he became very
interested in cars, and, together with friends owned a series of old wrecks.
His father was keen for him to go to Cambridge. However, in his recollections
he said that he found the problems for the scholarship exams extremely tedious
(and he added that by then he was enjoying life too much to work on them), so
he was not successful. Consequently, he went to Edinburgh University, a
decision that he never regretted.

Two important influences on Tom’s future career while an undergraduate in
Edinburgh were Robin Schlapp and Andrew Nesbitt, who taught classical
mathematical physics. Also important was the teaching in pure mathematics in
the Mathematics Department under A.C. Aitken, who was known for his great
facility for mental arithmetic, being able to extract cube roots of twelve-
digit numbers in seconds. Tom remarked that it was no accident that a
considerable number of the leading mathematical physicists in the UK were
educated in Edinburgh during this period. Two other near contemporaries in
Edinburgh were David Olive (FRS 1987) and Keith Moffat.

Halfway through Tom’s undergraduate course, Max Born (FRS 1939) retired and
was replaced by Nicholas Kemmer (FRS 1956), who gave inspiring lectures
completely without notes (but with the distracting habit of balancing a row of
pieces of chalk end on end). Tom continued on to a PhD in Edinburgh; his
supervisor was John Polkinghorne (FRS 1974). His PhD thesis consisted of two
parts: 1\. Schwinger’s Action Principle 2\. Dispersion Relations for Inelastic
Scattering Processes.

Figure 4: Tom and Anne Graduation (courtesy of the Kibble family)

During Tom’s education in Edinburgh he was for some time a member of the
Student’s Representative Council, and it was through this that he met Anne
Allan, who was also a member. See Fig. 4. They became engaged in June 1954 and
married when he was in the middle of his PhD, in July 1957. They had three
children: Helen, Alison and Robert.

###  1.3 Caltech

Following the award of his PhD, Tom received a Harkness Fellowship to spend a
year in the United States. Tom and Anne went to Caltech, where Tom got to know
Dick Feynman (ForMemRS 1965) and Murray Gell-Mann (ForMemRS 1978). Upon
returning to the UK in 1959 as a NATO Fellow, Tom joined the Theoretical
Physics Group at Imperial College.

##  2 IMPERIAL COLLEGE LONDON

###  2.1 Quantum Field Theory and the influence of Abdus Salam

In 1957 the Head of the Physics Department at Imperial, P. M. S. Blackett (FRS
1933), later Lord Blackett (PRS 1965-1970), recruited the Pakistani physicist
Abdus Salam (FRS 1959) from Cambridge to be founding member of The Theoretical
High Energy Physics Group. Salam remained as Professor there until his death
in 1996, sharing the 1979 Nobel Prize with Sheldon Glashow and Steven Weinberg
(ForMemRS 1981) for unifying the weak and electromagnetic forces. As Tom
himself recalls [Kibble 1997] “It was a very exciting place to be. We had lots
of visitors: Steven Weinberg, Murray Gell-Mann, Ken Johnson, Lowell Brown,
Stanley Mandelstam, John Ward (FRS 1965), to name but a few. The year I
arrived, 1959, was also the year Salam became the youngest Fellow of the Royal
Society at the age of 33.”

Figure 5: Chris Hull, Tom Kibble, Michael Duff, Ahmed Salam, Saba Manzoo with
Abdus Salam’s Nobel Medal
(courtesy of Imperial College)

In the late 50s and early 60s, however, the prevailing mood in fundamental
physics was not favourable to quantum field theories other than Quantum
Electrodynamics (QED). The success of QED, pioneered by Richard Feynman,
Julian Schwinger, Shinichiro Tomanaga and Freeman Dyson (FRS 1952) relied on
the smallness of the coupling constant and hence a sensible perturbative
expansion.This would not be the case for the strong interactions in the Yukawa
theory, however, as the squared coupling constant would be about 15. In Tom’s
own words [Kibble 1997]: “Salam was convinced from an early date–certainly
well before he came to Imperial College—that the fundamental theory would be a
gauge theory. This was not a popular view. Many people thought that field
theory had had its day, particularly for the strong interaction, where it was
to be superseded by S-matrix theory. This was an appealingly economical idea:
everything was supposed to follow from fundamental principles of covariance,
causality and unitarity”. But Salam was undeterred and under his influence,
and that of P. T. Matthews (FRS 1963), the construction of unified gauge
theories was a major theme of the work at Imperial. Tom Kibble’s interest in
gauge field theories extended throughout his career. In 1961, he wrote an
influential paper on the relation between gravity and a gauge theory of the
Poincare group [Kibble 1961]. Another significant contribution was a series of
four papers [Kibble 1968a, Kibble 1968b, Kibble 1968c, Kibble 1968d] devoted
to resolving the notoriously tricky problem of defining an S-matrix in
theories with massless particles such as QED. There has been a recent revival
of interest in this problem and Tom’s resolution, involving coherent states,
continues to be frequently cited. See, for example [Hannesdottir 2020] .

Tom concludes “I shall always look back on these early years as the most
exciting of my life. There was a wonderful atmosphere at the College; I cannot
think of anywhere I would rather have been. I shall always be immensely
grateful to Abdus Salam for making it possible for me to participate in these
memorable developments.” The inspiration was reciprocated: Salam’s 1968
electro-weak unification relied crucially on the spontaneous-symmetry-breaking
mechanism that Tom pioneered [Kibble 1964a, Kibble 1967].

###  2.2 Spontaneous symmetry breaking

The first example of a gauge theory beyond QED was the SU(2) non-abelian gauge
theory proposed by Yang and Mills [Yang 1954] and by Salam’s student Shaw
[Shaw 1955]. Although originally intended for the strong interactions, this
formed the basis of Salam’s unification of the weak and electromagnetic
forces. One of the key problem for the weak interactions was their short
range, which implies that the force carrier particles must be massive, as
opposed to the naturally massless excitations of gauge theories.

In 1964-65, Gerry Guralnik and Dick Hagen were visitors. This led to a key
collaboration with Tom which borrowed ideas from P. Anderson (ForMemRS 1980)
in condensed matter physics in the context of the Ginzburg-Landau model of
superconductivity. This early model (now derivable from the full Bardeen-
Cooper-Schrieffer theory as an effective theory) involves a scalar order
parameter field $\phi(t,{\bf x})$ representing the wave function of a
condensate of Cooper pairs, i.e. bound state pairs of electrons. The
Hamiltonian for the $\phi$ field is

$H=\int d^{3}x\left[\frac{1}{2m}{\bf D}\phi^{\ast}\cdot{\bf
D}\phi+V(\phi)\right]$

where ${\bf D}\phi=\nabla\phi-2ie{\bf A}\phi$ is the electromagnetic covariant
derivative. Expanding in a power series
$V(\phi)=\alpha\phi^{\ast}\phi+\frac{1}{2}\beta(\phi^{\ast}\phi)^{2}$ when
$\phi$ is small. Such a potential has a “sombrero” shape shown in Fig. 6 if
$\alpha<0\,,\ \beta>0$.

Figure 6: Ginzburg-Landau symmetry breaking “Sombrero” potential

The minima of $V(\phi)$ do not occur at the symmetrical point $\phi=0$ but
rather at points in the “rim” where $\phi^{\ast}\phi=-\frac{\alpha}{\beta}$.
This shift to a minimum in the “rim” also produces a term $-\frac{2\alpha
e^{2}}{\beta m}{\bf A}\cdot{\bf A}$ giving a mass to the electromagnetic
vector field $\bf A$. Consequently, the photon becomes massive in the broken-
symmetry phase of a superconductor [Anderson 1963].

The idea that a similar mechanism could also give masses to elementary
fermions in a relativistic field theory had been suggested by Y. Nambu [Nambu
1960]. However, the Ginzburg-Landau model involves an effective scalar-field
order parameter which is clearly not a fundamental field of the
superconducting theory. The model was also non-relativistic, and the
complications imposed by relativity seemed at the time to be crucial.
Moreover, giving a mass to the photon was something definitely not wanted in a
fundamental relativistic theory.

###  2.3 Goldstone modes

Nambu and Jona-Lasinio proposed in 1961 a four-fermion interaction model based
on a spinor field $\psi$ with both an ordinary $U(1)$ rigid phase symmetry
$\psi\to e^{i\alpha}\psi$ and also a rigid chiral $U(1)$ phase symmetry
$\psi\to e^{\tilde{\alpha}\gamma_{5}}\psi$ with a Lagrangian interaction term
${\cal L}_{\rm int}=g[(\bar{\psi}\psi)^{2}-(\bar{\psi}\gamma_{5}\psi)^{2}]$
[Nambu 1961a, Nambu 1961b]. In this model, a condensate of the $\psi$ field
occurs: $\langle\bar{\psi}\psi\rangle\neq 0$. The model successfully gives a
mass to the fermion, but it also gives rise to a massless pseudoscalar mode.
Nambu and Jona-Lasinio proposed that, given additional soft explicit chiral
symmetry breaking, the pseudoscalar mode could in the end have a small mass
and could be identified with the pion. (Indeed, such explicit chiral symmetry
breaking is understood now to arise from quark masses in the underlying QCD
theory.)

The fact that spontaneous symmetry breaking of rigid symmetries inevitably
leads to the existence of massless modes is the content of Goldstone’s
Theorem, named after Jeffrey Goldstone (FRS 1974) of which a proof was given
in 1962 by Goldstone, Salam and Weinberg [Goldstone 1962]. This is illustrated
by the two modes of a complex scalar sketched in Fig. 6. The radial mode
clearly experiences a restoring force, but the tangential “rolling mode” does
not. In a relativistic theory, the radial mode experiencing a restoring force
corresponds to a massive particle, while the rolling mode corresponds to a
massless particle.

The seeming inevitability of Goldstone modes arising in consequence of
symmetry breaking posed a real obstacle to the implementation of the symmetry-
breaking idea in relativistic theories, since no such massless bosons were
known in physics. Attempts to circumvent the occurrence of such massless
Goldstone modes by the addition of explicit symmetry-breaking mass terms were
not acceptable when considered at the level of fundamental theory because such
terms give rise to non-renormalizability, i.e. uncontrollable behaviour at
high energies. Such behaviour could be contemplated in an approximate
effective theory (such as in pion physics) but not at the level of a
fundamental theory of microphysics.

###  2.4 The Brout-Englert-Higgs-Guralnik-Hagen-Kibble mechanism

The resolution of the Goldstone-mode problem came within a short period in
1964 from the work of three independent groups: Brout & Englert; Higgs;
Guralnik, Hagen & Kibble.

The key point was the recognition that Goldstone’s theorem could be evaded in
a gauge theory. Peter Higgs (FRS 1980) had previously shown that, for a gauge
theory formulated in radiation gauge, the lack of explicit relativistic
Lorentz invariance arising from the gauge choice renders Goldstone’s theorem
inapplicable.

The three groups approached the symmetry-breaking problem from different
perspectives:

  * 1.

Brout and Englert, in the first paper to be published, based their argument on
a calculation of vacuum polarization in lowest-order quantum perturbation
theory about the symmetry-breaking vacuum state [Brout 1964]

  * 2.

Higgs’s treatment was purely classical and was similar to the Ginzburg-Landau
model of superconductivity (which he did not know at the time), but in a
relativistic model [Higgs 1964]

  * 3.

Guralnik, Hagen and Kibble used a quantum operator formalism, concentrating on
the role of the current conservation law and studying the precise way in which
the Goldstone theorem could be evaded [Kibble 1964a]

The key conclusion of this fundamental development was that in a gauge theory,
the Goldstone mode can be absorbed into the longitudinal mode of the vector
gauge field, rendering it massive. The existence of a residual scalar mode,
which was discussed only in Higgs’s paper, was not considered important at the
time.

###  2.5 Tom Kibble’s 1967 paper, the massless photon and the Standard Model

The three BEHGHK 1964 papers addressed the key problem of how to avoid
Goldstone’s theorem and generate masses for vector fields by spontaneous
symmetry breaking. It was hoped that this could lead to a renormalizable
theory with controlled ultraviolet behaviour. (This was later shown by Gerard
’t Hooft and Martinus Veltman [Veltman 1972].) However, another key issue
remained for a successful application to particle physics: how can one ensure
that some gauge fields remain massless, and in particular the photon field?

The three 1964 symmetry-breaking papers had attracted little attention at the
time. Meanwhile, in Glashow [Glashow 1961] and Salam and Ward (unaware of
Glashow’s work) [Salam 1964] had proposed a unified model of the weak and
electromagnetic interactions based on the gauge group ${\rm SU}(2)\times{\rm
U}(1)$, but without any explanation for the generation of mass needed for the
vector particles responsible for the weak interactions.

What was needed to pull these diverse elements together to form what we know
know as the Standard Model was a deeper understanding of the general structure
of spontaneous symmetry breaking in gauge theories. In 1967, Tom Kibble wrote
a magisterial paper [Kibble 1967] laying out the complete mathematical
structure of spontaneous symmetry breaking and the Higgs effect. It provided
the complete explanation of which vector gauge fields become massive and which
remain massless after spontaneous symmetry breaking and the Higgs effect.

This paper was acknowledged by Steven Weinberg in his 1967 paper [Weinberg
1967] as settling the issue the issue of which gauge fields remain massless.
This was crucial to the creation of a unified model of the weak and
electromagnetic interactions, as the quantum of the electromagnetic field, the
photon, had to remain massless. This model, stemming from work by Sheldon
Glashow [Glashow 1961] and culminating in the 1967-68 papers of Weinberg and
Salam [Salam 1968], led to the award of the Nobel Prize for Physics in 1979.
See Fig. 5.

A major part of what is now known as the Standard Model, this unification of
weak and electromagnetic interactions is based upon the electroweak gauge
group ${\rm SU}(2)\times{\rm U}(1)$, which has four Lie algebra generators.
The corresponding four vector gauge fields couple to a Higgs complex scalar
field gauge-symmetry doublet with a Lagrangian potential term similar to the
“sombrero” potential sketched in Fig. 6. The Higgs complex doublet contains
four real fields. Three of these belong to the broken-generator coset $({\rm
SU}(2)\times{\rm U}(1))/{\rm U}(1)_{\rm em})$, where the unbroken
electromagnetic subgroup ${\rm U}(1)_{\rm em}$ mixes one combination of ${\rm
SU}(2)$ generators with the ${\rm U}(1)$ “hypercharge” factor of the original
gauge group prior to symmetry breaking. These erstwhile Goldstone fields are
absorbed by the Higgs effect into masses for three of the vector gauge fields.
The gauge field for the fourth gauge group generator, corresponding to the
unbroken ${\rm U}(1)_{\rm em}$ subgroup, remains massless. This is the
electromagnetic photon field $A_{\mu}$.

To date, the electroweak Standard Model is one of the most precisely tested
and verified achievements of elementary particle physics.

###  2.6 Cosmic strings

In 1976, Tom once more wrote an extremely influential paper on the
consequences of spontaneous symmetry breaking, entitled ‘The Topology of
Cosmic Domains and Strings’ [Kibble 1976]. The focus now was on the
topological structure of solutions and domain formation during phase
transitions.

When applied to proposed extensions of the unification idea to include also
the strong interactions, spontaneous symmetry breaking leads to nonsingular
magnetic “monopole” solutions, as found by Gerard ‘t Hooft [’t Hooft 1974 ]
and Alexander Polyakov [Polyakov 1974 ] in an $SU(2)$ model with a triplet
Higgs field in 1974\. These involve nontrivial maps from the 2-sphere at
spatial infinity to the vacuum manifold $SU(2)/U(1)$. Tom’s 1976 paper
considered the cosmological implications of more general “topological
defects”: topologically stabilised strings and domain walls as well as
monopoles. These possibilities are classified by the homotopy groups
$\pi_{k}{(\cal M})$ of the Higgs vacuum manifold, ${\cal M}$, i.e. by maps
from an $S^{k}$ sphere at spatial infinity to ${\cal M}$. If $\pi_{0}({\cal
M})\neq 0$, one can have domain walls; if $\pi_{1}({\cal M})\neq 0$, one can
have string solutions and if $\pi_{2}({\cal M})\neq 0$, one can have
monopoles. Kibble realised that these structures could condense as the
universe cooled from the hot conditions prevailing in the big-bang, and might
therefore have striking effects on the development of large-scale structure in
the universe.

Finite-temperature corrections induce changes to the effective potential for
the Higgs scalar fields, with an unbroken gauge symmetry $G$ being restored
above some critical temperature $T_{c}$. Below this temperature, phase
transitions will take place, producing a phase domain structure separated by
such topological defects. The values of the symmetry breaking order parameter
can be roughly constant in widely separated domains, but can be in conflict
where domains meet. This causes formation of a topological defect, in the
center of which the order parameter rises to its unbroken symmetry value, with
a consequent concentration of energy. In the Big Bang origin of the universe,
gauge symmetries would be restored by the initial high temperatures, with a
rapid thermal freeze-out quench as the temperature decreased below $T_{c}$.

The implications of $\pi_{0}({\cal M})\neq 0$ domain walls are very bad for
cosmology, since they would be very massive and would cause an impossibly
large anisotropy in the 3°K cosmic microwave background [Kibble 1976].
Consequently, the Higgs vacuum manifold must be connected. This constraint on
models of particle physics heralded a very fertile interaction between
cosmology and fundamental particle theory. Similarly there is a danger of an
over-abundance of monopoles; a problem nowadays addressed by theories of
cosmic inflation.

Cosmic strings, on the other hand, do not suffer from the same over-dominance
problems as domain walls and monopoles. Over the years Tom and co-workers
including Turok [Kibble 1982] Lazarides [Kibble 1982], Shafi [Kibble 1982],
Copeland [Kibble 1993], Hindmarsh [ Kibble 1995], Vilenkin [Kibble 1995],
Davis [Kibble 2005] and Vaschaspati [Kibble 2015] considered models for
structure formation seeded by cosmic strings. Kibble’s vision has thereby
provided an extraordinary link between the macroscopic and microscopic
features of our universe. See Fig. 7.

Figure 7: Cosmic string network (courtesy of Carlo Contaldi)

###  2.7 String solitons, M-theory and D-branes

The developments in Kibble’s cosmic strings, non-perturbative macroscopic
solutions of grand unified quantum field theories, and perturbative
superstrings, candidates for the fundamental microscopic constituents of
matter incorporating gravity, proceeded along different lines (although in
1985 Witten (ForMem 1999) speculated that fundamental strings produced in the
early universe might be stretched to macroscopic scales which however get
diluted away by inflation [Witten 1985]).

However, things began to change with the realisation that topological defects
such as monopoles, strings and domain walls appear as solitonic solutions of
the string equations which, in the low energy limit, are those of
supergravity. Loosely speaking, these solitons are non-singular field
configurations which solve the source-free field equations, carry a non-
vanishing topological “magnetic” charge and are stabilised by a topological
conservation law. They are intrinsically non-perturbative with a mass
depending on the inverse of the coupling constant. See the reviews [Duff 1995]
111 This was co-authored by one of us (MJD) while attending the six-month long
workshop Topological Defects organised by Tom Kibble at the Isaac Newton
Institute, Cambridge, in 1994. and [Stelle 1997]. They are to be compared and
contrasted with the elementary field configurations which are singular
solutions of field equation with a source term and carry a non-vanishing
Noether “electric” charge.

Moreover, since string theory allows ten spacetime dimensions, and its non-
perturbative completion M-theory allows eleven, they admit as solutions a
menagerie of higher-dimensional objects in addition to the fundamental
F-string, such as M-branes, Dirichlet-branes (D-branes) and Neveu-Schwarz
branes (NS-branes). They behave like strings if all but one of their spatial
dimensions are wrapped around the extra dimensions of spacetime. As Tom Kibble
remarked, “string theory cosmologists have discovered cosmic strings lurking
everywhere in the undergrowth”. In particular Sarangi and Tye [Sarangi 2002]
predicted the production of cosmic superstrings in the last stages of brane
inflation and Polchinski [Polchinski 2005] realised that the expanding
universe could not only stretch F and D-strings to galactic size but, with
large extra dimensions, they could be light enough to survive inflation. Tom
continued to make significant contributions to this new cosmic stringy lease
of life [Kibble 2010].

Cosmic strings are not the dominant source of density perturbations. Precision
measurements of the cosmic microwave background (CMB) suggest that initial
random gaussian fluctuations, subsequently amplified by inflation, are the
dominant source and that cosmic strings cannot account for more than about 10%
of CMB structure. However, some such component could still have observable
consequences. Gravitational lensing and gravitational radiation also provide
possible observational features of cosmic strings.

###  2.8 Condensed matter systems

Wojciech Zurek pointed out that Tom’s description of topological defects also
apply to the phase transition of normal fluid helium to superfluid helium
[Zurek 1985]. This became known as the“Kibble-Zurek mechanism”. Such a
correspondence is found also in other condensed matter systems, for example
liquid crystals [Kibble 2002, Kibble 2007]. Indeed, Tom was involved in an
experiment to search for defect formation in the B phase of He-3 where the
experimental conditions mimic the cosmological big bang [Kibble 1996]. His
work inspired several experimental teams to search for defects formed in phase
transitions in condensed matter systems, for example Lancaster (He-3 and
He-4), Grenoble and Helsinki (superfluid systems), US (liquid crystals). These
effects have been experimentally confirmed in the context of vortex formation
in superfluid Helium 3 [Kibble 1996, Kibble 2002, Kibble 2007].

##  3 LEADERSHIP, MENTORING AND SOCIAL RESPONSIBILITY

In 1970 Kibble became Professor of Theoretical Physics at Imperial, and held
the position of head of the Department of Physics from 1983 to 1991. He was an
outstanding teacher and his undergraduate textbook “Classical Mechanics ” (now
in a 5th edition co-authored with Frank H. Berkshire) is widely regarded as
the best of its kind. He was also one of the winners of a competition set by
the then Secretary of State William Waldegrave to explain the Higgs Boson to
the general public on a single sheet of A4.

He was elected Fellow of the Royal Society in 1980 and served as its vice-
president in 1988-89. At a fairly difficult time, in 1984, he chaired the
SERC’s Nuclear Physics Board. He received a CBE in 1998 and a Knighthood in
2012. His numerous awards are listed in section HONOURS AND AWARDS.

Kibble was concerned about the nuclear arms race and took leading roles in
several organisations promoting the social responsibility of science. These
included the British Society for Social Responsibility in Science, Scientists
against Nuclear Arms, Scientists for Global Responsibility and the Martin Ryle
Trust.

##  4 PERSONALITY

Brilliant physicist, inspirational teacher and leader though he was, Tom
conducted himself with an amazing degree of modesty and humility. He also
showed unfailing kindness to the colleagues and students with whom he came
into contact. Tom and his wife Anne frequently invited the whole Theory Group
to parties at their home in Richmond.

A colleague of Tom at Imperial College, Anne-Christine Davis, now Emeritus
Professor of Mathematical Physics at DAMTP Cambridge, writes [Davis 2020];
“Tom was a mentor to many younger physicists and a strong supporter of women
physicists, helping not just those in his immediate vicinity but others in the
wider community. At times when one needed a sympathetic ear, or good advice,
Tom was there and was at times instrumental in helping female physicists. His
dedication to helping and mentoring younger physicists and in particular
female physicists was recognised with the 2005 Mentoring lifetime achievement
award, NESTA/Nature. Tom also had a good sense of humour, sometimes dry
humour, sometimes even wicked! At Imperial Tom liked most to be surrounded by
others in his research group, in particular the younger members and would make
a point of joining them for lunch and tea rather than sitting with senior
colleagues.”

In a moving eulogy [Kibble 2020], his son Robert provided some anecdotes
giving us more glimpses of his personality. One recalls his being introduced
to someone who said “I remember you teaching me thirty years ago, and I’ve
just come back to finish my PhD” Tom congratulated him, shook his hand, and
afterwards asked who that had been, to be told that it was Queen guitarist
Brian May. Another recalls that while travelling on a bus in his 70s, a
pickpocket stole his wallet. Tom leapt off the bus and chased him down the
street, eventually forcing him to drop it. Tom was a passionate supporter of
many causes, frequently ending up running the organisations he joined.
(Scientists against Nuclear Arms and the Richmond Rambling Association, to
name but two.) A little known fact is that Tom was an accomplished dancer.

##  5 DISCOVERY OF THE HIGGS BOSON: NOBEL DILEMMA

Figure 8: Tom Kibble at the CMS Detector, CERN (courtesy of Imperial College)

As we have seen, one of Kibble’s most important pieces of work was his study
of the spontaneous symmetry-breaking mechanism whereby vector particles can
acquire a mass accompanied by the appearance of a massive scalar boson. This
mechanism, put forward independently by Brout and Englert [Brout 1964], by
Higgs [Higgs 1964] and by Guralnik, Hagen and Kibble [Kibble 1964] in 1964
lies at the heart of the Standard Model and all modern unified theories of
fundamental particles. It was vindicated in 2012 by the discovery of the Higgs
boson at CERN; the massive W and Z vector bosons having already been
discovered at CERN in 1982 and 1983, respectively. Fig. 8 shows Tom standing
next to the Compact Muon Solenoid (CMS) which, together with ATLAS, detected
the Higgs boson. This presented a much-debated dilemma for the Nobel
Committee, notwithstanding Brout’s demise in 2011, as the Prize can be awarded
to at most three. However, Kibble was also sole author of a 1967 paper,
focusing on the non-abelian generalisation. Indeed, in early papers, the
mechanism was known as the “Higgs-Kibble effect”. See, for example, the 1972
paper by ‘t Hooft and Veltman [‘t Hooft 1972] for which they won the 1999
Nobel Prize. Higgs and Kibble were the joint winners of the 1981 Hughes medal
of the Royal Society and the 1984 Rutherford Medal and Prize of The Institute
of Physics.

Tom’s 1967 paper laid out the complete mathematical structure of spontaneous
symmetry breaking, settling the issue of which gauge fields become massive and
which remain massless. Private discussions with Salam informed his thinking on
the problem. For a gauge field theory model based on a non-abelian gauge group
$G$ and a scalar Higgs sector producing spontaneous symmetry breaking down to
a subgroup $H$, vector fields corresponding to the coset $G/H$ become massive,
while the vectors corresponding to the unbroken subgroup $H$ remain massless.
When the Standard Model came along shortly afterwards, Kibble’s paper was seen
to explain not only why the W and Z acquire a mass but, equally crucial, why
the photon does not. His 80th birthday was marked by Imperial College in March
2013 with a symposium day. In his evening public lecture, Nobel Laureate
Steven Weinberg [Weinberg 2013] ended by saying “Tom Kibble showed us why
light is massless”. Indeed Higgs said that Kibble should have shared the 2013
Nobel Prize awarded to Englert and himself “because of what he wrote in 1967”
[Higgs 2012]. Kibble himself maintained a dignified modesty throughout in
keeping with the honesty and integrity for which he was justly famous.

##  6 HONOURS AND AWARDS

1980 Fellow of the Royal Society

1981 Hughes Medal of the Royal Society (with P.W. Higgs)

1984 Rutherford Medal & Prize, Institute of Physics (with P.W. Higgs)

1991 Fellow of the Institute of Physics

1993 Guthrie Medal & Prize, Institute of Physics

1998 CBE

2005 Mentoring lifetime achievement award, NESTA/Nature

2009 Dirac Medal IOP

2010 Sakurai Prize for Theoretical Physics

2012 Royal Medal, Royal Society

2012 Honorary Fellow IOP

2012 Einstein Medal

2013 Knighthood

2014 Royal Medal, Royal Society of Edinburgh

2016 Isaac Newton Medal IOP

##  7 REFERENCES TO OTHER AUTHORS

[Yang 1954] Conservation of isotopic spin and isotopic gauge invariance. Yang,
C. N. and Mills, R. L.
Phys, Rev., 96 191-195 (doi:10.1103/PhysRev.96.191)

[Shaw 1954] Imperial College PhD thesis

[Nambu 1960] Dynamical theory of elementary particles suggested by
superconductivity. Nambu, Y.
In Proceedings of ICHEP 60, 858-866

[Nambu 1961a] Dynamical Model of Elementary Particles Based on an Analogy with
Superconductivity, I. Nambu, Y. and Jona-Lasinio, G. Phys.Rev. 122 (1961)
345-358 (doi:10.1103/PhysRev.122.345)

[Nambu 1961b] Dynamical Model of Elementary Particles Based on an Analogy with
Superconductivity, II. Nambu, Y. and Jona-Lasinio, G. Phys.Rev. 124 (1961)
246-254 (doi:10.1103/PhysRev.124.246)

[Glashow 1961] Partial Symmetries of Weak Interactions. Glashow, S.
Nucl. Phys. 22 (1961), 579-588 (doi:10.1016/0029-5582(61)90469-2)

[Goldstone 1962] Broken symmetries. Goldstone J., Salam, A. and Weinberg, S.
Phys.Rev. 127 965-970 (doi:10.1103/PhysRev.127.965)

[Anderson 1963] Plasmons, gauge invariance, and mass. Anderson, P. W.
Phys. Rev. 130 439-442 (doi:10.1103/PhysRev.130.439)

[Brout 1964] Broken Symmetry and the Mass of Gauge Vector Mesons. Brout, R.
and Englert, F.
Phys. Rev. Lett. 13 (1964) 321-323 (doi:10.1103/PhysRevLett.13.321)

[Higgs 1964] Broken symmetries, massless particles and gauge fields. Higgs, P.
Phys.Lett. 12 (1964) 132-133 (doi:10.1016/0031-9163(64)91136-9)

[Salam 1964] Electromagnetic and weak interactions. Salam, A. and Ward, J.
Phys. Lett. 13 (1964),168-171 (doi10.1016/0031-9163(64)90711-5)

[Weinberg 1967] A Model of Leptons. Weinberg, S.
Phys. Rev. Lett. 19 (1967), 1264-1266 (doi:10.1103/PhysRevLett.19.1264)

[Salam 1968] Weak and electromagnetic interactions. Salam, A. In Elementary
Particle Physics: Relativistic Groups and Analyticity, N. Svartholm, Ed.,
Almquvist and Wiksell, Stockholm (1968) 367, Proceedings of the Eighth Nobel
Symposium.

[Veltman 1972] Regularization and renormalization of gauge fields. ’t Hooft,
G. and Veltman, M.
Nuclear Physics B, 44(1), 189-213 (doi:10.1016/0550-3213(72)90279-9)

[’t Hooft 1974] Magnetic monopoles in unified gauge theories. ’t Hooft, G.
Nucl. Phys. B 79 276-284 (doi:10.1016/0550-3213(74)90486-6)

[Polyakov 1974] Particle Spectrum in the Quantum Field Theory. Polyakov, A. M.
JETP Letters. 20 (6): 194-195 ( ISSN 0370-274X)

[Zurek 1985] Cosmological experiments in superfluid helium? Zurek, W.
Nature 317 (6037): 505-508 ( doi:10.1038/317505a0)

[Witten 1985] Cosmic superstrings. Witten, E.
Phys. Lett. B 153, 243-246 (doi:10.1016/0370-2693(85)90540-4).

[Stelle 1997] BPS branes in supergravity. Stelle, K. S.
In Trieste 1997, High energy physics and cosmology 29-127

[Sarangi 2002] Cosmic string production towards the end of brane inflation.
Sarangi, S. and Tye, S. H.
Physics Letters B. 536 (3-4): 185 (doi:10.1016/S0370-2693(02)01824-5. S2CID
14274241)

[Copeland 2004] Cosmic F- and -D strings. Copeland, E., Myers, R. and
Polchinski, J.
JHEP 06 013-013 (doi: 10.1088/1126-6708/2004/06/013)

[Polchinski 2005] Cosmic superstrings revisited. Polchinski, J.
Int. J. Mod. Phys. A 20, 3413-3415 (doi:10.1142/S0217751X05026686).

[Weinberg 2012]
http://www3.imperial.ac.uk/newsandeventspggrp/imperialcollege/eventssummary/event_21-1-2013-16-35-46

[Higgs 2012a] (The Scotsman, 13 November 2013)

[Higgs 2012b] http://www.telegraph.co.uk/science/science-news/10443449/Peter-
Higgs-Tom-Kibble-should-have-shared-Nobel-Prize-with-me.html

[Hannesdottir 2020] An S-matrix for massless particles. Hannesdottir, H. and
Schwartz, M. D.
Phys Rev D101 105001. (doi:10.1103/PhysRevD.101.105001)

[Davis 2020] Private Communication. Davis, A-C.

[Kibble 2020] Private Communication. Kibble, R.

##  8 BIBLIOGRAPHY

[Kibble 1961] Lorentz invariance and the gravitational field.
J. Math. Phys. 2, 212-221 (doi: 10.1063/1.1703702)

[Kibble 1964a] (with Guralnik, G. S. and Hagen, C. R.) Global Conservation
Laws and Massless Particles. Phys.Rev.Lett. 13 (1964) 585-587
(doi:10.1103/PhysRevLett.13.585)

[Kibble 1964b] (With Brown, L. S.) Interaction of intense laser beams with
electrons.
Phys. Rev. 133, A705-19 (doi:10.1103/Phys. Rev.133.A)

[Kibble 1967] Symmetry-breaking in non-Abelian gauge theories.
Phys. Rev. 155 (1967), 1554-1561 (doi:10.1103/Phys Rev.155.1554)

[Kibble 1968a] Coherent Soft-Photon States and Infrared Divergences. i.
Classical Currents.
J. Math. Phys. 9 no. 2 315-324 (doi: 10.1063/1.16645820)

[Kibble 1968b] Coherent Soft-Photon States and Infrared Divergences. ii. Mass-
Shell Singularities of Green’s Functions. Phys. Rev. 173 1527-1535 (doi:
10.1103/PhysRev.173.1527)

[Kibble 1968c] Coherent Soft-Photon States and Infrared Divergences. iii.
Asymptotic States and Reduction Formulas. Phys. Rev. 174 1882-1901
(doi:10.1103/PhysRev.174.1882)

[Kibble 1968d] Coherent Soft-Photon States and Infrared Divergences. iv. The
Scattering Operator.
Phys. Rev. 175 1624-1640 (doi: 10.1103/PhysRev.175.1624)

[Kibble 1976] Topology of cosmic domains and strings.
J. Phys. A: Math. & Gen. 9, 1387 (doi: 10.1088/0305-4470/9/8/029)

[Kibble 1979] Geometrization of quantum mechanics.
Commun. Math. Phys. 65, 189-201 (doi: 10.1007/BF01225149 )

[Kibble 1980] Some Implications of a Cosmological Phase Transition.
Phys. Rept. 67, 183 (doi: 10.1016/0370-1573(80)90091-5)

[Kibble 1982] (with Turok, N.) Selfintersection of cosmic strings.
Phys.Lett.B 116, 141-143 (doi: 10.1016/0370-2693(82)90993-5)

[Kibble 1982] (With Lazarides, G. and Shafi, Q.) Walls bounded by strings.
Phys. Rev. D26 435-439 (doi: 10.1103/PhysRevD.26.435)

[Kibble 1986] (With Stelle, K. S.) Gauge theories of gravity and supergravity.
In Progress in quantum field theory, ed. H. Ezawa & S. Kamefuchi (Amsterdam:
North-Holland, 57-81. Festschrift for H. Umezawa

[Kibble 1993] (with Austin, D. and Copeland, E.) Evolution of cosmic string
configurations.
Phys. Rev. D48 5594-5627 (doi: 10.1103/PhysRevD.48.5594)

[Kibble 1995] (With Hindmarsh, M. B.) Cosmic strings.
Reports on Progress in Physics 58, 477-562 (doi: 10.1088/0034-4885/58/5/001)

[Kibble 1995] (with Vilenkin, A.) Phase equilibration in bubble collisions.
Phys.Rev.D 52 (1995) 679-688 (doi:10.1103/PhysRevD.52.679)

[Kibble 1996] (With Ruutu, V .M., Eltsov, V. B., Gill, A. J., Krusius, M.,
Makhlin, Yu. G., Plaçais, B., Volovik, G. E. and Xu Wen), Vortex formation in
neutron-irradiated superfluid 3He-B as an analogue of cosmological defect
formation. Nature 382 (1996) 334-336 (doi:10.1038/382334a0)

[Kibble 1997] Recollections of Abdus Salam at Imperial College. In The Abdus
Salam Memorial Meeting, eds Ellis, Hussain, Kibble, Thompson, Virasoro, World
Scientific 1999

[Kibble 2002] Symmetry breaking and defects. In Patterns of Symmetry Breaking,
ed. H. Arodz, J. Dziarmaga & W.H. Zurek, NATO Science II, 127, pp. 3-36,
Proceedings of NATO Advanced Study Institute, Cracow, September 2002

[Kibble 2005] (With Davis, A-C.) Fundamental cosmic strings.
Contemporary Physics, 46, 313-322 (doi:10.1080.00107510500165204)

[Kibble 2007] Phase transition dynamics in the lab and the universe.
Physics Today 60, no. 9, 47-53 (doi:10.1063/1.2784684)

[Kibble 2009] Englert-Brout-Higgs-Guralnik-Hagen-Kibble mechanism.
Scholarpedia 4(1):644

[Kibble 2010] (With Copeland, E. J.) Cosmic strings and superstrings.
Proc. Roy Soc. A 466, 623-657 (doi:10.1098.rspa.2009.0591)

[Kibble 2015] (With Vachaspati, T.) Monopoles and strings.
J. Phys. G42, 094002 (doi:10.1088/0954-3899/42/9/094002)

##  9 ACKNOWLEDGEMENTS

We thank Carlo Contaldi, Anne Davis, Graziela De Nadai, Philip Diamond, John
Ellis, Chris Isham, Robert Kibble, Alison Martin, Arttu Rajantie, Ray Rivers
and Helen Wilson, for help in preparing this memoir.

##  10 AUTHOR PROFILES

  * 1.

Michael Duff

Michael Duff (FRS 2009) is Emeritus Professor of Theoretical Physics and
Senior Research Investigator at Imperial College London. He was formerly Abdus
Salam Professor of Theoretical Physics and Principal of the Faculty of
Physical Sciences. His time in the Theory Group overlapped with that of Tom
Kibble for a total of 23 years. He is recipient of the 2004 Meeting Gold
Medal, El Colegio Nacional, Mexico, the 2017 Paul Dirac Gold Medal and Prize,
Institute of Physics, UK and the 2018 Trotter Prize. He is a visitor to The
Institute of Quantum Science and Engineering and was inducted as 2019 Faculty
Fellow of the Hagler Institute for Advanced Study, both at Texas A&M
University.

  * 2.

Kellogg Stelle

Kelly Stelle is Professor of Theoretical Physics at Imperial College London
and formerly Head of the Theoretical Physics Group. He is the recipient of the
2006 Humboldt Research Award, Alexander von Humboldt Foundation, and the 2020
John William Strutt, Lord Rayleigh Medal and Prize, Institute of Physics.

[6.5cm] [6.5cm]

Michael Duff FRS (Portrait by Beyond the Pixels.) Kellogg Stelle (courtesy of
Imperial College)