# Dataset of gold nanoparticle sizes and morphologies extracted from
literature-mined microscopy images

Akshay Subramanian1,2, Kevin Cruse2,3, Amalie Trewartha2,
Xingzhi Wang2,4, Paul Alivisatos2,3,4,5, Gerbrand Ceder2,3,*

###### Abstract

The factors controlling the size and morphology of nanoparticles have so far
been poorly understood. Data-driven techniques are an exciting avenue to
explore this field through the identification of trends and correlations in
data. However, for these techniques to be utilized, large datasets annotated
with the structural attributes of nanoparticles are required. While
experimental SEM/TEM images collected from controlled experiments are reliable
sources of this information, large-scale collection of these images across a
variety of experimental conditions is expensive and infeasible. Published
scientific literature, which provides a vast source of high-quality figures
including SEM/TEM images, can provide a large amount of data at a lower cost
if effectively mined. In this work, we develop an automated pipeline to
retrieve and analyse microscopy images from gold nanoparticle literature and
provide a dataset of 4361 SEM/TEM images of gold nanoparticles along with
automatically extracted size and morphology information. The dataset can be
queried to obtain information about the physical attributes of gold
nanoparticles and their statistical distributions.

1 Indian Institute of Technology Roorkee, Roorkee, Uttarakhand 247667, India

2 Materials Sciences Division, Lawrence Berkeley National Laboratory,
Berkeley, California 94720, United States

3 Department of Materials Science and Engineering, University of California,
Berkeley, Berkeley, California 94720, United States

4 Department of Chemistry, University of California, Berkeley, Berkeley,
California 94720, United States

5 Kavli Energy NanoScience Institute, Berkeley, California 94720, United
States

* Corresponding author: Gerbrand Ceder<EMAIL_ADDRESS>
##  1 Background and Summary

Scanning Electron Microscopy (SEM) and Transmission Electron Microscopy (TEM)
are two of the most widely used imaging techniques for nanoparticles. A
variety of qualitative and quantitative information about the structural
characteristics of nanoparticles can be retrieved from the analysis of these
images. In particular, information about the physical attributes of
nanoparticles such as sizes and morphologies that have direct bearing on the
properties of the material can be especially valuable. [1, 2, 3, 4] The
factors controlling these attributes are poorly understood, and so aggregating
this information on a large scale and identifying trends in this data is key
to better understanding the synthesis and properties of nanoparticles. Data-
driven and statistical analysis techniques can potentially play an important
role in achieving this.

One possible avenue for acquisition of large numbers of images is through the
published scientific literature, which provides a vast source of high-quality
figures, including SEM/TEM images that can supply a large amount of data at a
low cost if effectively mined. However, the complex structure and lack of
uniformity in the way images are presented in literature makes the task of
effectively retrieving and utilizing this information challenging. [5]

Over the last few years, there has been significant progress in the
development of techniques to mine and analyse microscopy images present in
scientific literature. Mukaddem et al. [6] developed ImageDataExtractor, a
pipeline to automatically mine microscopy images from literature and perform
statistical analysis of particle sizes. Kim et al. [7] created a tool for the
size analysis of nanoparticles in SEM images, distinguishing between two broad
classes of nanoparticles: core-only and core-shell nanoparticles. Hiszpanski
et al. [8] utilized this tool to gather insights from nanoparticle literature.
Schwenker et al. [9] developed an automated pipeline to create a database of
images extracted from literature that are ’self-labelled’ with keywords
retrieved from captions.

While a suite of techniques have been developed to extract microscopy images
from literature and analyse particle sizes in them, there has been relatively
less progress in development of techniques to identify nanoparticle
morphologies (for example sphere, rod, cube etc.) at the particle-level in the
extracted microscopy images. Morphology identification tools have been
developed recently for experimental microscopy images using unsupervised
clustering approaches [10, 11], but they have not been designed to do so on
more noisy images such as those from literature which contain artifacts and
have high variability. Moreover, they have been developed as an additional
step that is carried out after particle segmentation, which results in the
accumulation of errors over the two steps. This lack in robust morphology
identification tools is expected to have also contributed to the scarcity in
publicly available morphology-annotated datasets of microscopy images.

In this work, we

  1. 1.

develop a fully-automated, deep learning-based pipeline to retrieve and
analyse microscopy images from scientific literature. The pipeline includes a
unified Mask-RCNN [12] model that can accurately segment nanoparticles
(including overlapping particles), and simultaneously identify human-
interpretable morphologies of the segmented particles at the particle-level.
Through this we attempt to bridge the gap in morphology identification tools
in past work.

  2. 2.

provide a dataset of 4361 SEM/TEM images of gold nanoparticles along with
automatically extracted size and morphology information. Through this, we
attempt to address the need for publicly accessible morphology and size-
annotated data of microscopy images. While the dataset described in this paper
is a current snapshot, it will be updated dynamically as and when
modifications are made.

We believe that the importance of the techniques utilized in this work will be
amplified as robotic synthesis of nanoparticles becomes used in the future,
and can also be be extended/improved in several directions. One potential
extension that may be possible is to analyse the correlations between
synthesis recipes as described in the text [13], and the synthesis outcomes
(size, shape etc.) as extracted from images of scientific publications.

##  2 Methods

###  2.1 Image Extraction Pipeline

Our image extraction pipeline consists of six stages: 1) paper parsing, 2)
sub-figure separation, 3) isolation of particulate microscopy images, 4)
detection and interpretation of labels, scales and bars, 5) nanoparticles
segmentation and morphology classification, and 6) size measurement. The
techniques used in each stage are detailed in the following sections.

####  2.1.1 Paper Parsing

Parsers written using the Beautiful Soup Python library are used to
automatically retrieve image URLs, captions and figure titles (for example,
Figure 1 in this paper) from the HTML sources of papers. The scientific
publications used in this work are drawn from the following four publishers:
Elsevier, The Royal Society of Chemistry, Nature Publishing Group and
Springer, each of which use their own HTML syntax/style. Hence, separate
parsers had to be written for each publisher. In total, 131,103 articles were
parsed in this stage.

These articles were obtained through a two-step natural language processing
pipeline. First, from a database of 4,973,165 full-text materials science
articles [14], 811,905 nanomaterial articles were identified using regular
expression matching for any word in the full text starting with ”nano”. Next,
we used term frequency-inverse document frequency (TF-IDF) vectorization to
represent the uniqueness of each word in each article, where every element of
the TF-IDF vector of a given article represents the frequency of a word token
normalized by the number of documents in which that word token appears across
the entire corpus. For this step, we identified articles whose TF-IDF values
for the words ”gold” or ”Au” were larger than any of ”silver”, ”Ag”, ”copper”,
”Cu”, ”palladium”, ”Pd”, ”platinum”, or ”Pt”, similar to the nanomaterial
article processing pipeline developed in Hiszpanski et. al [8]. This final
step yielded the 130,692 gold nonmaterial articles used for TEM/SEM image
extraction.

Once figures from all papers are parsed, those that are likely to contain
SEM/TEM images of gold nanoparticles are selected with regular expression
(regex) filters and then downloaded. We apply three categories of regex
filters, based on: 1) type of microscopy image; 2) composition; and 3)
morphology of particles. The keywords belonging to each category are shown in
Table 1. For a figure to be downloaded, we require its captions to match at
least one keyword from each of the three categories. In addition to the
keywords mentioned in the table, we also include grammatical variants of each
by only matching for the stems of each keyword (stemming). We also note that
while we include ”aunr” as a keyword, we do not include ”aunp”, ”auns”, ”aunc”
and ”aunt” even though papers use them to refer to nanoparticles, nanospheres,
nanocubes and nanotriangles respectively. This is because we observed that the
same abbreviations are sometimes also used to refer to other morphologies. For
example, aunc is used for gold nanocrystals, auns is also used for gold
nanostars and aunp, which refers to gold nanoparticles, is used as a unified
term that can refer to any morphology. So, including these terms results in
the collection of microscopy images with undesired morphologies, and hence a
loss in precision. The downloaded images are carried forward to the next
stage.

####  2.1.2 Sub-figures Separation

A majority (nearly 80%) of the filtered images are composite figures. This
makes the localization and separation of sub-figures a necessary step in the
pipeline. For localization, we use the sub-figure localization algorithm that
was developed by Tsutsui et al. [15], which is based on the YOLOv2 object
detection network [16]. Running this algorithm on the figures extracted in the
previous stage gives us predicted sub-figure locations. These are then cropped
out, separated and passed on to the Image Classification stage. Figure 1(b)
shows a sample prediction of the sub-figure localization algorithm on an
extracted image.

Two broad classes of approaches have been utilized in past literature for this
task of sub-figure separation. The first class of approaches defines a sub-
figure as an entity in a composite figure that can be uniquely identified by a
label (i.e. a, b, c etc.), for example, work by Schwenker et al. [9]. In their
approach, labels in the composite figure are first localized and these
locations are used to guide the detection of sub-figures. The second class of
approaches simply treats any image present within the composite figure as a
sub-figure, for example, work by Mukaddem et al. [6]. In their approach, the
position of labels are not utilized in the process of sub-figure detection.

In cases where clusters of images fall under the same sub-figure label, the
former approach is expected to identify the entire cluster as a single sub-
figure, while the latter approach is expected to identify each image in the
cluster as a separate sub-figure. In our work, we use the latter class of
approaches, since our aim is to extract individual TEM/SEM images and clusters
of them often fall under the same subfigure label.

An advantage of the first class of approaches however, is their inherent
ability to match the separated subfigures with relevant portions of the
caption. This task is significantly more challenging in the latter approach,
specifically in cases where more than one image fall under the same label.

####  2.1.3 Isolation of Particulate Microscopy Images

This stage consists of two binary classifier models: Classifier-1 and
Classifier-2, which are sequentially applied to the extracted sub-figures.
Classifier-1 first identifies whether a given image is a microscopy (SEM/TEM)
image or not. In the case it is identified as one, Classifier-2 then
determines whether it contains nanoparticles or not. Through the sequential
application of these classifiers on the entire database, we are able to
isolate SEM/TEM images that contain nanoparticles in them.

Datasets to train and evaluate these classifiers were prepared by manually
annotating a chosen set of sub-figures with relevant class labels. More
details about the datasets used for each task are shown in Table 2.

We used the same model architecture and training procedure for both
classification tasks. A ResNet-50 [17] deep learning model, which was pre-
trained on the ImageNet dataset [18], was fine-tuned on the training set for
500 epochs, and the best performing model on the validation set was chosen.
The Stochastic Gradient Descent (SGD) optimizer was employed with learning
rate of 6.5e-3 during training. To reduce the effects of imbalance in class
frequencies on model training, we utilized weighted sampling of examples to
ensure that the model “saw” an equal number of examples from each class during
every batch of training. Model preparation and training were done using the
PyTorch deep learning framework [19] and tuning of hyperparameters was done
using the RayTune framework [20].

####  2.1.4 Detection and Interpretation of Labels, Scales and Bars

This stage consists of four major steps:

  1. 1.

A YOLOv4 [21] object detection network is used to locate labels, scales and
bars in the SEM/TEM images shortlisted during the classification stage. The
detected objects are then cropped and separated. A sample prediction is shown
in Figure 1(d).

  2. 2.

The cropped scales and labels are magnified and passed through an SRCNN super-
resolution model [22].

  3. 3.

Labels and scales are read using the Tesseract [23] OCR reader.

  4. 4.

The horizontal lengths (in pixels) of the cropped bars are then measured and
mapped to the text (digit and unit) read from the corresponding scales.

Since the SRCNN inference step (step 2) requires the cropped label/scale to be
passed through a deep learning model, it is a relatively time-consuming step,
and becomes a bottleneck if the label/scale is already clearly legible. Hence,
we perform this step only if directly performing the OCR reading step (step 3)
is unsuccessful. Success of the OCR reading step is determined by verifying if
the read text satisfies certain criteria. For scales, this includes conditions
such as (i) both a unit (nm or $\mu$m) as well as a number must be present in
the read text and (ii) the read number should be divisible by 5 if it is
greater than 10. We impose condition (ii) to improve the precision of read
values since a large majority of scales tend to either be single digit numbers
or double/triple digit numbers that are divisible by 5. For labels, the read
text must consist of either a single letter or a letter followed by a digit in
order to be considered successful. The Tesseract OCR reader tends to perform
best when dark text is located on a light background. Hence, for every failed
read attempt of the OCR reader, we repeat the step after replacing the
scale/label image with its color-inverted version. We observed that this
improves the extraction recall by improving performance on light-colored
scales and labels.

The YOLOv4 object detection network [21] mentioned in step 1 requires a
labelled dataset to be prepared for model training. Details on the dataset can
be found in Table 2. Annotation of ground truth bounding boxes and classes
were performed using the LabelImg [24] software. Model training was carried
out for 6000 iterations on the training set and the best performing model on
the validation set was chosen. Training and evaluation were performed using
the Darknet deep learning framework [25] and the model achieved a mean Average
Precision (mAP@50) of 0.880 (0.963 on scales, 0.962 on labels, and 0.716 on
bars) on the test set. The lower score on bars results from the high level of
variation in the way that bars are presented in literature. For example, some
papers use horizontal I-shaped lines, while others use lines calibrated with
precision markers instead of bars. In order to reduce the effect of this score
on performance, we use the horizontal length of the detected scale as an
approximation of the bar length in cases where the model fails on bar
detection but is successful in scale detection. Since bars are often present
within scale boxes, we observed that lengths of scale boxes and bars are
nearly equal in a majority of cases, and hence this can be considered a good
approximation.

Past efforts on the task of label and scale detection have mostly used
classical threshold-based segmentation techniques which make use of common
attributes such as the rectangular shape and white color of label and scale
boxes to identify them [6, 7]. By instead using the YOLOv4 model for this
task, we preclude any such prior assumptions on the size, shape, and color of
scales and labels. Hence, unlike classical approaches that require separate
rules to be written for labels, scales, and bars, the deep learning approach
is able to detect all three objects in a single forward pass without the need
for specification of any object-specific rules. Our approach is similar to the
one used by Schwenker et al. [9].

####  2.1.5 Nanoparticles Segmentation and Morphology Classification

For this task, we use the Mask-RCNN [12], an instance segmentation model that
is capable of performing both the tasks of segmentation and object-level
classification. We use its segmentation capability to perform nanoparticle
segmentation in microscopy images and its classification capability to perform
particle-level morphology identification. For the task of morphology
identification, we consider only four morphologies in this work - sphere, rod,
cube, and triangular prism since these are the shapes that occur most
frequently in our database. Figure 1(e) shows the prediction of the Mask-RCNN
model on a sample image.

A dataset to train and evaluate the Mask-RCNN model was prepared by annotating
microscopy images with segmentation masks and particle morphologies. More
details on the dataset can be found in Table 2. Since each image contained
nearly thirty particles on average, more than 5000 instances were annotated in
total. All annotation was done using the LabelBox software [26]. We found it
important to (i) include negative examples (images with no nanoparticles) and
(ii) include both low and high resolution images in the training data, in
order to improve the robustness of the trained model to the large variety of
images observed in literature. We also included 131 experimental TEM images to
increase the size of low frequency classes such as triangles. Of these 131
images, 114 are TEM images of gold nanoparticles that we synthesized [10], and
17 are microscopy images from the LLNL-MI-812379 release [7]. We release this
annotated dataset of 131 SEM/TEM images publicly on our GitHub repository.

For model training, ImageNet pre-trained weights were used as an
initialization. Training was then carried out for 300 epochs, of which the
first 50 involved training of only the network heads and the following 250
involved fine-tuning of the entire network. The model with the best
performance on the validation set was chosen. The final model achieved a mean
Average Precision (mAP@50) of 0.76 on the test set. While the segmentation and
classification performances on all 4 morphologies were good, we did observe
certain cases where cubes were mis-predicted as spheres, especially when the
particles are very small in proportion to the size of the image. We also
observed that the unique characteristics of nanoparticles as compared to
objects present in standard computer vision tasks required certain
hyperparameters to be tuned carefully. In particular, the sizes and aspect
ratios of anchor boxes that are used in the Region Proposal Network were found
to be the most important and needed to be tuned in accordance with the sizes
and aspect ratios seen in TEM images of nanoparticles. In addition, we found
data augmentation to significantly improve the performance of our model. The
augmentation strategies utilized include vertical and horizontal flipping,
rotation and Gaussian blurring of images. By augmenting the training data with
these transformed versions of the images, we improved the robustness of the
model to various orientations and degrees of noise in SEM/TEM images. We
believe that this is especially helpful in our case since the size of the
training dataset is relatively small.

Past work has primarily used classical image processing techniques for the
task of nanoparticle segmentation, such as Thresholding, Hough transforms and
the Watershed algorithm. [27, 28, 29] These techniques typically require a
significant amount of user-tuned parameters. A drawback with these classical
techniques is their inability to separate overlapping particles when the
degree of overlap is high, which is a very common characteristic of TEM/SEM
images containing nanoparticles. By instead using the Mask-RCNN, we
significantly improve performance on such cases. Figure 3(a) shows a
comparison of performances shown by a ”classical” algorithm (watershed
algorithm) and the Mask-RCNN on a TEM image from our database. While the
classical approach fails to distinguish between particles that overlap, the
Mask-RCNN model is clearly able to classify them as separate instances. The
implementation of the Watershed algorithm used is available on our GitHub
repository.

While there have been recent approaches [30, 31, 32, 33] that have also
utilized deep learning approaches to the task of nanoparticle segmentation and
thereby perform well on overlapping particles, we believe a key feature of our
approach using the Mask-RCNN model is its added ability to identify particle
morphologies. In addition, by incorporating both segmentation and morphology
identification into a single forward pass, we prevent errors from being
accumulated between the two tasks.

###  2.2 Size Measurement

We measure sizes differently for particles belonging to different
morphologies. A common initial step for all morphologies is to locate the
centroid of the particle. We then simulate lines originating from the centroid
and intersecting the boundaries of the particle. Constraints are then imposed
on these lines in order to shortlist those that span the dimensions we are
interested in measuring. For example, in order to measure the length of a rod,
we would simply choose the longest line (which would connect the centroid of
the rod to the rod tip) and double its length. Similarly, the width can be
obtained by doubling the length of the shortest line. The dimensions measured
for each morphology are: 1) rod: length and width, 2) cube: side length, 3)
triangular prism: height, and 4) sphere: diameter (averaged over all
diameters). These measured dimensions are then converted into the unit that
has been read from the scale to arrive at the final magnitude of the
dimension.

###  2.3 Web Application

To illustrate the capabilities of the described image extraction and analysis
techniques, we have developed a web application for automated analysis of TEM
and SEM images. This allows users to easily experiment with our algorithms on
custom images using a Graphical User Interface (GUI). The web application
supports all image analysis techniques mentioned so far and additionally
allows users to view statistics such as particle size distributions. It is
accessible at https://github.com/CederGroupHub/TEMExtraction-webapp.

##  3 Data Records

The complete dataset consists of 4361 literature-mined microscopy (SEM and
TEM) images of gold nanoparticles along with extracted particle sizes,
morphologies, and metadata corresponding to each image. Metadata includes
fields such as the DOI of the parent paper, URL of the parent composite
figure, and a unique hash for each image.

The dataset is publicly available as a JSON file (https://doi.org/10.6084/
m9.figshare.17019836.v2) and an associated Python script (https://github.com/
CederGroupHub/TEMExtraction). The JSON file contains a mapping between image
names and their corresponding particle sizes, morphologies, and image metadata
(complete schema shown in Table 3). The associated Python script can be used
to download all the microscopy images onto a local system. It achieves this by
1) Retrieving and downloading all parent composite figures from the URLs
present in the JSON file 2) Separating sub-figures from composite figures, and
3) Isolating the subset of images that belong to the dataset.

The set of papers that were used in this work were shortlisted from a much
larger set of Materials Science papers. More details on how this was achieved
are given in Section 2.1.1.

##  4 Technical Validation

###  4.1 Extraction Accuracy

All major steps in the pipeline have metric scores greater than 80%, as can be
seen in Table 2.

We noted that the recall of the OCR reading step (step 2 in Section 2.1.4) was
particularly low (30-40%), i.e., only 30-40% of the scales and labels in
microscopy images are successfully read by the OCR reader. We observed that
this low recall arises from the inability of the OCR reader to perform
”successfully” (see Section 2.1.4 for definition of ”success”) on cropped
labels/scales having low resolution or low contrast between text and
background (black text on a dark background for example). A high percentage of
cropped labels/scales have low resolutions since they often occupy a very
small portion of the parent microscopy image.

###  4.2 Morphology Distribution

Figure 3(b) illustrates the distribution of images among the four
morphologies. The percentages shown beside each section refer to the
fractional percentage of images that have a majority of particles belonging to
a given morphology. For example, 58.68% of the images have spheres as their
majority morphology. The pie chart indicates that there is a large imbalance
in the morphologies present in literature images, with spheres being most
common, followed by rods, cubes and finally triangles.

###  4.3 Impurities

Each SEM/TEM image can contain nanoparticles of one or more morphology
classes. When we explore the dataset to try and identify trends in the co-
occurrence of various morphologies, we observe that spheres co-occur with
other morphologies most often. This observation matches with intuition since
spheres are often formed as a byproduct during the shape controlled synthesis
of more unique nanoparticle morphologies such as gold nanorods. [34, 35, 36,
10]

We then explored the fractional distribution of co-occurring morphologies in
each image. The fraction of spheres in an image can be treated as an inverse
measure of purity of synthesis since spheres are often the undesired
byproducts that are formed in the synthesis of a target morphology. Figure
3(c) illustrates the distributions of sphere fractions that rods and triangles
co-occur with. We have not included a violin plot for cubes, as the
performance of the morphology classification step was relatively low for
cubes, in particular in distinguishing cubes and spheres. It can be seen that
the majority of images that have triangles or rods in them have less than 40%
of spherical impurities, and the median fractions of spherical impurities fall
below 0.3 in both cases. Thus, our method can potentially serve as an
objective measure of the purity of the product of a synthesis. The development
of such a measure would allow future researchers to compare the shape purity
of a synthesis to those found in existing literature, enabling them to
evaluate the quality of a newly developed synthetic method.

###  4.4 Size/Shape Distribution

We next looked at the distribution of particle sizes in our dataset. Figure 4
shows size distribution histograms for each of the four morphologies. The
measured dimensions are chosen depending on the morphology (as detailed in
Section 2.2).

We note that the peak position in each histogram is in general more reliable
than the range/spread. This is because of errors in the pipeline that are
introduced because of the presence of certain special classes of microscopy
images in the dataset. These errors lead to noise in values in the histograms
corresponding to very high or very low sizes. In particular, TEM/SEM figures
sometimes have zoomed-in/magnified images embedded within them, which also
tend to have their own scales and bars. This class of figures are often used
when authors need to describe characteristics of nanoparticles at both coarse
and fine levels. A special characteristic of these figures is that multiple
magnifications of nanoparticles and multiple scales/bars exist in a single
image. The current version of our pipeline is unable to differentiate between
particles of embedded and parent figures and also between scales of embedded
and parent figures. We simply measure the sizes of all particles in an image
and perform unit conversion based on the value read from the single scale that
has been identified with the highest confidence by the Label, Scale and Bar
Detector. Hence, depending on which scale is chosen by the detector, sizes of
some particles in this class of images will be overestimated or
underestimated.

It is also interesting to note that dimensionless quantities are unaffected by
this issue of over/underestimation, since they remain unaffected by changes in
magnification and scale readings. Aspect ratio is one such example. Figure 4
(a) shows the distribution of aspect ratios of rods in the dataset. For such
cases, both the peak position as well as the range/spread of the histogram can
be treated as reliable measures of what is seen in literature. Another
property of dimensionless quantities is that they are unaffected by the errors
introduced in the scale detection and OCR reading steps. Hence, aspect ratio
measurements, for instance, can be carried out for all images in the dataset,
even in cases where scales are not identified in them. This is why Figure 4(a)
has higher frequencies than the other three histograms.

##  5 Usage Notes

The full dataset is available publicly at as a JSON file and an associated
Python script. The latter is required to download the microscopy images on a
local system. The annotated training dataset used for training the Mask-RCNN
model (see Section 2.1.5) is also available publicly in the same GitHub
repository in a similar format (JSON and accompanied Python script). Detailed
instructions to install all required libraries are provided in the README. The
JSON file can be queried using a language of choice (for example Python,
Matlab, R and Wolfram Mathematica).

##  6 Code Availability

All code used to retrieve and analyse microscopy images from literature is
publicly available at https://github.com/CederGroupHub/TEMExtraction, and
instructions to install all required dependencies and run the pipeline are
detailed in the README. All machine learning frameworks used are open source:
Tensorflow [37], Keras [38], PyTorch [19] and Darknet [25].

##  7 Acknowledgements

This work was funded by the U.S. Department of Energy, Office of Science,
Office of Basic Energy Sciences, Materials Sciences and Engineering Division
under Contract No. DE-AC02-05-CH11231 (D2S2 program KCD2S2). We thank Timothy
Vollmer, Anna Sackmann and Rachael Samberg (Science Data and Engineering
Librarians at UC Berkeley) for helping us obtain Text and Data Mining
agreements with the specified publishers. We also thank Maria Chan, Weixin
Jiang and other group members for the valuable discussions on compound figure
separation techniques, as well as Sam Gleason, Jakob Dahl, Caitlin McCandler
and other members of the Alivisatos, Persson, Jain and Sutter-Fella groups for
insightful discussions about gold nanoparticle synthesis and advise on
analysis of the data.

##  8 Author Contributions

A.S. developed the image extraction pipeline, nanoparticle size and shape
identification algorithms, and analyzed the data. K.C. developed the text
mining pipeline to isolate gold nanoparticles papers from the larger set of
materials science papers. A.T. provided guidance on the development of the
pipeline, size/shape identification algorithms and data analysis. X.W.
generated experimental TEM images as training data for the segmentation model.
P.A. supervised the project. G.C. developed the approach, and supervised the
project. All authors discussed the results and wrote the final manuscript.

##  9 Competing Interests

The authors declare no competing interests.

## References

  * [1] Tatsiana Mironava et al.  “Gold nanoparticles cellular toxicity and recovery: Effect of size, concentration and exposure time”  In _Nanotoxicology_ 4.1, 2010, pp. 120–137  DOI: 10.3109/17435390903471463
  * [2] Irshad A. Wani and Tokeer Ahmad  “Size and shape dependant antifungal activity of gold nanoparticles: A case study of Candida”  In _Colloids and Surfaces B: Biointerfaces_ 101 Elsevier B.V., 2013, pp. 162–170  DOI: 10.1016/j.colsurfb.2012.06.005
  * [3] Kenichi Niikura et al.  “Gold nanoparticles as a vaccine platform: Influence of size and shape on immunological responses in vitro and in vivo”  In _ACS Nano_ 7.5, 2013, pp. 3926–3938  DOI: 10.1021/nn3057005
  * [4] B. Chithrani, Arezou A. Ghazani and Warren C.W. Chan  “Determining the size and shape dependence of gold nanoparticle uptake into mammalian cells”  In _Nano Letters_ 6.4, 2006, pp. 662–668  DOI: 10.1021/nl052396o
  * [5] Olga Kononova et al.  “Opportunities and challenges of text mining in aterials research”  In _iScience_ 24.3 Elsevier Inc., 2021, pp. 102155  DOI: 10.1016/j.isci.2021.102155
  * [6] Karim T. Mukaddem, Edward J. Beard, Batuhan Yildirim and Jacqueline M. Cole  “ImageDataExtractor: A Tool to Extract and Quantify Data from Microscopy Images”  In _Journal of Chemical Information and Modeling_ 60.5, 2020, pp. 2492–2509  DOI: 10.1021/acs.jcim.9b00734
  * [7] Hyojin Kim, Jinkyu Han and T. Jin Han  “Machine vision-driven automatic recognition of particle size and morphology in SEM images”  In _Nanoscale_ 12.37 Royal Society of Chemistry, 2020, pp. 19461–19469  DOI: 10.1039/d0nr04140h
  * [8] Anna M. Hiszpanski et al.  “Nanomaterial Synthesis Insights from Machine Learning of Scientific Articles by Extracting, Structuring, and Visualizing Knowledge”  In _Journal of Chemical Information and Modeling_ 60.6, 2020, pp. 2876–2887  DOI: 10.1021/acs.jcim.0c00199
  * [9] Eric Schwenker et al.  “EXSCLAIM ! – An automated pipeline for the construction of labeled materials imaging datasets from literature”, pp. 1–17
  * [10] Xingzhi Wang et al.  “AutoDetect-mNP: An Unsupervised Machine Learning Algorithm for Automated Analysis of Transmission Electron Microscope Images of Metal Nanoparticles”  In _JACS Au_ 1.3, 2021, pp. 316–327  DOI: 10.1021/jacsau.0c00030
  * [11] Byoungsang Lee et al.  “Statistical Characterization of the Morphologies of Nanoparticles through Machine Learning Based Electron Microscopy Image Analysis”  In _ACS Nano_ 14.12, 2020, pp. 17125–17133  DOI: 10.1021/acsnano.0c06809
  * [12] Kaiming He, Georgia Gkioxari, Piotr Dollar and Ross Girshick  “Mask R-CNN”  In _Proceedings of the IEEE International Conference on Computer Vision_ 2017-Octob, 2017, pp. 2980–2988  DOI: 10.1109/ICCV.2017.322
  * [13] Kevin Cruse et al.  “Text-mined AuNP Synthesis Recipes Dataset” Dataset  figshare, https://doi.org/10.6084/m9.figshare.16614262.v3, 2021
  * [14] Olga Kononova et al.  “Text-mined dataset of inorganic materials synthesis recipes”  In _Sci Data_ 6.203, 2019  DOI: 10.1038/s41597-019-0224-1
  * [15] Satoshi Tsutsui and David J. Crandall  “A Data Driven Approach for Compound Figure Separation Using Convolutional Neural Networks”  In _Proceedings of the International Conference on Document Analysis and Recognition, ICDAR_ 1, 2017, pp. 533–540  DOI: 10.1109/ICDAR.2017.93
  * [16] Joseph Redmon and Ali Farhadi  “YOLO9000: Better, faster, stronger”  In _Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017_ 2017-Janua, 2017, pp. 6517–6525  DOI: 10.1109/CVPR.2017.690
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ 2016-Decem IEEE, 2016, pp. 770–778  DOI: 10.1109/CVPR.2016.90
  * [18] Jia Deng et al.  “ImageNet: A large-scale hierarchical image database”  IEEE, 2009, pp. 248–255  DOI: 10.1109/cvprw.2009.5206848
  * [19] Adam Paszke et al.  “PyTorch: An imperative style, high-performance deep learning library”  In _Advances in Neural Information Processing Systems_ 32.NeurIPS, 2019  arXiv:1912.01703
  * [20] Richard Liaw et al.  “Tune: A Research Platform for Distributed Model Selection and Training”, 2018  arXiv: http://arxiv.org/abs/1807.05118
  * [21] Alexey Bochkovskiy, Chien-Yao Wang and Hong-Yuan Mark Liao  “YOLOv4: Optimal Speed and Accuracy of Object Detection”, 2020  arXiv: http://arxiv.org/abs/2004.10934
  * [22] Chao Dong, Chen Change Loy, Kaiming He and Xiaoou Tang  “Image Super-Resolution Using Deep Convolutional Networks”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 38.2 IEEE, 2016, pp. 295–307  DOI: 10.1109/TPAMI.2015.2439281
  * [23] Ray Smith  “An overview of the tesseract OCR engine”  In _Proceedings of the International Conference on Document Analysis and Recognition, ICDAR_ 2, 2007, pp. 629–633  DOI: 10.1109/ICDAR.2007.4376991
  * [24] “LabelImg”, 2015  URL: https://github.com/tzutalin/labelImg
  * [25] Joseph Redmon  “Darknet: Open Source Neural Networks in C”, http://pjreddie.com/darknet/, 2013
  * [26] “LabelBox”, 2021  URL: https://labelbox.com
  * [27] D.. Groom et al.  “Automatic segmentation of inorganic nanoparticles in BF TEM micrographs”  In _Ultramicroscopy_ 194.June, 2018, pp. 25–34  DOI: 10.1016/j.ultramic.2018.06.002
  * [28] Yingchao Meng, Zhongping Zhang, Huaqiang Yin and Tao Ma  “Automatic detection of particle size distribution by image analysis based on local adaptive canny edge detection and modified circular Hough transform”  In _Micron_ 106.August 2017 Elsevier, 2018, pp. 34–41  DOI: 10.1016/j.micron.2017.12.002
  * [29] Mohsen Mirzaei and Hossein Khodabakhshi Rafsanjani  “An automatic algorithm for determination of the nanoparticles from TEM images using circular hough transform”  In _Micron_ 96 Elsevier Ltd, 2017, pp. 86–95  DOI: 10.1016/j.micron.2017.02.008
  * [30] Fang Zhang et al.  “Spherical nanoparticle parameter measurement method based on mask r-cnn segmentation and edge fitting”  In _ACM International Conference Proceeding Series_ , 2019, pp. 205–212  DOI: 10.1145/3373509.3373590
  * [31] M. Frei and F.. Kruis  “Image-based size analysis of agglomerated and partially sintered particles via convolutional neural networks”  In _Powder Technology_ 360 Elsevier B.V., 2020, pp. 324–336  DOI: 10.1016/j.powtec.2019.10.020
  * [32] Yuanyi Wu, Mengxing Lin and Sohrab Rohani  “Particle characterization with on-line imaging and neural network image analysis”  In _Chemical Engineering Research and Design_ 157.Ld Institution of Chemical Engineers, 2020, pp. 114–125  DOI: 10.1016/j.cherd.2020.03.004
  * [33] Batuhan Yildirim and Jacqueline M. Cole  “Bayesian Particle Instance Segmentation for Electron Microscopy Image Quantification”  In _Journal of Chemical Information and Modeling_ 61.3, 2021, pp. 1136–1149  DOI: 10.1021/acs.jcim.0c01455
  * [34] Tai Hwan Ha, Hee Joon Koo and Bong Hyun Chung  “Shape-controlled syntheses of gold nanoprisms and nanorods influenced by specific adsorption of halide ions”  In _Journal of Physical Chemistry C_ 111.3, 2007, pp. 1123–1130  DOI: 10.1021/jp066454l
  * [35] Leonid Vigderman, Bishnu P. Khanal and Eugene R. Zubarev  “Functional gold nanorods: Synthesis, self-assembly, and sensing applications”  In _Advanced Materials_ 24.36, 2012, pp. 4811–4841  DOI: 10.1002/adma.201201690
  * [36] Xingchen Ye et al.  “Using binary surfactant mixtures to simultaneously improve the dimensional tunability and monodispersity in the seeded growth of gold nanorods”  In _Nano Letters_ 13.2, 2013, pp. 765–771  DOI: 10.1021/nl304478h
  * [37] Martın Abadi et al.  “TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems” Software available from tensorflow.org, 2015  URL: https://www.tensorflow.org/
  * [38] François Chollet  “Keras”  GitHub, https://github.com/keras-team/keras, 2015
  * [39] Subramaniam Jayabal and Ramasamy Ramaraj  “Bimetallic Au/Ag nanorods embedded in functionalized silicate sol-gel matrix as an efficient catalyst for nitrobenzene reduction”  In _Applied Catalysis A: General_ 470 Elsevier B.V., 2014, pp. 369–375  DOI: 10.1016/j.apcata.2013.10.056
  * [40] Bruno P. Crulhas, Naira P. Ramos, Gustavo R. Castro and Valber A. Pedrosa  “Detection of hydrogen peroxide releasing from prostate cancer cell using a biosensor”  In _Journal of Solid State Electrochemistry_ 20.9, 2016, pp. 2427–2433  DOI: 10.1007/s10008-016-3182-y
  * [41] Yuelei Si et al.  “What is the predominant electron transfer process for Au NRs/TiO2 nanodumbbell heterostructure under sunlight irradiation?”  In _Applied Catalysis B: Environmental_ 220 Elsevier B.V., 2018, pp. 471–476  DOI: 10.1016/j.apcatb.2017.08.024
  * [42] Wei Lu et al.  “Photoacoustic imaging of living mouse brain vasculature using hollow gold nanospheres”  In _Biomaterials_ 31.9 Elsevier Ltd, 2010, pp. 2617–2626  DOI: 10.1016/j.biomaterials.2009.12.007

##  10 Figures & Tables

Category |  Keywords
---|---
1 |  TEM, SEM
2 |  Gold
3 |  sphere, cube, rod, triangle, prism, aunr
Table 1: Regular Expression Keywords corresponding to each category, utilized as a filter in the Paper Parsing stage. To improve the precision of the collected figures, only those figures whose captions match at least one keyword from each of the three categories, are downloaded. Stemming is utilized to include grammatical variants of each keyword. |  Classifier-1 |  Classifier-2 |  Label, Scale and Bar Detector |  Particle Segmentation
---|---|---|---|---
Task |  Image Classification |  Image Classification |  Object Detection |  Instance Segmentation + Object Classification
Classification Classes |  Microscopy, Non-Microscopy |  Particulate, Non-Particulate |  Label, Scale, Bar |  Rod, Sphere, Cube, Triangle
Dataset Size(no. of images) |  1425 |  383 |  640 |  426
Metric Score on Test Data |  0.98 (F1 score) |  0.91 (F1 score) |  0.88 (mAP@50) |  0.77 (mAP@50)
Table 2: Illustration of datasets and models used for training all machine learning models in the pipeline. All datasets were divided into train, validation and test splits before model training.  Figure 1: Schematic representation of all major steps in the extraction pipeline. (a) Information such as caption, DOI and Hash are parsed during the Paper Parsing stage. (b) Sub-figures are identified and separated from their parent composite figures. (c) Two binary classifiers are applied sequentially on sub-figures to isolate microscopy (SEM & TEM) images that contain nanoparticles in them. (d) Labels, Scales and Bars are located and read with an OCR reader. (e) Nanoparticle segmentation and identification of particle morphologies is performed using a Mask-RCNN model. (f) Sizes of segmented particles are measured. The dimensions measured are chosen depending on the morphology of the particle. Figures shown in sub-figures (b), (d) and (e) show actual predictions made by models from the pipeline. Numbers within/beside predictions in sub-figures (b) and (d) are the prediction confidences. The sample figure shown in (b) has been adapted with permission from ref. [39]. Copyright 2014 Elsevier. The UV-vis spectrum shown in (c) has been adapted with permission from ref. [40]. Copyright 2016 Springer Nature. The TEM image shown in (c) and (d) has been adapted with permission from ref. [41]. Copyright 2018 Elsevier.  Figure 2:  Illustration of the results table displayed on the web application when a sample TEM image is uploaded. (a) Segmentation tab (b) Scale and Label Detection tab (c) Particle Size Distribution tab. Separate Size Distribution histograms are plotted for each measured dimension for all morphologies identified in the image. The sample image shown in (a) and (b) has been adapted with permission from ref. [41]. Copyright 2018 Elsevier.  Data Description |  Data Key Label |  Data Type |  Nested Keys and Types
---|---|---|---
DOI of the parent paper |  DOI |  _string_ |  _
A unique identifier of the image |  Hash |  _string_ |  _
URL of the parent composite figure |  Composite_Figure_URL |  _string_ |  _
Title of the parent composite figure |  Composite_Figure_Title |  _string_ |  _
Morphology class of majority of the particles in the image |  Main_class |  _string_ |  _
Morphology classes that are present in lower frequency |  Minority_classes |  _list of strings_ |  _
Morphology-specific size measurements of all identified particles and units |  Size |  Object(_dict_) |  Unit: _string_ Measurement: Object(_dict_)a
Sub-figure label associated with the image |  Label |  _string_ |  _
Information read from the scale |  Scale |  Object(_dict_) |  digit: _string_ unit: _string_ bar_length: _string_
Table 3: Illustration of the dataset schema. DOI, Hash and Composite_Figure_URL are metadata fields that give general information about the source and nature of the image. Main_class, Minority_class and Size are fields that have been populated by the image analysis steps of the pipeline. The Unit and Measurement sub-fields of the Size field contain the measurement unit (nanometer or micron) and magnitudes of size measurements respectively. The digit, bar_length and unit sub-fields of the Scale field contain the magnitude of the scale reading, horizontal length of the measurement bar and unit of measurement respectively. Schema of Object(_dict_)a is shown in Table 4.  Data Key Label |  Data Type |  Nested Key and Type
---|---|---
rod |  Object(_dict_) |  length: _list of floats_ width: _list of floats_
sphere |  Object(_dict_) |  diameter: _list of floats_
triangle |  Object(_dict_) |  height: _list of floats_
cube |  Object(_dict_) |  side: _list of floats_
Table 4: Schema of the object linked to the Measurement sub-field. It contains
separate nested objects for each of the four morphologies. Each nested object
contains fields with the measured dimension values. Figure 3: (a) Figure
illustrating a comparison between the Watershed segmentation and the Mask-RCNN
algorithms on segmentation of overlapping nanoparticles. The original TEM
image is shown on the left, the results of the Watershed algorithm are shown
in the middle, and the results of the Mask-RCNN are shown on the right. The
image shown has been adapted with permission from ref. [42]. Copyright 2010
Elsevier (b) Pie-chart showing fractional presence of various morphologies in
the extracted dataset. The percentage shown beside each section can be
interpreted as the percent of microscopy images in the dataset that contain a
particular morphology in majority. Sections corresponding to spheres,
triangles, rods and cubes represent 2491, 129, 1455 and 170 data points
respectively. (c) Violin plots showing the fraction of spheres that co-occur
with rods and triangular prisms. Horizontal lines intersecting the plots mark
the positions of the medians and extrema. The plot corresponding to rod (blue)
represents 325 data points while the plot corresponding to triangle (red)
represents 36 data points.  Figure 4: Size distribution histograms for various
dimensions of rods, cubes, triangles and spheres. Distributions of (a) Aspect
Ratios of rods (b) Lengths and Widths of rods (c) Diameters of spheres (d)
Side Lengths of cubes (e) Heights of triangular prisms in the dataset. Median
sizes from each microscopy image have been used in order to minimize the
effects of outliers/mis-predictions which may have very large/small sizes in
comparison to other particles in the image. x-axes of histograms (b), (c) and
(d) have been truncated to make the peak positions more clearly visible, since
peak positions are more reliable than the range/spread (More details in
Section 4.4).


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Pattern Recognition Lab, Friedrich-Alexander-Universität
Erlangen-Nürnberg, Erlangen, Germany
11email<EMAIL_ADDRESS>
# Novel Artistic Scene-Centric Datasets for Effective Transfer Learning in
Fragrant Spaces

Shumei Liu 11 0009-0001-7509-2275 Haiting Huang 11 0009-0002-9159-6139 Mathias
Zinnen 11 0000-0003-4366-5216 Vincent Christlein 11 0000-0003-0455-3799

###### Abstract

Olfaction, often overlooked in cultural heritage studies, holds profound
significance in shaping human experiences and identities. Examining historical
depictions of olfactory scenes can offer valuable insights into the role of
smells in history. We show that a transfer-learning approach using weakly
labeled training data can remarkably improve the classification of fragrant
spaces and, more generally, artistic scene depictions. We fine-tune
Places365-pre-trained models by querying two cultural heritage data sources
and using the search terms as supervision signal. The models are evaluated on
two manually corrected test splits. This work lays a foundation for further
exploration of fragrant spaces recognition and artistic scene classification.
All images and labels are released as the ArtPlaces dataset at
https://zenodo.org/doi/10.5281/zenodo.11584328.

###### Keywords:

Transfer learning Fragrant spaces Scene classification Olfaction

##  1 Introduction

Olfaction is an essential sensory modality that adds richness and depth to our
life experiences, influencing our perception, behavior, and well-being in
profound ways. Smell is also deeply intertwined with cultural traditions,
rituals, and practices, as it shapes our experiences and identities in diverse
cultural contexts. For instance, incense and fragrances have historically held
pivotal positions in religious practices, medicine, and social relations
across Western civilizations from ancient Egypt to the eighteenth century.
Nevertheless, olfaction and its aesthetic potential have traditionally been
neglected and devalued, and the human sense of smell was viewed as a redundant
evolutionary remnant [12]. Given the prevailing emphasis on visual-centric
perspectives, locating pertinent information about smell in modern digital
cultural heritage remains a critical task[3].

In recent years, an increasing number of scholars and institutions have
started to challenge the “olfactory gaze”, unveiling the significant role of
smell, and pioneering novel approaches to the study of history and culture
through olfaction-centric exploration[2, 1, 3, 13]. The paradox of smell lies
in its dual nature as both our most instinctive sense and one that is
particularly difficult to articulate. Physical smell perception is inevitably
absent when experiencing visual mediums such as artworks. The automatic
extraction of scent-related cues thus has to rely on proxies such as smell-
related objects, gestures, scenes or iconography which indirectly suggest the
presence of olfactory dimensions in a painting [16]. While the recognition of
smell-related objects [7, 18, 19, 20], olfactory gestures [17], and even
emotions [10] have been addressed previously, the classification of fragrant
spaces has not yet been explored.

The primary challenge arises from the absence of published datasets annotated
with scene information for historical artworks, particularly those specific to
artistic fragrant spaces. Consequently, direct training for our classification
tasks is not possible. Numerous datasets have been curated specifically for
scene classification in real-world environments, _e.g_., Scene15[8, 9], MIT
Indoor67[11], SUN[14], and also multi-million-item scene-centric datasets such
as Places365[15]. However, we observe a significant performance drop when we
apply models trained on these large-scale photographic data to our artistic
target data. To bridge this gap, we leverage the overlap between fragrant
places and the large number of scene categories annotated in Places365. This
allows us to use photographic data for pre-training while querying open
heritage data to enable fine-tuning with artistic imagery. Using the query
terms as weak labels, we create the ArtPlaces dataset, consisting of 4623
artworks.

We evaluate our approach in two ways: First, we specifically assess the
model’s performance in classifying fragrant spaces. To this end, we evaluate
the models using a small set of smell-related artworks Fragrant Spaces,
manually annotated with Places365 labels. Secondly, to achieve a more reliable
evaluation for general scene classification, we extend Fragrant Spaces with
747 images from ArtPlaces and manually correct the weak labels to obtain a
reliable test set.

To the best of our knowledge, this study represents the first attempt to
identify fragrant places in artworks. In particular, our contributions are as
follows:

  * •

We construct the ArtPlaces dataset, consisting of 3804 weakly labeled artworks
for training, and 975 manually labeled artworks for evaluating artistic scene
recognition systems. All annotations and images are published on
Zenodo.111https://zenodo.org/records/11584328

  * •

We apply four different classifier architectures and analyze their performance
with respect to different pre-training, fine-tuning, and evaluation schemes.

##  2 The ArtPlaces Dataset

###  2.1 Pre-Training Data

All our models are pre-trained on the standard benchmark of Places365
(Places365-Standard) [15], using the predefined splits. Places365 consists of
10 million photographs, annotated with 365 scene categories and serves as a
standardized benchmark for evaluating scene recognition algorithms.

###  2.2 Data Collection

####  2.2.1 Artistic Scene-Centric Datasets.

We create two weakly annotated datasets by retrieving images from the APIs of
the Rijksmuseum collection222https://data.rijksmuseum.nl/object-metadata/api/
and Wikidata333https://query.wikidata.org/. The respective query terms are
used as weak (_i.e_. semi-automatically generated) labels which serve as
supervision signals during fine-tuning. Additionally, we create a manually
labeled dataset of olfaction-related artworks to test the algorithms’
capability to classify fragrant spaces.

##### RASD.

The Rijksmuseum is the national museum of the Netherlands, which provides a
treasure trove of data with extensive descriptions of over 500,000 art
historical objects, a vast collection of object photographs and the complete
library catalogue. The Rijksmuseum offers multiple access points for its
dataset. We use the APIs and Image requests service provided by the platform
to access the structured metadata
(OAI_DC)444https://data.rijksmuseum.nl/object-metadata/ and images of the
Rijksmuseum collection.

We select and restrict metadata information such as object type, material,
technique, etc. and query the images according to search terms matching the
365 Places365 category names. The resulting weakly labeled images comprise the
Rijksmuseum Artistic Scene-Centric Dataset (RASD).

Table 1: Selected Wikidata Properties Property | Description | Value
---|---|---
P18 | image | image
P31 | instance of | painting item
P180 | depicts | Places365-Standard categories

##### WASD.

The second intermediate dataset is derived from Wikidata. Wikidata serves as a
freely accessible knowledge base that functions as a centralized storage
facility for structured data spanning diverse fields. It consists of 1.54
billion item statements as of early
2023555https://en.wikipedia.org/wiki/Wikidata, and the size is continuously
growing as more data is added and updated by contributors from around the
world. Each item within is uniquely identified by a QID and described through
its respective statements, which consist of a property and corresponding
value.

We use the MediaWiki API and the SPARQL endpoint provided by the Wikidata
Query Service to query the data. To identify depictions of specific scenes, we
match the relevant QID for each Places365 category name and build SPARQL
queries using properties from the WikiData ontology (_cf_. Tab. 1).

Specifically, we associate the scene labels with the property P180, which
indicates events portrayed in any form of art. The responses comprise the URLs
of the desired images, facilitating their download. Finally, the Wikidata
Artistic Scene-Centric Dataset (WASD) is compiled, each labeled with a scene
category.

We split validation sets from both weakly labelled datasets, RASD and WASD, to
allow the analysis of models trained on only one of the subsets and a cross-
dataset comparison. Furthermore, we transformed the multi-label problem into
multiple single-label classification problems to allow comparison with single-
label classification algorithms. We provide example images for RASD and WASD
in Fig. 1.

##### Fragrant-Places.

The third source subset is created by manually reviewing all artworks of the
ODOR dataset [20] and assigning Places365 labels where applicable. This
Fragrant-Places subset consists of 228 artworks and covers 35 Places365
labels. Figure 2 shows exemplary image samples for some of the most common
categories.

Figure 1: Image samples of the artistic scene-centric datasets. Figure 2:
Image samples of the Fragrant-Spaces dataset.

####  2.2.2 Final Data Split.

We combine the three source datasets to derive the ArtPlaces dataset as
follows: First, we combine the RASD and WASD subsets to generate a joint
weakly supervised dataset. We refer to this combination as ArtPlaces-raw. From
this combination, we split
$88\text{\,}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}$
of the data for training, resulting in ArtPlaces-train. The labels of the
remaining
$12\text{\,}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}$
of ArtPlaces-raw were manually corrected and then merged with Fragrant-Places,
resulting in ArtPlaces-test. Additionally, we keep a separate version of the
Fragrant-Spaces test set.

This leaves us with three final data subsets:

  1. 1.

_ArtPlaces-train_ : Is the weakly labeled training split, obtained by
combining parts of RASD and WASD.

  2. 2.

_Fragrant-Spaces_ : The primary objective of the Fragrant-Spaces test set is
to evaluate the models’ ability to detect fragrant spaces in olfaction-related
artworks. As it is based on the ODOR dataset [20], we can assume that all of
the images have some relation to olfaction. This focused approach allows us to
measure how well the models identify and classify fragrant environments, which
is crucial for their application in automated smell-reference extraction.

  3. 3.

_ArtPlaces-test_ : In contrast, the ArtPlaces-test split is designed to
provide a broader evaluation of scene classification capabilities. It does not
specifically focus on olfaction-related images but aims to assess the general
scene classification capabilities. It provides more images and covers more of
the Places365 categories, thus enabling a more reliable evaluation. This
dataset offers a robust framework for testing scene recognition performance at
the expense of a specific focus on smell-related scenes.

Table 2 gives an overview of all subsets described above and Fig. 3
illustrates their creation process.

Figure 3: Illustration of the dataset creation process. ArtPlaces-raw is derived from the weakly labeled RASD and WASD subsets. From ArtPlaces-raw, ArtPlaces-train is split, and in conjunction with Fragrant-Spaces ArtPlaces-test. Table 2: Complete overview of the datasets and subsets used during this work. Images denote the number of images in the respective subset, coverage the number of Places365 labels covered, and labels reports on how the labels were obtained. Dataset | Size | Coverage | Manual Labels
---|---|---|---
RASD |  0704 |  098 |
WASD | 3691 | 124 |
Fragrant-Spaces |  0228 |  035 | ✓
ArtPlaces-raw | 4395 | 155 |
ArtPlaces-train | 3804 | 120 |
ArtPlaces-test |  0975 | 174 | ✓

###  2.3 Source Data Analysis

(a) RASD

(b) WASD

(c) Fragrant-Spaces dataset

Figure 4: Dominant categories in different datasets, contributing separately
$90\text{\,}\mathrm{\char 37\relax}$ of each data volume.

The source datasets are summarized in the top three columns of Tab. 2. WASD
has a larger collection of artworks, comprising 3691 images, compared to
RASD’s total of 704 pieces. Both weakly labeled datasets are considerably
larger than the manually labeled Fragrant-Places subset.

Diversity. Both weakly supervised datasets encompass a wide range of scene
types, with more than one hundred categories each. Figure 4 indicates examples
of the more frequent scenes. Although gathered from different sources, the
datasets show a similar composition of scene categories.

Multi-labeling. We preserve the complete scene information by marking the
datasets with multiple labels, effectively enabling multi-label approaches to
be applied to the dataset in future work.

Class Imbalance. Despite capping the query results to 100 results per class,
the collected data shows a significant class imbalance. When sorting the
categories by size, the 90th percentile classes of RASD and WASD are ranked
48th and 49th separately, indicating that half of the categories encompass
most of the data.

###  2.4 Label Quality & Correction

####  2.4.1 Quality Evaluation.

We assess the quality of our dataset samples using random sampling.
Considering the size of our data, we decided to use stratified sampling. This
involves the random selection of $5\text{\,}\mathrm{\char 37\relax}$ samples
from each class within the original datasets, ensuring that each class is
represented in the sample set according to its original proportion. Then we
perform a detailed manual inspection, including visual validation, and source
and background verifications.

We repeat this procedure before and after the corrections applied during the
creation of ArtPlaces-test and report the results in Tab. 3. Despite being
automatically generated, WASD provides excellent quality outputs with an
accuracy estimation of $97.2\text{\,}\mathrm{\char 37\relax}$, and only 6
artworks were found to have a wrong label. RASD has relatively more mistakes
with original accuracy estimation being $81.0\text{\,}\mathrm{\char
37\relax}$.

Table 3: Quality Evaluation of Artistic Scene-Centric Datasets

Dataset | Samples | Mislabeled Samples | Acc. Est. (Before) | Acc. Est. (After)
---|---|---|---|---
RASD |  063 | 12 | $81.0\text{\,}\mathrm{\char 37\relax}$ | $93.7\text{\,}\mathrm{\char 37\relax}$
WASD | 217 |  06 | $97.2\text{\,}\mathrm{\char 37\relax}$ | $97.2\text{\,}\mathrm{\char 37\relax}$

####  2.4.2 Error Analysis.

The labeling errors can be split into five different groups:

  * •

Noisy Contextual Information. Refers to the noisy background information that
affects the context in which the label is extracted. The Rijksmuseum includes
documentation or references in each artwork’s description, in which also
appear words like “art gallery”, “art school”, _etc_., that belong to
Places365 categories and can be identified by mistake. This problem occurs
mainly among RASD samples, with three cases.

  * •

Text Recognition Error. Indicates the issue where the system misreads or
misidentifies the label due to the presence of similar words or a wrong
combination of multi-word labels in the text. Three RASD samples are
mistakenly labeled, _e.g_., word “bare” is mistakenly extracted as the label
“bar”.

  * •

Missing references. Wikidata offers a list of descriptive words of objects or
scenes for its painting items, along with their references. We found four WASD
samples labeled without clear reference clues or feasible manual
identification.

  * •

Unknown Reasons. Two sorts of mislabeling are discovered with unidentified
causes. One of the Places365 categories, “corn field”, may be mostly
misassigned in RASD. All the five samples from this category are found to be
misclassified. Other concerned labels are “fishpond” and “pond” in WASD. All
samples related to these categories are currently tagged with both labels
simultaneously. However, two cases out of them are checked probability not
suitable for the specific “fishpond”.

  * •

Inadequate Filtering. The error arises because the filters used for the query
are not strict enough, which occurs in RASD. Instead of representing an
artwork, a sample corresponds to a photo of a physical object, satisfying one
selected combination of object type and label filters.

####  2.4.3 Data Rectification.

  * •

Eliminate Noise. RASD is relatively small, making it possible to manually
filter out noisy data, _i.e_. photos of non-artworks.

  * •

Remove Duplicates. Scene identification is naturally a multi-label issue,
therefor in our datasets exist duplicate images marked with different labels.
We process a systematic cleansing of the redundant images while maintaining
the multi-label information.

  * •

Exclude Problematic Label Data. “corn field” is the only label experiencing a
significant issue with systematic misassignment at scale. All its samples from
RASD are wrongly labeled, in which exist a considerable quantity as well. In
order to uphold the standards of training and analysis, we have decided to
eliminate the entire category in RASD, specifically 69 label instances (while
retaining the five corrected samples).

By implementing the above strategies, we address the mislabeling concerns that
have the most significant impact through manual correction. These measures are
mainly applied to RASD, in which half of the mismatching issues are resolved,
and the revised accuracy estimate achieves $93.7\text{\,}\mathrm{\char
37\relax}$ (only four mislabeled cases found in the second round sampling).
The remaining error types are more dispersed and have relatively minor
impacts, and will be concerned in future work.

##  3 Methodology

###  3.1 Experimental Setup

We employ the widely-used ResNet50[4], ResNet18[4], and DenseNet161[5]
classification networks. All of these have pre-trained weights available for
the Places365-Standard
dataset.666https://github.com/CSAILVision/places365/tree/master?tab=readme-ov-
file

Additionally, we evaluate the recent WaveMix [6] method. It comprises a series
of self-similar and resolution-preserving WaveMix blocks, inside which a
multi-level two-dimensional discrete wavelet transform (2D-DWT) is utilized
for lossless down-sampling and token-mixing while maintaining image fidelity.
The system strategically reorganizes spatial information guided by image
priors, such as scale-invariance, shift-invariance, and sparseness of edges.
This approach incorporates multi-resolution information, facilitating
efficient learning with fewer parameters and layers, resulting in a rapid
expansion of the receptive field compared to traditional CNN layers.

All models are trained using cross-entropy loss with Stochastic Gradient
Descent with Momentum as optimizer. Additionally, we apply geometric, spatial
and color-based in-place data augmentation.

##  4 Results and Discussion

We conduct three sets of experiments to evaluate  (i) the models’ capability
to recognise fragrant spaces in historical artworks,  (ii) the broader scene
classification in artistic depictions, and  (iii) the impact of fine-tuning
dataset properties on classification performance.

We evaluate the four baseline models described above in each experimental
setup, pre-trained on Places365-standard.

###  4.1 Fragrant Spaces Recognition

Table 4: Classification performance of models pre-trained on Places365 when
evaluated on Places365-val and Fragrant-Places.

| Places365-val |  | Fragrant-Places
---|---|---|---
| Top-1 acc. | Top-5 acc. | F1 |  | Top-1 acc. | Top-5 acc. | F1
ResNet18[4] | $53.86\text{\,}\mathrm{\char 37\relax}$ | $83.95\text{\,}\mathrm{\char 37\relax}$ | $53.18\text{\,}\mathrm{\char 37\relax}$ |  |  0$2.63\text{\,}\mathrm{\char 37\relax}$ | $12.72\text{\,}\mathrm{\char 37\relax}$ |  0$2.76\text{\,}\mathrm{\char 37\relax}$
ResNet50[4] | $55.09\text{\,}\mathrm{\char 37\relax}$ | $85.10\text{\,}\mathrm{\char 37\relax}$ | $54.37\text{\,}\mathrm{\char 37\relax}$ |  |  0$1.75\text{\,}\mathrm{\char 37\relax}$ | $14.04\text{\,}\mathrm{\char 37\relax}$ |  0$3.18\text{\,}\mathrm{\char 37\relax}$
DenseNet161[5] | $56.55\text{\,}\mathrm{\char 37\relax}$ | $86.18\text{\,}\mathrm{\char 37\relax}$ | $55.78\text{\,}\mathrm{\char 37\relax}$ |  |  0$4.39\text{\,}\mathrm{\char 37\relax}$ | $17.54\text{\,}\mathrm{\char 37\relax}$ |  0$5.13\text{\,}\mathrm{\char 37\relax}$
WaveMix[6] | $51.87\text{\,}\mathrm{\char 37\relax}$ | $82.51\text{\,}\mathrm{\char 37\relax}$ | $51.04\text{\,}\mathrm{\char 37\relax}$ |  |  0$2.63\text{\,}\mathrm{\char 37\relax}$ | $18.42\text{\,}\mathrm{\char 37\relax}$ |  0$2.85\text{\,}\mathrm{\char 37\relax}$

We measure the capability to recognize fragrant spaces in historical artworks
using the Fragrant-Spaces subset of ArtPlaces. When we directly apply models
pre-trained on Places365 to the Fragrant-Spaces test set, we see a large
performance decrease as listed in Tab. 4. Compared to the performance levels
exhibited by the pre-trained models on the Places365 validation set, the
classification accuracies in our target domain are extremely low, with top-5
accuracies falling below $20\text{\,}\mathrm{\char 37\relax}$, and top-1
accuracies even less than $5\text{\,}\mathrm{\char 37\relax}$. This stark
contrast underscores a substantial disparity in performance between the scene
classification for real-world images and that for artistic fragrant spaces.
The pre-trained models, typically trained on large-scale datasets of real-
world images, are not well-suited for capturing the nuances of artistic or
abstract spaces. Artistic depictions of fragrant spaces may contain elements
deviating from typical features seen in real-world scenes. These spaces could
include abstract compositions, unconventional lighting, non-representational
forms, and other artistic elements that may not be adequately represented in
the learned features of a conventional scene classification model.

Table 5: Fragrant-Spaces classification performance of models with and without
fine-tuning on ArtPlaces-train.

Model | Fine-Tuning | Top-1 acc. | Top-5 acc. | F1(macro) | F1(weighted)
---|---|---|---|---|---
ResNet18[4] |  |  0$2.63\text{\,}\mathrm{\char 37\relax}$ | $12.72\text{\,}\mathrm{\char 37\relax}$ | $2.76\text{\,}\mathrm{\char 37\relax}$ |  0$3.91\text{\,}\mathrm{\char 37\relax}$
ResNet18[4] | ✓ | $18.86\text{\,}\mathrm{\char 37\relax}$ | $43.42\text{\,}\mathrm{\char 37\relax}$ | $6.40\text{\,}\mathrm{\char 37\relax}$ | $16.48\text{\,}\mathrm{\char 37\relax}$
ResNet50[4] |  |  0$1.75\text{\,}\mathrm{\char 37\relax}$ | $14.04\text{\,}\mathrm{\char 37\relax}$ | $3.18\text{\,}\mathrm{\char 37\relax}$ |  0$2.02\text{\,}\mathrm{\char 37\relax}$
ResNet50[4] | ✓ | $19.74\text{\,}\mathrm{\char 37\relax}$ | $42.11\text{\,}\mathrm{\char 37\relax}$ | $8.97\text{\,}\mathrm{\char 37\relax}$ | $18.01\text{\,}\mathrm{\char 37\relax}$
DenseNet161[5] |  |  0$4.39\text{\,}\mathrm{\char 37\relax}$ | $17.54\text{\,}\mathrm{\char 37\relax}$ | $5.13\text{\,}\mathrm{\char 37\relax}$ |  0$5.42\text{\,}\mathrm{\char 37\relax}$
DenseNet161[5] | ✓ |  24.12 % |  46.93 % |  9.79 % |  20.91 %
WaveMix[6] |  |  0$2.63\text{\,}\mathrm{\char 37\relax}$ | $18.42\text{\,}\mathrm{\char 37\relax}$ | $2.85\text{\,}\mathrm{\char 37\relax}$ |  0$3.13\text{\,}\mathrm{\char 37\relax}$
WaveMix[6] | ✓ | $16.23\text{\,}\mathrm{\char 37\relax}$ | $42.98\text{\,}\mathrm{\char 37\relax}$ | $7.76\text{\,}\mathrm{\char 37\relax}$ | $16.19\text{\,}\mathrm{\char 37\relax}$

However, after fine-tuning with our weakly supervised ArtPlaces-train dataset,
all models display an immense performance increase. Table 5 shows that all
four models fine-tuned on the larger combined dataset exhibit an increase in
all metrics. These promising outcomes highlight the effectiveness of this
methodology in improving performance in artistic fragrant spaces. The fine-
tuned DenseNet161[5] outperforms other models, providing the highest top-1
accuracy of $24.12\text{\,}\mathrm{\char 37\relax}$ and top-5 accuracy of
$46.93\text{\,}\mathrm{\char 37\relax}$.

Similarly, after fine-tuning, the macro-average F1 scores increase by
$4.9\text{\,}\mathrm{\char 37\relax}$ in average. The weighted average
F1(micro) scores improve by an even larger margin of
$12.3\text{\,}\mathrm{\char 37\relax}$ in average. In our context, the
weighted-average evaluation leads to better results, because it takes class
imbalance into the calculation, by considering the performance of each class
but placing higher weight on classes with more samples. The outcomes of larger
classes have a greater influence on the overall average.

Even though the finetuned models have seen a great performance improvement,
there are still elements preventing further increases in accuracy. Unlike the
training set, Fragrant-Space is single-labeled, which can exclude diverse
information and reduce the hit rate. For example, an artwork named “Wooded
landscape with herdsmen and their flock crossing shallow water” describes
woods, a pond, and a path, but a single label only covers the last one, which
however rarely appears in ArtPlaces-train dataset and therefore is hard to
learn. The two largest categories that were significantly misclassified,
“forest path” and “bar”, are both those with insufficient coverage in the
training set. Artistic scenes can be inherently ambiguous, because of their
unreality and abstraction. Food is an important theme in smell-related
artworks, in still life paintings frequently present a full table of fruits,
vegetables, and utensils. It is hard to say whether this extracted fragmentary
scene belongs to a dining room or a kitchen. Another limitation is manual
labels can also make errors. Some scenes in historical artworks are not easy
to visually identify, _e.g_., a goldsmith’s workshop was mistakenly labeled as
a “bar”.

(a) Based on dominant categories of RASD

(b) Based on dominant categories of WASD

(c) Based on the dominant categories of Fragrant-Spaces

Figure 5: Comparison of the number of images per category between datasets
RASD, WASD, and Fragrant-Spaces. In each polar coordinate, the radial axis
measures the number of images per category, and the angular axis represents
the sorted dominant categories.

###  4.2 Artistic Scene Recognition

We measure the general scene recognition capability using the larger
ArtPlaces-test subset, which contains a broader range of images than Fragrant-
Places. The pre-trained models are still displaying low accuracies when tasked
with classifying general artistic scenes, as shown in Tab. 6 . But both the
pre-trained and fine-tuned models demonstrate increased performance on the
ArtPlaces-test dataset in comparison to the smaller Fragrant-Spaces test
set(_cf_. Tab. 7).

ArtPlaces-test contains a much larger number of categories compared to
ArtPlaces-train, because of the manually modified and introduced labels.
Categories that do not appear in the training set cannot be identified.
However, incorporating a diverse range of labels in the test set improves its
representativeness, leading to more effective evaluation. The mature DenseNet
structures exhibit notably enhanced accuracy rates, with a top-1 accuracy of
$29.73\text{\,}\mathrm{\char 37\relax}$ and a top-5 accuracy of
$57.40\text{\,}\mathrm{\char 37\relax}$.

Table 6: Classification performance of models pre-trained on Places365 when
evaluated on Places365-val and ArtPlaces-test.

| Places365-val |  | ArtPlaces-test
---|---|---|---
| Top-1 acc. | Top-5 acc. | F1 |  | Top-1 acc. | Top-5 acc. | F1
ResNet18[4] | $53.86\text{\,}\mathrm{\char 37\relax}$ | $83.95\text{\,}\mathrm{\char 37\relax}$ | $53.18\text{\,}\mathrm{\char 37\relax}$ |  |  0$6.95\text{\,}\mathrm{\char 37\relax}$ | $25.17\text{\,}\mathrm{\char 37\relax}$ |  0$5.58\text{\,}\mathrm{\char 37\relax}$
ResNet50[4] | $55.09\text{\,}\mathrm{\char 37\relax}$ | $85.10\text{\,}\mathrm{\char 37\relax}$ | $54.37\text{\,}\mathrm{\char 37\relax}$ |  |  0$8.31\text{\,}\mathrm{\char 37\relax}$ | $28.02\text{\,}\mathrm{\char 37\relax}$ |  0$6.41\text{\,}\mathrm{\char 37\relax}$
DenseNet161[5] | $56.55\text{\,}\mathrm{\char 37\relax}$ | $86.18\text{\,}\mathrm{\char 37\relax}$ | $55.78\text{\,}\mathrm{\char 37\relax}$ |  |  0$8.77\text{\,}\mathrm{\char 37\relax}$ | $29.73\text{\,}\mathrm{\char 37\relax}$ |  0$7.16\text{\,}\mathrm{\char 37\relax}$
WaveMix[6] | $51.87\text{\,}\mathrm{\char 37\relax}$ | $82.51\text{\,}\mathrm{\char 37\relax}$ | $51.04\text{\,}\mathrm{\char 37\relax}$ |  |  0$9.23\text{\,}\mathrm{\char 37\relax}$ | $29.73\text{\,}\mathrm{\char 37\relax}$ |  0$5.75\text{\,}\mathrm{\char 37\relax}$

Table 7: ArtPlaces-test classification performance of models with and without
fine-tuning on ArtPlaces-train.

Model | Fine-Tuning | Top-1 acc. | Top-5 acc. | F1(macro) | F1(weighted)
---|---|---|---|---|---
ResNet18[4] |  |  0$6.95\text{\,}\mathrm{\char 37\relax}$ | $25.17\text{\,}\mathrm{\char 37\relax}$ |  0$5.58\text{\,}\mathrm{\char 37\relax}$ |  0$8.13\text{\,}\mathrm{\char 37\relax}$
ResNet18[4] | ✓ | $23.12\text{\,}\mathrm{\char 37\relax}$ | $50.23\text{\,}\mathrm{\char 37\relax}$ |  0$8.49\text{\,}\mathrm{\char 37\relax}$ | $18.82\text{\,}\mathrm{\char 37\relax}$
ResNet50[4] |  |  0$8.31\text{\,}\mathrm{\char 37\relax}$ | $28.02\text{\,}\mathrm{\char 37\relax}$ |  0$6.41\text{\,}\mathrm{\char 37\relax}$ |  0$9.05\text{\,}\mathrm{\char 37\relax}$
ResNet50[4] | ✓ | $26.99\text{\,}\mathrm{\char 37\relax}$ | $55.24\text{\,}\mathrm{\char 37\relax}$ | $11.98\text{\,}\mathrm{\char 37\relax}$ | $22.63\text{\,}\mathrm{\char 37\relax}$
DenseNet161[5] |  |  0$8.77\text{\,}\mathrm{\char 37\relax}$ | $29.73\text{\,}\mathrm{\char 37\relax}$ |  0$7.16\text{\,}\mathrm{\char 37\relax}$ | $10.04\text{\,}\mathrm{\char 37\relax}$
DenseNet161[5] | ✓ |  29.73 % |  57.40 % |  12.39 % |  24.82 %
WaveMix[6] |  |  0$9.23\text{\,}\mathrm{\char 37\relax}$ | $29.73\text{\,}\mathrm{\char 37\relax}$ |  0$5.75\text{\,}\mathrm{\char 37\relax}$ | $10.72\text{\,}\mathrm{\char 37\relax}$
WaveMix[6] | ✓ | $24.26\text{\,}\mathrm{\char 37\relax}$ | $52.73\text{\,}\mathrm{\char 37\relax}$ | $11.64\text{\,}\mathrm{\char 37\relax}$ | $20.83\text{\,}\mathrm{\char 37\relax}$

###  4.3 Subset Performance Analysis

To analyze the impact of the fine-tuning datasets, we evaluate the performance
of models trained on only one of the respective subsets and perform a cross-
dataset evaluation.

Table 8: Performance analysis relative to fine-tuning with the different
source subsets.

(a) Classification performance on Fragrant-Spaces of models fine-tuned on RASD
and WASD, respectively.

Model | Fine-tuned on RASD |  | Fine-tuned on WASD
---|---|---|---
Top-1 acc. | Top-5 acc. | F1 |  | Top-1 acc. | Top-5 acc. | F1
ResNet18[4] | $18.86\text{\,}\mathrm{\char 37\relax}$ | $39.91\text{\,}\mathrm{\char 37\relax}$ |  0$4.61\text{\,}\mathrm{\char 37\relax}$ |  | $19.74\text{\,}\mathrm{\char 37\relax}$ | $35.53\text{\,}\mathrm{\char 37\relax}$ | $3.38\text{\,}\mathrm{\char 37\relax}$
ResNet50[4] | $18.86\text{\,}\mathrm{\char 37\relax}$ | $38.60\text{\,}\mathrm{\char 37\relax}$ |  0$6.76\text{\,}\mathrm{\char 37\relax}$ |  | $20.18\text{\,}\mathrm{\char 37\relax}$ | $39.47\text{\,}\mathrm{\char 37\relax}$ | $4.70\text{\,}\mathrm{\char 37\relax}$
DenseNet161[5] | $28.95\text{\,}\mathrm{\char 37\relax}$ | $44.74\text{\,}\mathrm{\char 37\relax}$ | $10.13\text{\,}\mathrm{\char 37\relax}$ |  | $19.74\text{\,}\mathrm{\char 37\relax}$ | $42.11\text{\,}\mathrm{\char 37\relax}$ | $5.39\text{\,}\mathrm{\char 37\relax}$
WaveMix[6] | $17.98\text{\,}\mathrm{\char 37\relax}$ | $40.79\text{\,}\mathrm{\char 37\relax}$ |  0$5.04\text{\,}\mathrm{\char 37\relax}$ |  | $14.91\text{\,}\mathrm{\char 37\relax}$ | $36.40\text{\,}\mathrm{\char 37\relax}$ | $3.22\text{\,}\mathrm{\char 37\relax}$

(b) Classification accuracy on dataset RASD Model | Pretrained | Fine-tuned on WASD
---|---|---
Top-1 acc. | Top-5 acc. | Top-1 acc. | Top-5 acc.
ResNet18[4] |  0$5.24\text{\,}\mathrm{\char 37\relax}$ | $19.64\text{\,}\mathrm{\char 37\relax}$ | $15.73\text{\,}\mathrm{\char 37\relax}$ | $37.27\text{\,}\mathrm{\char 37\relax}$
ResNet50[4] |  0$6.29\text{\,}\mathrm{\char 37\relax}$ | $20.02\text{\,}\mathrm{\char 37\relax}$ | $17.73\text{\,}\mathrm{\char 37\relax}$ | $40.32\text{\,}\mathrm{\char 37\relax}$
DenseNet161[5] |  0$6.48\text{\,}\mathrm{\char 37\relax}$ | $20.97\text{\,}\mathrm{\char 37\relax}$ | $18.02\text{\,}\mathrm{\char 37\relax}$ | $38.80\text{\,}\mathrm{\char 37\relax}$
WaveMix[6] |  0$7.34\text{\,}\mathrm{\char 37\relax}$ | $18.88\text{\,}\mathrm{\char 37\relax}$ | $20.88\text{\,}\mathrm{\char 37\relax}$ | $43.57\text{\,}\mathrm{\char 37\relax}$

(c) Classification accuracy on dataset WASD Model | Pretrained | Fine-tuned on RASD
---|---|---
Top-1 acc. | Top-5 acc. | Top-1 acc. | Top-5 acc.
ResNet18[4] | $10.07\text{\,}\mathrm{\char 37\relax}$ | $28.92\text{\,}\mathrm{\char 37\relax}$ |  0$9.61\text{\,}\mathrm{\char 37\relax}$ | $28.26\text{\,}\mathrm{\char 37\relax}$
ResNet50[4] | $10.92\text{\,}\mathrm{\char 37\relax}$ | $30.87\text{\,}\mathrm{\char 37\relax}$ | $12.68\text{\,}\mathrm{\char 37\relax}$ | $33.71\text{\,}\mathrm{\char 37\relax}$
DenseNet161[5] | $11.37\text{\,}\mathrm{\char 37\relax}$ | $31.56\text{\,}\mathrm{\char 37\relax}$ | $14.60\text{\,}\mathrm{\char 37\relax}$ | $36.02\text{\,}\mathrm{\char 37\relax}$
WaveMix[6] |  0$9.86\text{\,}\mathrm{\char 37\relax}$ | $28.08\text{\,}\mathrm{\char 37\relax}$ |  0$9.18\text{\,}\mathrm{\char 37\relax}$ | $26.27\text{\,}\mathrm{\char 37\relax}$

Table 8(a) compares the classification performance on Fragrant-Spaces relative
to the fine-tuning dataset used. Despite the size disparity between the RASD
and WASD datasets, the performances of models fine-tuned on them are
relatively similar. Their top-1 accuracies are also very close to those of
models fine-tuned on the combined ArtPlaces-train set. But the top-5
accuracies are not as high compared to the latter.

However, when testing on each other, the outcomes demonstrate noticeable
differences. Models fine-tuned on RASD and evaluated on WASD exhibited
significant enhancements, with an average increase of
$20.11\text{\,}\mathrm{\char 37\relax}$ and $11.75\text{\,}\mathrm{\char
37\relax}$ in top-5 and top-1 accuracy, respectively (_cf_. Tab. 8(b)). By
contrast, when fine-tuned on WASD and tested on RASD, their performances
remain relatively consistent before and after fine-tuning (_cf_. Tab. 8(c)).

To investigate the underlying cause of this phenomenon, we conduct a detailed
analysis of the dataset compositions.

As shown in Fig. 5(a), WASD encompasses most of the dominant categories
(_i.e_., including $90\text{\,}\mathrm{\char 37\relax}$ data) in RASD, and
contains a fair amount of data for training. Conversely, RASD includes about
three-quarters of the dominant categories in WASD (_cf_. Fig. 5(b)), moreover
only $18\text{\,}\mathrm{\char 37\relax}$ of these categories possess more
than 20 images, which is too limited compared to the size of the test set. The
insufficient availability of representative training data in this scenario
hinders the potential performance improvements.

We further compare all three datasets (_cf_. Fig. 5(c)) and find that RASD and
WASD both cover the majority of the top 20 categories in the small Fragrant-
Spaces set (RASD $65\text{\,}\mathrm{\char 37\relax}$, WASD
$75\text{\,}\mathrm{\char 37\relax}$), and they are also complementary,
providing a combined coverage of $80\text{\,}\mathrm{\char 37\relax}$ along
with a sufficient quantity of instances. Accordingly, models fine-tuned on the
merged dataset exhibit a large performance boost, emphasizing the crucial role
of size and diversity of fine-tuning datasets.

##  5 Conclusion

This work demonstrates that both fragrant places and general artistic scene
depictions can effectively be recognized using a weakly labeled transfer
learning approach. We based our method on two open cultural heritage data
sources: The Rijksmuseum and Wikidata. We constructed a fine-tuning training
set of 3,804 artworks using the query terms as weak labels. To evaluate the
model’s ability to recognize fragrant spaces we manually labeled images from
the ODOR dataset [20] with Places365 labels. Additionally, we combined this
Fragrant Places test set with a manually corrected split from the queried
artworks to evaluate broader artistic scene classification performance.

A cross-dataset analysis suggests that successful transfer training requires
the availability of fine-tuning datasets with sufficient representative
categories and instances.

However, compared to natural image datasets such as Places365 [15] or SUN [14]
our dataset is small. Expanding the dataset by adding additional data sources,
effectively broadening the scope of covered categories holds significant
potential for future work. Our results show that even around 5,000 artworks
lead to a notable increase in performance. We expect this improvement to
become more pronounced when scaling up the approach by processing more data
sources. A dataset quality assessment revealed high variability in the quality
of the weak labels: While the Wikidata subset achieved a high-quality level,
there is room for improvement in the Rijksmuseum labels. Conducting similar
analyses for additional data sources will be beneficial, allowing corrective
measures where necessary. This work opens up the space for additional research
efforts to explore methods to advance the classification of olfactory spaces
in artworks, and possibly broaden the scope to achieve a more general
classification of artistic scene depictions.

## References

  * [1] Ali, H., Paccosi, T., Menini, S., Mathias, Z., Pasquale, L., Kiymet, A., Raphaël, T., van Erp, M.: Musti-multimodal understanding of smells in texts and images at mediaeval 2022. In: Proceedings of MediaEval 2022 CEUR Workshop (2022)
  * [2] Bembibre, C., Strlič, M.: From smelly buildings to the scented past: an overview of olfactory heritage. Frontiers in Psychology 12, 718287 (2022)
  * [3] Ehrich, S.C., Verbeek, C., Zinnen, M., Marx, L., Bembibre, C., Leemans, I.: Nose-first. towards an olfactory gaze for digital art history. In: CEUR Workshop Proceedings. vol. 3064. CEUR Workshop Proceedings (2021)
  * [4] He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition (2015). https://doi.org/10.48550/arXiv.1512.03385
  * [5] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700–4708 (2017)
  * [6] Jeevan, P., Viswanathan, K., S, A.A., Sethi, A.: WaveMix: A Resource-efficient Neural Network for Image Analysis (2023)
  * [7] Kim, S., Park, J., Bang, J., Lee, H.: Seeing is Smelling: Localizing Odor-Related Objects in Images. In: Proceedings of the 9th Augmented Human International Conference. AH ’18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3174910.3174922, https://doi.org/10.1145/3174910.3174922
  * [8] Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06). vol. 2, pp. 2169–2178. IEEE (2006)
  * [9] Oliva, A., Torralba, A.: Modeling the shape of the scene: A holistic representation of the spatial envelope. International journal of computer vision 42, 145–175 (2001)
  * [10] Patoliya, V., Zinnen, M., Maier, A., Christlein, V.: Smell and emotion: Recognising emotions in smell-related artworks (2024), https://arxiv.org/abs/2407.04592
  * [11] Quattoni, A., Torralba, A.: Recognizing indoor scenes. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 413–420. IEEE (2009)
  * [12] Shiner, L.: Art scents: exploring the aesthetics of smell and the olfactory arts. Oxford University Press (2020)
  * [13] van Erp, M., Tullett, W., Christlein, V., Ehrhart, T., Hürriyetoğlu, A., Leemans, I., Lisena, P., Menini, S., Schwabe, D., Tonelli, S., Troncy, R., Zinnen, M.: More than the Name of the Rose: How to Make Computers Read, See, and Organize Smells. The American Historical Review 128(1), 335–369 (2023). https://doi.org/10.1093/ahr/rhad141
  * [14] Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: Large-scale scene recognition from abbey to zoo. In: 2010 IEEE computer society conference on computer vision and pattern recognition. pp. 3485–3492. IEEE (2010)
  * [15] Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 Million Image Database for Scene Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(6), 1452–1464 (2018). https://doi.org/10.1109/TPAMI.2017.2723009
  * [16] Zinnen, M.: How to see smells: Extracting olfactory references from artworks. In: Companion Proceedings of the Web Conference 2021. pp. 725–726 (2021)
  * [17] Zinnen, M., Hussian, A., Tran, H., Madhu, P., Maier, A., Christlein, V.: SniffyArt: The Dataset of Smelling Persons. In: Proceedings of the 5th Workshop on analySis, Understanding and proMotion of heritAge Contents. pp. 49–58 (2023). https://doi.org/10.1145/3607542.3617357
  * [18] Zinnen, M., Madhu, P., Bell, P., Maier, A., Christlein, V.: Transfer learning for olfactory object detection. In: Digital Humanities Conference, 2022. pp. 409–413. Alliance of Digital Humanities Organizations (2022), https://arxiv.org/abs/2301.09906
  * [19] Zinnen, M., Madhu, P., Kosti, R., Bell, P., Maier, A., Christlein, V.: ODOR: The ICPR2022 ODeuropa Challenge on Olfactory Object Recognition. In: 2022 26th International Conference on Pattern Recognition (ICPR). pp. 4989–4994 (2022). https://doi.org/10.1109/ICPR56361.2022.9956542
  * [20] Zinnen, M., Madhu, P., Leemans, I., Bell, P., Hussian, A., Tran, H., Hürriyetoğlu, A., Maier, A., Christlein, V.: Smelly, dense, and spreaded: The Object Detection for Olfactory References (ODOR) dataset. Expert Systems with Applications p. 124576 (2024). https://doi.org/https://doi.org/10.1016/j.eswa.2024.124576, https://www.sciencedirect.com/science/article/pii/S095741742401443X