11institutetext: Cornell University, Computer Science Department
22institutetext: Delft University of Technology, Perceptual Intelligence Lab
∗22email<EMAIL_ADDRESS>

# Insights From A Large-Scale Database of Material Depictions In Paintings

Hubert Lin∗ 11 Mitchell Van Zuijlen 22 Maarten W.A. Wijntjes 22 Sylvia C. Pont
22 Kavita Bala 11

###### Abstract

Deep learning has paved the way for strong recognition systems which are often
both trained on and applied to natural images. In this paper, we examine the
give-and-take relationship between such visual recognition systems and the
rich information available in the fine arts. First, we find that visual
recognition systems designed for natural images can work surprisingly well on
paintings. In particular, we find that interactive segmentation tools can be
used to cleanly annotate polygonal segments within paintings, a task which is
time consuming to undertake by hand. We also find that FasterRCNN, a model
which has been designed for object recognition in natural scenes, can be
quickly repurposed for detection of materials in paintings. Second, we show
that learning from paintings can be beneficial for neural networks that are
intended to be used on natural images. We find that training on paintings
instead of natural images can improve the quality of learned features and we
further find that a large number of paintings can be a valuable source of test
data for evaluating domain adaptation algorithms. Our experiments are based on
a novel large-scale annotated database of material depictions in paintings
which we detail in a separate manuscript.

###### Keywords:

Artistic Material Depictions Large-Scale Data Segmentation Classification
Interpretability Domain Adaptation

##  1 Introduction

Deep learning has enabled the development of high performing recognition
systems across a variety of image-based tasks [15, 17, 50]. These systems are
often trained on natural photographs with applications in real world
recognition like self-driving. Furthermore, applying recognition systems to
large collections of images can also reveal cultural trends or give us insight
into the visual patterns in the world (e.g. [32, 29, 26]). Human-created
images, such as paintings, are particularly interesting to analyze from this
perspective. Artistic depictions can reveal insights into culturally relevant
ideas throughout time, as well as insights into human visual perception
through the realism depicted by skilled artists.

Whereas most computer vision systems focusing on digital art history are
concerned with _object_ recognition (e.g., [11]), it is the depiction of
_space_ and _materials_ that visually characterized the course of art history.
The depiction of space has had considerable attention in scientific literature
[34, 46, 20, 37] while recently the depiction of materials has gained
scientific interest [5, 47, 38, 48]. Therefore, it is interesting to
investigate the interplay between deep learning systems designed for natural
image analysis and the rich visual information found in paintings, especially
with respect to artistic depictions of materials.

The remainder of this paper is organized into three parts. In Section 2, we
briefly describe the dataset that subsequent experiments are based on. In
Section 3, we explore how deep learning systems that have primarily been
developed for use on natural photographs can be used to analyze paintings.
Specifically, we explore (a) segmentation and (b) detection of materials in
paintings. Recognition of materials in paintings can be useful for digital art
history as well as general public interest. In Section 4, we explore how
paintings can be a useful source of data from which better recognition systems
can be built. Specifically, we investigate (c) the generalizability and
interpretability of classifiers trained on paintings, and we investigate (d)
the role that a large-scale painting dataset can play in evaluating visual
recognition models.

##  2 Dataset

All experiments in this paper utilize data from the Materials in Paintings
(MIP) dataset, a large-scale annotated dataset of material depictions in
paintings. Extensive details and analysis of this database will be available
in a separate manuscript. For context and completeness, we summarize a few
relevant details here. The dataset consists of 19K high resolution paintings
downloaded from the online collections of international art galleries, which
span over 500 years of art history. The galleries with corresponding number of
paintings are: The Rijksmuseum (4,672), The Metropolitan Museum of Art
(3,222), Nationalmuseum (3,077), Cleveland Museum of Art (2,217), National
Gallery of Art (2,132), Museo Nacional del Prado (2,032), The Art Institute of
Chicago (936), Mauritshuis (638), and J. Paul Getty Museum (399). The
distribution of paintings by year is shown in Fig. 1. The dataset includes
crowdsourced extreme click [35] bounding box annotations over 15 material
categories, which are further delineated into 50 finegrained categories. Fig.
2 shows a few examples of the annotated bounding boxes available in the
dataset.

Figure 1:  Year Distribution of Paintings in Dataset. Each bin equals 20
years. There are peaks in the paintings in the 1700s and 1900s. The former
corresponds to the European golden ages; it is less clear what explains the
latter peak. Figure 2:  Examples of Annotated Bounding Boxes. Left to Right:
Liquid, Fabric, Ceramic, Metal, and Food.

##  3 Using Computer Vision to Analyze Paintings

Research in computer recognition systems have focused primarily on natural
images. For example, semantic segmentation benchmarks (of objects, ‘stuff’, or
materials) [21, 25, 14, 7, 2, 3] emphasize parsing in-the-wild photos, with
applications in robotics, self-driving, and so forth. However, the analyses of
paintings can also benefit from the use of visual recognition systems.
Paintings can encode both cultural and perceptual biases, and being able to
analyze paintings at scale can be useful for a variety of scientific
disciplines including digital art history and human visual perception.

In this section, we explore the effectiveness of interactive segmentation
methods (which can be used to select regions of interest in photographs for
the purpose of image editing or data annotation) when applied to paintings. We
also explore how well an _object_ bounding box detector can be finetuned to
detect _materials_ depicted in unlabelled paintings, which could be used for
content-based retrieval.

###  3.1 Extracting Polygon Segments with Interactive Segmentation

Polygon segmentation masks are useful for reasoning about boundary
relationships between different semantic regions of an image, as well as the
shape of the regions themselves. However, annotating segmentations is
expensive and many modern datasets rely on expensive manual annotation methods
[2, 25, 51, 7, 10]. Recent work has focused on more cost effective annotation
methods (e.g. [24, 31, 4, 27]). The use of interactive segmentation methods
that transform sparse user inputs into polygon masks can ease annotation
difficulty. For paintings, it is unclear whether these methods (especially
deep learning methods trained on natural images) would perform well. Semantic
boundaries in paintings likely have a different, and more varying structure
than in photos. Paintings can have ambiguous or fuzzy boundaries between
objects or materials [33] which can potentially be problematic for color-based
methods. This can be due to variations in artistic style which can emphasize
different aspects of depictions – for example, Van Gogh uses lines and edges
to create texture, but such edges could potentially appear as boundaries to a
segmentation model. In this experiment, we study the difficulty of segmenting
paintings and whether innovations are necessary for existing methods to
perform well.

####  3.1.1 Experimental Setup.

We experiment with GrabCut [40] (an image-based approach) and DEXTR [31] (a
modern deep learning approach). We evaluated the performance of these methods
against 4.5k high-quality human annotated segmentations from [52]. The inputs
to these methods are generated from the extreme points of the regions we are
interested in. We use a variant of the GrabCut initialization proposed in
[35], as well as a rectangular initialization for reference. For DEXTR, we
consider models pretrained on popular object datasets [25, 14] as a starting
point.

####  3.1.2 Results.

We found that both GrabCut and DEXTR perform quite well on paintings.
Surprisingly, DEXTR transfers quite well to materials in paintings despite
being trained only with natural photographs of objects. The performance of
DEXTR can be further improved by finetuning on COCO with a smaller learning
rate (10% of original learning rate for 1 epoch). Finetuning DEXTR on Grabcut
segments or iteratively finetuning with output of DEXTR does not seem to yield
further improvements. The performance is summarized in Table 1, and samples
are visualized in Fig. 3.

Segmentation mean IOU (%)
Grabcut Rectangle Grabcut Extr DEXTR Pascal-SBD DEXTR COCO DEXTR Finetuned
44.1 72.4 74.3 76.4 78.4
DEXTR Finetuned IOU By Class (%)
Animal Ceramic Fabric Flora Food 76.9 86.8 79.1 77.0 87.5 Gem Glass Ground
Liquid Metal 74.4 83.2 69.6 73.0 75.5 Paper Skin Sky Stone Wood 86.1 78.9 78.5
81.7 67.4

Table 1:  Segmentation Performance. Grabcut Extr is based on [35] with small
modifications: (a) minimum cost boundary is computed with the negative log
probability of a pixel belonging to an edge; (b) in addition to clamping the
morphological skeleton, the extreme points centroid and extreme points are
clamped; (c) GC is computed directly on the RGB image. DEXTR [31] is
pretrained on Pascal-SBD and COCO. Note that Pascal-SBD and COCO are natural
image datasets of objects, but DEXTR transfers surprisingly well across both
visual domain (paintings vs. photos) and annotation categories (materials vs.
objects). Figure 3:  Extreme Click Segmentations. Left to right: Original
Image, Ground Truth Segment, Grabcut Extr Segment, DEXTR COCO Segment. Both
Grabcut and DEXTR use extreme points as input. For evaluation, the extreme
points are generated synthetically from the ground truth segments. In
practice, extreme clicks can be crowdsourced. Bottom-right corner shows the
IOU for each segmentation.

###  3.2 Detecting Materials in Unlabeled Paintings

To allow the public to view and interact with art collections, museums and
galleries provide extensive online functionality to search and navigate
through the collections. Currently, to our knowledge, no online collections
allows online visitors to query the collection for depicted _materials_ within
painting, which can be of interest to the public. Furthermore, depiction of
materials plays a crucial role in characterizing art history. Detecting
materials within novel paintings will be particularly beneficial to digital
art historians who study materials such as stone [1, 13] or skin [6, 22].
Having access to specific materials can also digital art historians to compare
these depictions directly with respect to painting style or technique. We
experiment with automatic bounding box detection to ease access to material
depictions in unlabelled collections.

####  3.2.1 Experimental Setup.

We train a FasterRCNN [39] bounding box detector to localize and label
material boxes with on 90% of annotated paintings in the dataset, and evaluate
on the remaining 10%. Default COCO hyperparameters from [49] are used. Given
the non-spatially-exhaustive nature of the annotations, many detected bounding
boxes will not be matched against labelled ground truth boxes. However, the
dataset is exhaustively annotated at an image level, and therefore, we report
image-level accuracies. This can be interpreted as the accuracy of the model
in tagging each image with the types of materials present. The validity of
each localized box can be further quantified through a user study, but we did
not perform this study at this time.

####  3.2.2 Results.

Table 2 shows the performance. We found that the FasterRCNN model is able to
accurately detect materials in paintings by finetuning on the annotated
bounding boxes directly without any changes to the network architecture or
training hyperparameters. It is certainly promising to see that an algorithm
designed for object localization in natural images can be readily applied to
material localization in paintings. A qualitative sample of detected bounding
boxes is given in Fig. 4. To improve the spatial-specificity of the detected
materials, it can be interesting to train an instance detector like MaskRCNN
on segments extracted using methods discussed in the previous section. It
would also be useful to combine material recognition with conventional object-
based detection to extract complementary forms of information that improve the
ability for users to filter data by their specific needs.

Class Accuracy (%) (Mean = 83.3%)
Animal Ceramic Fabric Flora Food 85.6 92.7 66.0 85.0 94.9 Gem Glass Ground
Liquid Metal 88.4 91.3 86.5 86.4 70.7 Paper Skin Sky Stone Wood 92.4 70.2 89.4
74.8 74.9

Table 2:  Image-level Detection Accuracy. Bounding boxes are detected with
FasterRCNN trained on paintings. Because the dataset is not exhaustively
annotated spatially, image-level accuracy is reported instead of box precision
and recall. Overall, images are tagged with the correct materials with high
accuracy. Figure 4:  Detected materials in Unlabeled Paintings. Automatically
detecting materials can be useful for content retrieval and for filtering
online galleries by viewer interests.

##  4 Using Paintings to Build Better Recognition Systems

In recent work for machine perception systems, art has been used in various
ways. Models that learn to convert photographs into painting-like or sketch-
like images have been studied extensively for their application as a tool for
digital artists [18]. Recent work has shown that such neural style transfer
algorithms can also produce images that are useful for training robust neural
networks [16]. Artworks have also been used directly to evaluate the
robustness of neural networks under “domain shifts” in which a model trained
to recognize objects from photographs are shown artistic depictions of such
objects instead [23, 36].

We use the MIP dataset of material depictions in paintings to explore two
directions. First, we hypothesize that the perceptually focused depictions of
artists can allow neural networks to learn better cues for classification. We
find that learning from paintings can improve the interpretability of the cues
it uses for its predictions. In a second experiment, we investigate the
utility of the MIP dataset as a benchmark for computer vision models under
domain shifts when more data is available than is typical in existing domain
adaptation benchmark datasets. We find that existing domain adaptation
algorithms can fail to behave as expected in this setting.

###  4.1 Learning Robust Cues for Finegrained Fabric Classification

The task of distinguishing between images of different semantic content is a
standard recognition task for computer vision systems. Increasing attention is
being given to “fine-grained” classification where a model is tasked with
distinguishing images of the same broad category (e.g. distinguishing
different species of birds or different types of flora [45, 44, 43]). Fine-
grained classification is particularly challenging for deep learning systems.
Such a task depends on recognizing specific attributes for each finegrained
class; in comparison, classifiers can perform well on coarse-grained
classification by relying on context alone. We hypothesize that the painted
depictions of materials can be beneficial for this task. Since some artistic
depictions focus on salient cues for perception through perceptual shortcuts
[30, 8, 12], it is possible that a network trained on such artwork is able to
learn a more robust feature representation by focusing on these cues.

####  4.1.1 Experimental Setup.

We experiment with the task classifying cotton/wool versus silk/satin. The
latter can be recognized through local cues such as highlights on the cloth;
such cues are carefully placed by artists in paintings. To understand whether
artistic depictions of fabric allow a neural network to learn better features
for classification, we train a model with either photographs or paintings.
High resolution photographs of cotton/wool and silk/satin fabric and clothing
(dresses, shirts) are downloaded and manually filtered from publicly available
photos licensed under the Creative Commons from Flickr. In total, we
downloaded roughly 1K photos. We sample cotton/wool and silk/satin samples
from our dataset to form a corresponding dataset of 1K paintings.

##### Generalizability of Classifiers.

Does training with paintings improve the generalizability of classifiers? To
test cross-domain generalization, we test the classifier on types of images
that it has not seen before. A classifier that has learned more robust
features will perform better on this task than one that has learned to
classify images based on more spurious correlations. We test the trained
classifiers on both photographs and paintings.

##### Interpretability of Classifier Cues.

Are the cues used by each classifier interpretable to humans? We produce
evidence heatmaps with GradCAM [41] from the feature maps in the network
before the fully connected classification layer. We extract high resolution
feature maps from images of size 1024 $\times$ 1024 (for a feature map of size
32 $\times$ 32). The heatmaps produced by GradCAM show which regions of an
image the classifier uses as evidence for a specific class. If a classifier
has learned a good representation, the evidence that it uses should be more
interpretable for humans. For both models, we compute heatmaps for test images
corresponding to their ground truth label. We conduct a user study on Amazon
Mechanical Turk to find which heatmaps are preferred by humans. Users are
shown images with regions corresponding to heatmap values that are above 1.5
standard deviations above the mean. Fig 5 illustrates an example. Our user
study resulted in responses from 85 participants, 57 of which were analyzed
after quality control. For quality control, we only kept results from
participants who spent over 1 second on average per task item.

Figure 5:  Classifier Cues. Left to Right: Original Image, Masked Image
(Painting Classifier), and Masked Image (Photo Classifier). The unmasked
regions represent evidence used by the classifiers for predicting “silk/satin”
in this particular image.

####  4.1.2 Results.

We find that the classifier trained with paintings exhibits better cross-
domain generalization, and uses cues that humans prefer over the photo
classifier. This suggests that paintings can improve the robustness of
classifiers for this task of fabric classification.

##### Generalizability of Classifiers.

In Table 3, the performance of the two classifiers are summarized. We find
that both classifiers perform similarly well on the domain they are trained
on. However, when the classifiers are tested on cross-domain data, we find
that the painting-trained classifier performs better than the photo-trained
classifier. This suggests that the classifier trained on paintings has learned
a more generalizable feature representation for this task.

##### Interpretability of Classifier Cues.

Overall, we find that the classifier trained on paintings uses evidence that
is better aligned with evidence preferred by humans (Table 4 and Fig. 6). Due
to domain shifts when applying classifiers to out-of-domain images, we would
expect the cues selected by the painting classifier to be preferable on
paintings, and the cues selected by the photo classifier to be preferable on
photos. Interestingly, this does not hold for photos of satin/silk (column 2
of Table 4) – we find that users equally prefer the evidence selected by the
painting classifier to the evidence selected by the photo classifier. This
suggests that either (a) the painting classifier has learned the “correct”
human-interpretable cues for recognizing satin/silk, or (b) that the photo
classifier has learned to classify satin/silk based on some spurious
contextual signals. We asked users to elucidate their reasoning when choosing
which set of cues they preferred. In general, users noted that they preferred
the network which picks out regions containing the target class. Therefore, it
seems that the network trained on paintings has learned better to distinguish
fabric through the actual presence of such fabrics in the image over other
contextual signals.

Taken together, our results provide evidence that a classifier trained on
paintings can be more robust than a classifier trained on photographs. It
would be interesting to explore this further. A limitation of this study is
the relatively small number of data samples, and very limited number of
material types (two: cotton/wool and silk/satin) that we explored. Are there
other materials or objects which deep neural networks can learn to recognize
better from paintings than photographs?

|  Photo $\rightarrow$ Photo |  Painting$\rightarrow$ Painting
---|---|---
MEAN F1 Score |  79.6% |  80.5%
|  Photo $\rightarrow$ Painting |  Painting$\rightarrow$ Photo
MEAN F1 Score |  49.5% |  57.8%
Table 3:  Classifier Generalization. Classifiers are trained to distinguish cotton/wool from silk/satin. One classifier is trained on photographs and another classifier is trained on paintings. Both classifiers perform similarly well on images of the same type they were trained on, but the classifier trained on paintings performs better on photographs than vice versa. This suggests that the features learned from paintings are more generalizable for this task on this set of data. |  Cotton/Wool Photos |  Silk/Satin Photos |  Cotton/Wool Paintings |  Silk/Satin Paintings |  MEAN
---|---|---|---|---|---
Photo Classif. Preferred |  64.7 $\pm$ 3.5% |  48.9 $\pm$ 3.1% |  26.8 $\pm$ 2.5% |  39.1 $\pm$ 2.1% |  44.9 $\pm$ 1.9%
Painting Classif. Preferred |  35.3 $\pm$ 3.5% |  51.1 $\pm$ 3.1% |  73.2 $\pm$ 2.5% |  60.9 $\pm$ 2.1% |  55.1 $\pm$ 1.9%
Table 4:  Human Agreement with Classifier Cues. On average, humans prefer the
cues used by the painting-trained classifier to make its predictions over the
cues used by the photo-trained classifier. Interestingly, the human judgements
also indicate that the painting-trained classifier uses cues that are just as
good to the cues used by the photo-trained classifier for silk/satin photos
despite never seeing a silk/satin photo during training (column 2). A
pictorial representation of the results is given in Fig. 6.  Figure 6:  Human
Agreement with Classifier Cues. Pictorial representation of user study results
from Table 4. The y-axis represents how often humans prefer the cues from a
classifier trained on the same domain as the test images. It is clear that
humans prefer the painting classifier for paintings more than they prefer the
photo classifier for photos. Interestingly, the painting and photo classifiers
are equally preferred for silk/satin photos despite the painting classifier
never seeing a photo during training (bar 2).

###  4.2 Benchmarking Unsupervised Domain Adaptation

In unsupervised domain adaptation (UDA), models are trained on a ‘source’
dataset with annotated labels as well as an unlabeled ‘target’ dataset. The
goal is to train a model which performs well on unseen target dataset samples.
Existing domain adaptation benchmark datasets for classification focus
primarily on object recognition and tend to be limited in number of data
samples, with most class categories containing on the order of 1000 samples or
fewer (for example, refer to Table 1 of [36]). In contrast, the dataset we use
here has the unique properties of (a) focusing primarily on material
classification and (b) containing on the order of 10-30K for 9 of the 15
annotated classes (e.g. fabric, wood), with the remainder in the range of
2K-5K (e.g. ground). This positions this data as a valuable addition for
benchmarking for UDA algorithms.

####  4.2.1 Experimental Setup.

For this study, we focus on a family of domain adaptation algorithms which aim
to explicitly minimize feature discrepancy across the source and target
domains. Existing work has shown that class-conditional UDA in which labels
are estimated for target domain samples during training can be better than
class-agnostic UDA where adaptation is performed without using any estimated
label information at all. We choose CDD [19] and MMD [28, 42] as
representative methods for class-conditional and class-agnostic discrepancy
minimization. All methods are trained with default settings from publicly
available source code for CDD, which includes the use of domain batch
normalization [9]. We selected 10 material categories: ceramic, fabric,
foliage, glass, liquid, metal, paper, skin, stone, and wood. For our painting
dataset, we sampled as-class-balanced-as-possible from these classes to form a
dataset with 10K samples and a dataset with 60K samples. A corresponding
photograph dataset is constructed from Opensurfaces/MINC/COCO with 10K and 60K
samples as well.

####  4.2.2 Results.

We find that the studied domain adaptation algorithms can indeed behave
differently than would be expected from results on existing benchmark
datasets. This is could due to more data being available or a more difficult
domain shift than conventional adaptation benchmark datasets.

##### Effect of Dataset Size.

Results are summarized in Table 5. With the conventional 1K samples per class,
we confirm domain adaptation yields gains over source-only as found on
existing benchmark datasets. In contrast to results on existing benchmarks
however, we find that class-conditional adaptation does not necessarily
outperform class-agnostic adaptation. We hypothesize this occurs due to
failures in target label estimation for the class-conditional case – we
discuss this further below. Next, with 6K samples per class (which is
6$\times$ more data samples per class than conventional UDA benchmarks), we
find that source-only (i.e., no adaptation) performs very competitively. In
fact, source-only strictly outperforms adaptation for Painting to Photo
transfer in this data regime! This result suggests that domain adaptation is
useful in lower data regimes, but source-only is a competitive alternative
when more data is available. We leave a deeper exploration of this phenomenon
to future work.

##### Effect of Class Label Estimation.

As found above, class-conditional adaptation can underperform class-agnostic
adaptation despite utilizing more information. As class-conditional adaptation
depends on estimated target labels, large domain shifts that hamper label
estimation can harm adaptation. To confirm this, we consider two experiments:
CDD with intraclass discrepancy minimization only (instead of both intraclass
minimization and interclass maximization), and CDD with ground truth labels
(i.e., perfect label estimation). Results are in Table 6. In both cases, we
see performance improves. In the case where perfect label estimation is
assumed, then CDD does outperform intraCDD and MMD as found on existing
datasets. Therefore, estimating class labels for domain adaptation is useful
in practice, but only if the labels are estimated sufficiently well.

ResNet18 _1K imgs/class per domain_ |  Photo $\rightarrow$ Painting |  Painting $\rightarrow$ Photo
---|---|---
Source-Only |  35.2 |  46.9
MMD |  46.1 (+10.9) |  56.4 (+9.5)
CDD |  40.5 (+5.3) |  57.4 (+10.5)
ResNet18 _6K imgs/class per domain_ |  Photo $\rightarrow$ Painting |  Painting $\rightarrow$ Photo
Source-Only |  38.7 |  53.6
MMD |  43.5 (+4.8) |  51.5 (-2.1)
CDD |  35.6 (-3.1) |  49.4 (-4.2)
Table 5: Effect of Dataset Size. UDA from photo (source) to painting (target) and painting (source) to photo (target). Source-only refers to a reference baseline where no adaptation is used. The gap between source-only and UDA decreases as data samples increases from 1K images per class to 6K images per class. Furthermore, in contrast to behavior found on existing benchmark datasets, the class-conditional method of CDD does not necessarily outperform the class-agnostic counterpart MMD.  ResNet18 _1K imgs/class per domain_ |  Photo $\rightarrow$ Painting |  Painting $\rightarrow$ Photo
---|---|---
MMD |  46.1 (+10.9) |  56.4 (+9.5)
IntraCDD |  44.4 (+9.2) |  58.5 (+11.6)
CDD |  40.5 (+5.3) |  57.4 (+10.5)
IntraCDD w/ GT labels |  57.6 (+22.4) |  64.4 (+17.5)
CDD w/ GT labels |  61.6 (+26.4) |  72.5 (+25.6)
Table 6:  Effect of Class Label Estimation. Reducing the reliance class label
estimation improves class-conditional UDA when label estimation for target
data is poor. We find that IntraCDD (which considers only intraclass
discrepancy) outperforms CDD (which considers both intraclass and interclass
discrepancy). Using ground truth (GT) labels with CDD (i.e., assuming perfect
class label estimation) recovers performance gains over intraCDD and MMD. MMD
does not require class label estimation, and so its performance not suffer in
the case of poor label estimation.

##  5 Conclusion

In this paper, we explored how modern deep learning tools developed for
natural images can be used to analyze paintings, and in turn, how paintings
can be used to improve deep learning systems in a series of experiments. Our
findings suggest that progress in visual perception for natural images can
benefit systems used for fine art analysis, and having access to the visual
information encoded in paintings can be fruitful for building more
generalizable perception systems.

##  6 Acknowledgements

This work was funded in part by NSF (CHS-1617861 and CHS-1513967), NSERC
(PGS-D 516803 2018), and the Netherlands Organization for Scientific Research
(NWO) project 276-54-001.

## References

  * [1] Augart, I., Saß, M., Wenderholm, I.: Steinformen. De Gruyter, Berlin, Boston (2018). https://doi.org/https://doi.org/10.1515/9783110583618, https://www.degruyter.com/view/title/535173
  * [2] Bell, S., Upchurch, P., Snavely, N., Bala, K.: OpenSurfaces:A Richly Annotated Catalog of Surface Appearance. ACM Transactions on Graphics 32(4), 1 (2013). https://doi.org/10.1145/2461912.2462002, http://dl.acm.org/citation.cfm?doid=2461912.2462002
  * [3] Bell, S., Upchurch, P., Snavely, N., Bala, K.: Material recognition in the wild with the Materials in Context database. Computer Vision and Pattern Recognition (CVPR) (2015)
  * [4] Benenson, R., Popov, S., Ferrari, V.: Large-scale interactive object segmentation with human annotators. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 11700–11709 (2019)
  * [5] Beurs, W.: ”De groote waereld in’t kleen geschildert, of Schilderagtig tafereel van’s weerelds schilderyen. Kortelijk vervat in ses boeken: Verklarende de hooftverwen, haare verscheide mengelinge in oly, en der zelver gebruik…”. By Johannes en Gillis Janssonius van Waesberge (1692)
  * [6] Bol, M., Lehmann, A.S.: Painting skin and water: towards a material iconography of translucent motifs in early netherlandish painting. In: Rogier van der Weyden in context: papers presented at the Seventeenth Symposium for the Study of Underdrawing and Technology in Painting held in Leuven, 22-24 October. pp. 215–228. Peeters (2012)
  * [7] Caesar, H., Uijlings, J., Ferrari, V.: Coco-stuff: Thing and stuff classes in context. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1209–1218 (2018)
  * [8] Cavanagh, P.: The artist as neuroscientist. Nature 434(7031), 301–307 (2005). https://doi.org/10.1038/434301a
  * [9] Chang, W.G., You, T., Seo, S., Kwak, S., Han, B.: Domain-specific batch normalization for unsupervised domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7354–7362 (2019)
  * [10] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3213–3223 (2016)
  * [11] Crowley, E.J., Zisserman, A.: The state of the art: Object retrieval in paintings using discriminative regions. In: British Machine Vision Conference (2014)
  * [12] Di Cicco, F., Wijntjes, M.W., Pont, S.C.: Understanding gloss perception through the lens of art: Combining perception, image analysis, and painting recipes of 17th century painted grapes. Journal of Vision 19(3), 1–15 (2019). https://doi.org/10.1167/19.3.7
  * [13] Dietrich, R.: Rocks depicted in painting & sculpture. Rocks & Minerals 65(3), 224–236 (1990)
  * [14] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. International journal of computer vision 88(2), 303–338 (2010)
  * [15] Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Martinez-Gonzalez, P., Garcia-Rodriguez, J.: A survey on deep learning techniques for image and video semantic segmentation. Applied Soft Computing 70, 41–65 (2018)
  * [16] Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., Brendel, W.: Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231 (2018)
  * [17] Hafiz, A.M., Bhat, G.M.: A survey on instance segmentation: state of the art. International Journal of Multimedia Information Retrieval pp. 1–19 (2020)
  * [18] Jing, Y., Yang, Y., Feng, Z., Ye, J., Yu, Y., Song, M.: Neural style transfer: A review. IEEE transactions on visualization and computer graphics (2019)
  * [19] Kang, G., Jiang, L., Yang, Y., Hauptmann, A.G.: Contrastive adaptation network for unsupervised domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4893–4902 (2019)
  * [20] Kemp, M., et al.: The science of art: Optical themes in western art from brunelleschi to seurat (1990)
  * [21] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Duerig, T., et al.: The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint arXiv:1811.00982 (2018)
  * [22] Lehmann, A.S.: Fleshing out the body: The’colours of the naked’in workshop practice and art theory, 1400-1600. Nederlands Kunsthistorisch Jaarboek 59, 86 (2008)
  * [23] Li, D., Yang, Y., Song, Y.Z., Hospedales, T.M.: Deeper, broader and artier domain generalization. In: Proceedings of the IEEE international conference on computer vision. pp. 5542–5550 (2017)
  * [24] Lin, H., Upchurch, P., Bala, K.: Block annotation: Better image annotation with sub-image decomposition. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5290–5300 (2019)
  * [25] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 8693 LNCS(PART 5), 740–755 (2014). https://doi.org/10.1007/978-3-319-10602-1_48
  * [26] Lin, Z., Sun, J., Davis, A., Snavely, N.: Visual chirality. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12295–12303 (2020)
  * [27] Ling, H., Gao, J., Kar, A., Chen, W., Fidler, S.: Fast interactive object annotation with curve-gcn. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5257–5266 (2019)
  * [28] Long, M., Cao, Y., Wang, J., Jordan, M.: Learning transferable features with deep adaptation networks. In: International conference on machine learning. pp. 97–105. PMLR (2015)
  * [29] Mall, U., Matzen, K., Hariharan, B., Snavely, N., Bala, K.: Geostyle: Discovering fashion trends and events. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 411–420 (2019)
  * [30] Mamassian, P.: Ambiguities and conventions in the perception of visual art. Vision Research 48(20), 2143–2153 (2008). https://doi.org/10.1016/j.visres.2008.06.010
  * [31] Maninis, K.K., Caelles, S., Pont-Tuset, J., Van Gool, L.: Deep Extreme Cut: From Extreme Points to Object Segmentation (2017), http://arxiv.org/abs/1711.09081
  * [32] Matzen, K., Bala, K., Snavely, N.: Streetstyle: Exploring world-wide clothing styles from millions of photos. arXiv preprint arXiv:1706.01869 (2017)
  * [33] M.H. van Eikema Hommes: The contours in the paintings of the oranjezaal, huis ten bosch’ (2005)
  * [34] Panofsky, E.: Perspective as symbolic form. Princeton University Press (1927/2020)
  * [35] Papadopoulos, D.P., Uijlings, J.R.R., Keller, F., Ferrari, V.: Extreme clicking for efficient object annotation. International Journal of Computer Vision (2017). https://doi.org/10.1109/ICCV.2017.528, http://arxiv.org/abs/1708.02750
  * [36] Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., Wang, B.: Moment matching for multi-source domain adaptation. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1406–1415 (2019)
  * [37] Pirenne, M.H.: Optics, painting & photography. Cambridge University Press
  * [38] Pottasch, C.: Frans van mieris’s painting technique as one of the possible sources for willem beurs’s treatise on painting. Art & Perception pp. 1 – 17 (13 Jun 2020). https://doi.org/https://doi.org/10.1163/22134913-bja10013, https://brill.com/view/journals/artp/aop/article-10.1163-22134913-bja10013/article-10.1163-22134913-bja10013.xml
  * [39] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems. pp. 91–99 (2015)
  * [40] Rother, C., Kolmogorov, V., Blake, A.: ” grabcut” interactive foreground extraction using iterated graph cuts. ACM transactions on graphics (TOG) 23(3), 309–314 (2004)
  * [41] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
  * [42] Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T.: Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 (2014)
  * [43] Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., Belongie, S.: The inaturalist species classification and detection dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8769–8778 (2018)
  * [44] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset (2011)
  * [45] Wei, X.S., Wu, J., Cui, Q.: Deep learning for fine-grained image analysis: A survey. arXiv preprint arXiv:1907.03069 (2019)
  * [46] White, J.: The birth and rebirth of pictorial space. Cambridge, MA (1957)
  * [47] Wiersma”, L.: ”‘colouring’ — material depiction in flemish and dutch baroque art theory”. ”Art & Perception” pp. ”1 – 23” (”22 Apr 2020”). https://doi.org/”https://doi.org/10.1163/22134913-bja10005”, ”https://brill.com/view/journals/artp/aop/article-10.1163-22134913-bja10005/article-10.1163-22134913-bja10005.xml”
  * [48] Wijntjes, M.W.A., Spoiala, C., de Ridder, H.: Thurstonian scaling and the perception of painterly translucency. Art & Perception pp. 1 – 24 (04 Sep 2020). https://doi.org/https://doi.org/10.1163/22134913-bja10021, https://brill.com/view/journals/artp/aop/article-10.1163-22134913-bja10021/article-10.1163-22134913-bja10021.xml
  * [49] Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2 (2019)
  * [50] Ye, M., Shen, J., Lin, G., Xiang, T., Shao, L., Hoi, S.C.: Deep learning for person re-identification: A survey and outlook. arXiv preprint arXiv:2001.04193 (2020)
  * [51] Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 633–641 (2017)
  * [52] van Zuijlen, M.J., Pont, S.C., Wijntjes, M.W.: Painterly depiction of material properties. Journal of vision 20(7), 7–7 (2020)